DIVA: Dataset Derivative of a Learning Task
Authors: Yonatan Dukler
Alessandro Achille
Giovanni Paolini
Avinash Ravichandran
Marzia Polito
Stefano Soatto
Venue: ICLR 2022 Poster (2022)
Keywords: Leave one out cross validation
AutoML
dataset optimization

Abstract: We present a method to compute the derivative of a learning task with respect to a dataset. A learning task is a function from a training set to the validation error, which can be represented by a trained deep neural network (DNN). The ``dataset derivative'' is a linear operator, computed around the trained model, that informs how perturbations of the weight of each training sample affect the validation error, usually computed on a separate validation dataset.  Our method, DIVA (Differentiable Validation) hinges on a closed-form differentiable expression of the leave-one-out cross-validation error around a pre-trained DNN. Such expression constitutes the dataset derivative. DIVA could be used for dataset auto-curation, for example removing samples with faulty annotations, augmenting a dataset with additional relevant samples, or rebalancing. More generally, DIVA can be used to optimize the dataset, along with the parameters of the model, as part of the training process without the need for a separate validation dataset, unlike bi-level optimization methods customary in AutoML. To illustrate the flexibility of DIVA, we report experiments on sample auto-curation tasks such as outlier rejection, dataset extension, and automatic aggregation of multi-modal data.

Published as a conference paper at ICLR 2022
DIVA: DATASET DERIVATIVE OF A LEARNING TASK
Yonatan Dukler' 12,, Alessandro Achille', Giovanni Paolini', Avinash Ravichandran',
Marzia Polito', Stefano Soatto"
1 Amazon Web Services,
taachille, paoling, ravinash, mpolito, soattos) @amazon. com
2 Department of Mathematics, University of California, Los Angeles ydukler@matte edu
ABSTRACT
We present method compute the derivative of learning task with respect a dataset. learning task is a function from a training set to the validation error, which can be represented by trained deep neural network (DNN). The "dataset derivative" is a linear operator, computed around the trained model, that informs how perturbations of the weight of each training sample affect the validation error, usually computed on separate validation dataset. Our method, DIVA (Differentiable Validation) hinges on a flosed-form differentiable expression of the leave-one-ord out cross- validation error around a pre-trained DNN. Such expression constitutes the dataset derivative. DIVA could be used for dataset auto-curation, for example removing samples with faulty annotations, augmenting a dataset with additional relevant samples, or rebalancing. More generally, DIVA can be used to optimize the dataset, along with the parameters of the model, part of the training process without the need for a separate validation dataset, unlike bi-level optimization methods customary in AutoML. To illustrate the flexibility of DIVA, we report experiments on sample auto-curation tasks such as outlier rejection, dataset extension, and automatic aggregation of multi-modal data.
INTRODUCTION
Consider the following seemingly disparate questions. (i) Dataset Extension: Given relatively small training set, but access to large pool of additional data, how to select from the latter samples to augment the former? (ii) Datasel Curation: Given a potentially large dataset riddled with annotation errors, how to automatically reject such outlier samples? (iii) Dataset Reweighting: Given a finite training set, how to reweight the training samples to yield better generalization performance?
These three are examples of Dataset Optimization. In order to solve this problem with differentiable programming, one can optimize a loss of the model end-to-end, which requires differentiating the model's 's loss with respect to the dataset. Our main contribution is an efficient method to compute such a dataset derivative. This allows learning an importance weight a; for each datum in a training dataset D, extending the optimization from the weights w of a parametric model such as deep neural network (DNN), to also include the weights of the dataset.
As illustrated in the following diagram, standard optimization in machine learning works by finding the weights wa that minimize the training loss Trgain (w, Da)) = Σ; a, (x(((((x,) y1) on given (weighted) dataset D& (dark box). We solve a more general learning problem (light box) by jointly optimizing the dataset D, in addition to w. To avoid the trivial solution a = 0, is customary in AutoML to optimize D. by minimizing the validation error computed a disjoint dataset. This makes for inefficient use of the data, which has to be split between training and validation sets. Instead, leverage closed-form expression of the leave-one-out cross-validation error to jointly optimize the model and data weights during training, without the need to create a separate validation set.
"Wok conducted Amazon Weh Services.

---

Published as a conference paper at ICLR 2022
Da
(W, Da))
Wl
(Wa)
The intermediate block in the diagram (which finds the optimal weights wa for the the training loss on Da) is usually non-differentiable with respect to the dataset, or the derivative prohibitively expensive compute. DIVA leverages recent progress in deep learning linearization Achille et al. (2020), to derive closed-form expression for the derivative of the final loss (validation error) with respect to the dataset weights. In particular, Achille et al. (020)) have shown that, by replacing cross-entropy with least-squares, replacing ReLu with leaky- Re,u, and performing suitable pre- conditioning, the linearized model performs on par with full non-linear fine-tuning. We also leverage a classical result to compute the leave-one-out loss of a linear model in closed-form (Rifkin & Lippert 2007 Green Silverman, 1993). This allows us to optimize the LOO loss without requiring separate validation set, setting DIVA apart from bi-level optimization customary in AutoMI..
To illustrate the many possible uses of the dataset derivative, we run experiments with a simplified version of DIVA to cleanup dataset of noisy annotations, extend training set with additional data from an external pool, identify meaningful data augmentation, and to perform multi-modal expansion using CLIP model (Radford et al. 202Ip.
Rather than using the full lincarization of the model derived by Achille et al. 2020) we restrict the gradient to its last layer, cognizant that we are not exploiting the full power of LQF and thereby obtaining only a lower-bound of performance improvement. Despite that restriction, our results show consistent improvements from dataset optimization, at the modest computational cost of forward pass over the dataset to optimize the importance weights.
To summarize, our main contributions are:
1. We introduce method to compute the dataset derivative in closed form, DIVA.
2. We illustrate the use of DIVA to perform dataset optimization by minimizing directly the leave-one-out error without the need for an explicit validation dataset.
3. We perform experiments with simplified model that, despite not using the full power of the linearization, shows consistent improvements in dataset extension, re-weighting, outlier rejection and automatic aggregation of multi-modal data.
Our method presents several limitations. The dataset derivative of a learning task is computed around a point represented by a pre-trained model. It only allows local optimization around this point. Moreover, we only compute restriction of the linearization to the dimensions spanned by the last few layers. In general, this yields suboptimal results compared to full global optimization from scratch, one could compute that scale. Nonetheless, the linearized setting is consistent with the practice of fine-tuning treitrained models in light of the results of Achille et al. (2020), see also (Radford et al.
202T Rezende al. 2017 Mormont al. 2018 Hosny et al. 2018
RELATED WORK
AutoML. State of the art performance in image classification tasks often relies large amount
of human expertise in selecting models and adjusting the training settings for the task at hand (Li
al. 2020ap. Automatic machine learning (AutoML) (Feurer et al. 2019 He al. 202T) aims
to automate model selection (Cawley & Talbot 2010) and the training settings by instead using meta-algorithms for the different aspects of the learning settings. Such methods follow a bi-level
optimization framework. optimizing the training settings in the outer level, and traditional model optimization in the inner level (Jenni Favaro 2018). AutoML has focused on achieving better
results via automatic model selection (Deshpande et al. 2021 Feurer et al. 2019) including neural
architecture search (NAS) (Zoph Le 2016 Elsken et al. 2019) Liu et al. 2019). Other important
AutoML topics include hyper- hyper-parameter selection (Li al.] 2017 Akiba et al. 2019) and data augmentation (Cubuk et al. 2018 Lim et al., 2019 Chen et al. 2020 Behl et al. 2020), which closer our settings of optimizing the dataset weights. Since the main signal for model's performance is the final validation loss, which requires full optimization the model for each evaluation, AutoMI. approaches often incur a steep computational costs. Alternatively, other methods
2

---

Published as a conference paper at ICLR 2022
follow alternating optimization of the criteria, such as the work of Ren et al. (2018) that approximates full network optimization with a single SGD step to learn to reweight the training set dynamically. Differentiable AutoM alleviates outer-optimization costs while optimizing the final validation error via differentiable programming, by utilizing proxy losses and continuous relaxation that enable differentiation. Different approaches to differentiable AutoML include differentiable NAS (Liu et al. 2018, Wu et al.] 2019)), data augmentation (Liu et al. 202T; Li et al., 20205), and hyper-paramett optimization (Andrychowicz et al, 2016). The DIVA dataset derivative follows the differentiable AutoML framework by enabling direct optimization of the dataset with respect to the final validation error of the model.
Importance sampling, While our dataset optimization problem may seem superficially similar to importance sampling, the optimization objective different. Importance sampling aims to reweight the training set to make it more similar to the test distribution or to speed up convergence. On the other hand, DIVA objective is to optimizes a validation loss of the model, even this requires making the training distribution significantly different from the testing distribution. Importance sampling methods have a long history in the MCMC machine learning literature where the sampling is conditioned on the predicted importance of samples (Metropolis & Ulam 1949 Liu 2008). In deep learning, importance sampling methods have been studied theoretically for linearly-spaaree
data (Byrd & Lipton 2019) and recently in more generality (Xu et al.) (02T). Furthermore, there
exist many importance sampling heuristics in deep learning training including different forms of hard
sample mining (Shrivastava et al. 2016 Xue et al. 2019 Chang et al. 2017), weighting based on
focal loss (Lin et al. (0177), re-weighting for imbalance. Cui et al. 2019] Huang et al., 2019 Dong
2019). We emphasize that DIVA's optimization of al. 2017) and gradient based scoring (Li al.
the sample weights is not based on a heuristic but is rather differentiable AutoML method driven by optimization of a proxy of the test error. Further, DIVA allows optimization of the dataset weights with respect to an arbitrary loss and also allows for dataset extension computation.
LOO based optimization. Leave-one-out cross validation is well established in statistical learning (Stone 1977). In ridge regression, the LOO model predictions for the validation samples have closed- expression that avoids explicit cross validation computation (Green & Silverman 1993
Rifkin & Lippert 2007) enabling efficient and scalable unbiased estimate of the test error. Efficient
LO0 has been widely used as a criterion for regularization (Pedregosa et al. 2011 Quan et al.
2010 Birattari et al., 1999| Thapa et al. 2020), hyper- prpee-pprameter selection (Hwang & Shim 2017
and optimization (Wen et al. 2008). Most similar to our dataset derivative are methods that: (1) optimize a restricted set of parameters, such kernel bandwidth, in weighted least squares (Cawley
2006:
Hong et al. 2007) (2) locally weighted regression methods (memorizing regression) (Atkeson
al. 1997 Moore et al., 1992). or (3) methods that measure the impact of samples based on LOO
predictions [Brodley & Friedl 1999; Nikolova et al. 2021).
Dataset selection & sample impact measures. Koh & Liang 2017) measure the effect of changes of a training sample weight on final validation loss through per-sample weight gradients, albeit without optimizing the datasst and requiring separate validation set. Their proposed expression for the per-sample gradient, however, does not scale easily to our problem of dataset optimization. In contrast, proposition 5 we introduce an efficient closed-form expression for the derivative of the whole datasets. Moreover, in proposition
g] we show how to optimize the weights with respect to cross- vooss-valiotionn loss which does not require a separate set. In Pruthi ct al. 2020), the authors present sample-imper impact measure for interpretability based on validation set; for dataset extension, Yan et al. (2020) presents a coarse dataset extension method based on self-supervised learning. Dataset distillation and core selection methods aim to decrease the size of the dataset (Wang al. 2018)
by selecting a representative dataset subset (Hwang et al. 2020 Jeong et al.) 2020; Coleman et al.
2019 Joneidi et al. 2020 Trichet & Bremond] 2018 Killamsetty et al. (021). While DIVA is
capable of removing outhiers. in this work we do not approach dataset selection from the perspective
of making the dataset more computationally tractable by reducing the number of samples.
3 METHOD
In supervised learning, we a parametrized model fw( to predict a target output y given input x coming from a joint distribution (x, y) ~ TT. Usually, we are given a training set D = ((x; y: 1 with samples (x. y) assumed to be independent and identically distributed (i.i.d)) according to T.
3

---

Published as a conference paper at ICLR 2022
The training set D is then used to assemble the empirical risk for some per-sample loss L
N
Ltrain (w; D)-
(x:), Mi),
which is minimized to find the optimal model parameters Wpi
WD = argmin Luain (w; D).
The end goal of empirical risk minimization is that weights will also minimize the test loss, computed using separate test set. Nonetheless DD is often biased and differs from the distribution T. In addition, from the perspective of optimization, different weighting of the training loss samples can enable or inhibit good learning outcomes of the task (Lin al. 2017).
Dataset Optimization. In particular, it may not be the case that sampling the training set ' i.i.d. from T is the best option to guarantee generalization, nor it realistic to assume that D is a fair sample. Including in-distribution samples that are too difficult may negatively impact the optimization, while including certain distiiiiiibtttionn examples may aid the generalization on T. It not uncommon, for example, to improve generalization by training on a larger dataset containing samples coming from other sources, or generating ouitor-distribution samples with data augmentation. We call Dataset Optimization the problem of finding the optimal subset of samples, real synthetic, to include or exclude from a training set 'D in order to guarantee that the weights Wp trained on D will generalize much as possible.
Differentiable Dataset Optimization. Unfortunately, a naïve brute-force search over the 2 possible subsets of D is unfeasible. The starting idea of DIVA is to instead solve optimization problem that can however be optimized end-to-end. Specifically, we parameterize the choice of samples in the augmented dataset through weights a; which can be optimized by gradient descent along with the weights of the model. Let (..,..., , aN) be the vector of the sample weights and denote the corresponding weighted dataset by Da. The training loss on Da is then defined as:
general continuous
of non-negative continuous sample
N
(n(w; Da)
Σ ai (x.))))) i=1
Note that if all 's are either 0 or 1, we are effectively selecting only a subset of 'D for training. As we will show, this continuous generalization allows to optimize the sample selection in differentiable way. In principle, we would like to find the sample weights a* argmin, ID (w,) that lead to the best generalization. Since we do not have access to the test data, in practice this translates to optimizing with respect to (unweighted) validation loss L
a' argmin, Lva))
We can, of course, compute a validation loss using a separate validation set. However, as we will see in Section 3.3 we can also use leave-one-out cross-validation loss directly on the training set, without any requirement of a separate validation set.
In order efficiently optimize a by gradient-descent, we need to compute the dataset derivative V Lv (wa)). By the chain rule, this can be done by computing V. However, the training function a - Wa that finds the optimal weights W. of the model given the sample weights may be non-trivial to differentiate or may not be differeentable at all (for example, it may consist of thousands of steps of SGD). This would prevent us from minimizing a end-to-end.
In the next section, we show that if, instead of linearizing the wa end-to-end in order to compute the derivative, we linearize the model before the optimization step, the derivative can both be written in closed-form and computed efficiently, thus giving us tractable way to optimize a.
{a} Semple imporiante
W Opinoolwiggtt
L
Evaluation
Trail perforaacce extimata
V (W
DIVA dotaset derivative
Figure 1: The DIVA dataset derivative computed end-to-end from the final validation loss
4

---

Published as a conference paper at ICLR 2022
3.1 LINEARIZATION
In real-world applications. the parametric model fw (x) is usually a deep neural network. Recent work (Achille et al. 2020 Mu al. 2020) have shown that in many cases, a deep neural network can be transformed to an equivalent linear model that can be trained on simple quadratic loss and still reach a performance similar to the original model. Given model fw (x), let Wo denote an initial set of weights, For example, wo could be obtained by pre-training on a large dataset such as ImageNet (if the task is image classification). Following Achille et al. (020); Mu al. (2020), we consider a linearization (x(xx of the network (x) given by the first-order Taylor expansion of fw (x) around Wo:
N (x) fwo (x) Ww.fo (x) ww).
Intuitively, if fine-tuning does not move the weights much from the initial pre-trained weights Wo- then für (x) will remain a good approximation of the network while becoming linear in w (but still remaining highly non-lineat with respect to the input x). Effectively, this is equivalent to training a linear classifier using the gradients 2; == Ww (x,) as features (Mu et al. 2020).
Although Ali (x) is a linear model, the optimal weights w. may still be complex function of the training data, depending on the loss function used. AAhille et al. (2020) showed that equivalent performance can be obtained by replacing the empirical cross-entropy with the regularized least- squares loss:
LD (w)
Σ sit y + A/|w||2
(3)
where y denotes the one-hot encoding vector of the label yı. In Achille et al. (2020) it is shown that linearized models are equivalent from the standpoint of performance on most standard tasks and classification benchmarks, and better the low-data regime, which is where the problem of "dataset augmentation" is most relevant. The advantage of using this loss is that the optimal weights w can now be written in closed-form as
w'
(Y Swo (X));
(4)
where - 1i...... 2N] is the matrix of the Jacobians be applied with no changes to linearization of the full network, for simplicity in our experiments we restrict to linearizing only the last layer of the network. This is equivalent to using the network as a fixed feature extractor and training a linear classifier on top the last-layer features, that is, 2, fwo (x;) are the features at the penultimate layer.
V Vw.fw, (xx). While our method can
3.2 COMPUTATION OF THE DATASET DERIVATIVE
We now show that for linearized models we can compute the derivative V. wa in closed-form. For the a-weighted dataset, the objective in eq. (B) with L2 loss for the linearized model is written as,
N
Wa = argmin LD(w; D.) argmin
allw 2. - yall² + Al|w||/
(5)
where Vw ww (x) as in the previous section. Note that al/w z, 2 ||w 2 yf
where Z = / az, and y = Vay; Using this, we can reuse eq. (4) to obtain the following closed-form solution for w.:
W, = (Z AI XI -Z D.Y.
(6)
where we have taken D., diag(a). In particular, note that w. now a differentiable function of a. The following proposition gives a closed-for flosed-form expression for the derivative.
Proposition 1 (Model-Dataset Derivative V W.). For the ridge regression problem equation 5 and
W. defined as in equation define
C =
-1
Then the Jacobian of w. with respect to is given by
V w. ZC, ((I ZC, Z Da) Y),
(8)
5

---

Published as a conference paper at ICLR 2022
Figure 2: Examples of the reweighting done by DIVA. (Left) Samples from the FGVC Aircraft classification dataset that are up- up-weighted by DIVA and (Right) samples that are down-weighted because they increase the test error. Down-weighted samples tend to have planes in non-canonical poses, multiple planes, or not enough information classify the plane correctly.
Where we write A o B E RRxxxx for the batch-wise outer product of A E Rexm and B e Bonk along the common dimension k, i... (A o B) iikk = dizbik
The Jacobian V. would be rather large to compute explicitly. Fortunately, the end-to-end gradient (w)) the final validation loss can still be computed efficiently, as we now show. Given validation dataset v the validation loss is:
Leal [ea( EE..DED. ((fwa (xi), y1).
(9)
The following gives the expression to optimize a end-to-end with respect to the validation loss. Proposition 2 (Validation Loss Dataset Derivative). Define as the loss function derivative with respect to the network outputs as,
0e of (f(x1), v1),
0e
L
of (f(xN).
Then the dataset derivative importance weights with respect final validation is given by Vo (Wa) = ZC, x (1
(10)
3.3 LEAVE-ONE-OUT OPTIMIZATION
{a:) It is common in AutoMI. to optimize the hyper-parameteri with respect to separate validation set. However, using a separate validation may not practical in limited data settings, which are main focus of dataset optimization. To remedy this, we now show that minimizing a leave-one-out cross-validation loss that only requires a training set:
instead optimize by
[a)o(a) E (x,)= v:),
(11)
where wa are the optimal weights obtained by training with the loss eq. 5 the entire dataset except for the i-th sample (x,, This may seem counter-intuitive, since we are optimizing the weights of the training samples using validation loss defined the training set itself. It is useful to recall that w minimizes the a- wewghttted Lg loss on the training set (minus the i-th example):
arg min L Lp'(w (w, Da)) = arg min Σ (x))f - 2 + 2/
W α
(12)
w
jott
Meanwhile, minimizes the unweighted validation loss in eq. degenerate solutions for a.
This prevents the existence of
Computing JaLoo naively would require training n classifiers, but fortunately, in the of linear classifier with the Lez loss, a more efficient closed-form solution exists (Green & Silverman, 1993
Rifkin & Lippert 2007). Generalizing those results the case of weighted loss, able to derive the following expression.
Proposition 3. Define
R., Z DD. (Z D.Z (
V D

---

Published as a conference paper at ICLR 2022
Then a-weighted LOOV predictions defined in eq.
(22 admit closed-fome solution:
Ra, R.. dd.
(13)
diag( VD, R...
where diag(A) -(a=)) between vectors is element-wise.
[an1,-... ... Gan) denotes the vector containing the diagonal of A, and the division
Note that the prediction fw., (2.) the i-th sample when training all the other samples is differentiable function of Composing eq. (13 in eq. (II) we compute the derivative Li (a)). which allows us to optimize the cross-validation loss with respect to the sample weights, without the need of separate validation set. We give the closed-form expression for VaLLoo in the Appendix.
3.4 DATASET OPTIMIZATION WITH DIVA
We can now apply the closed-form expressions for VaLva(()) and (al for differentiable dataset optimization. We describe the optimization using Tvaa, but the same applies to L..0.
DIVA Reweight. The basic task consists reweighting the samples of an existing dataset order to improve generalization. This can curate a dataset by reducing the influence of outliers or wrong labels, or by reducing possible imbalances. To optimize the dataset weights, we use gradient descent in the form:
(14) It is important to notice that Lua is an unbiased estimator of the test loss only at the first step, hence optimizing using eq. (14) for multiple steps can lead to over-fitting (see Appendix). Therefore, we apply only 1-3 gradient optimization steps with a relatively large learning rate 17 = 0.1. This early stopping both regularizes the solution and decreases the wall-clock time required by the method. We
a + a a a
initialize so that a; 1 for all samples.
DIVA Extend. The dataset gradient also allows us to extend an existing dataset. Given a core dataset D ((x, yi ) and an external (potentially noisy) data pool = [(x; ûc) N we want to find the best samples from & to add to D. For this we merge DD and & in single dataset and initialize for samples of (so that initially the weighted a such that a; = 1 for samples DD and a; (a) () to find the top k samples of & that have the largest dataset matches D). We then compute Lva negative value of V Lv) i.e.. the samples that would give the largest reduction in validation error if added to the training set and add them to D. This is repeated until the remaining samples in E all have positive value for the derivative (adding them would not further improve the performance).
Detrimental sample detection. The i-th component of Va Laal specifies the influence of the i-th sample on the validation loss. In particular, (Va > 0 implies that the sample increases the validation loss, hence is detrimental (e.g.. it is mislabeled or overly represented in the dataset). We can select the set of detrimental examples by thresholding VaLval
Detrimental(c) = {i: (Va 2c}.
(15)
RESULTS
For our models use standard residual architectures (ResNet) models pre-trained on ImageNet
Deng et al. 2009) and Places365 ZZou et al. 2017). For our experiments on dataset optimization we consider datasets that are smaller than the large scale datasets used for pre-training as we believe they reflect more realistic conditions for dataset optimization. For our experiments we use the CUB-200
(Welinder et al. 2010), FGVC-Aireraft, (Maji et al., 2013), Stanford Cars Krause et al. 2013). Caltech-256 (Griffin et al., 2007). Oxford Flowers 102 Niissack & Zisserman, 2008), MIT-67 Indoor Quattoni & Torralbal 2009). Street View House Number (Netzer et a [21 1). and the Oxford Pets 2012) visual recognition and classification datasets. In all experiments,
the network as a fixed feature extractor, and train a linear classifier on top of the network features using the weighted L2 loss eq. 5D and optimize the weights using DIVA.
Parkki et al.
Dataset AutoCuration. We use DIVA Reweight to optimize the importance weights samples from several fine-grain classification datasets. While the datasets have already been manually curated
7

---

Published as a conference paper at ICLR 2022
Dataset
Original
DIVA Reweight
Chang et al.
2017
Ren et al.
2018
Gain
Aircrafts
57.58
54.64
70.48
81.82 (80.62)
+2.94
Cub-200
39.30
36.93
57.85
72.35 (75.35)
+2.36
MIT 67
32.54
31.27
37.84
64,48 (58.06)
+1.27
Oxford Flowers
20.23
19.16
22.82
48.80 (55.46)
+1.07
Stanford Cars
58.91
56.31
75.87
83.09 (84.50)
+2.56
Caltech-256
23.98
21.29
37.52
58.44 (52.77)
+2.69
Table I: Test error of DIVA Reweight to curate several fine-grain classification datasets. We use ResNet-34 pretrained ImageNet as feature extractor and train linear classifier top of the last layer. Note that DIVA Reweight can improve performance even on curated and noiseless datasets whereas other reweighting methods based on hard-coded rules may be detrimental in this case.
by experts to exclude dustr--distribttion or mislabeled examples, we still observe that in all cases DIVA can further improve the test error of the model (Table D. To understand how DIVA achieves this, in Figure 2 we show the most up- up-weighted (left) and down-weighted (right) examples on the FGVC Aireraft classification task Maji al. (2013). We observe that DIVA tends give more weight to clear, canonical examples, while it detects as detrimental (and hence down-weights) examples that contain multiple planes (making the label uncertain), that do not clearly show the plane, or show non-canonical poses. We compare DIVA Reweight with two other re-weighting approaches: Ren al. (2018). that applies re- weightnngg using information extracted from separate validation gradient step, and Chang et al. (2017). which reweighs based on the uncertainty of each prediction
(dhreshold-closeness weighting scheme). For Ren al. (2018), we set aside 20% of the training samples as validation for the reweight step, but use all samples for the final training (in parentheses). We notice that both baselines under-perform with respect to DIVA on noiseless datasets.
Dataset extension. We test the capabilities of DIVA Extend extend dataset with additional samples the distribution. In Figure f and Table 2 (in the Appendix), we observe that DIVA is able to select the most useful examples and reaches optimal performance generalization error using significantly less samples than the baseline uniform selection. Moreover, we notice that DIVA identifies smaller subset of samples that provides better test accuracy than adding all the pool samples to the training set.
Detrimental sample detection. To test the ability of DIVA to detect training samples that detrimental for generalization, we introduce wrong labels in the dataset. In Section 3.4 we suggest detecting detrimental examples by finding where Lo positive. To verify this, in Figure we plot the histogram of the derivatives for correct and mislabeled examples. We observe that most mislabeled examples have positive derivative. In particular, we can directly classify an example as mislabeled if the derivative is positive. In Figure 3 we report the F1 score and AUC obtained in mislabeled sample detection task using the DIVA gradients.
tub200 samples delected by dive score
549
Dataset
Fl-score (e
AUC
CEN pcare
O
Cub200
0.87
0.98
303
Aircrafts
0.68
0.90
MIT Indoor-67
0.98
Stanford Cars
0.75
01.33
Saltech-256
0.92
01.99
329
Oxford Flowers
0.83
0.97
+0.3
-2.
Figure 3: (Left) Distribution of LOO DIVA gradients for correctly labelled and mislabelled samples in CUB-200 dataset (20% of the samples are mislabeled by replacing their label uniformly at random). (Right) DIVA for outlier rejection. We use DIVA on ResNet-34 network linearization and detect mislabelled samples (outliers) in a dataset present with 20% label noise. Selection is based on Va (Wa))(a)))))))))))))))()())))))) Wa)); >
8

---

Published as a conference paper at ICLR 2022
Al Alrorsts
clhedd25G
033200
212
253
31.
31.
315
Figure 4: DIVA Extend. We show the test error achieved by the model samples selected from a dataset pool using either DIVA Extend (red line) or uniform sampling (blue line). The pool set matches the same distribution as the training set. In all cases DIVA Extend outperforms uniform sampling and identifies subsets of the pool set with better performance than the whole pool. We also note that using only a subset selected by DIVA as opposed to using the whole pool, actually improves the test accuracy.
extend dataset with
Multi-modal learning. Recent modti models such as CLIP (Radford al. 2021 can embed both text and images in the same vector spaces. This allows to boost the performance on few-shot image classification tasks by also adding to the training set textual descriptions of the classes, such as the label name. However, training on label names may also hurt the performance, for example if the label name is not known by the CLIP model. To test this, we create a few-shot task by selecting 20 images per class from the 225 and use DIVA Extend to select an increasing number of labels to add to the training set. In Figure (right) of the Appendix, we show that DIVA can select the beneficial label embeddings to add in order to improve the few-shot test performance. However, when forced to add all labels, including detrimental ones, the test error increases.
Data augmentation. To further test the versatility of DIVA, qualitatively evaluate DIVA Reweight on the task of tuning the probabilities with which we apply a given data augmentation procedure. Let 11.-..,4 'k be set of data augmentation transformations. Let D be the result of applying the data augmentation ' to D. We can create an augmented dataset D = D U DDo U U D'x, by merging all transformed datasets. We then apply DIVA Reweight on D to optimize the weight a of the samples. Based on the updated importance weights we estimate the optimal probability with which to apply the transformation f4 as P = (EieD' az)/(E, a;). In particular we select common data augmentation procedures. horizontal flip and vertical flip, and we tune their probability on the Street View House Number, Oxford Flowers and the Oxford Pets classification tasks. We observe that DIVA assigns different probabilities to each transformation depending on the task (Figure 5 in Appendix): on the number classification task DIVA penalizes both vertical and horizontal flips, which may confuse different classes (such 2 and 5, 6 and 9). On an animal classification task (Oxford Pets) DIVA does not penalize horizontal flips, but penalizes vertical flips since they are out of distributions. Finally, on Flowers classification DIVA gives equal probability all transformations (most flower pictures are frontal so all rotations and flips are valid).
5 DISCUSSION
In this work we present a gradient-based method to optimize a dataset. In particular we focus on sample reweighting, extending datasets, and removing outliers from noisy datasets. We note that by developing the notion of a dataset derivative we are capable of improving dataset quality in multiple disparate problems in machine learning. The dataset derivative we present is given in closed-form and enables general reweighting operations on datasets based on desired differentiable validation losses. In cases where a set-aside validation loss is not available we show the use of the loave-one-out framework enables computing and optimizing dataset "for free" and derive the first closed-form dataset derivative based on the LOO framework.
9

---

Published as a conference paper at ICLR 2022
REFERENCES
Alessandro Achille, Aditya Golatkar, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. Lqf: Linear quadratic fine-tuning. arXiv preprint arXiv:2012.11140, 2020.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-gencration hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference knowledge discovery &amp; data mining, pp. 2019.
Marein Andrychowiez, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. arXiv preprint arXiv: Saov::600744 2016.
Christopher G Atkeson, Andrew W Moore, and Stefan Schaal. Locally weighted learning. Lazy learning, pp. 11-73, 1997.
Harkirat Singh Behl, Atilim Güney Baydin, Ran Gal, Philip HS Torr, and Vibhav Vineet. Autosimu- (quic(q))ckl)) learning synthetic data generation. European Conference on Computer Vision, pp. 255-271. Springer, 2020.
Mauro Birattari, Gianluca Bontempi, and Hugues Bersini. Lazy learning meets the recursive least squares algorithm. Advances in neural information processing systems, pp. 375-381, 1999.
Carla Brodley and Mark Friedl. Identifying mislabeled training data. Journal of artificial intelligence research, 11:131-167, 1999.
Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning? In International Conference Machine Learning, pp. 872-881. PMLR, 2019.
Gavin C Cawley. Leave-one-out cross-validation based model selection criteria for weighted Is-svms. In The 2006 IEEE international joint conference on neural network proceedings, pp. 1661-1668. IEEE, 2006.
Gavin C Cawley and Nicola LC Talbot. On over-fitting in model selection and subsequent selection bias in performance evaluation. The Journal of Machine Learning Research, 11 11:2079-2101 2010.
Haw-Shiuan Chang, Erik Learned-Milln and Andrew McCallum. Active bias: Training more accurate neural networks by emphasizing high variance samples. Advances in Neural Information Processing Systems, 30: 1002-1011, 2017.
Chih-Yang Yang Chen, Che-Han Chang, and Edward Y Chang. Hypemetwork--aseed augmentation. arXiv preprint arXiv:2006.063320 2020.
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. In International Conference Learning Representations, 2019.
Ekin D Cubuk, Barret Zoph. Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint 1rXiv:1805.09501 2018.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2268-9277, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. Ieve, 2009.
Aditya Deshpande, Alessandro Achille, Avinash Ravichandran, Hao Li, Luca Zancato, Charless Fowlkes, Rahul Bhotika, Stefano Soatto, and Pietro Perona. linearized framework and new benchmark for model selection for fine-tuning. arXiv preprint arXiv:2102.00000 2021.
10

---

Published as a conference paper at ICLR 2022
Qi Dong, Shaogang Gong, and Xiatian Zhu. Class rectification hard mining for imbalanced deep learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1851-1860, 2017.
Thomas Elsken, Jan Hendrik Metzen, Frank Hutter, et al. Neural architecture search: A survey. Mach. Learn. Res., 20(55):1-21, 2019.
Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Tobias Springenberg, Manuel Blum, and Frank Hutter. Auto-skleam: efficient and robust automated machine learning. In Automated Machine Learning, pp. 1 113-134. Springer, Cham, 2019.
Peter Green and Bernard W Silverman. Nonparametric regression and generalized linear models: roughness penalty approach. Cre Press, 1993.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
Xin He, Kaiyong Zhao, and Xiaowen Chu. Automl: A survey of the state-of-the-arr, Knowledge-
Based Systems, 212:106622, 2021.
Xia Hong, Sheng Chen, and Chris J Harris. A kernel-based two-class classifier for imbalanced data sets. IEEE Transactions on neural networks, 18(1):28-41, 2007.
Khalid M Hosny, Mohamed A Kassem, and Mohamed M Foaud. Skin cancer classification using deep learning and transfer learning. In 2018 9th Cairo International Biomedical Engineering Conference (CIBEC), pp. 90-93. IEEE, 2018.
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Deep imbalanced learning for face recognition and attribute prediction. IEEE transactions pattern analysis and machine intelligence,
2019.
Changha Hwang and Jooyong Shim. Geographically weighted least squartes-spportt vector machine.
Journal of the Korean Data and Information Science Society, 2811)2227- 2017.
Myunggwon Hwang, Yuna Jeong, and Wonkyung Sung. Data distribution search to select core-set for machine learning. In Proceedings of the 9th International Conference on Smart Media &amp; Applications (SMA 2020), Jeju, Korea, pp. 17-19, 2020.
Simon Jenni and Paolo Favaro. Deep bilevel learning. In Proceedings of the European conference on computer vision (ECCV), pp. 618-633, 2018.
Yuna Jeong, Myunggwon Hwang, and Wonkyung Sung. Dataset distillation for core training set construction. 2020.
Mohsen Joneidi, Saced Vahidian, Ashkan Esmaeili, Weijia Wang, Nazanin Rahnavard, Bill Lin, and Mubarak Shah. Select to better learn: Fast and accurate deep learning using data selection from nonlinear manifolds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2819-7829 2020.
KrishnaTeja Killamsetty, Durga Sivasubramanian, Baharan Mirzasoleiman, Ganesh Ramakrishnan, Abir De, and Rishabh K. Tyer. GRAD-MATCH: A gradient matching based data subset selection for efficient learning. CoRR, abs/2103.00123, 2021. https::: http://wwww...o//////////0
03123
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning, pp. 1885-1894. PMLR, 2017.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- ane-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
Buyu Li, Yu Liu, and Xiaogang Wang. Gradient harmonized single-stage detector. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 8577-8584, 2019.
11

---

Published as a conference paper at ICLR 2022
Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Rethinking the hyperparameters for fine- ane-tuning. arXiv preprint arXiv:2002.11770, 2020a.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-basee approach to hyperparameter optimization. The Journal of Machine Learning Research, 18(1):6765-66) 6816, 2017.
Yonggang Li, Guosheng Hu, Yongtao Wang. Timothy M. Hospedales, Neil Martin Robertson, and Yongxin Yang. DADA: differentiable automatic data augmentation. 2020b.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment, arXiv preprint arXiv: 90.99000003999 2019.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Aoming Liu, Zehao Huang, Zhiwu Huang, and Naiyan Wang. Direct differentiable augmentation search. Xrviv preprint vrXiv:2104.044 04282, 2021.
Chenxi Liu, Liang-Chine Chich Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L Yuille, and Li Fei- Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 82-92, 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search, arXiv preprint 1rXiv: 09055, 2018.
Jun Liu. Monte Carlo strategies in scientific compuring. Springer Science & Business Media, 2008.
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aireraft. Technical report, 2013.
Nicholas Metropolis and Stanislaw Ulam. The monte carlo method. Journal of the American statistical association, 44(247)3335-341 1949.
Andrew W. Moore, Daniel J. Hill, and Michael P. Johnson. An empirical investigation brute force to choose features, smoothers and function approximators. In Computational Learning Theory and Natural Learning Systems. MIT Press, 1992.
Romain Mormont, Pierre Geurts, and Raphaël Marée. Comparison deep transfer learning strategies for digital pathology. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pp. 2262-22712 2018
Fangzhou Mu, Yingyu Liang, and Yin Li. Gradients as features for deep representation learning. arXiv preprint a arXiv:2004.05529 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Natalia Nikolova, Rosa M Rodríguez, Mark Symes, Daniela Toneva, Krasimir Kolev, and Kiril Tenekedjiev. Outlier detection algorithms over fuzzy data with weighted least squares. International Journal of Fuzzy Systems, pp. 1-23, 2021.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics &amp; Image Processing, pp. 722-729. IEEE, 2008
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Fabian Pedregosa, Gaël Varoquaux. Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathicu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-leam: Machine learning in python. the Journal of machine Learning research, 12:2825-2830. 2011.
12

---

Published as a conference paper at ICLR 2022
Garima Pruthi, Frederick Liu, Mukund Sundararajan. and Satyen Kale. Estimating training data influence by tracking gradient descent. arXiv preprint arXiv:2002.08488, 2020.
Tingwei Quan, Xiaomao Liu, and Qian Liu. Weighted least squares support vector machine local region method nonlinear time series prediction. Appl. Soft Comput., 10(2):62--666, March 2010. ISSN 1568-4946.
Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 420. IEEE, 2009.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual 2021.
models from natural language supervision. arXiv preprint arXiv:2103.000
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International Conference Machine Learning, pp. 4334-4343. PMLR, 2018.
Edmar Rezende, Guilherme Ruppert, Tiago Carvalho, Fabio Ramos, and Paulo De Geus. Malicious software classification using transfer learning of resnet-50 deep neural network. In 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 1011-1014 IEEE, 2017.
Ryan M Rifkin and Ross A Lippert. Notes on regularized least squares. 2007.
Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard example mining. In Proceedings of the IEEE conference on computer vision and pattem recognition, pp. 761-769, 2016.
Mervyn Stone. Asymptotics for and against cross-validatte Biometrika, pp. 29-35, 1977.
Mishal Thapa, Sameer B Mulani, and Robert W Walters. Adaptive weighted least-squares polynomial chaos expansion with basis adaptivity and sequential adaptive sampling. Computer Methods in Applied Mechanics and Engineering, 360: 12759, 2020.
Remi Trichet and Francois Bremond. Dataset optimization for real-time pedestrian detection. IEEE access, 6:7719-7727, 2018.
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint 181iv: 10059,, 2018.
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report ONS-TR-2010-001. California Institute of Technology, 2010.
Wen Wen, Zhifeng Hao, and Xiaowei Yang. heuristic weight-setting strategy and iteratively updating algorithm for weighted least-squares support vector regression. Neurocomputing, 71 (16- (16-18):3096-310) 2008
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10734-10742, 2019.
Da Xu, Yuting Yc, and Chuanwei Ruan. Understanding the role of importance weighting for deep learning. In International Conference on Learning Representations, 2021. URI. https
//openreveew.o net forum?id= Www. eeHMM
Jiabin Xue, Jiqing Han, Tieran Zheng, Jiaxing Guo, and Boyong Wu. Hard sample mining for the improved retraining of automatic speech recognition. arXiv preprint 190400900.00033 2019.
Xi Yan, David Acuna, and Sanja Fidler. Neural data server: A large-scale search engine for transfer learning data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattrr 2020.
Recognition, pp.
13

---

Published as a conference paper at ICLR 2022
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning, arXiv preprint 1.1017,6 2016.
14


Reviews:
[{"review_id": "eAjhu7SFulc", "reviewer": "Reviewer_9tXK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The paper is well-written and easy to follow.\n- Learning sample weights without a separate validation dataset is interesting and useful since it enables us to use the full dataset as training data. Although there are several methods to learn sample weights as described in section 2, they assume a separate validation dataset.\n- The authors derive the closed-form expression of the derivative of the loss w.r.t. sample weights and it is attractive in terms of efficiency.\n\nCons.\n- As described in the Introduction, the proposed method assumes pre-trained models to obtain the closed-form expression of the derivative, which might restrict its applicability in some cases. Is it possible for the proposed method to optimize model (neural network) parameters and sample weights simultaneously? \n- Although the authors conducted experiments in multiple tasks, it is not sufficient. For example, comparison methods are a little naive. Comparison with other methods was conducted only in the dataset autocuration experiment.\n\nOther comments and questions:\n- Although the authors explicitly derive the Jacobian of $w_{\\alpha}$ in eq. (8), is it sufficiently fast by using automatic differentiation of $w_{\\alpha}$ (eq. (5))?\n- How important is the initialization of sample weights $\\alpha$?\n- Although the proposed method learns sample weights without additional validation data, is it also possible to tune its hyperparameters such as optimization iteration numbers without the validation data?\n- It is better to describe how many trials were performed in each experiment. \n- It is unclear which loss was used ($L_{val}$ and $L_{LOO}$)  in the main paper.\n\nTypo:\n- second line of Introduction: how to select from from -> how to select from.\n- $z_i$ -> $x_i$ in eq. (13)?\n- in Figure 4, the meaning of blue and red lines would be opposite\n", "summary_of_the_paper": "This paper proposes a method for dataset optimization that learns sample weights without a separate validation dataset.\nBy using squared loss and linearization around a pre-trained model, derivative w.r.t. the sample weights can be obtained as closed-form, which realizes efficient end-to-end learning. In addition, this paper uses leave-one-out cross-validation (LOOCV) loss instead of validation loss on a separate dataset, which enables to conduct evaluation without a separate validation dataset.\nThe experiments are conducted with multiple tasks (Dataset AutoCuration, Dataset extension, detrimental sample detection, multi-modal learning, and data augmentation).", "summary_of_the_review": "Although the motivation of this work is interesting and the paper has some technical novelty, experiments can be improved.\nThus, I'm slightly leaning towards the reject side at this moment.\n\n---After rebuttal---\nThanks for the response.\nI think there is room for improvement in the experiment (comparison with existing methods for each task), \nbut considering the promise of the proposed method in a number of tasks and other reviews, I decide to increase my score to 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "auwuiOXxMdF", "reviewer": "Reviewer_2K7t", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positives\n+ The problem formulation and closed form solution to the dataset derivative are potentially interesting and generally useful for deep learning\n+ The work achieves good results with respect to other previous works, and experiments on dataset expansion are very interesting\n\nNegatives\n\nMy concerns with the work are to do with the methodology and experimental design choices are around comparison with sufficient baselines.\n\nMethodology: The proposed experiments in the paper all use the deep network as a feature extractor and the last linear layer of the network to make decisions on classification. Thus, we are in a pretraining regime (where there is no dataset derivative being used) and a downstream task where we extract the pretrained features and do linear classification. Given this, the experimental / methods section (sec. 3) is very terse and complicated since it talks about linearizing the model etc. where the model actually that is used is infact already linear. I understand it might have been written that way with generality in mind, but the experiments should then also reflect that and demonstrate that the model / approach does indeed work for the deep learning case (say training from raw pixels) or something more general like that. \n\nComparison to baselines: A number of important comparisons to baselines / discussion around those approaches seem to be missing in this current work. \n\n1) The paper mentions that it is difficult to compare to Koh and Liang (2017) which directly addresses the problem of outlier detection and also the problem of dataset reweighting for better genrealization by estimating the influence of upweighting a data sample by epsilon. The paper mentions that the work was not compared to because of computational reasons, but I am struggling to see how that work is much more expensive than the current work or the work of Ren et.al. (2018). For example, given the task of dataset reweighting, it appears to me that the work of Koh and Liang requires the following to be computed: 1) the hessian of the data at the optimum solution on the training set (unweighted), 2) the gradient of the loss with respect to the parameters at the training datapoints, and 3) the gradient of the loss with respect to the parameters at the test datapoints. 1) and3) are required for the proposed approach as well, and 2) is not that expensive to compute for a Mean Square Error Regression. Secondly, unlike the claim in the paper Koh and Liang can be used to do dataset reweighting by computing the influence of each training datapoint on the validation set, and keeping around some top-K training samples and checking how well one does based on them.  (*)\n\n2) Another important paper in this literature which is missing (and should really serve as a baseline) is that of [A]. I think this work should definitely be compared to the current framework. (*)\n\n3) As far as comparison to Ren. et.al. (2018) is concerned, the key difference is that the current work proposes to linearize the classifier function (whereas Ren et.al. (2018)) use a linear approximation to the actual loss function, since they do gradient descent whose variational form is essentially \\min_{\\theta’) L(\\theta’) + || \\theta - \\theta’||_2^2 which is then approximated as \\min_{\\theta’} L(\\theta) + d/d\\theta (L(\\theta)) (\\theta - \\theta’) + ||\\theta - theta’||_2^2, solving which gives us the update equation for SGD. The second difference is that while Ren et.al. (2018) solve the update for the dataset in a single step and the inner loop problem in a single step (not fully), the current paper considers a linear approximation to the original problem and solves the two problems exactly. It would be nice to understand given this where exactly the gains of the work over that of Ren. et.al. (2018) come from. This can be achieved in two ways:\n\n1) Solving the current problem sub-optimally by only doing one step of gradient descent and computing inexact solutions in the linear case, even though we know exact solutions (derived in this work exist). This would help to contextualize how much benefit we have from the current approach.\n2) Solving a problem that is non-linear (as opposed to fixed feature extraction and linear classification) such that the linear approximation using taylor series actually comes into play in the sense that there are higher-order terms which are being ignored. In this setting it would be important to compare the current approach to Ren. et.al. This would tell us whether one benefits in this problem domain by solving the correct problem approximately (which Ren. et.al. do) or solving an approximate problem correctly (which this paper does) and what the tradeoff is between these modes. (*)\n\n[A]: Lorraine, Jonathan, Paul Vicol, and David Duvenaud. 2019. “Optimizing Millions of Hyperparameters by Implicit Differentiation.” arXiv [cs.LG]. arXiv. http://arxiv.org/abs/1911.02590.", "summary_of_the_paper": "The paper proposes a novel method for computing the derivative of the weights assigned to datapoints in a dataset with respect to the generalization error of a learning task. The core idea being that one can take a pretrained model, and compute its derivative with respect to a fine-tuning task/ dataset and understand which datapoints are important and which datapoints are not important. This can then be used for tasks such as 1) outlier detection, 2) dataset expansion, and 3) dataset reweighting. Comparisons of the proposed work to other previous work reveals that the current work achieves better results on dataset reweighting. \n", "summary_of_the_review": "This is a promising method to the well studied problem of identifying the weighting of the datapoints in a learning task that aid generalization on a given task. The current method while promising is missing some vital comparisons which limit the ability of a reader to assess the true impact of the work, along with experiments which make a lot of the methods sections and details in it seem like overly complicated detail. Given this I do not think the current paper is ready for publication. Important points for the rebuttal are marked with a (*).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WASIgR9NjKn", "reviewer": "Reviewer_7aEq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, I think this is a good paper.\n\nI think it addresses a longstanding and open question -- how do we know we are feeding our models the right data during training to best set them up for success?  This paper adds a new tool in the arsenal for figuring out which data is best for a given learning task.  I believe the focus of this paper would be of interest to a broad audience.\n\nThe discussion section ends very abruptly.\n\nI would like to see this method extended to apply in cases where the model class in question is not a DNN.  The Dataset Extension, Curation, and Reweighting capabilities would also be of great use for models trained on tabular data.\n\nOne thing that feels missing in the paper is a good real-world example of how this method could make positive impact.  i.e. an immediate thought that comes to mind could be a scenario where multiple domain experts label data, but inter-rater reliability may be low - using DIVA to downweight samples which ultimately confuse the model seems useful in a case like this, which often arises in clinical contexts.\n\nA relationship that I would like to see the authors explore in the discussion section is overfitting of the model.  DIVA reduces the weights of hard/confusing images (figure 3.4).  An alternate hypothesis for why the updated model performs better on test data is that the easy/canonical images can be adequately classified by a simpler model.  Side-by-side, the simple model trained on easy data may generalize better than a complex model trained on hard data.  Ruling this out would make DIVA appear stronger, as this would mean it is impossible to improve test set accuracy by simply feeding a model the most canonical images during training.", "summary_of_the_paper": "The authors define a data set derivative, which can be used to update weights of individual samples in training data such that a trained model may increase its performance on test data.  In addition to re-weighting samples within a training data set, the authors also describe how the same techniques can be used to select new data points from a pool of potential candidates and used to estimate which samples are likely to be mislabeled in data.", "summary_of_the_review": "I think the paper is in good enough shape to warrant publication with only minor changes.  I believe the focus of this paper would be of sheer interest to a broad audience.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "PeWmDlrrG_0", "reviewer": "Reviewer_3qBW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**List strong and weak points of the paper. Be as comprehensive as possible.**\n\n**Pros:**\n\n* The introduction is well written and motivates the necessity of DIVA and discusses its limitations.\n* the related work sectiton does a great job at showing the similarities and differences compared to other approaches.\n* Data quality is often a major problem and solved with overparametrized models. Tackling this issue with a closed-form solution is great.\n* I think it's an important contribution to raise awareness for the limitations of only one predefined validation set. For computational reasons, deep learning is often not using the previously well established concept of k-fold cross-validation, and this paper offers an attractive alternative for more robust evaluation of neural networks. \n\n**Cons:**\n\n* There are a few question left open, see questions below.\n* The empirical behavior off DIVA is only shown on image data. We do not know how it performs in other domains or types of datasets.\n* Minor/Typos:\n  * Introduction line 2: \"from from\"\n  * Introduction last paragraph before summary: \"LQF\" should be defined\n  * Figure 5 (right): \"Test Error (\\\\%)\" remove \"\\\\\"\n  * Equation 13: \",.\" -> only \",\" should be there?\n\n**Clearly state your recommendation (accept or reject) with one or two key reasons for this choice.**\n\nI think this is a valuable and interesting contribution and I vote for accept. Going beyond a single fixed validation in deep learning set is important and often neglected and DIVA proposes an efficient approximation for a better estimate of the generalization error. \n\n**Ask questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment.**\n\n\n* it would be helpful to have maybe a piece of pseudocode describing how the application of DIVA works. \n  * Is it necessary to train the linear layer?\n  * Does this layer represent the classification task?\n* The paper only shows results for classification tasks. Is it possible to apply DIVA for regression or other tasks in general? Would be helpful to add some clarification and at least mention other tasks.\n\n* the authors mention that \n  \n  *For our experiments on dataset optimization we consider datasets that are smaller than the large scale datasets used for pre-training as we believe they reflect more realistic conditions for dataset optimization*\n\n  it seems that computational efficiency could also be an important factor with large data sets, right? \n\n* the idea of efficient leave-one-out errors is also well known in the kernel machine literature and it seems that some of the results are similar to the results for efficient leave-one-out crossvaliation for e.g. kernel ridge regression or kernel logistic regression (see e.g. [Cawley and Talbot, \"Efficient approximate leave-one-out cross-validation\nfor kernel logistic regression\", Machine Learning Journal, 2008](https://link.springer.com/content/pdf/10.1007/s10994-008-5055-9.pdf)). I don't think this similarity weakens the contribution of this submission, but it could be helpful to comment on the relationship between the dual variables in kernel machines and the importance weights in the DIVA approach\n\n* sorry if I missed that, but would it be possible to include a direct comparison of the approximation and the true LOO error? The SGD based optimization seeems a bit unconventional, with the step size and very early stopping. Maybe some of the solutions from the kernel literature or matrix algebra, e.g. using cholesky decompositions, would be helpful for more efficient optimization of DIVA? \n\n**Provide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.**\n\nA detailed description how to apply DIVA would improve replicability.\n\nAlso it would be helpful to see the computational complexity of DIVA as a function of the data set size. \n\nAnd I'm not sure I fully understand the argument with DIVA not needing a validation set. Leave-one-out cross-validation (LOO CV) is using validation data, it's just split differently. Ideally every model should report k-fold cv'd results, and the estimate of the generalization error will become better the larger k. The fact that often researchers use a predefined validation set is probably more due to the high computational complexity of doing CV properly. So I think I wouldn't emphasize that aspect too much as an advantage of DIVA, that it's applicable when there is no validation set - k-fold CV/jackknife/LOO would also help in that case.", "summary_of_the_paper": "The paper introduces an approach to calculate the derivative of data samples' influence on the validation error. For this, the authors use a recent model linearization method and leverage closed-form leave-one-out loss calculation for linear models. Besides the theoretical derivation, they present consistent improvements in dataset extension, re-weighting, outlier rejection and automatic aggregation of multi-modal data. ", "summary_of_the_review": "This paper is a valuable and interesting contribution. It raises awareness for the importance of going beyond a single validation split for evaluation of neural networks and presents an interesting closed form solution for the leave-one-out (LOO) error that is, to the best of my knowledge, novel. The similarities of the proposed approach with efficient LOO cross-validation approaches for kernel machines suggests that there could be synergies that would allow to improve the optimization procedure of DIVA. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has been independently reviewed by four expert reviewers. Two of them recommended straight acceptance, one of them assesses this work as marginally acceptable after increasing their score as a result of the author's rebuttal, and the last reviewer considers this paper marginally below the acceptance threshold. While the reviewers agree on the importance of the  targeted problem and relative novelty of the presented work, the main points of criticism involve empirical evaluations - its methodology, experimental design, missing relevant and important comparisons. Since the authors have addressed most of those concerns in their rebuttal, I am leaning towards recommending acceptance of this work for ICLR.
Decision: Accept (Poster)