Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper tackles the problem of finding subsets of a data set on which to apply data augmentation, such that the performance is as close as possible to augmenting the whole data set. First, the paper models data augmentation as additive perturbations of the data and provides a theoretical analysis of the effect of perturbations on the singular values of the network Jacobian. Then, it proposes a method to select a subset of the data that aims at matching the alignment of the Jacobian of the fully augmented data with the residual vector. Finally, the paper evaluates the proposed method on several data sets and architectures, comparing with selecting random subsets and max-loss, a previously proposed method.
strengths_and_weaknesses: 
As main strength, I highlight that this paper provides a sound theoretical analysis of how additive perturbations of the inputs impact the eigenspace of the Neural Tangent Kernel (NTK), or the singular values of the model Jacobian. The analysis concludes that input perturbation change proportionally the smaller singular values, while the larger singular values are affected less.

Another strength of the paper is the breadth of the empirical evaluation, in which CIFAR-10, SVHN, Caltech256, Tiny ImageNet and ImageNet are used as benchmarks, though with large differences in the depth of the analysis across data sets.

Finally, regarding strengths, I would like to point out that the paper is clearly written, it is mostly easy to follow and the structure is reasonable.

Regarding weaknesses, my first concern is about the generality of modelling data augmentation as additive perturbations, following the work by Rajput et al. (2019). While I acknowledge that simplifying data augmentation as something that can be easily modelled mathematically is currently necessary, and I do not reject the significance of the results of this paper because of this assumption, I do argue that it is important to also acknowledge the limitation of this modelling assumption. In my opinion, this limitation is barely discussed and, on the contrary, the paper argue that the results can be extended to more complex augmentations, while this is barely analysed in the paper, besides a short section and one figure in the appendix. Data augmentation comprises a range of complex transformations that, in the case of natural images, encode properties of the human visual system. Take, for instance, the case of horizontal flips, which largely preserve the label of the image by applying a transformations that is fundamentally different to additive perturbations. However, the paper makes this strong modelling assumption but then uses the term "data augmentation" to talk about the conclusions from using additive perturbations. As a comparison, Rajput et al. (2019) do acknowledge this limitation and discuss the need to extend the theoretical analyses to practical augmentation methods.

Another concern has to do with the overall motivation for the proposed method, that is for the need to select subsets of data on which to apply data augmentation. On the second sentence of the abstract, the authors write that "the most effective augmentation techniques become computationally prohibitive for even medium-sized datasets". This statement is extended in the introduction, where the authors cite several papers on so-called _automatic data augmentation_. First, these methods are indeed disproportionally expensive in computational terms, while providing only marginal improvements (in the best case) with respect to traditional, simple, cheap data augmentation techniques (Pérez and Wang, 2017). The paper also mentions that performing data augmentation increases "the training time by orders of magnitude". However, it has been shown that the largest benefits of data augmentation are achieved by very simple and cheap transformations such as horizontal flips and translations, and more complex transformations only marginally improve the performance further (Hernández-García and König, 2018). Such simple transformations largely increase the performance without increasing the training time. By way of illustration, the previously cited paper shows that training with 50 % of CIFAR-10 _and_ data augmentation on the full available set achieves more than 95 % of the _full_ accuracy in about half the training time. Therefore, given this result, is it really necessary to develop methods to select subsets given a full data set? In fact, the present paper does not stick to so-called automatic augmentation methods, but also studies CutOut, which is computationally very cheap to perform. Again, if CutOut is considered a strong augmentation method, then what is the need for improving the computational efficiency?

Related to the concerns discussed above, my last main concern is related to the effectiveness of the proposed method given the results and the degree to which the conclusions and claims reflect the results. For example, the conclusion from Table 1 is that "the improvement from augmenting the coresets is significantly larger than random data points". However, a close inspection of the results in the table casts some doubts about the strength of the statement. On CIFAR-10 and ResNet20, the improvement of augmenting the coresets with respect to no augmentation (right-most column) is better than augmenting random sets with 0.1 and 0.2 % subsets. With 0.5 and 1 % subsets, the improvement is about the same. On Caltech256 with ResNet18, the improment on coresets is actually worse than on random sets with 5 and 10 % of the data, and about the same in the rest of the cases. Furthermore, the trend is the opposite than on CIFAR-10, which is confusing. Therefore, I would humbly suggest to adjust the claims regarding this set of results. In particular, these sentences:

* "the augmented coresets outperform augmented random subsets by a large margin": this is not true, as discussed above.
* "the improvement from augmenting the coresets is significantly larger than random data points": this is not true either, by extension.
* "We see that while the non-augmented coreset performs worse than random in most cases": this is also not true, by inspecting the table.

In Table 4, we also see that coresets provide better improvement on average than max-loss, but not consistently in all cases and by a small margin. In particular, the averages across data sets and architectures with 5, 10 and 30 % sets are: 

|          | 5 %    | 10 %   | 30 %   | 
|----------|--------|--------|--------|
| max-loss | 33.2 % | 45.7 % | 74.8 % |
| coresets | 34.7 % | 49.7 % | 76.6 % |

While coresets indeed provide better improvement on average, the difference is very small, especially if we take into account that that these numbers are the improvement of the performance from no augmentation to full augmentation. The small margin becomes relevant since the speed up of coresets is smaller than that of max-loss, but the authors claim that "[w]hile our method yields similar or slightly lower speed-up to the max-loss policy and random approach respectively, our resulting accuracy outperforms these two approaches". I would kindly suggest to adjust this claim too.

References

* Pérez and Wang. [The effectiveness of data augmentation in image classification using deep learning](https://arxiv.org/abs/1712.04621). 2017.
* Hernández-García and König. [Data augmentation instead of explicit regularization](https://arxiv.org/abs/1806.03852). 2018
questions: Besides the concerns discussed in the previous section, I would like to ask the authors whether they could elaborate on how the theoretical analysis and the conclusions contribute to our understanding of how data augmentation (or additive perturbations) impact overfitting and generalisation. In particular, I refer to Section 4.3, where this conclusions are mentioned. However, I either do not understand completely the relationship between the results and overfitting and generalisation, or the connection is just weak.

I have another comment or suggestion regarding a sentence in the introduction that I find a bit misleading: 

"We demonstrate the effectiveness of our approach applied to training ResNet18, ResNet20, ResNet32, ResNet50, Wide-ResNet on CIFAR10, CIFAR10-IB, SVHN, Caltech256, TinyImageNet, and ImageNet compared to random and max-loss baselines"

This gives the impressions that multiple architectures are tested all on multiple data sets, while in fact, only one or maximum two architectures are tested per data sets and the evaluation on some of the data sets, for instance ImageNet, is quite limited.

Finally, a few minor typing errors: 

* In Table 1, the heading of the first column is Model/Dataset, but it should read Dataset/Model.
* In page 8, "We use [37] for..." reads weirdly.
* I would use "Table" instead of just "Tab.", which reads more easily and uses just one more character.
limitations: I have discussed the limitations above. In particular, regarding limitations, I kindly encourage the authors to more explicitly and openly discuss the limitations of the paper. I believe it is a good paper, the results are interesting and relevant, but it is important to discuss, in my opinion, the limitation of modelling data augmentation as additive perturbation, the fact that simple data augmentation methods are effective and computationally cheap and that the empirical results of coresets provide only moderate benefits compared to other methods.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper first studies the relationship between data augmentation and the singular values of Jacobian in an over-parametrized network. Then, the authors propose a method to select a subset of data augmented training data to speed up the training of the neural networks. Empirically, the proposed approach achieves improved results than the random data argumentation subset selection and speeds up the training on standard image classification tasks.
strengths_and_weaknesses: The topic of the paper is interesting and relevant to the Neurips community. The technical details of the paper look correct, and the authors properly cite important relevant works. The empirical analysis in the paper is detailed, and the authors provide code to reproduce experiments in the paper. It is convincing that the proposed approach can select a subset of augmented data helpful in improving the training accuracy (with reduced dataset) and speed up training.

However, I am still unclear about the core motivation and application of the paper. More specifically, in what cases selecting a subset of the training data is useful (especially when using the full augmentation typically achieves the best test accuracies)? It would be helpful if the authors provided a specific use case of the proposed approach. Moreover, the authors did not fully describe the limitations of their proposed approach. Does the proposed approach work for smaller networks (under-parameterized) or when the dataset size is small? Discussing these limitations in the paper would be helpful to the readers.
questions: Comments:
* As mentioned above, I wonder if there is a useful application for their proposed approach.
* CIFAR-10 baseline (with standard data augmentation) for ResNet20 seems to be lower than the reported values in the literature, where you can above 95% test accuracy.

Minor comments:
* In Equation 3, the eigenvalue $\lambda$ is not defined.
* In Algorithm 1, it would be helpful if variables $V_c$ and $T_i$ were defined.


** After Rebuttal **
I acknowledge that I have read the author's response and other reviews. Thank you for the clarification on the use-cases of the proposed approach. As noted by other reviewers, I believe that including these points more clearly would strengthen the paper. 

Regarding the accuracy of the CIFAR-10 baseline, I apologize for the confusion. I thought 95 percent accuracy can be obtained on ResNet-20 on CIFAR-10. 
limitations: I believe that the work does not have any potential negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper provides analysis to show that the effect of data augmentations can be explained by looking at the singular values of the network jacobian. Based on this, an algorithm is proposed to find a subset of the data points on which data augmentations can be done to best approximate the effect of doing augmentations on the whole dataset. Experiments show that the proposed algorithm works better than selecting the subset randomly or according to max-loss. 
strengths_and_weaknesses: Strengths:
1. I like the analysis part which focuses on the effect of data augmentation on the singular values of the network jacobian, which is a novel observation and motivates the proposed selection algorithm.
2. Extensive experiments show that the proposed selection algorithm works better than the random selection and the selection according to max-loss.

Weaknesses:
I have two major concerns:
1. The problem itself is not well motivated enough, i.e., why do we need to select a subset to perform data augmentation. Although some augmentation techniques are computationally expensive, there are many ``easy'' augmentations (that are also commonly used like random cropping, flipping, permutation, etc.) and these augmentations are not computationally prohibitive.
2. Even if we leave the motivation concern aside, the current experiments do not support the authors' claim. The authors claim that by selecting a subset to perform augmentations, we can speed up the overall training time by several times. Ideally the experiments should compare the training time of (1) train the network using the whole dataset while performing augmentations on all data and (2) traing the network using the whole dataset while only performing augmentations on the selected subset. If the training of (2) is several times faster than (1), then the authors' claim is well supported. However, the current experiments only show that training on a selected subset is much faster than training on the whole dataset. This speedup is a result of fewer training steps and has nothing to do with the data augmentation.


questions: Please see Weaknesses
limitations: The authors included some discussions about the limitations.
review_title: null

Meta-review: metareview: This work demonstrates that it can be sufficient to apply data-augmentation only on a core-set of the data to achieve accuracy comparable to augmenting the full dataset. These findings are supported by theoretical arguments in the NTK framework, and by empirical evaluation. The proposed method can provide a trade-off between the training time and the accuracy when data-augmentation is costly. 

The reviewers noted that the restriction to additive perturbations might be a limitation of the proposed approach and suggested to incorporate a discussion of these limitations and possible use cases in the final version of the paper (such as also promised in the rebuttal).
confidence: Less certain
Decision: Accept