Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors study both theoritically and empirically the sample efficiency of training with data augmented samples on a set of augmentations such as translation, shearing, rotation and zoom. They derive mathematical tools for analytically studying data augmentations and its effects on loss. As a result they derive an explicit regularizer that produces a model that is the equivalent of using DA on non-trivial transformations (the first of its kind). 

The authors also study the number of DA samples needed for a model to properly absorb the information given by a data augmentation strategy on a particular image, as well as the number of DA samples needed in a setting where we have thousands of unique images for a model to fully absorb such information. They find that for a single image that number is close to 10 ^ 4, and for datasets with thousands of images, it can be around 50 data augmented versions of each image. 

The authors also study the variance of a given loss and architecture applied on a particular data augmentation strategy and rediscover a popular reqularizer called TangentProp.
strengths_and_weaknesses: Originality:

The investigation is both interesting and insightful, and follows a novel direction in how it conducts an analytical study on data augmentations and attempts its best to come up with a theoretical framework for studying the effects of data augmentation on models and losses.

Quality:

The technical rigour of the work is of high quality, but the writing can vary in quality, sometimes making it harder to follow what the authors are trying to communicate. 

Experiments are strictly done on MNIST, which is a majour concern as to whether any of the empirically supported conclusions hold.

Clarity:

The writing quality of the paper is mediocre, but the authors have definitely excelled in their mathematical communication in the paper. Furthermore, the figures presented would benefit from additional textual descriptions of what they portray. 

Significance:

The conclusions regarding the sample efficiency of data augmentation strategies can be useful for people in the field if applied on more datasets and models. Furthermore, I am not sure how TangentProp can help improve machine learning models in any additional ways as a result of this study. 

The significance of the study is quite uncertain. 
questions: I recommend the following:
- Repeat your experiments on more datasets and models to see whether any of the conclusions related to them hold.
- Better motivate/communicate what your findings mean for future research and/or current machine learning practices. 
- Rewrite parts of your paper with an aim towards more clarity and more conciseness. 
limitations: Technical limitations were discussed above. I don't see any potential negative societal impacts.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper provides a theoretical analysis of data augmentation (DA) methods.
- It introduces a novel integral transform formulation (called Data-Space Transform (DST)) for describing a class of DA, e.g. translation or rotation.
- It provides an explicite formula for expectation and variance of transformed images
- It provides an explicite formula for regularisation that has the same effect on the loss as training with infinite DA samples
- It provides a theoretical connection with TangentProp

strengths_and_weaknesses: Strengths:

The paper provides some insights of the effect of DA:
- How it has the same effect as an explicite regulariser
- It can explicitly recover the expectation and variance of the transformed samples
- It provides interesting connections between the variance of the MC estimate and the model’s Jacobian matrix, which leads to the TangentProp regulariser (that minimises the variance).

The theoretical analysis looks original to me.

The paper discusses a very important topic, as understanding the effects of DA and regularisation could help design systems that generalise better.

Weaknesses:

The presentation could be clearer:
- fig 1 does not help understanding. One has to zoom a lot to see the details. I had to actually read the text to understand the image. Perhaps a much lover resolution image (6x6 max) depicting something very simple like a circle would be better. Then the reader could understand the transformation matrix. Also there should be an indication in the figure how the image is flattened.
- figures are referenced out of place, eg. fig 1 in page 5 and some in captions in other figures.
- It is not clear how the linear model for MNIST in fig 5 was trained. I am assuming. The training set is created with N DA samples. For all N, the number of epoch are the same. For the black line the explicite regularisation is used.

The proposed method has many limitations:
- The class of DA is limited to the cases where they can be expressed as a linear function on images. Many relevant DA falls outside of this (e.g. jpeg compression).
- It is hard to use in practice, as deriving the expectation and variance of a bit more complicated DA (e.g. some non-linear warp) is a very had manual task.
- For non-linear models the method relies on 3rd order Taylor expansion.
- Even in the linear regression case, for a bit complicated DA the method requires the spectral decomposition of a very large system (a matrix of size n^2 x n^2). It is not discussed how this computation can be done efficiently (I assume it can be done for simple DA). If it cannot be done, then what are the trade-offs?

It is not clear to me how practical the explicite regulariser is. See Questions.

Overall I rate the significance of the method not very high because of the limitations, that I think are severe.

EDIT:
The rebuttal changed my mind, I think the paper is worth reading despite the limitations of the method in its current form. The significance is in the theory and that is valuable. 

questions: 
Does the method help reduce the number of steps to train a model in a non-linear case?

Does it help generalisation in the regime with small data-set in the non-linear case? Fig 7 did not show results with explicit regularisation.

Please explain the experiments on fig 5 and 7: what is the exact loss, training set, number of epochs and relevant hyperparameters for each scenario.
limitations: 
The authors addressed some limitations (see weaknesses) except for the efficiency of the method.

Please discuss how the explicate regularisation can be computed efficiently, i.e the spectral decomposition. Without this the method cannot work on large images.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a novel operator called Data-Space Transform (DST) to express data augmentation (DA) and use it to investigate three topics: 1) equivalent regularizer induced by each DA; 2) the number of DA samples for a Monte-Carlo (MC) approach to correctly estimate the information of the DA; 3) sensitivity analysis of loss/model pairs under a DA policy. The third topic led to a rediscovery of TangentProp from first principles.
strengths_and_weaknesses: Strengths: 

1. The paper is generally well written and motivated. 

2. The new transform operator proposed is intuitive.

3. One of the key discoveries enabled by DST is a closed-form expected loss for DAs (for linear models and MSE loss), which allows for measuring the number of samples needed to correctly estimate the information conveyed by each DA.

4. The rediscovery of TangentProp from principled derivation provides a nice link to practical use cases beyond the linear model and MSE loss case.

5. Limitations of the work including various assumptions and approximations are discussed in the paper.

Weaknesses:

Major:

1. As shown in figure 5, it seems that even for the linear model and MSE loss combination where an analytical solution is available, the classic MC sampling DA method can outperform the expected loss method with large enough training sets (e.g. N=3000 for MNIST), which suggests that there are aspects not captured by the closed-form loss. Although it’s hard to evaluate whether some of the effects as error bars are not included in this figure, the trend shows that increasing the size of the training set could lead to larger performance lead for the multi-sample DA method over the expected loss approach. Can the authors 1) provide error bars, 2) comment on why MC DA may outperform the expected loss method for the linear model, and 3) conduct experiments with larger training sets?

Minor:

1. Results for non-linear models seem to be limited to small augmentations. Many state-of-the-art augmentations are rather aggressive, how well does the results generalize to these aggressive augmentations?

2. Figure 3 caption, missing space before “Fourth column”

3. Line 243: “than” => “as”

4. Line 263: remove “as”

questions: Section 4 (as well as the title)  suggests that the simple translation augmentation needs around 1000 to 10000 samples to correctly estimate the MSE loss. In the 2012 Alexnet paper, by counting the number of unique (fixed-sized) crops of sampling a 224x224 patch from a 256x256 image, a 1024x gain in the number of training samples is estimated. I understand that this work investigates the problem in a different framework, but can the authors comment whether these two estimates are related in some way?
limitations: The authors discuss various limitations of this work in several sections which include important mathematical assumptions and approximations. Overall I believe the limitations are well discussed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper aims to provide a theoretical framework for analyzing the role of data augmentations (DA) for learning tasks involving both linear and linear models. To this end, it first proposes a framework by which image transformations are modeled as matrix-vector multiplications using data-space transforms. 

This framework is subsequently used to derive the analytical form of the first two moments of an augmented sample. Next, the MSE loss under a linear model is considered in which an exact form of an explicit DA regularizer is derived. An exact expression of the optimal solution under infinite DA sampling is derived, which generalizes the standard Least-Squares solution which is known when no DA is applied. The expression depends exactly on the expectation and variance of transformed samples. For the non-linear case, it is mentioned that the Taylor approximation can generalize from the linear setting.  

Next, the paper analyses DA's sampling efficiency and loss sensitivity. An empirical evaluation shows the DA's sampling strategy is inefficient and requires many samples to converge to a good solution. The sensitivity of a given loss and model under a DA is analyzed leading to a derivation of a popular regularization strategy of TangentProp. 
strengths_and_weaknesses: Strengths:

1. The paper proposes a novel set of mathematical tools for analyzing DA in a theoretical setting, which has thus been mainly analyzed through an experimental lens. These tools could be important in analyzing proposed DA schemes and so are an important and original contribution. 
2. The authors use these tools to derive an explicit regularizer induced by each DA as well as derive the sensitivity, of a given loss and model under a DA policy. These are useful properties for understanding the role of DA as a regularization. 
3. The analysis of the variance (sensitivity) leads to a first-principles derivation of a TangentProp as a regularizer which minimizes the loss variance. This is significant as it demonstrates the applicability of the framework in deriving and understanding useful regularizers. 

The paper is mostly clear, however many of the proofs are left for supplementary, and at least an intuition could be useful to include in the main text.  

Weaknesses:
1. The analysis provided in Sec.4.1 is given for a single image and for a particular transformation. It is unclear if this holds for a more general case and in particular for a wide range of natural images and transformations. For a more general setting, can an expression in terms of the number of images n be given in order to closely approximate estimate the MSE loss from the augmented samples?
2. The analysis provided in the paper in Sec.3.4 and Sec.4.2 is exact for linear models. It is unclear how the derivation is affected under non-linear models even for small transformations. Further, in practice, one often performs large transformations (e.g. in SSL models) which are crucial to their success. Hence it is unclear if the framework provided in this paper can explain the non-linear or large transformation setting well, limiting the significance of the work. 

****
Typo, line 176, Gaussian blue

 
questions: 1. How do the derivation provided in Sec.3.4 and 4.2 change assuming a non-linear model? In particular for which range of transformations is the approximation sufficiently close to the linear case?
2. How does the analysis provided in Sec.4.1 generalizes for different transformations and images?
limitations: The authors mention the limitation regarding the third-order approximation of non-linear models. However, this is not probably characterized. In which cases (range of transformations, images) is it ok to make such assumptions?
review_title: null

Meta-review: metareview: This paper studies data augmentation through the lens of an explicit regularizer, deriving a closed form regularization term corresponding to the effect of data augmentation in the case of linear models and MSE, and analyzes its properties with respect to convergence, sample efficiency and stability. Data augmentation is a core technique in deep learning that is poorly understood and has not, to my knowledge, been the subject of much rigorous analysis, so this work has the potential to be quite influential in our understanding of a fundamental deep learning practice.

Reviewers were generally positive on the approach and the theoretical underpinnings. Several were concerned about generalization of the approach to nonlinear models, which the authors explain proceeds by means of a Taylor expansion. The main shortcoming according to reviewers was experiments limited to MNIST, which the authors remedied with extension of their experiments to additional datasets (EMNIST and Fashion MNIST, to be precise).

With scores uniformly deep within acceptance territory with the exception of yaYp who promises to "accept this paper with flying colours" if the new results are incorporated into the main paper, this seems like an obvious candidate for acceptance.

P.S. The technique of deriving a closed form penalty is reminiscent of marginalized dropout (Wager et al, 2013), a work I would suggest deserves citation in the camera ready.
confidence: Certain
Decision: Accept