Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This contribution is aimed at automated data augmentation. A continuous search space is defined using simple image operators, and thus enables learning of augmentation parameters. A randomised algorithm is proposed for parameter learning.
main_review: Data augmentation is an important topic and automated learning of augmentation methods and parameters can improve deep learning.  
In this paper the authors engineer a randomized algorithm for learning to augment.

The  originality could be better; there a numerous examples of continuous parameterizations of image augmentation starting with the work of Hauberg et al.  2016: "Dreaming more data: Class-dependent distributions over diffeomorphisms for learned data augmentation." In Artificial Intelligence and Statistics (pp. 342-350). PMLR, and later papers citing this one. Also the literature on generative modeling (GAN etc) is missing, see e.g. Antoniou et al 2017. Data augmentation generative adversarial networks. arXiv preprint arXiv:1711.04340.

The quality of the work is good, has a fine motivation and good experimental evaluation. Promising results. However, I recommend in a revised version to do a much more careful literature search and place the work more specifically in the literature on learned augmentations.

The clarity of the report is excellent, it is well written.

Overall the significance is limited by the novelty and also, if I may recommend to focus a bit more on analysis, to understand to the limitations of learning data augmentation.
limitations_and_societal_impact: See above - no immediate adverse societal impact
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors pose the problem of learning data augmentation as a continuous optimization problem and introduces a Bayesian approach for learning and sampling augmentations. The method allows for optimizing over a space of infinitely many possible transformations in contrast to the majority of automated data-augmentation approaches (ADA) which are limited by discretization of the solution space. The values of the proposed approach over prior works in terms of prediction accuracy and speed have been demonstrated on a range of benchmarks. 
main_review: Strengths
- Clarity: The manuscript is very well-written, and its contributions are clearly communicated with respect to prior works. 
- Convincing empirical results:  the method has been compared against a range of relevant prior methods on multiple image classification benchmarks and demonstrates convincingly that it can attain a better accuracy-speed trade-off.
- Solid method contribution:  the proposed method is carefully constructed by mixing different ingredients, all of which are important in different ways. Firstly, differentiable parameterization of transformations is a necessary first step to pose the ADD as a continuous optimization problem. Secondly, the single-step approximation used in eq. (9) offers a computationally efficient approximation to the bi-level optimization problem of ADD. Lastly, the use of Stochastic Gradient Langevin Dynamics allows ones to approximately sampling multiple data-augmentation strategies from the posterior distribution instead of resorting to a point-estimate of the optimal data-augmentation strategy, a key to ensuring diversity of relevant transformations. 


Weaknesses
- There is little insight into when the proposed method may break or may not work as well as other competing methods. How does the efficacy of the proposed method vary with the size of the validation dataset? When would the single-step approximation be assumed in eq. (9). Is there any relevant form of data augmentation that may be hard to parametrize in a differentiable way? If so and if such transformations were to be critical for some applications, then wouldn't the competing approaches based on discrete optimization be more preferable?  
- It is unclear what the benefits of the Bayesian approach are. For example, one could set the noise term $\eta=0$ in eq.(13) and I suspect that the method may still work reasonably well (not shown in the manuscript). The noise injection may potentially improve the diversity of sampled augmentations, but I feel this should be shown to substantiate the benefits of the Bayesian approach.   


Minor comments
- line 103: "differential" => "differentiable"
- line 124: "we pick k<< n for simplicity" -> what do we lose from this assumption? 
- line 182: "pseudo one-step update" -> why pseudo? 
- line 204: "second-order derivative … computed more efficiently" => efficient in time or space? 
- line 232: no space after '.' 
- line 255: "Tab. 3 and Tab. 2" => Tab. 2 and Tab. 3
limitations_and_societal_impact: The main limitations in my view are the lack of insights into the limitations of the proposed approach as elaborated above. 
NA for societal impact.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary:     Authors present a continuous formulation of data augmentation for
    images. The formulation is used to define a Bayesian model for augmentation
    parameters. The Bayesian model does not rely on a bi-level optimization
    problem as commonly used for optimizing data augmentation. Instead, only a
    single gradient step approximates a full optimization of a network with a
    given set of augmentation parameters. Gradient step is taken starting from
    pretrained weights determined through initial training using random
    augmentation. Samples from the posterior distribution are gathered using
    Langevin dynamics MCMC with random batches. Samples are used to train
    the ultimate networks. Experiments show that the proposed model outperforms
    existing techniques in terms of accuracy of the final models at a fraction
    of computation time. 
main_review:     The proposed technique is novel to the best of my knowledge. The continuous
    formulation, which is emphasized a bit too much perhaps in the article, is
    necessary for using Langevin dynamics MCMC during inference. The article
    deviates quite a bit from common literature on this topic and comes up with
    a novel method that yields great results at the fraction of the compute
    time. 

    Experimental results are quite convincing. Authors only evaluate using
    rather small datasets but I believe this is the common strategy taken by
    most of the works in this domain, therefore, it is not a major drawback. 

    The article is mostly well written. The method and experiments are clearly
    explained. The presentation can be improved but I do not think this is a
    major drawback.

    Data augmentation is an essential part for training neural networks. A
    computationally light method for determining best augmentation parameters
    would be very valuable in practice. 

    My questions are mostly about the experimental setup: 
    1. Authors show in Table 1 that the random sampling of the augmentation
       parameters already yields better results than some of the methods that
       optimize for the parameters. This is rather unexpected and
       not intuitive. Can they provide an explanation for this? 
    2. It is unclear to me whether authors start all the augmentation methods
       from some pretrained model. Using the pretrained model is essential, as I
       understand it, for the approximation given in Equation 9. Without a
       pretrained model, the proposed approach would not work most
       probably. Such pretraining may also help the other methods for
       determining optimal augmentation parameters, both in terms of final
       accuracy and computation time. Have authors started the other methods
       from a pretrained model?  
    3. It is somewhat surprising that a single gradient step approximates a full
       optimization so well in this work. Can authors elaborate on this? 


      --- After the update ---
    I thank the authors for the responses. I keep my opinion that this is good work. 
limitations_and_societal_impact:     I do not see a negative societal impact of this work. On the contrary, this
    work may lead to reduction in training times and save energy. 

    Authors claim they discuss the limitations of the work in Section 6 but I
    fail to see that in the text. The main limitation is the scope of the
    experiments. Authors could have demonstrated the model on larger set of
    problems, e.g., segmentation. They could have also used larger
    datasets. However, I do not see them as crucial limitations.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents an automated data augmentation technique that optimizes a distribution over a continuous augmentation space. Augmentation is formulated as learning a continuous mapping from an augmentation vector in R^d to an image transformation that composes three general families/modes of image transformations (color adjustment, image filtering, and image warping).   The proposed method uses a bilevel optimization approach where the primary learning task is trained on an augmented training set and the augmentation learning task is trained on the validation set. To make the optimization tractable, the inner level training is replaced by a one-step gradient update on an augmented mini-batch with model parameters being shared across the bilevel optimization iterations, i.e., incrementally updated during the bilevel optimization. A comprehensive set of experiments, including ablation experiments and comparisons to SOTA methods for automated data augmentation, demonstrate the efficacy and efficiency of the proposed method. 
main_review: The paper presents an efficient method for automated data augmentation that is learned end-to-end with the primary learning task without compromising performance. The paper moves away from restrictive assumptions about the discrete augmentation space, including predefined/handcrafted augmentations, and poses automated augmentation as a continuous optimization problem. This enables diversity in the augmented samples and avoids ad-hoc approximations when computing gradients in the discrete space cases. Experiments focus on natural image classification tasks, but the method is general to be adapted to other learning tasks and image modalities.

The work can be improved by addressing the below concerns:

- The method relies on the predefined semantic groupings of augmentation modes (color adjustment, image filtering, and image warping) that assume natural images, but the method is generic enough to be applied to other image modalities by considering modality-specific parameterized transformations. This should be made clear in the discussion section.

-  It would be insightful if a discussion is included on the choice of MCMC versus variational inference given the Bayesian formulation of the problem.

- The learned augmentation relies heavily on the generalization of the model on the validation set. There is no machinery in the proposed method that guarantees the generation of challenging augmented samples for the classification task (e.g., samples that can be easily misclassified, near the decision boundary).

- How sensitive/robust the proposed method under data shift and outliers?

- How sensitive/robust the proposed method under a severely limited training budget (e.g., medical imaging)? For instance, in Algorithm 1, a single augmented sample is added to the training data at each bilevel iteration, this won't be applicable in high-dimension-low-sample-size scenarios.







limitations_and_societal_impact: The authors did not acknowledge any potential negative societal impact of their work.
review_title: null

Meta-review: This paper is concerned with augmentation design, i.e. finding a set of parameters for generating random image variants via color adjustment / filtering / warping to aid generalization in models based on images. The basic idea is to reduce augmentations to a finite-dimensional continuous space, then learn a distribution over the parameters. State of the art methods [5] use bi-level optimization to learn this distribution. Instead, this paper proposes a Bayesian posterior where the likelihood of any set of hyperparameters can be computed in terms of scores on a held-out validation set. That construction, if defined exactly, would still involve a double-loop. However, essentially by "warm-starting" the inner-optimization, this is avoided and a single-loop sampling algorithm is obtained.

Reviewers were generally positive about this paper, and found it clear. The major concerns were that the experimental setup could be improved, discussion of related work could be more thorough, there could be more discussion of when the method might fail, and that it isn't entirely clear what the benefits of the Bayesian approach is. (The paper argues that it has the benefit of avoiding a double-loop optimization, but it appears that the idea of warm-starting the inner optimization could be applied to other methods as well, so it isn't entirely clear what is providing the benefit here.)

I found the central construction difficult to understand. (Reviews simply referred to it as a Bayesian model, but I wanted to understand what this model was, so I read the paper in detail.) After extensive discussion with the authors I am satisfied that there is a coherent model being defined. However, I strongly suggest that the notation of the paper be reconsidered to first explicitly state the model in the form of the equations (i), (ii), and (iii) that were given in the discussion below.
Decision: Accept (Poster)