Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a method to learn data invariance along with model training. The method avoids architectural modification and bilevel optimization, which makes it easy to use in many scenarios.
strengths_and_weaknesses: The technical details are generally good and easy to follow; however, I am concerned about the motivation and theoretical foundation of the paper.
questions: My primary concern is the claim that the method can recover the *true* data invariance. It ideally would require two steps: (1) all possible types of invariance are considered in the augmentation module, and (2) the distribution is properly learned for each type of invariance. However, these two steps can hardly be satisfied. For the first step, (1a) it is impossible to enumerate all types of invariance; (1b) not all types of invariance are differentiable (e.g., cutoff); (1c) the paper does not discuss how to select the invariances among all possible combinations (e.g., using validation). For the second step, (2a) the scalar amplitude itself is insufficient to characterize an unparameterized distribution, and (2b) there is no theory in the paper that the recovered distribution matches the true distribution (Indeed, the learned distribution degenerates to identical mapping without regularizer). In summary, I can hardly agree that the learned augmentation matches the *true* invariance.

It also makes the motivation of the paper unclear. One reason to use augmentation is to boost performance --- however, the proposed method still falls behind the fixed augmentation. Another reason to use augmentation is to boost robustness against invariance attack --- however, such robustness is not systematically evaluated in the paper.

My last minor concern regards computational complexity. As mentioned in the paper, the model in inference requires more than one sample per example. I wonder if the authors could provide an analysis of the tradeoff between computational complexity and model accuracy/uncertainty. 
limitations: Not applicable.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The author proposed an augmentation layer to learn data invariance from the training data. The proposed method is applicable to any differentiable transformation, and the augmentation layer can learn a weighted sum of different transformations that can best captures the data invariance. Sufficient experiments are conducted to verify the proposed arguments.
strengths_and_weaknesses: Pros:

The paper proposed an idea that the data invariance can be learnt in an end to  end fashion. Greatly eased the learning process comparing to bi-level optimization methods.
Great amount of experiments are conducted on both synthetic data and benchmark datasets. Experiments are well designed to demonstrate the effectiveness of proposed methods.

Cons:
Overall, I think the paper shares very similar idea comparing to the Augerino paper.  The main difference is that the proposed method applied learnable weights to multiple transformations. Thus I consider the novelty of this paper as an incremental variation of existing works.

The author proposed a selective regularizer that is different from the one used in the Augerino work. But the difference of the proposed regularizer is basically injecting the newly proposed learnable weight parameters to the regularizer proposed in Augerino, making the proposed method more like a mirror design of the existing work.

The author explored the proposed augmentation method beyond computer vision tasks and emphasized the novelty of this extension. I do not see this point as something unique for this proposed method. In fact, there shouldn’t be obstacles for other existing methods to be applied to tasks beyond computer vision. It would be great if the author can further explain on this point if I missed/misunderstand something.

some minor points:
in line 123, the author mentioned : Fortunately, even when C is small, f˜ is an unbiased estimator of f. Can the author further explain on the reason?
Figure 4 could be a bit hard to read, the position of each label could be a bit confusing. Might be better to reformat the figure.
questions: see Strengths And Weaknesses
limitations: There is no negative social impact of this work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose a module that learns the invariances (and their magnitudes) present in a dataset while training a neural network. The module applies a convex combination of augmentations before passing to a trunk network, where the outputs are aggregated. To prevent trivial augmentations (such as identity), the authors apply regularization to the magnitude predictions of their module. The authors test their module on synthetic and real-world data, comparing to other learned invariance approaches.
strengths_and_weaknesses: Strengths
* The paper is well written and clearly presents its methodology and experiments.
* The synthetic experiment shows the ability to uncover ground truth invariances from a set, and the Mario&Iggy experiment demonstrates the benefit of the selected magnitude prediction regularization. 

Weaknesses
* The proposed approach is an incremental improvement to that proposed by (Augerino, Benton et al. 2020). Both rely on aggregating the output of a network across augmentation samples to approximate an expectation, and both require some form of regularization to prevent learning identity for the transformation magnitude. 
* The paper is limited in experimental validation. Some experiments do not adequately compare to related methods (e.g., Augerino on the EEG dataset)
* The authors should reference (“Learning Invariances using the Marginal Likelihood,” van der Wilk et al. 2018). It seems the aggregation module proposed by the author is similar to the Bayesian approach in the mentioned work.
* Since the authors are comparing to learned augmentation approaches, they are missing many related works that should be cited from that literature. Just a few, for example: (RandAugment, Cubuk et al. 2019), (Faster AutoAugment, Hataya et al. 2019), (Cutout, DeVries et al. 2017), (Adversarial AutoAugment, Zhang et al. 2019), (AugMix, Hendrycks et al. 2020). The authors should at least include a comparison to RandAugment in their experiments, such as Figure 6b.
* Regarding test accuracy, many recent methods that have been proposed to learn an augmentation policy perform comparable or worse than the simple and computationally quick approach offered by RandAugment. Justification of learning the dataset invariances would be strengthened by also comparing to a network that implicitly learns invariances by training on RandAugment.
questions: * Why do the authors not compare to Augerino for the sinusoidal dataset?
* Is there any way for the authors to apply their method on a dataset where a set of plausible transformations T_Q are not known beforehand? This would be closer to the problem setup of a priori learning of augmentations. 
* How does AugNet compare to related methods on a larger scale image dataset, like CIFAR-10 or ImageNet?
* Why does AugNet learn to apply a single augmentation at each layer? Wouldn’t it be preferable to capture all invariances using fewer parameters?
limitations: * The problem set-up states that the authors are interested in “data invariances [that] are not always known a priori”, but the authors still rely on pre-specifying which augmentations they will apply (denoted as a set T_Q). The propose methodology seems to be limited in scenarios where a set of transformations is not known a priori.
* Without more experimental comparison and validation, it is difficult to assess the efficacy of the proposed method
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper recognizes the difficulties of current bilevel optimization of automatic data augmentation (ADA) methods. 
The authors propose AugNet, by adding learnable augmentation layers and an aggregation module to a base trunk model, and making network training and augmentation learning a single level end-to-end optimization problem.
Experiments across two simulated tasks, one image classification task, and one physiological data classification task are conducted to show the empirical versatility of AugNet.

strengths_and_weaknesses: Strengths:
1. Learning data augmentation policies is an important topic in training generalizable deep neural networks. The authors provide a good review of related work and identify some important limitations of previous methods such as AA, FAA, and ADDA. Thus the topic and motivation of this work are well justified.
2. Some mathematical intuitions are discussed in the paper in Section 3. Practical approximations are mentioned for Proposition 3.1.
3. Basic experimental details such as number of epochs, learning rates, and weight decay are generally well documented, choices of hyperparameters are mostly sensible (see weaknesses for exceptions).
4. Error estimates are provided for Figure 5 Top and Figure 6 (a).
5. A range of synthetic and real classification tasks are tested.

Weaknesses:
Major:
Generally, empirical evaluation is weak and unconvincing at several levels which makes it hard to believe the proposed method has much practical advantage over existing methods.

1. Although the tasks are somewhat diverse, they are all small or toy problems. It is a well known trend that data augmentation tends to work better when training data is limited and its performance boost may diminish when the training data is abundant. It is important to empirically show how well AugNet scales to larger (both in size and complexity) datasets and tasks.

2. Similar to tasks, the models used in the experiments have very low capacities (especially for Section 4.3). Not testing AugNet with larger and more realistic models would significantly limit its potential usefulness.

3. For the Mario&Iggy experiments:

  3.1. Qualitatively, both mu regularization and mu-omega regularization work well for Augerino and AugNet as they all roughly learn the optimal Pi/4 rotation.

  3.2. It is unclear how sensitive AugNet is to lambda. Experiments with different lambda are missing.

  3.3. Since C is set to 10 for Section 4.3 and 4 for other experiments, it is reasonable C is at least a somewhat important hyperparameter. 
Experiments to show how C affects performance are missing. 

4. The Cifar-10 experiments are very problematic in many ways. 

  4.1. AugNet simply performs poorly by falling behind even the base augmentation that was developed about a decade ago (from the AlexNet paper) which has 0 GPU hours of computation cost since it was handcrafted. Spending 1 extra GPU hour to learn an inferior augmentation policy is hardly a win.

  4.2. The gap in top-1 accuracy between AugNet (91.9%) vs Fixed Augment (93.6%) is significant as it could correspond to a 2x or 3x model size difference, this shouldn’t be understated.

  4.3. Since AugNet is supposed to improve upon methods like AA, it is natural to show results with AA and RA, since Pytorch has official implementations of searched policies on Cifar and ImageNet. These methods should significantly boost performance further.

  4.4. Figure B.5 shows that layer 2 learns TraslateY but with magnitude 0, which means identity transform, which suggests the failure of the proposed regularization. The same issue can be found in Figure B.6 layer 1.

  4.5. Additionally, other learned invariances with non-zero weights have magnitudes of 1, which also suggests that AugNet is only learning binary choices for both weights and magnitudes in this task. That also contradicts with what the authors propose regarding AugNet learning both weights and (continuous) magnitude.

5. Most accuracies in the MASS task differ by less than 1%, it is hard to evaluate if any median trends observed are statistically significant without error bars.

6. The limitations of this work are not properly discussed. 

Minor:
Overall I think some math can be more concise or moved to the appendix, as this paper does not focus on proposing new theories. This will free up some space to discuss the limitations of this work and potentially more experimental evaluations of AugNet.

questions: While itemized suggestions are laid out in the “weaknesses” section, here’s some higher level suggestions.

Overall I think this work is well motivated and has potential to make an impact if the experimental validation was much more convincing.

To me the first thing the authors could do without adding additional tasks and models is getting more solid results on existing tasks. As I mentioned in “weaknesses” error bars should be reported in all results to determine whether any effects are statistically significant, this should be the basis of experimental validation. It will also be nice to see stronger empirical performance across the existing tasks. As is, the improvements of AugNet over prior art is marginal and qualitative at best, with significant performance drop for Cifar. Important experiments such as performance vs lambda are missing.

The paper would also benefit a lot by going beyond the toyish task/model combinations. It is important to see whether AugNet generalizes to larger tasks and models for it to be more useful practically.

limitations: The authors discussed various limitations of previous work such as the bilevel optimization nature of ADAs. While there are mentions such as how practical transforms deviate from the group assumption, the authors did not have a clear discussion of the limitations of AugNet.
review_title: null

Meta-review: metareview: The decision for this paper was a hard one. I pondered the scores with respect to the engagement of the different reviewers.  I believe the initial scores were due to a misunderstanding of the limitations of the baseline model Augerino, and how the proposed method solves some of the failures and limitations of Augerino (e.g. being able to model only affine transformation). I also find the authors expanded their experiments in a convincing manner during the rebuttal period. We encourage the authors to *improve the clarity of their contributions* in their final version, and to include all additional experiments that were ran during the rebuttal period. 

confidence: Less certain
Decision: Accept