Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes to select parameters for augmentation through a training procedure (without validation data, like the prior work on Augerino). It does so by introducing a Bayesian model selection framework for selecting the ideal set of augmentations for a neural network. Effort in making this model tractable is notable and the evaluation shows some benefits over strong baselines.
strengths_and_weaknesses: Strengths:
* The paper is clear and reads well, though the treatment of technical derivations is quite long.
* Model does sensible behavior (i.e., Figure 2).
* Model has improvements over Augerino (e.g., Table 1).
* Diversity of datasets, models, and augmentations seems acceptable, but can be still strengthened (e.g., Table 1).

Weaknesses:
* One baseline is non-invariant, which seems like a weak baseline, since there is often a significant performance boost by just using augmentations in a simple way. Augerino is a stronger baseline, but the results are much less noticable against that baseline (e.g., Figure 3).
* It's not clear nor intuitive why this method works. Some examples of failure modes of Augerino that were addressed would be enlightening.

Post Rebuttal:
Many of the paper's strengths are due to being a Bayesian treatment of the problem of setting augmentation parameters, which avoids hyperparameter tuning. There is a convincing case that it would be used in the future as a building block as this approach works on a standard set of benchmarks and is otherwise sensible. The evaluation can be strengthened (as always, e.g., scaling the approach on more difficult datasets), but I think the paper has sufficient contribution as is, and so I am increasing my score.
questions: * What parts of the design of your model are entirely novel and what parts are adapted from prior work (e.g., how KFAC, Laplace-GGN, etc. fit in with prior work)?
* What are the computational concerns with this method? How does it compare with Augerino in terms of training and inference time?
* What is the significance of the baseline used in experiments?
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a gradient-based method for automatically selecting the data augmentation parameters without the validation data. The trick is to parameterize the data augmentation scheme, approximate the marginal likelihood with respect to these data augmentation parameters and optimize them during training. The authors propose schemes such as KFAC approximation and explicit automatic differentiation to make the computation of differentiable marginal likelihood more tractable. Empirically, the proposed approach is capable of recovering invariances presented in the data on MNIST, FashionMNIST, and CIFAR10.
strengths_and_weaknesses: Strengths:
* The topic is interesting and relevant to the Neurips community.
* The paper is well-written, and the technical details look correct to me. 
* While limited to small datasets and models, the experiments demonstrate that the proposed approach can successfully recover invariances presented in the model and can help improve generalization.

Weaknesses:
* There are several specific limitations of the proposed approach. To give some specific examples: (1) the proposed approach does not scale to large datasets and models, (2) the method requires task-specific data augmentation parameterization (e.g., affine transformation for image classification tasks), and it is not straightforward to apply the proposed method in other domains, and (3) the proposed algorithm is challenging to implement on standard deep learning frameworks and introduce additional computational overhead that scales quadratically with model parameters and linearly to the number of datasets.
* While I believe that the toy experiments presented in the paper demonstrate the proposed approach's effectiveness, the proposed method's scalability is still questionable.
* It would be helpful if the authors also compared their algorithms with other data augmentation adaptation methods that make use of validation datasets. To my understanding, these methods should be much more efficient than the proposed approach, although it requires a validation dataset. If we select 20% of training data as validation data, shouldn't the prior algorithms also be able to learn these invariances?


** Post-Rebuttal **

Thank you for the author's response. I acknowledge that I have read the rebuttal and other reviewers' comments. As the author addressed all my concerns in the paper, I increased my score.
questions: Comments:
* In the introduction, the authors state that choosing the correct transformation is expensive and requires trial and error. However, the proposed approach also requires selecting the parameterization of $\eta$, which also requires some prior knowledge of the dataset and task. For example, in the case of image classification, affine invariance parameterization looks like a reasonable choice, but how should we select the parameterization for other tasks (e.g., language modelling). It would be helpful if the authors addressed these issues.
* In Table 2, is the model evaluated on the standard CIFAR-10 datasets, or is it a corrupted version (e.g., scaled and translated)? The test accuracy for the baseline method seems to be a little lower than I expected.
* As the paper introduces a lot of notations, it would be helpful to have a table of notations summarizing all notations used in the paper in the Appendix.
* As the authors mentioned in the checklist, releasing the code would be helpful.
* What additional hyperparameters does the proposed method require, and how sensitive are they? Given a new dataset, can the proposed algorithm work without putting a lot of effort into tuning these hyperparameters?

Minor Comments:
* Some colours in Figure 2 (e.g. vertical & horizontal scaling) are too similar, and it is difficult to distinguish them in the plot when printed.
* In line 301, there is a missing bracket after App. B.
limitations: The work does not have a potential negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors extend previously proposed techniques for a Laplace approximation for Bayesian model selection to learn and incorporate invariances while training a deep network. To make this approximation computationally feasible, the authors propose techniques to apply a Kronecker-Factored approximation. The authors extensively evaluate their method on many datasets, with ablations and comparisons to another method that learns invariances while training. 
strengths_and_weaknesses: Strengths
* The authors have very illustrative experiments (Figure 2 is very well-presented) that show that the models they train are learning the invariances present in the dataset. Training a network to capture the invariances present in a dataset is of significant interest for the machine learning community. Furthermore, they test their method on a wide variety of datasets/network architectures. 
* The authors clearly describe the novelty of their approach in comparison to Immer et al 2021. They describe how the aforementioned work would be computationally infeasible for their proposed task.

Weaknesses
* One goal of this work is “obtain a model that is invariant to a set or distribution of transformations” (lines 16-17) as measured by test accuracy in Table 1. In the case where one wants s network with invariance to the natural variations in a dataset, the authors should justify their approach over simply training with augmented samples, as done in (RandAugment, Cubuk et al. 2019). For example, can the authors’ approach provide benefit over RandAugment on ImageNet using their six candidate augmentations? It should be stated that the authors’ proposed method has significance outside this scenario, since they also identify which invariances are present (as opposed to augmentation approaches). Nevertheless, such a comparison is important for the community to understand which method should be used for this given circumstance.
* Figure 4 of the appendix is very nice for measuring performance of different approximation methods. To strengthen their work, the authors should consider giving empirical runtime results as well so that practitioners can understand the empirical performance/run speed tradeoffs. 
questions: * Does the baseline get trained with any augmentations applied?
* Have the authors considered using a generative model in place of the six hand-picked invariances (e.g., a disentangled variational auto-encoder like the Beta-VAE from Higgins et al 2011)? This would take the work one step closer to true a priori invariance learning. 
limitations: * From the experiments the authors have presented, it appears that their methods requires the set of candidate invariances to be specified beforehand. This is limited for datasets where the invariances are not known beforehand.
* The authors have not released code.
review_title: null

Meta-review: metareview: After reading the submission and its reviews, my understanding is that the submission proposes to use a tractable Laplace approximations (General Gauss-Newton and K-FAC) to learn suitable invariances/augmentation for the considered neural network. The derivations of this paper and the description of the method are clear and well motivated. While the experiments illustrate the concept and the soundness of the method, they remains small scale and are limited to sets of parameterized augmentations. Nonetheless, I recommend this submission for acceptance.
confidence: Certain
Decision: Accept