Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Paper proposes to automatically learn which transformations to augment during training without introducing biases, for example, when the augmentations are not appropriate for the task. Authors rewrite the problem as a semi-infinite constrained problem similar to a recent adversarial robustness framework. Proposed approach learns to sample from a distribution over the transformations using a Metropolis-Hastings approach. 
strength_and_weaknesses: Strengths:

1. Automatically learning data augmentations for a particular task is a significant and important challenge which helps not just in-distribution but also for out-of-distribution robustness of models. 
2. Not requiring differentiable transformations (unlike baselines) can be helpful. 
3. Typically, the tradeoff between the original task loss and the augmentation loss is done via hyperparameter tuning, the proposed approach instead maximizes the relevant dual variable (whose advantages are discussed in the paper). 

Weaknesses:

1. I think the paper can be improved with a more formal treatment of the problem and method. 
   1. The problem is not formally defined, for example, the paper never discusses formally the true invariances of the data distribution. Essentially, the task of an automatic data augmentation method will be to learn transformations that hold for the true (unknown) data generation process.
   2. Degree of invariance and approximate invariance are used loosely and not well-defined. 
   3. The paper is motivated solely on improving the sample complexity but no such bounds are provided; while data augmentation itself can lower the sample complexity, it is unclear if that continues to hold when learning augmentations automatically especially with an approximation from constrained optimization (CSRM) to unconstrained optimization (D-CERM). 

2. Empirical evaluation does not show case the claims of advantages of the proposed method vs the baselines.

   1. Experiments do not explicitly show cases when data augmentation results in biasing the model and is harmful, which is one of the main reasons for learning the appropriate distributions over the transformations. 

   2. Another advantage of the proposed method compared to the baselines is to allow non-differentiable transformations, but no such experiments are performed. 
   3. Overall, the empirical gain in performance is not significant compared to the baselines.



Other Questions/Comments:

1.  Introduction: “The use of invariance overcomes the need to search for a specific distribution over transformations”. I do not understand this statement. 
2. As opposed to the existing methods, $\gamma$ is treated as a dual variable (and optimized) which allows the tradeoff to be learnt directly. However, $\epsilon$ is a hyperparameter which is cross-validated which also trades-off the approximate invariance, thus leading to the same problem. Further, it is not clear from the paper, what cross-validation methodology is used (i.e., what is the training and test datasets for this cross-validation)?
3. There seems to be a single slack variable \epsilon that controls all different types of transformations (for example, rotations, translations, shear, etc) with no additional regularization to force toward invariance unlike [1].

3. I do not agree with the statement “sampling $\lambda^*_c$ accurately is not a concern given that we are interested in promoting approximate invariance” as we still need to characterize the kind of approximation. For instance, using a constraint level ($\epsilon$) is one way of approximation but it is different from not able to accurately estimating $\lambda_c^*$.

Minor/Typos:

1. Typos: “whithout”, “infromativeness”

2. $\lambda^*_c$ takes different inputs at different places in Section 4.2. 

   
**References**
[1] Benton, Gregory, et al. "Learning invariances in neural networks from training data." Advances in neural information processing systems 33 (2020): 17605-17616.


clarity,_quality,_novelty_and_reproducibility: Writing can be improved with a more streamlined and concise presentation. Some aspects of the framework (e.g., semi-infinite constrained learning and MCMC sampler required) exist in a previous work in adversarial robustness. 
summary_of_the_review: Paper can be improved with a more formal treatment of the problem and method (e.g., in defining the problem, discussing sample complexity of the proposed method). Empirical evaluation does not show case the claims of advantages of the proposed method vs the baselines (tasks where some augmentations are not appropriate, non-differentiable transformations).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a novel approach to learn a data augmentation method by formulating the task as invariance-constrained learning problem. They leverage Monte Carlo Markov Chain (MCMC) sampling to solve it, which achieves state-of-the-art results on CIFAR10 and CIFAR100 for specific wide neural network architectures.

strength_and_weaknesses: Strengths:
+ The overall presentation is clear, the background is explained in detail, and the respective terms and their high level interpretation are discussed.
+ The proposed data augmentation learning method seems to outperform the baselines on CIFAR10 and CIFAR100.
+ The presented problem formulation has the advantage that if a solution to the unconstrained problem is feasible, the presence of that constraint has no effect on the statistical problem.
-> Could this fact also be helpful to assess whether the learned transformations are consistent with the data distribution or impose an unwanted bias?

Weaknesses and open questions:
- Competing papers (e.g., Müller and Hutter, 2021) compare run times of different augmentation algorithms. Such an analysis is completely missing.
- The authors make multiple claims regarding the advantages of their formulation as constraint optimisation problem but these are not demonstrated experimentally.
(Can invariances be learned?)
- In contrast to competing methods, no experiments on ImageNet are provided. This is relevant for the question whether the proposed data augmentation method scales to large, complex datasets.
- The authors claim themselves that they fail to achieve large improvements in accuracy over baselines but this could be attributed to a stagnation in data augmentation research (Müller and Hutter, 2021) and 
could reflect limits of the benchmarking setup. In this case, the authors could also look for other datasets and demonstrate improvements of their method over benchmarks there.
- A general question for this line of research: Why are no synthetic experiments conducted to test the ability of automated data augmentation algorithms to recover symmetries in the data? It would be easy to generate data that is invariant to some transformations and test how many samples are required to learn the correct transformations. The claim that the relevant transformations are learned is otherwise unsupported.
- The data augmentation proposal seems to rely on a couple of hyper parameters that need to be tuned in practice.
- A couple of hand-crated baselines are discussed in the introduction but not compared with in the experiment section.

Points of minor critique and open questions:
- No guarantees with respect to the convergence of the sampler to the equilibrium distribution are provided. (This would, however, be a hard problem and might be too much to ask for.)
- The related work section is quite short and adds little value in addition the introduction.
- Figures 1 and 2 are a bit difficult to read for yellow markers.
- Considering that a lot space is left in the main paper, the conclusions and insights into the proposed method could be extended. In particular, moving the analysis of the sampled transformations could highlight additional features of the proposed method.

clarity,_quality,_novelty_and_reproducibility: The presentation is very clear and didactically well designed. While the proposed approach seems to be novel for learning data augmentation strategies, its benefits could be clarified and also be subject of more extensive experiments. Currently, the practical impact of the approach seems minor. Importantly, also an analysis of the computational complexity of the proposed method is missing (and could present a considerable downside of the approach).
summary_of_the_review: While the paper is well motivated and clearly written, some relevant analysis (like the runtime analysis, potentially additional experiments that highlight the specific advantages of the proposed approach, etc.) are missing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose to treat data augmentation in the context of a constrained optimization problem, where the constraints are defined using a set of desired invariances. The data augmentations are then chosen using an MCMC scheme based on the loss of the model (which is supposed to be invariant), while the strength of their influence is optimized using a Langrage multiplier.
strength_and_weaknesses: Strengths:
- The problem of tuning data augmentation to different tasks is important.
- The paper is clearly written and the theory seems correct.

Weaknesses:
- The experiments are a bit limited and inconclusive.

Major comments:
- How does the choice of just two MC steps in the experiments affect the performance? I would imagine that the samples stay quite close to the uniform proposal then instead of actually sampling from the right distribution.
- DeepAA seems like a quite strong baseline, it outperforms all other methods in one out of two experiments where it is included. Why is it not included in the other ones? I feel like it should be.
- Given that TA is designed to be a ridiculously simple approach, it performs quite well compared to the proposed method. How does it compare in terms of runtime? I would imagine that TA would be much faster, is that true?
- I appreciate the image classification experiment, but I think it would be illustrative to have an experiment with artificial invariances (similar to the ones in [1]), so one could discern to what extent the learned augmentations would correctly capture those.

Minor comments:
- The related work could mention [1]
- In Eq. 3, what is $h$? Should that be $f_\theta$?
- Sec. 3.2: a intance -> an instance

[1] https://arxiv.org/abs/2202.10638
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and is of high theoretical quality. The quality of the experiments is a bit lower. The method seems novel and the experiments seem reproducible.
summary_of_the_review: Overall, I think this is a really nice idea with a strong theoretical motivation. If the experiments could be improved, I would be happy to recommend acceptance; for now, I'm keeping my score slightly lower due to the inconclusive experiments.

UPDATE: Based on the changes made by the authors during the rebuttal, I have updated my score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper studies an interesting problem on how to automatically generate augmented data for learning invairance representations. The paper proposes a MCMC based methods, and then tested the results in some datasets. The paper seems to be on a novel track of goals, but needs further development.  
strength_and_weaknesses: - strength
    - the study of generating augmented samples for invariance seems very novel to me. 
    - the writing of the paper is reasonably good
- weakness
    - the algorithm still needs further development, it's hard for me to believe a MCMC based method is the best solution in this scenario, given the rich techniques developed in deep learning regime. 
    - the empirical scope is quite limited: the deepAA method is barely compared. 
clarity,_quality,_novelty_and_reproducibility: - clarity: 
    - good
- quality: 
    - the paper needs further development on its empirical end to be considered for publication. 
    - the MCMC based method can probably be further improved in terms of efficiency and performances with more modern techniques. 
- novelty:
    - another highly relevant work regarding section 3
        - Toward Learning Robust and Invariant Representations with Alignment Regularization and Data Augmentation
    - Algorithms 1 and 2 seem fairly standard to me, not sure where the novelty is from. 
- reproducibility: 
    - seems good. 
summary_of_the_review: The paper seems still in its preliminary stage, a little bit limited in technical novelties, and more importantly empirical validation. I'm happy to give it a more thorough read if the authors can provide more convincing and more comprehensive empirical evaluations.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: We appreciate the authors for responding to the review comments, adding more experiments, and engaging in discussion with some reviewers. Automating the otherwise manual data augmentation process is very useful and this work has contributed interesting ideas to this topic. Most of the existing automated data augmentation methods are so computationally demanding that it is simply infeasible to adopt them for practical applications. Gradient-based methods are computationally more attractive than others, such as those based on reinforcement learning, although they still cannot be considered as lightweight methods. The fact that the proposed method is based on MCMC sampling alerts us that efficiency, and hence practicality, is an important aspect that is worth considering. In their response to one of the reviewers, the authors admit that experiments involving larger datasets such as ImageNet exceed their current computational budget. This is in line with our concern that the proposed method may not be computationally feasible for practical applications. Among other comments and concerns, the authors are recommended to pay more attention to this aspect in further enhancing their method that hopefully can really be adopted by practitioners to make real-world impact.

justification_for_why_not_higher_score: The support for acceptance is not strong enough.

justification_for_why_not_lower_score: N/A
Decision: Reject