Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################################################################


Summary:
The paper seeks to find how to train with augmentation to train robust and invariant models. 
For this purpose, they evaluate if  models trained with augmentation are generalizable and their associated regularization techniques. 

##########################################################################



Pros:
1. The paper tackles an important issue of training well with augmentation.
1. The theoretical analysis is strong and interesting.  
2. The experiments are thorough and extensive. 
##########################################################################


Cons:
1. While a number of related work is discussed in the Related Work section, it has not been considered in the experiments section. The authors should compare to the methods, for example cosine similarity hasn't been compared to. 

2. The paper is not well written and hard to read. It mentions constant references to the appendix in the theory, without being complete in itself. It is hard to separate the paper's proposed techniques from baseline models, for example, VWA and RA are well known methods in the literature. 

3. Key experimental details are missing: how many seeds were the experiments run for? It misses standard deviation results as well.

4. Have the authors compared to constrastive learning methods, for example, 
A Simple Framework for Contrastive Learning of Visual Representations
Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton  
##########################################################################


Rebuttal Questions:
Please address the cons above.
##########################################################################




review_title: Weak Accept 

final_score: rating: 4: Ok but not good enough - rejection
review_content: Strongness:
a) The paper unifies existing assumptions enforced to achieve robustness into a framework, by summarizing them into a set of general assumptions A1-A6. It can incorporate some types of transformations and models. 
b) Among these generalization results, an unchanged term is the distance of representations between original samples and the ones after transformation. This is aligned with the conclusion in the literature. 
c) By considering specific types of transformations (rotation, contrast, and texture), the experimental results show that the RA and RWA can improve a lot compared to VA and VWA, which validates the effectiveness of representation regularization (i.e., theorem 3.3 and lemma 3.4). Moreover, compared with existing methods specifically designed to boost the robustness with respect to the transformations considered, the methods show a significant improvement. 
d) The paper is well organized and easy to understand. 

Weakness:
a) My most concern lies in the validity of the assumptions, specifically A2, A5, and A6. 
     i) For A2, consider the transformation set to be the l_infity additive noise for the adversarial attack, even the model specifically trained with robust (min-max) optimization, the label can still change. Besides, it seems that the framework cannot incorporate the transformations in the adversarial attack, which however is very important and cannot be ignored. 
    ii) For A5, I cannot get the intuition of why equality holds. For any distribution, the "=" (rather than "\approx") seems too strong. 
   iii) It seems that the purpose of A6 is for the convenience of the proof. Again, I cannot get the intuition of why the inequality holds especially when g(f(x; \theta)) \neq g(f(x'; \theta)), although I can understand that in this case, the transformation set is broad enough. But it is not sufficient to get this inequality since both sides are affected by the broadness of the transformation set. 
b) As stated in the i) in a), this paper may not incorporate the transformations in the scenario of adversarial attack. Besides, the definition of the invariance is only restricted in the scope of considered transformation; correspondingly, the robustness of the paper only lies in the interpolation rather than extrapolation [1]. It is more meaningful to consider the invariance and robustness for general out-of-distribution settings, such as [1] and [2]. 
c) For experimental results, the Cross-domain performance on ImageNet-Sketch seems too low to have meanings. Again, the invariance and robustness should be discussed in a broader perspective. Besides, how's the improvement margin if the batch-normalization, default augmentation are implemented? 
d) The theoretical analysis barely have some novel insights, since most of them are simple derivations of existing results. 



[1] Krueger, David, et al. "Out-of-distribution generalization via risk extrapolation (rex)." arXiv preprint arXiv:2003.00688 (2020).
[2] Arjovsky, Martin, et al. "Invariant risk minimization." arXiv preprint arXiv:1907.02893 (2019). 
review_title: This paper generalizes the existing generalization results with specific assumptions into a framework, with the aim of pursuing invariance and robustness. However, the scope of robustness is restrictive and the assumptions seem not reasonable for me. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: In order to improve the robustness of the learned models, prior work has proposed various data augmentation techniques and different ways of incorporating them into training. This work seeks to provide a general understanding of how we should train with augmented samples in order to learn robust and invariant models from both theoretical and empirical perspectives. More importantly, the authors showed that the regularization of the augmented samples in the training procedure can be inspired from the theoretical analysis since it directly suggests the ideal regularization.

Originality and significance:
The proved upper bound of the worst case generalization error and the suggestion of the ideal regularization is new as far as I know, despite it also leads to the regularization using the worst-case augmentation to introduce invariance as in (Yang et al., 2019). 
One issue, though, is that the set of possible data shifts is known beforehand and the best-performing RWA method must see all the augmentation functions. It immediately raises the question of how the method generalizes to unseen augmentations. For example, training with texture and rotation augmentations but testing with the contrast augmentation. This also happens in Section 4.2 when competing with advanced methods. In each experiment, the "ground-truth" augmentation is considered even for cross-domain classification. 
Also, the "Contrast" results of VMA in Table 1 look strange. The vanilla augmentation already works very well in the robustness test with a score of 0.9837, but the regularized version only gives a score of 0.4470. Why does this happen?

Clarity: 
It is not easy to follow, especially Section 3. There are in total 6 assumptions used in this work, but the authors decided to spend significant space on half of them and put the rest in the supplementary material without any explanation in the main text. Also, it might be better to give an informal illustration of the theoretical results before giving the main theorems so that it would be more likely to be appreciated by a larger community. 
On the other hand, there is a gap between the theoretical analysis and the empirical verification. As the authors pointed out, the theoretical analysis advocates the use of the l1 distance while the authors recommend to use the squared l2 norm due to "l1 norm is not differentiable everywhere" and better quantitative results. This raise several questions: i) is there a everywhere differentiable approximation to l1 norm which is better than the squared l2 norm, and ii) what distribution divergence does the squared l2 norm correspond to. Since the empirical analysis in this work in mainly driven by the theoretical work, some discussion regarding the use of squared l2 norm would make the work stronger, especially when "the ideal regularization" does not work well.
Some minor issues: i) I think the symbols used in the last sentence in A1 would be difficult to understand for most of the community, therefore it might be better to explain it in word; ii) I think it should be "argmin" instead of "argmax" in Theorem 3.3. We are aiming for the worst augmentation, but "argmax" would find the augmentation which gives the "best" result; iii) it is almost impossible to know the which competing methods are used in the experiment section without referring to the appendix. Even adding the reference to the methods in each table would help a lot. 



review_title: Promising theoretical analysis, but there is still unexplained gap between the theoretical and empirical analysis

Meta-review: The reviewers brought up many important concerns about this paper. On the positive side, the understanding of data augmentation is an important topic in deep learning, having good theoretical results is interesting here , and the experiments seem to do an okay job of backing up the theory. On the negative side, presentational issues make the paper difficult to follow and mischaracterize the results. A major issue is that some of the assumptions are hidden in the appendix and are not stated formally, and other assumptions are stated in a much weaker form, then made suddenly stronger when the theorems are stated. For example, Assumption 2 as stated holds trivially for any dataset as long as the possible data-augmented versions any two different examples are disjoint (just choose the discrete metric on the images of the examples under the data-augmentation function); however, in every theorem that uses A2, the distance chosen is restricted to be the L1 norm. Other Assumptions are stated strangely: for example, A1 says "i.e., for any $a_1(), a_2() \in A$, $a_1(x_1) ⫫ a_1(x_2)$ for any $x_1, x_2 \in X$ that $x_1 \ne x_2$. But what is the point of introducing $a_2$ if it's never used in the formula? And what is the meaning of the symbol ⫫? Normally, this is used for conditional independence, but there aren't any random variables in this formula ($a_1$ and $a_2$ are defined as just functions, not random functions, and $x_1$ and $x_2$ are just examples and aren't random variables either). This paper will be much stronger with these presentational issues cleared up.
Decision: Reject