On the Consistency Loss for Leveraging Augmented Data to Learn Robust and Invariant Representations
Authors: Haohan Wang
Zeyi Huang
Xindi Wu
Eric Xing
Venue:  (2021)
Keywords: robustness
invariance
data augmentation
consistency loss

Abstract: Data augmentation is one of the most popular techniques for improving the robustness of neural networks. In addition to directly training the model with original samples and augmented samples, a torrent of methods regularizing the distance between embeddings/representations of the original samples and their augmented counterparts have been introduced. In this paper, we explore these various regularization choices, seeking to provide a general understanding of how we should regularize the embeddings. Our analysis suggests how the ideal choices of regularization correspond to various assumptions. With an invariance test, we show that regularization is important if the model is to be used in a broader context than the in-lab setting because non-regularized approaches are limited in learning the concept of invariance, despite equally high accuracy. Finally, we also show that the generic approach we identified (squared $\ell_2$ norm regularized augmentation) performs better than several recent methods, which are each specially designed for one task and significantly more complicated than ours, over three different tasks.

Under review as a conference paper at ICLR 2021
ON THE CONSISTENCY Loss FOR LEVERAGING AUG- MENTED DATA TO LEARN ROBUST AND INVARIANT REPRESENTATIONS
Anonymous authors Paper under double-bling blind review
ABSTRACT
Data augmentation is one of the most popular techniques for improving the robust- ness of neural networks. In addition to directly training the model with original samples and augmented samples, torrent of methods regularizing the distance between embocdingssrpprsentttins of the original samples and their augmented counterparts have been introduced. In this paper, we explore these various regu- larization choices, seeking to provide general understanding of how we should regularize the embeddings. Our analysis suggests the ideal choices of regulariza- tion correspond to various assumptions. With invariance test, argue that regularization is important if the model is to be used in a broader context than accuracy-driver setting because rogularrar approaches are limited in learn- ing the concept of invariance, despite equally high accuracy. Finally, we also show that the generic approach we identified (squared lz norm regularized augmenta- tion) outperforms several recent methods, which are each specially designed for task and significantly more complicated than ours, over three different tasks.
INTRODUCTION
Recent advances in deep learning has delivered remarkable empirical performance over i.dd test data, and the community continues to investigate the more challenging and realistic scenario when models are tested in robustness over non-i.id data (e.g., [Be--Dvvid et al., 2010, [Szegedy et al, 2013). Recent studies suggest that one cause of the fragility is the model's tendency in capturing undesired signals (Wang et al. 2020), thus combating this tendency may be a key to robust models.
To help models ignore the undesired signals, data augmentation (i.e., diluting the undesired signals of training samples by applying transformations existing examples) is often used. Given its widely usage, we seek to answer the question: how should we train with augmented samples so that the assistance of augmentation can be taken the fullest extent to learn robust and invariant models?
In this paper, We analyze the generalization behaviors of models trained with augmented data and associated regularization techniques. We investigate set of assumptions and compare the worst- case expected risk over unseen data when i.i.d samples are allowed to be transformed according to a function belonging to family. We bound the expected risk with terms that be computed during training, so that our analysis can inspire how to regularize the training procedure. While all the derived methods have an upper bound of the expected risk, with progressively stronger assumptions, we have progressively simpler regularization, allowing practical choices to be made according to the understanding of the application. Our contributions of this paper are as follows:
We offer analyses of the generalization behaviors of augmented models trained with different regularizations: these regularizations require progressively stronger assumptions of the data and the augmentation functions, but progressively less computational efforts. For example, with sumptions pertaining to augmentation transformation functions, the Wasserstein distance over the original and augmented empirical distributions can be calculated through simple norm distance.
We test and compare these methods and offer practical guidance how to choose regularizations in practice. In short, regularizing the squared l distance logits between the augmented samples and original samples is favorable method, suggested by both theoretical and empirical evidence.

---

Under review as a conference paper at ICLR 2021
With invariance test, we argue that vanilla augmentation does not utilize the augmented sam- ples to the fullest extent, especially in learning invariant representations, thus may not be ideal unless the only goal of augmentation is to improve the accuracy specific setting.
RELATED WORK & KEY DIFFERENCES
Data augmentation has been used effectively for years. Tracing back to the earliest convolutional neural networks, we notice that even the LeNet applied on MNIST dataset has been boosted by mixing the distorted images the original ones (LeCun et al., 1998). Later, the rapidly growing machine learning community has seen a proliferate development of data augmentation techniques (e.g., flipping, rotation, blurring etc.) that have helped models climb the ladder of the state-of-the- art (one may refer to relevant survey (Shorten & Khoshgoftaar 2019) for details). Recent advances expanded the conventional concept data augmentation and invented several new approaches, such as leveraging the information in unlabelled data (Xie et al., 2019). automatically learning augmen- tation functions (Ho et al.) 2019 Hu et al., 2019) Wang et al. 2019c; Zhang et al. 2020 Zoph al.] 2019), and generating the samples (with constraint) that maximize the training loss along training Fawzi et al. 2016), which is later widely accepted as adversarial training (Madry et al. 2018).
While the above works mainly discuss how to generate the augmented samples, in this paper, we mainly answer the question about how to train the models with augmented samples. For example, instead of directly mixing augmented samples with the original samples, one consider regulariz- ing the representations (or outputs) of original samples and augmented samples to be close under a distance metric (also known as consistency loss). Many concrete ideas have been explored in dif- ferent contexts. For example, & distance and cosine similarities between internal representations in speech recognition (Liang et al., 2018). 8p, squared & distance between logits (Kannan et al. 2018), or KL divergence between softmax outputs (Zhang et al.) 2019a) in adversarially robust vision models, Jensen-Shannon divergence (of three distributions) between embeddings for texture invariant image classification (Hendrycks et al. 2020). These are but few highlights of the concrete and successful implementations for different applications out of a huge collection (e.g.. (Wu al., 2019 Guo al.,
2019 Zhang et al.] 2019b] (Shah et al. 2019 Asai & Hajishirzi 2020 Sajjadi et al.) 2016, Zheng
al. 2016 Xie et al. 2015)), and one can easily imagine methods permuting these three elements (distance metrics, representation outputs, and applications) to be invented. Even further, although we are not aware of the following methods in the context data augmentation, given the popularity of GAN (Goodfellow 2016) and domain adversarial neural network (Ganin et al., 2016). one can also expect the distance metric generalizes to a specialized discriminator (i.e. a classifier), which calculated (usually maximized) distance measure, Wasserstein-1 metric as an example (Arjovsky et al. 2017 Gulrajani et al. 2017)
can be intuitively understood
With this rich collection of regularizing choices, which one method should we consider in general? More importantly, do we actually need the regularization at all? These ques- tions are important for multiple reasons, especially considering that there are paper suggesting that these regularizations may lead to worse results (Jeong et al. 2019). In this paper, we answer the first question with a proved upper bound of the worst case generalization error, and our upper bound explicitly describes what regularizations are needed. For the second question, we will show that regularizations can help the model to learn the concept of invariance.
Key Differences:
There are also several previous discussions regarding the detailed understandings of data augmen- tation (Yang al. 2019, Chen et al. 2019, Garean König 2018 Rajput et al. 2019, Dao et 2019). among which, (Yang et al. 2019) is probably the most relevant as it also defends the usage of regularizations. However, we believe our discussions are more comprehensive and sup- ported theoretically, since our analysis directly suggests the ideal regularization. Also, empirically, we design an invariance test in addition to the worst-case accuracy used in the preceding work.
3 TRAINING STRATEGIES WITH AUGMENTED DATA
Notations (X, Y) denotes the data, where X E R and Y E {0, 1]m*k (one-hot vectors for classes). and f(, 0) denotes the model, which takes in the data and outputs the softmax (probabilities of the prediction) and 0 denotes the corresponding parameters. g() completes the prediction (i.e., mapping softmax to one-hot prediction). I(:,-) denotes generic loss function. a(-) denotes
2

---

Under review as a conference paper at ICLR 2021
transformation that alters the undesired signals of sample, i.e., the data augmentation method. a E A, which is the set transformation functions. P denotes the distribution of y). For any sampled (x, y), we can have (a(x), y), and we use Pa to denote the distribution of these transformed samples. r(;0)) denotes the risk of model 0. denotes the estimated term
3.1 WELL-BEHAVED DATA TRANSFORMATION FUNCTION
Despite the strong empirical performance data augmentation has demonstrated. it should be intu- itively expected that the performance can only be improved when the augmentation is chosen wisely. Therefore, before we proceed to analyze the behaviors of training with data augmentations, we need first regulate some basic properties of the data transformation functions used. Intuitively, we will consider the following three properties.
"Dependence-preseee preservation" with two perspectives: Label-wise, the transformation cannot alter the label of the data, which is central requirement of almost all the data augmentation practice. Feature-wise, the transformation will not introduce new dependencies between the samples.
"Efficiency": the augmentation should only generate new samples of the same label as minor perturbations of the original one. transformation violates this property, there should exist other simpler transformations that can generate the same target sample.
of the transformations. For example, if one needs the model to be invariant to rotations from 0 to 60%, we consider the vertices to be 0 rotation function (thus identity map) and 60 rotation function. In practice, one usually selects the transformation vertices with intuitions and domain knowledge.
"Vertices": There extreme
We now formally define these three properties. The definition will depend on the model, thus these properties are not only regulating the transformation functions, but also the model. We introduce the Assumptions A A1-3 corresponding to the properties.
A1: eepeeddencepreseevveee
the transformation function will not alter the dependency regarding the label (i.e., for any a() A, a(x) will have the same label as x) or the features (i.e., for any a1 (), a2() E A, (x1)) IL a1 (x(2) for any X1, X2 E X that x1 F X2).
A2: Efficiency: for Ō and any a() E A. f(a(x);;) 0) closer to x than any other samples under distance metric de(-,-), i.e. de (f(((x)); f(x 0)) & minx
A3: Vertices: For model 0 and transformation a(). we use P o.@ to denote the distribution of [a(()):) 0) for (x,y) P. "Vertices" argues that exists two extreme elements in A, namely at and a with certain metric d.(.,)). we have
(n at Gr P. Pa- sup dP 05
P og.0)
a1.agEA
Note that .:).) is a metric over two distributions and de is a metric over two samples. Also, slightly different from the intuitive understanding of "vertices" above, A3 regulates the behavior of embedding instead of raw data. All of our follow-up analysis will require A1 to hold, but with more assumptions held, we can get computationally lighter methods with bounded error.
3.2 BACKGROUND, ROBUSTNESS, AND INVARIANCE
One central goal of machine learning is to understand the generalization error. When the test data and train data are from the same distribution, many previous analyses can be sketched as:
rp (0) < Pr (0) + <(OO), m, n)
which states that the expected risk can be bounded by the empirical risk and function of hypothesis space (0) and number of samples n; accounts for the probability when the bound holds. ø() is function of these three terms. Dependent
the details of different analyses, different concrete examples of this generic term will need different assumptions. We use generic assumption A4 to denote the assumptions required for each example. More concrete discussions in Appendix A
3

---

Under review as a conference paper at ICLR 2021
Robustness In addition to the generalization error above, also study the robustness by following the established definition as the worst case expected risk when the test data is allowed to shifted to some other distributions by transformation functions in A. Formally, we study
Pp' 18 (x.))>> max ((g ((({((((x);)) 0)) y))
aniA
As (0) S Tp. (0), we only need to study B different assumptions and offer formalizations of the generalization bounds under each scenario. Our bounds shall also immediately inspire the development of methods in each scenario as the terms involved in our bound are all computable within reasonable computational loads.
We will analyze B in different scenarios involving
Invariance addition to robustness, we are also interested in whether the model learns to be in- variant the undesired signals. Intuitively, if data augmentation is used to help dilute the undesired signals from data by altering the undesired signals with a() E A. a successfully trained model with augmented data will map the raw data with various undesired signals to the same embedding. Thus, we study the following metric to quantify the model's 's ability in learning invariant representations:
I(0,P) P)
sup a1,0
dp P. Po2.8).
(4)
a1.6
dz() is distance over two dis- tributions, and we suggest use Wasserstein metric given its favorable properties (e.g.. see practical examples in Figure 1 of (Cuturi & Doucet 2014) or theoretical discussions in (Villani (008)). Due to the difficulties in assessing f(((x); 0) (as it depends on 0), we mainly study empirically, and argue that models trained with explicit regularization of the empirical counterpart of will have favorable invariance property.
where Pa.g to denote the distribution of f(a(x); 0) for (x
3.3 WORST-CASE AUGMENTATION (ADVERSARIAL TRAINING)
We consider robustness first. can written equivalently into the expected risk over a pseudo distribution P' (see Lemma 1 in (Tu et al. 2019)), which is the distribution that can sample the data leading the worst expected risk. Thus, equivalently, we can consider supprer(P,.) Tpx (0). With an assumption relating the worst distribution of expected risk and the worst distribution of the empirical risk (namely, A5, Appendix Ap, the bound of our interest (i.e., SUPPIeT(PPA) rp (0)
be analogously analyzed through supPrET(P.A) Ppr (0). By the definition of P', we can have:
Lemma 3.1. With Assumptions AI, A4, and A5, with probability at least 6, we have
sup Tp. (0) (P.A(
1 n
sup [g((f((((x)):) y) ((0)), (x.y) dEA
Σ
(5)
This result is straightforward follow-up of the preceding discussions. In practice, it aligns with the adversarial training (Madry et al. 2018), a method that has demonstrated impressive empirical successes in the robust machine learning community.
While the adversarial training has been valued by its empirical superiorities, it may still have the following two directions that can be improved: firstly, it lacks an explicit enforcement of the concept of invariance between the original sample and the transformed sample; secondly, it assumes that
elements of A are enumerable, thus (x.xy) SUPaEA ((((((((x);)) fy) is computable. The remaining discussions expand along these two directions.
3.4 REGULARIZED WORST-CASE AUGMENTATION
To force the concept of invariance, the immediate solution might be to apply some regularizations to minimize the distance between the embeddings learned from the original sample and the ones leameed from the transformed samples. We have offered summary of these methods in Section 2
To have a model with small invariance score, the direct approach will be regularizing the empirical counterpart of because of the computational difficulty of Wasserstein distance. Conveniently, we have the following result that links the & regularization the Wasserstein-1 metric in the context of data augmentation.
We notice that existing methods barely consider this regularization, probably
4

---

Under review as a conference paper at ICLR 2021
Proposition 3.2. With A2, and de (:))) in A2 chosen be l norm, for any E A, we have
Σ [1(x;;;) - f(a(xx): 0111 (l(
(6)
This result conveniently allows to use & norm distance replace Wasserstein metric, integrating the advantages of Wasserstein metric while avoiding practical issues such as computational com- plexity and difficulty pass the gradient back during backpropagation.
We continue to discuss the generalization behaviors. Our analysis remains in the scope of multi- class classification, where the risk is evaluated misclassification rate, and the model is optimized with cross-entropy loss (with the base chosen to be log base in cross-entropy loss). This setup aligns with A4, and should represent the modern neural network studies well enough.
Before we proceed, we need another technical assumption A6 (details in Appendix A). which can be intuitively considered tool that allows to relax classification error into cross-entropy error, so that we can bound the generalization eπror with the terms we can directly optimize during training.
We can now offer another technical result:
Theorem 3.3. With Assumptions Al, A2, A4, A5, and A6, and de(:,-) in A2 is E1 norm, with proba- bility at least the worst case generalization risk will be bounded as
sup Tp' (0) S (0) Σ 11f(xx;) f(x;:0)l==+ + ((0), (A
and ==al a(x), where arg M [(((x);;),
This technical result also immediately inspires the method to guarantee worst case performance, as well as explicitly enforce the concept of invariance. Notice that a arg max y fa(xx; 0) is simply selecting the augmentation function maximizing the cross-entropy loss, standard used by many worst case augmenting method (e.g., Madry et 2018)
3.5 REGULARIZED TRAINING WITH VERTICES
As A in practice is usually set with large number of (and possibly infinite) elements, we may not always able to identify the worst case transformation function with reasonable computational efforts. This limitation also prevents us from effective estimating the generalization error as the bound requires the identification of the worst case transformation.
Our final discussion is to leverage the vertex property of the transformation function to bound the worst case generalization error:
Lemma 3.4. With Assumptions A1-A6, and de(-.-) in A2 chosen as norm distance, d=(--)) in A3 chosen as Wasserstein-1 metric, assuming there is a a'() E A where Pp (0) = 1 (PP (0) +
Pp_- (o)), with probability at least we have:
sup rp: (0) S (T ( Pr. (0) Σ [xs(at(xs):) s(a (xx=xx);)))1 + (n, ()
This result inspires the method that can directly guarantee the worst case generalization result and can be optimized conveniently without searching for the worst-case transformations. However, this method requires good domain knowledge of the vertices of the transformation functions.
3.6 ENGINEERING SPECIFICATION OF RELEVANT METHODS
Our theoretical analysis has lead to a line of methods, however, not every method can effectively implemented, especially due to the difficulties of passing gradient back for optimizations. There- fore, to boost the influence of the loss function through backpropagation, we recommend to adapt the methods with the following two changes: 1) the regularization is enforeed on logits instead of softmax; 2) we use squared l norm instead of & norm because L norm is not differentiable everywhere. We discuss the effects of these compromises in ablation studies in Appendix
5

---

Under review as a conference paper at ICLR 2021
Also, in the cases where need to identify the worst transformation functions, iterate through all the transformation functions and identify the function with the maximum loss.
Overall, our analysis leads to the following main training strategies:
VA (vanilla augmentation): mix the augmented samples of vertex function to the original ones for training (original samples are considered as from another vertex following experiments).
VWA (vanilla worst-case augmentation): each iteration, identify the worst-case transformation functions and train with samples generated by them (also known as adversarial training).
RA (regularized augmentation); regularizing the squared &2 distance over logits between the orig- inal samples and the augmented samples of a fixed vertex transformation function.
RWA (regularized worst-case augmentation): regularizing the squared & distance over logits between the original samples and the worst-case augmented samples identified at each iteration.
EXPERIMENTS
We first use some synthetic experiments to verify our assumptions and inspect the consequences when the assumptions are not met (in Appendix C Then, in the following paragraphs, we test the methods discussed support our arguments in learning robustness and invariance. Finally, we show the power of our discussions by competing with advanced methods designed for specific tasks.
4.1 EXPERIMENTS FOR LEARNING ROBUST INVARIANT REPRESENTATION
Experiment Setup: We first test our arguments with two data sets and three different sets of the aug- mentations. We study MNIST dataset with LeNet architecture, and CIFAR dataset with ResNetl architecture. To examine the effects of the augmentation strategies, we disable all the heuristics that are frequently used to boost the test accuracy of models, such as the default augmentation many models trained for CIFARIO adopted, and the BatchNorm (also due to the recent arguments against the effects of BatchNorm in learning robust features (Wang et al. 20200). ) although forgoing these heuristics will result in lower overall performance than usually expects.
We consider three different sets of transformation functions: texture, rotation, and contrast. The details of these transformation functions and the experiment setup are in Appendix D
We consider three different evaluation metrics:
Clean: test accuracy on the original test data, mainly reported as a reference for other metrics.
Robustness: the worst accuracy when each sample can be transformed with a E A.
Invariance: A metric to test whether the models learns the concept of invariance (details to follow).
Invariance-fest: To whether a model can truly learns the concept of invariance within AA {a, (), (2,2))..... {.}}} of t elements, we design a new evaluation metric: for a sampled collection of data of the sample label i, denoted as ( we generate the transformed copies of it with A, resulting in (() a xx_) We combined these copies into dataset, denoted as 20) For every sample in 20) retrieve its nearest neighbors of other samples in (0), and calculate the overlap of the retrieved samples and {a, (x), az (x()) at [x)}. Since the identify map is in A, so the calculated overlap score will be in [1/t, 1]. The distance used is d(.,)) (1 f(: 0),, where Ø is the model are interested examine. Finally, report the averaged score for every label. Thus, a high overlap score indicates the prediction of model 0 is invariant to the augmentation functions in A. If we other distance functions, the reported values may differ, but notice that the rank of the methods compared in terms of this test barely changes.
We show the results in Table and Table (in Appendix) for MNIST and CIFARIO respec- tively. Table shows that RWA generally a superior method, in terms of the metrics, especially the invariance evaluation as it shows much higher invariance score than competing methods. We believe this advantage of invariance comes from two sources: regularizations and the fact that RWA has seen all the augmentation functions in A. In comparison, RA also has regularization but only
Results:

---

Under review as a conference paper at ICLR 2021
Texture
Rotation
Contrast
R
C
R
Base
0,9921
0.9860
0.9236
0,9921
0.2960
0.2056
0,9921
0.2699
0.2003
VA 0.9928
0.9906
0.9876
0.9884
0,9336
0.5628
0,9922
0.9837
0.4153
RA 0,9909
0.9904
1
0.9930
0.9525
0.6044
0.9936
0.9823
0.4166
VWA 0.9922
0.9903
0.9940
0.9466
0.9408
0.6284
0.536
0.4470
0.2210
RWA
0.9915
0.9911
0.9934
0.9882
0.8856
0.994
0.9893
0.8786
Table I: Results of MNIST data ("C" stands for clean accuracy, "R" stands for robustness, and "I"
stands for invariance score): invariance score shows big differences while accuracy does not.
sees the vertices in A. so the invariance score of RA not compatitable RWA. although better than VA. Table roughly tells the same story. More discussions are in Appendix D
Other results (Appendix scenarios, even in the out-of-domain test scenario where the transformation functions are not in A. RWA generally performs the best, although not the best in every single test. We also perform ablation test to validate the choice of squared & norm over logits in contrast to other distance metrics. Our choice performs the best the worst-case performance. This advantage is expected as our choice is validated by theoretical arguments as well as consideration of engineering convenience.
The strength of RWA can also be shown in several other different
Overall, the empirical performances align with our expectation from the theoretical discussion: while all methods discussed have bounded worst case performance, we do not intend to com- pare the upper bounds because smaller upper bounds do not necessarily guarantee smaller risk. However, worst case augmentation methods tend show better worst case performances because they have been augmented with all the elements A. Also, there is no clear evidence suggesting the difference between augmentation methods and its regularized versions in terms of the worst case performance, but it is clear that regularization helps to learn the concept of invariance.
4.2 COMPARISON TO ADVANCED METHODS
Finally, we also compete our generic data augmentation methods against several specifically de- signed methods in different applications. We use the four generic methods (VA, RA, VWA, and RWA) with generic transformation functions (A of "rotation", "contrast", , or "texture" used in the synthetic experiments). We compare our methods with techniques invented for three different top- ics of study (rotation invariant, texture perturbation, and cross-domain generalization), and each of these topics has seen a long line of method development. We follow each own tradition (e.g., rota- tion methods are usually tested in CIFAR10 dataset, seemingly due to the methods' computational requirements), test over each own most challenging dataset (e.g.. ImageNet-Sketch is the most re- cent and challenging dataset in domain generalization, although less studied), and report each own evaluation metric (e.g., methods tested with ImageNet-C are usually evaluated with mCE).
Overall, the performances of our generic methods outperform these advanced SOTA techniques. Thus, the main conclusion, as validated by these challenging scenarios, are (1) usage of data aug- mentation can outperform carefully designed methods; (2) usage of the consistency loss can further improve the performances; (3) regularized worst-case augmentation generally works the best.
Due to the limitation of space, we leave the background details of these experiments in Appendix where we introduce the detailed experiment settings, and explain the acronyms in Tables 2
Rotation-invariant Image Classification We test the models with nine different rotations including 0". Augmentation related methods only use the A of "rotation" in synthetic experiments, so the testing scenario goes beyond what the augmentation methods have seen during training. The results in Table 2 strongly endorses the efficacy of augmentation-baser methods. Interestingly, regularized augmentation methods probably with the benefit of learning the concept of invariance, tend to behave well in the transformations not considered during training. Also, RA outperforms VWA average.
Texture-perturbed ImageNet classification We also test the performance on the image classifica- tion over multiple perturbations. We train the model over standard ImageNet training set and test the model with ImageNet-C data (Hendrycks Dietterich, 2019) which a perturbed version of ImageNet by comupting the original ImageNet validation set with a collection of noises. The results

---

Under review as a conference paper at ICLR 2021
300
315
330
345
0
15
30
60
avg.
Base
0.2196
0.2573
0,3873
0,6502
0.8360
0.6938
0,4557
0.3281
0.2578
0,4539
ST
0.2391
0.2748
0.4214
0.7049
0.8251
0.7147
0,4398
0.2838
0.2300
0,4593
GC
0.1540
0.1891
0.2460
0.3919
0.5859
0.4145
0.2534
0.1827
0.1507
0.2853
ETN
0.3855
0.4844
0.6324
0.7576
0.8276
0.7730
0.7324
0.6245
0.5060
0.6358
VA 0.2233
0.2832
0.4318
0.6364
0.8124
0.6926
0.5973
0.7152
0.7923
0.5761
RA 0.3198
0,3901
0,5489
0.7170
0.8487
0.7904
0.7455
0,8005
0.8282
0.6655
VWA 0.3383
0.3484
0,3835
0,4569
0.7474
0.866
0,8776
0.8738
0.8629
0,6394
RWA
0.4012
0,4251
0.4852
0.6765
0.8708
0.8871
0.8869
0.8870
0.8818
0.7113
Table 2: Comparison to advanced rotation-invariant models. We report the test accuracy the test sets clockwise rotated, 00-60 and 300"-360" Average accuracy also reported. Augmentation methods only consider -6- clockwise rotations during training.
Base
SU
AA
MBP
SIN
AM
AMS
VA
RA
VWA
RWA
Clean
23.9
24.5
22.8
23
27.2
22.4
25,2
23.7
23.6
23.3
22.4
mCE
80.6
74.3
72.7
73.4
73.3
68.4
64.9
76.3
75.6
74.8
64.6
Table 3: Summary comparison to advanced models over ImageNet-C data. Performance reported (mCE) follows the standard ImageNet-C data: clean error and mCE are both the smaller the better.
Base
InfoDrop
HEX
PAR
VA
RA
VWA
RWA
Top-1 Top-5
0.1204
0.1224
0.1292
0.1306
0.1362
0.1405
0.1432
0.1486
0,2408
0,256
0,2564
0.2627
0.2715
0,2793
0.2846
0.2933
Table Comparison advanced cross-domain image classification models, over ImageNet- -Seeet dataset. We report top-1 and top-5 accuracy following standards on ImageNet related experiments.
are reported in Table S which shows that our generic method can outperform the current SOTA methods after continued finetuning process with reducing learning rates.
Cross-domain Imagehht-Sketch Classification We also compare to the methods used for cross- domain evaluation. with the most challenging setup in this scenario: train the models with standard ImageNet training data, and test the model over ImageNet-Sketch data (Wang et al. (019a), which is a collection sketches following the structure ImageNet validation set. Similarly, we only augment the samples with generic augmentation set (A4 of "contrast" in synthetic experiments, Appendix D) The results in Table again support the strength the correct usage of data augmentation.
CONCLUSION
In this paper, we conducted a systematic inspection to study the proper regularization techniques that provably related the generalization error of machine learning model, when the test distribution are allowed to be perturbed by family of transformation functions. With progressively more specific assumptions, we identified progressively simpler methods that can bound the worst case risk. We summarize the main take-home messages below:
Regularizing a norm distance between the logits of the originals samples and the logits of the augmented samples enjoys several merits: the trained model tend to have good worst cast per- formance, and can learn the concept of invariance (as shown in our invariance test). Although our theory suggests & norm, but we recommend squared & norm in practice considering the difficultics passing the (sub)gradient of & norm in backpropagation.
With the vertex assumption held (it usually requires domain knowledge to choose the vertex functions), one can use "regularized training with vertices" method and get good empirical per- formance in both accuracy and invariance, and the method is at the same complexity order of vanilla training without data augmentation. When we do not have the domain knowledge (thus not confident in the vertex assumption), we recommend "regularized worst-case augmenta- tion", which has the best performance overall, but requires extra computations to identify the worst-case augmentated samples at each iteration.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE transactions on pattern analysis and machine intelligence, 40 (12):2897-2905, 2018.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan, 2017.
Akari Asai and Hannaneh Hajishirzi. Logic-guided data augmentation and regularization for con- sistent question answering, 2020.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory learning from different domains. Machine learning, 79(1-2):151-155, 2010.
Sergey Bobkov and Michel Ledoux. One-dimensional empirical measures, order statistics, and Kantorovich transport distances, volume 261. American Mathematical Society, 2019.
Olivier Bousquet, Stéphane Boucheron, and Gábor Lugosi. Introduction to statistical learning the- ory. In Summer School on Machine Learning, pp. 169-207. Springer, 2003.
Shuxiao Chen, Edgar Dobriban, and Jane H Lee. A group-theoretic framework for data augmenta- tion, 2019.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International confer- machine learming, pp. 2990-2999, 2016.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Pruceedings of the IEEE conference on computer vision and pattern recognition, pp. 113-123, 2019.
Marco Cuturi and Arnaud Doucet. Fast computation of wasserstein barycenters. 2014.
Tri Dao, Albert Gu, Alexander Ratner, Virginia Smith, Chris De Sa, and Christopher Ré. A kernel theory of modern data augmentation. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, Califormia, USA, volume 97 of Proceedings of Machine Learning Research, pp. 1528-155 1537. PMLR, 2019.
Alhussein Fawzi, Horst Samulowitz, Deepak Turaga, and Pascal Frossard. Adaptive data aug- mentation for image classification. In 2016 IEEE International Conference Image Processing, ICIP 2016, Phoenix, AZ, USA, September 25-28, 2016, pp. 3688-3692 IEEE, 2016.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversaria training of neural net- works. The Journal of Machine Learning Research, 17(1):2096-203 2016.
Ian Goodfellow. Nips 2016 tutorial:
Generative adversarial networks.
arXiv preprint
17Xiv:1701.00166 2016.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans, 2017.
Hao Guo, Kang Zheng, Xiaochuan Fan, Hongkai Yu, and Song Wang. Visual attention consistency under image transforms for multi-label image classification. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 729- 739. Computer Vision Foundation IEEE, 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. Proceedings of the International Conference on Learning Represen- fations, 2019.

---

Under review as a conference paper at ICLR 2021
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: simple data processing method to improve robustness and uncertainty. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Alex Hernández-Garcht and Peter König. Data augmentation instead of explicit regularization, 2018.
Daniel Ho, Eric Liang, Xi Chen, Ion Stoica, and Pieter Abbeel. Population based augmentation: Ef- ficient learning of augmentation policy schedules. In Kamalika Chaudhuri and Ruslan Salakhutdi- nov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2731-2741, PMLR, 2019.
Zhiting Hu, Bowen Tan, Russ R Salakhutdinov, Tom M Mitchell, and Eric P Xing. Learning data manipulation for augmentation and weighting. In Advances in Neural Information Processing Systems, pp. 15738-1555 15749, 2019.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In Ad- vances in neural information processing systems, pp. 2017-2025, 2015.
Jisoo Jeong, Seungeui Lee, Jeesoo Kim, and Nojun Kwak. Consistency- based semi-supervised learning for object detection. In Advances Neural Information Processing Systems, pp. 10758- 10767, 2019.
Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing, 2018.
Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch-gordan nets: fully fourier space spherical convolutional neural network. In Advances in Neural Information Processing Systems, pp. 10117- 10126, 2018.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-2322 1998.
Davis Liang, Zhiheng Huang, and Zachary Lipton. Learning noise-invariant representations for robust speech recognition. In 2018 IEEE Spoken Language Technology Workshop (SL7), pp- 56 63-63.10 IEEE, 2018.
Percy Liang. Cs229t/stat231: Statistical learning theory (winter 2016), 2016.
Raphael Gontijo Lopes, Dong Yin, Ben Poole, Justin Gilmer, and Ekin Cubuk, Improv- ing robustness without sacrificing accuracy with patch gaussian augmentation. arXiv preprint arXiv:1906.026// 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Confer- ence Track Proceedings. OpenReview.net, 2018
Shashank Rajput, Zhili Feng, Zachary Charles, Po-Ling Loh, and Dimitris Papailiopoulos. Does data augmentation lead to positive margin? International Conference on Machine Learning. pp. 5321-5338 2019.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. In Advances in neural information processing systems, pp. 1163-1171, 2016.
Meet Shah, Xinlei Chen, Marcus Rohrbach, and Devi Parikh. Cycle-consistency for robust vi- sual question answering. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Connor Shorten and Taghi M Khoshgoftaar, A survey on image data augmentation for deep learning. Journal of Big Data, 6(1):60, 2019.
10

---

Under review as a conference paper at ICLR 2021
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.619,, 2013.
Kai Sheng Tai, Peter Bailis, and Gregory Valiant. Equivariant transformer networks. arXiv preprint arXiv:1901.11399 2019.
Zhuozhuo Tu, Jingwei Zhang, and Dacheng Tao. Theoretical analysis of adversarial learning: A minimax approach. In Advances in Neural Information Processing Systems, pp. 12259- 12259-12269, 2019.
Cédric Villani. Topics in optimal transportation. Number 58. American Mathematical Soc., 2003.
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
Haohan Wang, Songwei Ge, Eric P. Xing. and Zachary C. Lipton. Learning robust global represen- tations by penalizing local predictive power, 2019a.
Haohan Wang, Zexue He, Zachary C. Lipton, and Eric P. Xing. Learning robust representations by projecting superficial statistics out. 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019b.
Haohan Wang. Xindi Wu, Zeyi Huang, and Eric Xing. High frequency component helps explain the generalization of convolutional neural networks. In Computer Vision and Patlern Recognition (CVPR), 2020.
Yulin Wang, Xuran Pan, Shiji Song, Hong Zhang, Gao Huang, and Cheng Wu. Implicit semantic data augmentation for deep networks. Advances in Neural Information Processing Systems, pp. 12614-12623, 2019c.
Xindi Wu, Yijun Mao, Haohan Wang, Xiangrui Zeng, Xin Gao, Eric P. Xing, and Min Xu. Regu- larized adversarial training (RAT) for robust cellular electron cryo tomograms classification. In Illhoi Yoo, Jinbo Bi, and Xiaohua Hu (eds.), 2019 IEEE International Conference on Bioinformat- and Biomedicine, BIBM 2019, San Diego CA, USA, November 18-21, 2019, pp. 1-6. IEEE, 2019.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoe Le. Unsupervised data augmentation. arXiv preprint 1eXiiv/9944 /2848, 2019.
Saining Xie, Tianbao Yang, Xiaoyu Wang, and Yuanqing Lin. Hyper-class augmented and regular- ized deep learning for fine-gra -reingr image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2645-2654, 2015.
Fanny Yang, Zuowen Wang, and Christina Heinze-Deml. paariance-inducing regularization using worst-case transformations suffices to boost accuracy and spatial robustness. In Advances in Neural Information Processing Systems, pp. 14757-14768, 2019.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael Jor- dan. Theoretically principled trade-off between robustness and accuracy. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California. USA, volume 97 of Proceedings of Machine Learning Research, pp. 7472-7482, PMLR, 2019a
Richard Zhang. Making convolutional networks shift-invariant again. arXiv:1904. 1486, 2019.
arXiv preprint
Xinyu Zhang, Qiang Wang, Jian Zhang, and Zhao Zhong, Adversarial autoaugment. In International Conference on Learning Representations, 2020.
Zhirui Zhang, Shuangzhi Wu. Shujie Liu, Mu Li, Ming Zhou, and Tong Xu. Regularizing neural agreement. Proceedings of the AAAI Conference on
machine translation by Artificial Intelligence, volume 33, pp. 443-450 2019b.

---

Under review as a conference paper at ICLR 2021
Stephan Zheng, Yang Song. Thomas Leung, and Ian Goodfellow. Improving the robustness of deep neural networks via stability training. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 4480-4488, 2016.
Barret Zoph, Ekin D. Cubuk, Golnaz Ghiasi, Tsung- Yi Lin, Jonathon Shlens, and Quoe V. Le. Learning data augmentation strategies for object detection, 2019.
12


Reviews:
[{"review_id": "n4rdc02iBpk", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\n\nSummary:\nThe paper seeks to find how to train with augmentation to train robust and invariant models. \nFor this purpose, they evaluate if  models trained with augmentation are generalizable and their associated regularization techniques. \n\n##########################################################################\n\n\n\nPros:\n1. The paper tackles an important issue of training well with augmentation.\n1. The theoretical analysis is strong and interesting.  \n2. The experiments are thorough and extensive. \n##########################################################################\n\n\nCons:\n1. While a number of related work is discussed in the Related Work section, it has not been considered in the experiments section. The authors should compare to the methods, for example cosine similarity hasn't been compared to. \n\n2. The paper is not well written and hard to read. It mentions constant references to the appendix in the theory, without being complete in itself. It is hard to separate the paper's proposed techniques from baseline models, for example, VWA and RA are well known methods in the literature. \n\n3. Key experimental details are missing: how many seeds were the experiments run for? It misses standard deviation results as well.\n\n4. Have the authors compared to constrastive learning methods, for example, \nA Simple Framework for Contrastive Learning of Visual Representations\nTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton  \n##########################################################################\n\n\nRebuttal Questions:\nPlease address the cons above.\n##########################################################################\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TMprEtIwUnZ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strongness:\na) The paper unifies existing assumptions enforced to achieve robustness into a framework, by summarizing them into a set of general assumptions A1-A6. It can incorporate some types of transformations and models. \nb) Among these generalization results, an unchanged term is the distance of representations between original samples and the ones after transformation. This is aligned with the conclusion in the literature. \nc) By considering specific types of transformations (rotation, contrast, and texture), the experimental results show that the RA and RWA can improve a lot compared to VA and VWA, which validates the effectiveness of representation regularization (i.e., theorem 3.3 and lemma 3.4). Moreover, compared with existing methods specifically designed to boost the robustness with respect to the transformations considered, the methods show a significant improvement. \nd) The paper is well organized and easy to understand. \n\nWeakness:\na) My most concern lies in the validity of the assumptions, specifically A2, A5, and A6. \n     i) For A2, consider the transformation set to be the l_infity additive noise for the adversarial attack, even the model specifically trained with robust (min-max) optimization, the label can still change. Besides, it seems that the framework cannot incorporate the transformations in the adversarial attack, which however is very important and cannot be ignored. \n    ii) For A5, I cannot get the intuition of why equality holds. For any distribution, the \"=\" (rather than \"\\approx\") seems too strong. \n   iii) It seems that the purpose of A6 is for the convenience of the proof. Again, I cannot get the intuition of why the inequality holds especially when g(f(x; \\theta)) \\neq g(f(x'; \\theta)), although I can understand that in this case, the transformation set is broad enough. But it is not sufficient to get this inequality since both sides are affected by the broadness of the transformation set. \nb) As stated in the i) in a), this paper may not incorporate the transformations in the scenario of adversarial attack. Besides, the definition of the invariance is only restricted in the scope of considered transformation; correspondingly, the robustness of the paper only lies in the interpolation rather than extrapolation [1]. It is more meaningful to consider the invariance and robustness for general out-of-distribution settings, such as [1] and [2]. \nc) For experimental results, the Cross-domain performance on ImageNet-Sketch seems too low to have meanings. Again, the invariance and robustness should be discussed in a broader perspective. Besides, how's the improvement margin if the batch-normalization, default augmentation are implemented? \nd) The theoretical analysis barely have some novel insights, since most of them are simple derivations of existing results. \n\n\n\n[1] Krueger, David, et al. \"Out-of-distribution generalization via risk extrapolation (rex).\" arXiv preprint arXiv:2003.00688 (2020).\n[2] Arjovsky, Martin, et al. \"Invariant risk minimization.\" arXiv preprint arXiv:1907.02893 (2019). ", "extra_scores": {}, "extra_text": {}}, {"review_id": "2hyBUKIwKnp", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In order to improve the robustness of the learned models, prior work has proposed various data augmentation techniques and different ways of incorporating them into training. This work seeks to provide a general understanding of how we should train with augmented samples in order to learn robust and invariant models from both theoretical and empirical perspectives. More importantly, the authors showed that the regularization of the augmented samples in the training procedure can be inspired from the theoretical analysis since it directly suggests the ideal regularization.\n\nOriginality and significance:\nThe proved upper bound of the worst case generalization error and the suggestion of the ideal regularization is new as far as I know, despite it also leads to the regularization using the worst-case augmentation to introduce invariance as in (Yang et al., 2019). \nOne issue, though, is that the set of possible data shifts is known beforehand and the best-performing RWA method must see all the augmentation functions. It immediately raises the question of how the method generalizes to unseen augmentations. For example, training with texture and rotation augmentations but testing with the contrast augmentation. This also happens in Section 4.2 when competing with advanced methods. In each experiment, the \"ground-truth\" augmentation is considered even for cross-domain classification. \nAlso, the \"Contrast\" results of VMA in Table 1 look strange. The vanilla augmentation already works very well in the robustness test with a score of 0.9837, but the regularized version only gives a score of 0.4470. Why does this happen?\n\nClarity: \nIt is not easy to follow, especially Section 3. There are in total 6 assumptions used in this work, but the authors decided to spend significant space on half of them and put the rest in the supplementary material without any explanation in the main text. Also, it might be better to give an informal illustration of the theoretical results before giving the main theorems so that it would be more likely to be appreciated by a larger community. \nOn the other hand, there is a gap between the theoretical analysis and the empirical verification. As the authors pointed out, the theoretical analysis advocates the use of the l1 distance while the authors recommend to use the squared l2 norm due to \"l1 norm is not differentiable everywhere\" and better quantitative results. This raise several questions: i) is there a everywhere differentiable approximation to l1 norm which is better than the squared l2 norm, and ii) what distribution divergence does the squared l2 norm correspond to. Since the empirical analysis in this work in mainly driven by the theoretical work, some discussion regarding the use of squared l2 norm would make the work stronger, especially when \"the ideal regularization\" does not work well.\nSome minor issues: i) I think the symbols used in the last sentence in A1 would be difficult to understand for most of the community, therefore it might be better to explain it in word; ii) I think it should be \"argmin\" instead of \"argmax\" in Theorem 3.3. We are aiming for the worst augmentation, but \"argmax\" would find the augmentation which gives the \"best\" result; iii) it is almost impossible to know the which competing methods are used in the experiment section without referring to the appendix. Even adding the reference to the methods in each table would help a lot. \n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers brought up many important concerns about this paper. On the positive side, the understanding of data augmentation is an important topic in deep learning, having good theoretical results is interesting here , and the experiments seem to do an okay job of backing up the theory. On the negative side, presentational issues make the paper difficult to follow and mischaracterize the results. A major issue is that some of the assumptions are hidden in the appendix and are not stated formally, and other assumptions are stated in a much weaker form, then made suddenly stronger when the theorems are stated. For example, Assumption 2 as stated holds trivially for any dataset as long as the possible data-augmented versions any two different examples are disjoint (just choose the discrete metric on the images of the examples under the data-augmentation function); however, in every theorem that uses A2, the distance chosen is restricted to be the L1 norm. Other Assumptions are stated strangely: for example, A1 says "i.e., for any $a_1(), a_2() \in A$, $a_1(x_1) ⫫ a_1(x_2)$ for any $x_1, x_2 \in X$ that $x_1 \ne x_2$. But what is the point of introducing $a_2$ if it's never used in the formula? And what is the meaning of the symbol ⫫? Normally, this is used for conditional independence, but there aren't any random variables in this formula ($a_1$ and $a_2$ are defined as just functions, not random functions, and $x_1$ and $x_2$ are just examples and aren't random variables either). This paper will be much stronger with these presentational issues cleared up.
Decision: Reject