WeMix: How to Better Utilize Data Augmentation
Authors: Yi Xu
Asaf Noy
Ming Lin
Qi Qian
Li Hao
Rong Jin
Venue:  (2021)
Keywords: Data Augmentation
Data Bias
Non-convex Optimization
Deep Learning Theory

Abstract: Data augmentation is a widely used training trick in deep learning to improve the network generalization ability. Despite many encouraging results, several recent studies did point out limitations of the conventional data augmentation scheme in certain scenarios, calling for a better theoretical understanding of data augmentation. In this work, we develop a comprehensive analysis that reveals pros and cons of data augmentation. The main limitation of data augmentation arises from the data bias, i.e. the augmented data distribution can be quite different from the original one. This data bias leads to a suboptimal performance of existing data augmentation methods. To this end, we develop two novel algorithms, termed "AugDrop" and "MixLoss", to correct the data bias in the data augmentation. Our theoretical analysis shows that both algorithms are guaranteed to improve the effect of data augmentation through the bias correction, which is further validated by our empirical studies. Finally, we propose a generic algorithm "WeMix" by combining AugDrop and MixLoss, whose effectiveness is observed from extensive empirical evaluations.

Under review as a conference paper at ICLR 2021
WEMIX: How TO BETTER UTILIZE DATA AUGMEN- TATION
Anonymous authors Paper under double-blind review
ABSTRACT
Data augmentation a widely used training trick in deep learning to improve the network generalization ability. Despite many encouraging results, several recent studies did point out limitations of the conventional data augmentation scheme in certain scenarios, calling for a better theoretical understanding of data augmen- tation. In this work, we develop a comprehensive analysis that reveals pros and cons of data augmentation. The main limitation of data augmentation arises from the data bias, i.e. the augmented data distribution can be quite different from the original one. This data bias leads to a suboptimal performance of existing data augmentation methods. To this end, we develop two novel algorithms, termed "AugDrop" and "MixLoss", to correct the data bias the data augmentation. Our theoretical analysis shows that both algorithms feet of data augmentation through the bias correction, which is further validated by empirical studies. Finally, propose generic algorithm "WeMix" by combining ArgDrop and MixLoss, whose effectiveness is observed from exten- sive empirical evaluations.
guaranteed to improve the ef-
INTRODUCTION
Data augmentation (Baird, 1992: Schmidhuber. 2015) has been a key the success of deep learning in image classification (He et al., 2019), and is becoming increasingly common in other tasks such as natural language processing (Zhang et al., 2015) and object detection (Zoph et al., 2019). The data augmentation expands training set by generating virtual instances through random augmentation to the original ones. This alleviates the overfitting (Shorten & Khoshgoftaar, 2019) problem when training large deep neural networks. Despite many encouraging results, it is not the case that data augmentation will always improve generalization errors (Min et al., 2020; Raghunathan et al., 2020). In particular, Raghunathan et al. (2020) showed that training by augmented data will lead to smaller robust error but potentially a larger standard error. Therefore, it is critical to answer the following two questions before applying data augmentation in deep learning:
When will the deep models benefit from data augmentation?
How to better leverage augmented data during training?
Several previous works (Raghunathan et al., 2020; Wu et al., 2020; Min et al., 2020) tried to address the questions. Their analysis is limited to specific problems such as linear ridge regression therefore may not be applicable to deep learning. In this work, we aim answer the two questions from theoretical perspective under more general non-convex setting. We address the first question in more general form covering applications in deep learning. For the second question, we develop new effective than the conventional data augmentation approaches.
approaches that provably
Most data augmentation operations alter the data distribution during the training progress. This imposes data distribution bias (we simply use "data bias" in the rest of this paper) between the augmented data and the original data, which may make difficult to fully leverage the augmented data. To be more concrete, let us consider label-mixing augmentation (e.g., mixup (Zhang al., 2018; Tokozume et al., 2018)). Suppose we have n original data DD = {(x,, yz). = 1, 1,...,n) n). where the input-label pair (x,, y:) follows distribution Ixx (Pxx, Iy (-[x)), ) Ixx is the marginal distribution of the inputs and IPy (-Ix) is the conditional distribution of the labels given inputs; we
generate augmented data D { ((x, V.). 1,...,m), ,m), where (3,) Pxy (Pxx, (x(()))))

---

Under review as a conference paper at ICLR 2021
(+)). Given x Px, the data bias is defined 6y maxy.5" y lly yll. We will show that when the bias between D and D is large, directly training on the augmented data will not be effective as training the original data.
and Px Px but (-(()x)
Given the fact that augmented data may hurt the performance, the next question is how to design better learning algorithms to leash out the power of augmented data. To this end, we develop two novel algorithms to alleviate the data bias. The first algorithm, termed AugDrop, corrects the data bias by introducing constrained optimization problem. The second algorithm, termed MixL Loss, corrects the data bias by introducing a modified loss function. We show that, both theoretically and empirically, even with large data bias, the proposed algorithms can still improve the generalization performance by effectively leveraging the combination of augmented data and original data. We summarize the main contributions of this work as follows:
We prove that in a conventional training scheme, deep model can benefit from augmented data when the data bias is small.
We design two algorithms termed AugDrop and MixLoss that can better leverage aug- mented data even when the data bias is large with theoretical guarantees.
Based on our theoretical findings, we empirically propose a new efficient algorithm WeMix by combining AugDrop and MixLoss, which has better performances without extra train- ing cost.
2 RELATED WORK
A series of empirical works (Cubuk et al., 2019; Ho et al., 2019; Lim et al., 2019; Lin et al., 2019a; Cubuk et al., 2020; Hataya al., 2019) how learn a good policy of using different data aug- mentations have been proposed without theoretical guarantees. In this section, we mainly focus on reviewing theoretical studies data augmentation. For survey of data augmentation, refer readers to (Shorten & Khoshgoftaar, 2019) and references therein for a comprehensive overview.
Several works have attempted establish theoretical understandings of data augmentation from different perspectives (Dao et al., 2019; Chen et al., 2019; Rajput et al., 2019). Min et al. (2020) shown that, with more training data, weak augmentation can improve performance while strong augmentation always hurts the performance. Later on, Chen et al. (2020) study the gap between the generalization error (please see the formal definition in (Chen et al., 2020)) of adversarially-trainn models and standard models. Both of their theoretical analyses were built special linear binary classification model or linear regression model for label-preserving augmentation.
Recently, Raghunathan al. (2020) studied label-preserving transformation in data augmentation, which is identical to the first case in this paper. Their analysis is restricted to linear least square re- gression under noiseless setting, which is not applicable to training deep neural networks. Besides, their analysis requires infinite unlabeled data. By contrast, we do not require the original data to be unlimited. Wu et al. (2020) considered linear data augmentations. There are several major differ- ences between their work and ours. First, they focus on the ridge linear regression problem which is strongly convex, while we consider non-convex optimization problems, which is more applicable in deep learning. Second, we study more general data augmentations beyond linear transformation.
3 PRELIMINARIES AND NOTATIONS
We study a learning problem for finding classifier to map an input x E X onto a label y E y C RK where K is the number of classes. We assume the input-label pair yx is drawn from distribution Fxy = (Ixx, (-))) Since every augmented example (x is generated by applying certain transformation to either one or multiple examples, will assume that (x, is drawn from a slightly different distribution Pzyy- = (IPx 1 (-1x)), where P& is the marginal distribution on the inputs x and (- (x)) (we can write it as Iy for simplicity) is the conditional distribution of the labels y given inputs X. We sample n training examples (x;,y;), 1 1,..., n from distribution Pxy and training examples (x,, yz),
1, 1...., m from I Pxy. We assume that m > due to the
data augmentation. We denote by D = {(x,, y:), = I....,n] and D = (x, yt), i = 1,..., m) the dataset sampled from xx and P Pxy. respectively. We denote by T(x) the set of augmented data transformed from x. We use the notation E(x,y)>P>5 [-] to stand for the expectation that takes over a random variable (x. y) following a distribution P Pxy. We denote by Vwh(w) the gradient of a function h(w) in terms of variable w. When the variable to be taken gradient is obvious, use
2

---

Under review as a conference paper at ICLR 2021
the notation Vh(w) for simplicity. Let use || as the Euclidean norm for vector or the Spectral norm for a matrix.
The augmented data D can be different from the original data D in two cases, according to (Raghu- nathan et al., 2020). In the first case, often referred to as aabelpprseerving we consider
(y(+ = Py (-)x), VX E T(x) but Px + Px.
In the second case, often referred label-mixing, consider
Ixx Px but (-(x) + (-(())) T
Examples of aabel-preserving augmentation include translation, adding noises, small rotation, and brightness or contrast changes (Krizhevsky al., 2012; 2; Raghunathan et al., 2020). One important example of label-mixing augmentation is mixup (Zhang et al., 2018; Tokozume et al., 2018). Due to the space limitation, we will focus on the label-mixing case, and the related studies and analysis for the label-preserving case can be found in Appendix A. To further quantify the difference between original data and augmented data when Pxx = IPx and Iy + Iyy, we introduce the data bias by given Px following:
6y max y y-y
The equation in (3) measures the difference between the label from original data and the label from augmented data given input x. We aim to learn prediction function fxx;; w) RD x X I that is as close as possible to y, where w E TR is the parameter and is a closed convex set. We respectively define two objective functions for optimization problems over the original data and the augmented data as
[(w)= F(xx,) [ w))l, Σ(w) EXx,3) (e, ((: w))],
(4)
where l is a cross-entropy loss function which is given by
exp(Ji(x:w)) w))
fly, J(:; w))
Σ (xP: (), where px w) - log
(5)
Eド= exp(Js(x;; (x)
We denote by w. and w. the optimal solutions minw L(w) and min Z(w) respectively,
W. E arg min L(w), W. E arg min Z(w).
(6)
wER?
wER?
Taking L(w) example, we introduce some function properties used in our analysis.
Definition 1. The stochastic gradients of the objective functions L(w) is unbiased and bounded, if we have F((x,) [Vwl f(x; w))] VL((), and there exists a constant G >> O, such that VV w)l I G,Vx E X, Vw E R, where p(x;w) = (p) (x: w)) is a vector. Definition 2. L(w) is smooth with an I.-Lipehizz continuous gradient, if there exists constant L such that [|V[(w)- v((u)] < L-|w-wul ul|, Vw, u E Rb, equivalently, L(w) L(u) < (VL(u), w u) 1 |w 1/|2, Vw, u R. The above properties are standard and widely used in the literature of non-convex optimiza- tion (Ghadimi & Lan, 2013; Yan et al., 2018; Yuan et al., 2019; Wang et al., 2019; Li et al., 2020). We introduce an important property termed folyak-Lojasiewv the objective function L(w).
(PL. condition (Polyak, 1963) on
Definition 3. condition) L(w) satisfies the PL condition, if there exists a constant >> 0 such that 2p([(w)- L(w.)) < [VC[()] Vw R. where w. defined in (6).
The PL condition has been observed in training deep and shallow neural networks (Allen-Zih et al., 2019: Xie et al., 2017), and is widely used in many non-convex optimization studies (Karimi al., 2016: Li & Li, 2018: Charles & Papailiopoulos, 2018; Yuan et al., 2019; Li et al., 2020). It is also theoretically verified in (Allen-Zhu et al., 2019) and empirically estimated in (Yuan et al., 2019) for deep neural networks. It is worth noting that PI. condition is weaker than many conditions such as strong convexity, restricted strong convexity and weak strong convexity (Karimi et al., 2016).
Finally, we will refer to
& as condition number throughout this study. p
3

---

Under review as a conference paper at ICLR 2021
4 MAIN RESULTS
In this section, we present the main results for label-mixing augmentation satisfying (2). Due to the space limitation, we present the results of label-preserving augmentation satisfying (1) in Ap- pendix A. Since we have access to m > augmented data, is natural fully leverage the augmented data during training. But on the other hand, due to the data bias 6y, the prediction model learned from augmented data D could be even worse than training the prediction model di- rectly from the original data D, as revealed by Lemma (its proof can found in Appendix C) and its remark. Throughout this section, suppose that a mini-batch SGD is used for optimization, i.e. to optimize [(w), we have
myg
W1+1 Wt
"
(y ((x,
m2
where 7) is the step size, mo is the batch size, and (xk,t, yk.r), k L...., my are sampled from D. A similar mini-batch SGD algorithm can be developed for the augmented data.
Lemma 1. Assume that and E satisfy properties in Definition 1, 2 and 3, by setting 1j = 1/ and
mo 2 8 when t 2 L log 4(C(w)) -ow. C(w.))) we have
ja
E[[(w(+1) [(w.)) S / 1 066//1)))
(8)
where W141 is output of mini-batch SGD trained on D. 8y is defined in (3).
Remark: It is easy verify (see the details of proof in Appendix D) that if simply train the learning model by the original data D. we have
[[[(w)+)) ((w.)) S log(n)/ ( (np²))
Comparing the result in (9) with the result of (8) in Lemma it is easy show that, when the data bias is too large, i.c., 63 2 Q (og(m)))m) we have O (L log(n)/(mp²) This implies that training the deep model directly on the original data D is more effective than on the augmented data D. Hence, in order to better leverage the augmented data in the presence of large data bias (6 2 Q(x log(n)/n), where K L/μ), we need to come up with approaches that automatically correct the data bias. Below, we develop two approaches to correct the data bias. The first approach, termed "AugDrop", corrects the data bias by introducing constrained optimization approach, and the second approach, termed "MixLoss", addresses the problem by introducing a modified loss function.
S ((6//p) p).
4.1 AugDrop: CORRECTING DATA BIAS BY CONSTRAINED OPTIMIZATION To address this challenge, we propose constrained optimization problem, i.c.
min L(w) s.t. ž(w) - Z (.W) 5/ WERD
(10)
a positive constant, W. defined in (6). The key idea is that by utilizing the aug- mented data to constrain the solution in a small region, we will be able to enjoy smaller condition number, leading to a better performance in optimizing L(w). To make it concrete, we first define three important terms:
where y
To (2p), A(7) {w: [(w) -(w.) S $3).
(11)
p(7) max {[(w) L(w.) [L(( E A(7))
(12)
R²
We then present proposition about A(7) and μ(()), whose proof is included in Appendix E. Proposition 1. If- E [no. 8ru], we have w. E A(7) and p(() 2 p.
According to Proposition 1, by restricting our solutions in A(r), we have a smaller condition number (since u²)) 2 μ4) and consequentially smaller optimization error. It is worth mentioning that the restriction of solutions A(y) is reasonable due to the optimal solution w. E A(y). The idea of using augmentation transformation to restrict the candidate solution recognized by several
4

---

Under review as a conference paper at ICLR 2021
earlier studies, e.g. (Raghunathan et al., 2020). But none of these studies cast it into a constrained
optimization problem, a key contribution of our work.
The next question is how to solve the constrained optimization problem in (10). It is worth noting that neither [(w) nor [(w) is convex. Although multiple approaches can be used solve non- convex constrained optimization problems (Cartis et al., 2011; Lin et al., 2019b; Birgin & Martínez, 2020: Grapiglia & Yuan, 2019; Wright, 2001: O'Neill & Wright, 2020; Boob al., 2019; Ma al., 2019), they are too complicated to be implemented in deep learning. Instead, we present a simple approach that divides the optimization into two stages, which is referred to as AugDrop (Please see the details of update steps from Algorithm 2 in Appendix F).
Stage We minimize [(w) over the augmented data D. It runs a mini-batch SGD against D at least Ti iterations with the size of mini-batch being m. We denote by WTT+1 the final output solution of this stage.
Stage II. We minimize L(w) using the original data D. It initializes the solution W1;+1 and runs mini-batch SGD against D in n/my iterations with mini-batch size being m2- We notice that AugDrop is closely related to TSLA by (Xu et al., 2020) where the first stage trains the data with label smoothing and the second stage trains the data without label smoothing. However, they study the problem how to reduce the variance of stochastic gradient in using label smoothing. while we study how to correct bias in data augmentation by solving constrained optimization problem. The following theorem states that if we run this two stage optimization algorithm, we could achieve better performance since p(8r))) is larger than μ.. We include its proof in Appendix F.
Theorem 1. Define pe pu(8%))- Assume that L and L satisfy properties in Definition 1, 2 and 3,
set learing rate mH1 1/ in Stage I and learning rate 1/2 2npe log (Cp" [(w.)) in Stage
GLL
II for AugDrop. Let w be the initial solution in Stage I of AugDrop and WI,+2..... Wrinn/ma+ be the intermediate solutions obtained by the mini-batch SGD in Stage II of AugDrop. Choose
Ti 1 log 2(((w)) E(w.>>e
7881 = (1
2Ti 1 V 3log
2
2z and m2 = (1 /3oog 8y
2 %)
with probability 1- 8, we have We E A(870), Vt E {27 + 2,...,, Ti + +n/m² + 13 and
L
(1=
(4np)(C(w)) (1 - [(w.))
))
L (
[[C(W) [(w.)] Ampu?
00
(13)
1
where w- Wn/+n/mat1 and oy is defined in in (3).
Remark. Theorem 1 shows that all intermediate solutions w, obtained in Stage II of AugDrop sat- isfy the constraint C(w.)- - Z(w.) S 8%o, that to say, W. E A(8))). Based on Propo- sition 1, we will enjoy a larger pe than μ.. Comparing the result of (13) in Theorem with (9), training by using AugDrop will result in a better performance than directly training on D due to Me 2 n.. Besides, when the data bias is large, i.c., 67 2 Ωl log(n)/(npi) we know OoLlog()) (oog())((mp)))) S Ouuaz/ p2) < (62 p)), where the last inequality holds due to Mr. 2 μ. By comparing (13) with the result of (8) in Lemma 1, we know that training by using AugDrop has a better performance than directly training on D when the data bias is large. By solving a constrained problem, the DroD algorithm can correct the data bias and thus can enjoy an better performance.
4.2 MixLoss: CORRECTING DATA BIAS BY MODIFIED Loss FUNCTION
Without loss of generality, we set L(w,) = 0, common property observed in training deep neural networks (Zhang et al., 2016; Allen et al., 2019; Du et al., 2018; 2019; Arora et al., 2019; Chizat et al., 2019; Hastie et al., 2019; Yun et al., 2019; Zou et al., 2020). Since ||y - - < 6, for any y and y and given x, we define new loss function e, (5, / [3: w)) as
f(x: w))
| min [(z, f (x: w)). 6,
(14)
It has been shown that since the cross- cross-entropy loss ((z,-) is convex in terms of E D, then the minimization problem (14) is a convex optimization problem and has a closed form solution (Boyd & Vandenherghe, 2004). Using this new loss, we define new objective function L. (ww
La (w) = E (v, f(;; w))] = E
min 2 8
C(z, f(x; w))
(15)
5

---

Under review as a conference paper at ICLR 2021
Algorithm WeMix
1: Input: T, T2, stochastic algorithms A1. A2 (e.g., momentum SGD)
2: Initialize: w1 E IR, À E (0, 1). m/. 72 > 0
// First stage: Weighted Mixed Losses
3: for t . do
4: draw examples (x,, ) at random from training data o construct stochastic gradient of
5: generate augmented examples Vi.
construct stochastic gradient of La
6: compute stochastic gradient ge = (V. (x), + (1 A (Yi, [(x,; w.))
7: Wi-1 ( Bc, 771)
update one step of A1
8: end for
// Second stage: Augmentation Dropping
9:
for to = 1,7 ......, + T2 do
10:
draw examples (x,,, Vi.) random from training data
construct stochastic gradient of L
11:
compute stochastic gradient ge - Vi(y (xx, W.))
12: Wr+1 = A2 W.; 12)
update one step of A
13:
end for
Output: wT+
is easy to verify that (w. = O and therefore w. also minimizes La(w) (see Appendix G). In contrast, w., the minimizer of L(w), can be very different from w.. Hence, we can correct the data bias arising from the augmented data by replacing [(w) with (a(w), leading to the following optimization problem:
min (w( XC(w) + (1 (W),
(16)
WERD
where E (0, 1). Since Lo(w) shares the same minimizer with L(w) (see Appendix G), it is sufficient to optimize (w(.). instead of optimizing L(w). The main advantage of minimizing (w( over L(w) is that by introducing small X, we will be able to reduce the variance com- puting the gradient of [w), and therefore improve the overall convergence. More specifically, our SGD method is given as follows: at each iteration f, we compute the approximate gradient as g: AV [x;; w:)) (1 >) >> D (Va
of (Xt, Wt)), where (x,,y,) ) is an example sampled from D at iteration L. We refer to this approach as MixLoss (Please see the details of up- date steps from Algorithm in Appendix H). We then give the convergence result in the following theorem, whose proof is included in Appendix H.
mo
Theorem 2. Assume that f. Ladd
and La satisfy properties in Definition 2 and 3, by setting mn 2 1. log p(Fr)( XLLG"
-2( and pn X²
< 1 in MixLoss, we have
2T.
ALG2 nir2
(1+5log ny=(((w)) XLT.2
(ALlog(n/X
AL log(n/22)
E [[((w++)) [(w.)] S
SO
(17)
14p2
Remark. According to the results in (17) and (9), we know that O lognnA/n/)3)
(0, selected, leading to a better performance by using MixLoss compared with the performance trained on the original data D. For exam- ple, one can simply use X =.. On the other hand, when the data bias large where we know OL log(n)/ (nqx)) S O(62/p). Based on previous discussion, by choosing an appropriate X (0. (e.g., X O(u/T.), we will have < O(8²/u). Then by comparing (17) with (8), we know that training by using MixLoss has better performance than directly training on D when the data bias is large. Therefore, by solving the problem with a modified loss function, the MixLoss algorithm can enjoy a better performance by correcting the data bias.
O [og()))) when an appropriate
oy satisfying 54 2
O
4.3 WeMix: GENERIC WEIGHTED MIXED LOSSES WITH AUGMENTATION DROPPING AL.GORITHM
Inspired by previous theoretical analysis of using augmented data, we propose generic frame- work of weighted mixed losses with augmentation dropping that builds upon two algorithms, Aug- Drop and MixLoss. Algorithm describes our procedure in detail, which is referred to WeMix. consists of two stages, wherein the first stage it runs a stochastic algorithm Aj (c.g., momen- tum SGD, SGD) for solving weighted mixed losses (16) and the second stage it runs another/same

---

Under review as a conference paper at ICLR 2021
Table 1: Comparison of Testing Top-1 Aceuracy (mean + standard deviation, in %) using Different Methods on ResNet-18 over CIFAR-I0 and CIFAR-100 for mixup
Method
CIFAR-I00
CIFAR-II 10
without mixup
76.97 = 0.27
94.95 + 0.17
mixup
78.31 + 0.18
95.67 + 0.09
AugDrop (ours)
80.24 0.34
96.03
Mixl Loss (ours)
79.70 + 0.31
95.94 + 0.11
WeMix (ours)
80.61 + 0.10
96.11 + 0.11
MixLoss-s (ours)
79.53 + 0.13
95.87 + 0.14
WeMix-s (ours)
80.29 + 0.22
96.06 + 0.16
stochastic algorithm Ag (c.g., momentum SGD, SGD) for solving the problem over original data. The notation A(s-. is one update step of a stochastic algorithm A with learning rate 1j. For ex- ample, if we select SGD as algorithm A. then SGD we; Be, 7) = W; 7/.. The proposed WeMix is generic strategy where the subroutine algorithm A1/Az can be replaced by any stochastic algo- rithms such as stochastic versions of momentum methods (Polyak, 1964; Nesterov, 1983; Yan et al., 2018) and adaptive methods (Duchi al., 2011; Hinton et al., 2012: Zeiler, 2012; Kingma & Ba, 2015; Dozat, 2016; Reddi et al., 2018). We can also replace la by e to avoid solving a minimization problem. The last solution of the first stage will be used as the initial solution of the second stage. If A = 0 and la = É, then WeMix reduces to the AugDrop: while if T2 = O, WeMix becomes to MixLoss. For label-pres case, we only need to simply use la = (i.e, 6y = 0) in WeMix.
EXPERIMENTS
To evaluate the performance of the proposed methods, trained deep neural networks two benchmark data sets, CIFAR-10 and CIFAR-100' (Krizhevsky & Hinton, 2009) for the image clas- sification task. Both CIFAR-I0 and CIFAR-100 have 50,000 training images and 10,000 testing images of 32 32 resolutions. CIFAR- has 10 classes containing 6000 images each, while CIFAR- 100 has 100 classes. We use mixup (Zhang et al., 2018) as an example of table-mixing augmentation and Contrast as an example of lable-preserving augmentation and. For the choice of backbone, we use ResNet- 18 model (He al., 2016) mixup, and Ride-RessNet22--- modakis, 2016) is applied in the Contrast experiment following by (Cubuk et al., 2019; 2020). To verify our theoretical results, we compare the proposed AugDrop and MixLoss with two baselines, SGD with mixup/Contrast and SGD without mixup/Contrast (baseline). We also include WeMix in the comparison. The mini- batch size of training instances for all methods is 256 as suggested by He et al. (2019) and He et al. (2016). The momentum parameter of 0.9 is used. The weight decay with the parameter value is set to be 5 x 10-4, The total epochs of training progress is fixed as 200. Followed by (He et al., 2016; Zagoruyko & Komodakis, 2016), we use 0.1 as the initial learning rates for all algorithms and divide them by 10 every 60 epochs.
model (Zagoruyko Ko-
For AugDrop, we drop off the augmentation after s-th epoch, where & {150, 160, 170, 180, 190) is tuned. For example, if 8 = 160, then it means that we run the first stage of Arg 160 epochs and the second stage 40 epochs. For MixLoss, we tune the parameter by from {0.5, 0.05, 0.005, 0.0005) and the best performance is reported. For WeMix, we use the value of 6, with the best performance in MixLoss, and we tune the dropping off epochs s same as AugDrop. We fix the convex combi- nation parameter À = 0.1 both for MixLoss and WeMix. We use top-1 accuracy to evaluate the performance. All top-1 accuracy on the testing data set are averaged over 5 independent random trails with their standard deviations.
5.1 MIXUP
Given two examples yt) and (x) that are drawn at random from the training data, mixup creates a virtual training example as follows x' 8x, + (1 (1-33 y)x = By, (1-1)))) where 3 E [0, 1] is sampled from a Beta distribution 3(a, a). We use a = 1 in the experiments as suggested in (Zhang et al., 2018). In this subsection, we want to empirically verify that our theoretical findings for label-mixing augmentation in Section 4. The experimental results conducted on CIFAR-10 and CIFAR-100 are listed Table 1. We can see from the results that both AugDrop and MixLoss are
'https:/ ht/////www/ /ww.//wwwwwoo//ooo/// cs. www.ttoroooooooo wrriz//krri//c html
7

---

Under review as a conference paper at ICLR 2021
Table Comparison of Testing Top-1 Aceuracy (mean + standard deviation, in %) using Different
Methods on ResNet-18 over CIFAR-100 for mixup of three images and ten images
Method
images
10 images
Mixup
76.56 + 0.23
60.36 = 0.88
AugDrop
80.18 = 0.19
76.35 = 0.27
MixLoss
79.61 = 0.09
75.41 = 0.19
WeMix
80.41 + 0.22
78.08 + 0.11
Table 3: Comparison of Testing Top-1 Accuracy (mean + standard deviation, in %) using Different Methods on WideRessNet-28-11 over CIFAR-10 and CIFAR- 100 for Contrast Transformation
Method
CIFAR-100
CIFAR-10
without Contrast
78.07 + 0.27
95.51 + 0.14
Contrast
77.90 + 0.26
95.66 + 0.05
AugDrop (ours)
78.40 = 0.24
95.93 + 0.21
MixLoss (ours)
78.17 = 0.20
95.70 = 0.11
WeMix (ours)
78.79 0.18
95.81 + 0.11
better than two baselines, with and without mixup, which matches the theory found in Section 4. The performance of MixLoss is slightly worse than that of AugDrop, but they are comparable. Besides, the proposed WeMix enjoys both improvements, leading to the best performance among all algorithms although its convergence theoretical guarantee is unclear.
Next, we implement MixLoss and WeMix with by (i.e., use la = E), which are denoted by MixLoss-s and WeMix-s, respectively. We summarize the results in Table 1, showing that both MixLoss-s and WeMix-s drop performance, comparing with MixLoss and WeMix, respectively.
Besides, we use more than two images in mixup such as three and ten images and the results are shown in Table 2. Although the top-1 accuracy of mixup reduces dramatically, we find that the proposed WeMix can still improve the performance when it comparing with mixup itself, showing the robustness of WeMix.
5.2 CONTRAST
As a simple label- prbeeering augmentation, Contrast controls the contrast of the image. Its transfor- mation magnitude is randomly selected from uniform distribution [0.1, 1.9] following by (Cubuk et al., 2019). Despite its simplicity, we choose to demonstrate our theory for the considered case in Appendix A. The results of highest top-1 accuracy on the testing data sets for different methods are presented in Table 3. We find that by directly training on data with Contrast, it will drop the performance little bit. Even so, the result shows that AugDrop has better performance than two baselines, which is consistent with the theoretical findings for label- preserving augmentation in Ap- pendix A that we need use the data augmentation at the early training stage but drop the end of training. Although there is no theoretical guarantee for the label-preserve implement MoxLoss and WeMix by setting o, = 0, i.e., using l, = l' (14). The results show that Mixloss and WeMix are better than two baselines but are slightly worse than AugDrop.
transformation case, we
CONCLUSIONS AND FUTURE WORK
In this paper. we have studied how to better utilize data augmentation in training deep neural net- works by designing two training schemes with the first one switches augmented data to original data during the training progress and the second one training a convex combination of original loss and augmented loss. We have provided theoretical analyses of these two training schemes in non-convex smooth optimization setting, With the insights of theoretical results, we have designed generic algorithm WeMix that can well leverage data augmentation in practice. We have verified our theoretical finding throughout extensive experimental evaluations training ResNet and WideRes- Net models over benchmark data sets. Despite the effectiveness of WeMix, its theoretical guarantee still not fully understand. We would like leave this open problem future work.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. International Conference on Machine Learning, pp. 242-252 2019.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. arXiv preprint arXiv:/901.08884 2019.
Henry S Baird. Document image defect models. In Structured Document Image Analysis, pp. 546- 556. Springer, 1992.
EG Birgin and JM Martínez. Complexity and performance of an augmented lagrangian algorithm. Optimization Methods and Software, pp. 1-36, 2020.
Digvijay Boob, Qi Deng, and Guanghui Lan. Proximal point methods for optimization with 02730,
convex functional constraints. arXiv preprint
Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
Coralia Cartis, Nicholas IM Gould, and Philippe Toint. On the evaluation complexity of composite function minimization with applications to nonconvex nonlincar programming. SIAM Journal on Optimization, 21(4):1721 2011.
Zachary Charles and Dimitris Papailiopoulos. Stability and generalization of learning algorithms that converge to global optima. In International Conference on Machine Learning, pp. 745-754, 2018.
Lin Chen, Yifei Min, Mingrui Zhang, and Amin Karbasi. More data can expand the generalization gap between adversarially robust and standard models. arXiv preprint arX1v:2002.04725, 2020.
Shuxiao Chen, Edgar Dobriban, and Jane H Lee. Invariance reduces variance: Understanding data augmentation in deep learning and beyond. arXiv preprint arXiv:1907.10005 2019.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. In Advances in Neural Information Processing Systems, pp. 2937-2947, 2019.
Imre Csiszar and János Körner. Information theory: coding theorems for discrete memoryless sys-
lems. Cambridge University Press, 2011.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 113-123, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Tri Dao, Albert Gu, Alexander Ratner, Virginia Smith, Chris De Sa. and Christopher Re. kernel theory of modern data augmentation. In International Conference on Machine Learning, pp. 1528 1528-1537, 2019.
Timothy Dozat. Incorporating nesterov momentum into adam. 2016.
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International Conference on Machine Learning, pp- 1675- 1685, 2019.
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterizee neural networks. arXiv preprint 111.0:1810 2018.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal Machine Learning Research, 12:2121-2159, 2011.

---

Under review as a conference paper at ICLR 2021
Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochas- tic programming. SIAM Journal on Optimization, 23(4):2341 -266, 2013.
Saeed Ghadimi, Guanghui Lan, and Hongchao Zhang. Mini-bated stochastic approximation meth- ods for nonconvex stochastic composite optimization. Mathematical Programming, 155(1-2): 267-305, 2016.
Geovani N Grapiglia and Ya-xiang Yuan. On the complexity of augmented lagrangian method for nonconvex optimization. arXiv preprint arXiv: 1906. 05622, 2019.
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan Tibshirani. Surprises in high-
dimensional ridgeless least squares interpolation. arXiv preprint arXiv: (903::/9000080000 2019.
Ryuichiro Hataya, Jan Zdenek, Kazuki Yoshizoe, and Hideki Nakayama. Faster autoaugment:
Learning augmentation strategies using backpropagation. arXiv preprint arXiv:1911.0690 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 558-567, 2019.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. 2012.
Daniel Ho, Eric Liang, Xi Chen, Ion Stoica, and Pieter Abbeel. Population based augmentation: Efficient learning of augmentation policy schedules. In International Conference on Machine Learning, pp. 2731-2741, PMLR, 2019.
Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal- gradient methods under the polyak-tojasiewicz condition. Joint European Conference on Ma- chine Learning and Knowledge Discovery in Databuses, pp. 795-811. Springer, 2016.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International
Conference on Learning Representations, 2015.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Mas- ter's 's's thesis, Technical report, University of Tronto, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks, In Advances in Neural Information Processing Systems, pp. 1097-1105, 2012.
Xiaoyu Li, Zhenxun Zhuang, and Francesco Orabona. Exponential step sizes for non-convex opti- mization. arXiv preprint arXiv:2002.05273. 2020.
Zhize Li and Jian Li. A simple proximal stochastic gradient method for nonsmooth nonconvex optimization. In Advances in Neural Information Processing Systems, pp. 5564-557,, 2018.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. In
Advances in Neural Information Processing Systems, pp. 6665-6675,
Chen Lin, Minghao Guo, Chuming Li. Xin Yuan, Wei Wu, Junjie Yan, Dahua Lin, and Wanli Ouyang. Online hyper-parameter learning for auto-augmentation strategy. In Proceedings of the IEÉE International Conference on Computer Vision, pp. 6579-6588 2019a.
Qihang Lin, Runchao Ma, and Yangyang Xu. Inexact proximal-pone penalty methods for non- convex optimization with non-convex constraints. arXiv preprint 1rXiv:1908.11516 2019b.
Runchao Ma, Qihang Lin, and Tianbao Yang. Proximally constrained methods for weakly convex optimization with weakly convex constraints. arXiv preprint /008.0/9000011777 2019.
10

---

Under review as a conference paper at ICLR 2021
Yifei Min, Lin Chen, and Amin Karbasi. The curious of adversarially robust models: More data can help, double descend, or hurt generalization. arXiv preprint arXiv:2002.11080 2020.
Yurii Nesterov. A method of solving a convex programming problem with convergence rate O(1/k²). Soviet Mathematics Doklady, 17:372-376, 1983.
Yurii Nesterov. Introductory lectures convex optimization: a basic course. Applied optimization. Kluwer Academic Publ., 2004. ISBN 1-4020-7553-7.
Michael O'Neill and Stephen J Wright. A log-barrier newton-eg method for bound constrained optimization with complexity guarantees. IMA Journal of Numerical Analysis, 2020.
Boris T Polyak. Some methods of speeding up the convergence of iteration methods. USSR Com-
putational Mathematics and Mathematical Physics, 4(5):1-17, 1964.
Boris Teodorovich Polyak. Gradient methods for minimizing functionals. Zhurnal Vychislitel noi Matematiki Matematicheskoi Fiziki, 3(4):643-653, 1963.
Aditi Raghunathan. Sang Michael Xie, Fanny Yang. John Duchi, and Percy Liang. Understanding and mitigating the tradeoff between robustness and accuracy. arXiv preprint arXiv:2002. 10716, 2020.
Shashank Rajput, Zhili Feng, Zachary Charles, Po-Ling Loh, and Dimitris Papailiopoulos. Does data augmentation lead to positive margin? In International Conference on Machine Learning, pp. 5321-531355 15330,, 2019.
Sashank Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In International Conference Learning Representations, 2018.
Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural networks, 61:85-117, 2015.
Connor Shorten and Taghi M Khoshgoftaar, A survey on image data augmentation for deep learning. Journal of Big Data 6(1):60, 2019.
Yuji Tokozume, Yoshitaka Ushiku, and Tatsuya Harada. Between-class learning for image classifi- cation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp- 5486-5494, 2018.
Alexandre B Tsybakov. Introduction nonparametric estimation. Springer Science & Business Media, 2008
Zhe Wang, Kaiyi Ji. Yi Zhou, Yingbin Liang, and Vahid Tarokh. Spiderboost and momentum: Faster variance reduction algorithms. In Advances in Neural Information Processing Systems, pp. 2400-2413, 2019.
Stephen J Wright. On the convergence of the cartoo//log-baricc method. Mathematical Program- ming, 90(1):71-100, 2001.
Sen Wu, Hongyang Zhang. Gregory Valiant, and Christopher Ré. On the generalization effects of linear transformations in data augmentation. arXiv preprint arXiv:2005.00M: 2020.
Bo Xie, Yingyu Liang, and Le Song. Diverse neural network learns true target functions. In Artificial Intelligence and Statistics, pp. 1216-1222 2017.
Yi Xu, Yuanhong Xu, Qi Qian, Hao Li, and Rong Jin. Towards understanding label smoothing. arXiv preprint arXiv:2006. 11653, 2020.
Yan Yan, Tianbao Yang, Zhe Li, Qihang Lin, and Yi Yang. A unified analysis of stochastic momen- tum methods for deep learning. International Joint Conference on Artificial Intelligence, pp. 2955-2961. 2018
Zhuoning Yuan, Yan Yan, Rong Jin, and Tianbao Yang. Stagewise training accelerates convergence of testing error over sgd. Advances in Neural Information Processing Systems, pp. 2604-2614 2019.
11

---

Under review as a conference paper at ICLR 2021
Chulhee Yun, Suvrit Sra, and Ali Jadbabaie. Small relu networks powerful memorizers: tight analysis of memorization capacity. In Advances in Neural Information Processing Systems, pp- 15558-15569 2019.
Sergey Zagoruyko and Nikos Komodakis. 1rXiv:6605.07146, 2016.
Wide residual networks.
arXiv preprint
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 16 2016.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Xiang Zhang. Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. In Advances in neural information processing systems, pp. 649-657, 2015.
Barret Zoph, Ekin Cubuk, Golnaz Ghiasi, Tsung-Y Yi Lin, Jonathon Shlens, and Quoe V Le. Learn- ing data augmentation strategies for object detection. arXiv preprint 19..v::906.111777 2019.
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes over- parameterized deep relu networks. Machine Learming, 109(3)466--992, 2020.
12

---

Under review as a conference paper at ICLR 2021
A
MAIN RESULTS FOR LABEL-PRESERVING AUGMENTATION
We consider label-preserving augmentation case (1), that is,
P. (+lx) (-()),, VX E T(x but Px + Px.
covers many image data augmentations including translation, adding noises, small rotation, and brightness contrast changes (Krizhevsky et al., 2012; Raghunathan et al., 2020). It is worth mentioning that the compositions of aubel-preserving augmentation could also be presering.. Similar to the case of label-mixing augmentation, we measure the following difference between Px and Px by a KI. divergence:
(x((x) (xx(x)
8p = DKL. (Pxx Ex
log
(18)
Due to the data bias 6p, the prediction model learned from augmented data D could be even worse than training the prediction model directly from the original data DD revealed by the following lemma and its remark.
Lemma 2. fleseriggseerving augmentation) Assume that L and E satisfy properties in Definition I, 4 2 and 3, by setting n = I/L. L. and mo Z "ōpp when 2 to = log
(C(w.)-L(w.)) we have 28p
48p S 0
0p
3xx ([(w++1)) [(w.)]
(19)
where Wi41 is output of mini-bated SGD trained on D, 8p is defined in (18).
Proof. See Appendix I.1.
Comparing the result in (9) with the result of (19) in Lemma 2, it easy to show that, when the data bias too large, i.e., 6p 2 (L log(n)/(nii) we have (L log(n)/(pp²) S 3(ōp/P) ()). This implies that training the deep model directly the original data D is more effective than the augmented data D. Hence, in order better leverage the augmented data in the presence of large data bias (ōp 2 (K (nn(m)/n), where x = L/μ), we need to come up with an approach that automatically correct the data bias in D. Below, use AugDrop correct the data bias by solving a constrained optimization problem.
Remark:
A.1 AugDrop: CORRECTING DATA BIAS BY CONSTRAINED OPTIMIZATION
To correct data bias, we consider to solve the constrained optimization problem (10). The key idea is to shrink the solution in small region by using utilize augmented data to enjoy smaller condition number, leading an improved convergence in optimizing L(w). By introducing term that
Y1 8p /p.
we can present a proposition about A(7) and p(>), showing that we have a smaller condition number and consequentially smaller optimization error by restricting solutions to A(7).
Proposition 2. i E [n,47), we have w. E A(7) and u(7) 2 where A(7) and p(7) are defined in (11) and (12), respectively.
Proof. See Appendix 1.2.
The following theorem shows the convergence result of AugDrop for label-preserving augmentation.
Theorem 3. Define 7 M1 (u(4)))) Assume that L. and È satisfy properties in Definition 1, and 3, set learning rate Tn 1/L in Stage and learning rate Tp2 2npe log (8n2 ((n)(((())-((w.) G.T.
in Stage II for AugDrop. Let w, be the initial solution in Stage I of AugDrop and WT,+2, WTitn/mat1 be the intermediate solutions obtained by the mini-batch
SGD in Stage Il of AugDrop. Choose Ti = L log 2(2(w1) E(W..)) 111 mi
(1 (1+ 3log 2Ti
2
8.
8p
3p
13

---

Under review as a conference paper at ICLR 2021
and m2 (1+ + V log 2
or with a probability 8, we have We E A(4r)),
{Ii + 2,...., 11 n/ m2 + +13 and
L
G 8np2
(np2(((wW)) - [(w.))) ((. so
(
)) (20)
E. [[[(WW [(w.)] - 8mp2
log
G2L
np?
where w = Wi and öp is defined in (18).
Proof. See Appendix I.3.
Remark. Theorem 3 shows that all intermediate solutions w, obtained in Stage II of AugDrop sat- isfy the constraint [(w,)- [(w.) < 471, that to say, W; E A(4n). Based on Propo- sition 2, we will enjoy a larger je than μ.. Comparing the result of (20) in Theorem 3 with (9), training by using AugDrop will result in better performance than directly training on D due Mr. 2 μr. Besides, when the data bias is large, i.e., 8p > Q (og(n)/(m4)), we know OLL loo(oo))/// (n)/(np() S Ouudp/pp 222) & Ogōp/Uu /1), where the last inequality holds due to jr 2 p. By comparing (20) with the result of (19) in Lemma 2, we know that training by using AugDrop has better performance than directly training on D when the data bias is large. By solving a constrained problem, the AugDrop algorithm can correct the data bias and thus enjoy an better performance.
B TECHNICAL RESULTS FOR CROSS-ENTROPY LOSS
Lemma 3. Assume that L(w) = E[e(y, w))] satisfies property in Definition I, where é is a cross-entropy loss, then we have
IV f(x:w) - (x((F, Glyy - -1
(21)
and
(22)
Proof.
The objective function is
((w) = E(xx) [ fxx w))],
(23)
where the cross-entropy loss function é is given by
K
exp(fi(x: w))
f(y,f(x;w) = Σ
Yi log
(24)
Σ exp(fx (x: w))
1=1
Let set
exp(fi(x: w))
p(x; (pi (x; w),...,,,kr Px(( w)), pi(x;w) =- -
(25)
Σラ Σ exp(ff w/)
then the gradient of l with respective to w is
Vely, w)) (y, V)(x:w)).
(26)
Therefore, Vx X' and w e R we have
| Cy ey, f(x; w)) - V f(x
(y y, Vp(x; (p(x:w))]1
< Vp(x; w) ||y - F11
<Glly- -il.
(27)
and
I f(, w))l (y, Vp(x; w) S [[Vx((x ||yl| SG
(28)
where uses the facts that [/Vp(x; w)ll < G and ||yl| - ||>|| = 1, here || ||| is Euclidean norm (6) norm) and || [| is &, norm.
14

---

Under review as a conference paper at ICLR 2021
C
PROOF OF LEMMA
Proof. Recall that the update of mini-batch SGD given by
Wi+1 7/8r-
Let set the averaged mini-batch stochastic gradients of [(we) as
Ve (V (Xx, wx))
then by the Assumption of E satisfying the property Definition 1, we know that S(s)) (Ve (Vr,s, (Xx,, wr))] vā(w:),
Vi E ml)
(29)
and thus
E 3z) [r] = Vā(w;),
(30)
where we write g. ance ge given by
[ as E(x,, sr.)) I.. ..I [ for simplicity. Then the norm vari-
E( 11 - vi((w:)11
Ve ((. - v V((we))
(()
mo
(a)
mo
V [R Vā(W:)) 19
m
2
1=1
4G²
(31)
mh
where (a) is due (29) and the pairs Vt.1,. (N Vt.ma) are independently sampled from D; (b) is due to the facts that the Assumption of satisfying the property in Definition and Lemma 3, and then by Jensen's inequality, we also have V|V S G, implying that Ve(s, f(x; (R) vū(w) property in Definition 2, we have
4G2. On the other hand, by the Assumption of L satisfying the
F ( ([(w++)) - [(w:)]
(VC(Ww)), we+1 - we) + L _ / - -will will2
(x.)))))
(a) 77
(x. [[/]L(w,) - -IV - n/) ||
2
())
(IV vā(w))1 + (T)
[IVŽ(w.) - |VWL(
2
- (1 (Z) E (1
(0) [V.)). V2(
4G2
[V[(w))l
(32)
mo
where the (a) is due to the update of Wx+1 Wt ng. (b) is due (30): (c) is due 17 1/L and (31). By using the Assumption of E and satisfying the property in Definition I and Ixx Px have
VV((w:) vā(w:)) Fi V V (xx; (xx)) (V (K:
Ex 33 V.llV f(x; w.)) Vwl(F, w:)
GE (llly 511]
G5y,
(33)
15

---

Under review as a conference paper at ICLR 2021
where (a) uses Jensen's inequality; (b) is due to (3). By using the Assumption of C satisfying the
property in Definition 3 and (33), inequality (32) becomes
g. [C( ((2+1) [(w:)]
G
2gG
V
2
mo
2
G22
2g/
np (C(w,) [(w.)).
2
mo
which implies
E 3.) (((e+1)) - L(w+)]
G
2ny
s (1 mp)) (C(we) [(w.))
mg
1/ +
G1
$ (1 nu)' (C(w1) - f(w.))+ +
Σ na)
mo
Due to (1 - n4)² v exp( tnpp) and E1 (1 -m)) 1
when
mm 2
8z
and
L
4(C(wI) L(w.))
2 log
82G2
11
we know
62
E ([(e+)))) -(W
D PROOF OF (9)
We first put the full statement of (9) the following lemma.
Lemma 4. Assume that C satisfies the properties in Definition 1, 2 and 3, by set-
2nd log (nnn"([(wi) ((w))-C((we ((w.))) we have F(), [f(wn+1) L (w.)]
Ga + Eng"
ting 7)
S
GLL
L log Sny
( ((w.)) G²L
where wn+1 is output of SGD trained on D.
Proof.
By the Assumption of L. satisfying the property in Definition 2, have E(x...)) - L(w,).
<E (VL((W.) wat1 - -w,)] + 2 E(xm)ya) []| -w,ll
(a)
Ve (y sxxni w..)))] + 1 E(xnnya [I[V (yn, f(xii (x)::w)) [2]] ITT. 2 Exx. y..) VV (yn, ((x.; wx)) 1|2),
( mlIv<(w.))l
where (a) is due to the update of w W,, V f(x,i i w,));; (b) is due to the Assumption of satisfying the property in Definition that Fx(xy) [wl (y. f(x; w)) VL(w). By using the Assumption of L satisfying the property in Definition I that V (yy, fxx; w))ll < G and the Assumption of L satisfying the property in Definition 3, we have E Flx.,..](((ww4. [C(w,+1) L(w,)]
1" LG
-
2
LC2
2nu (L(wn) - L(w.)),
2
16

---

Under review as a conference paper at ICLR 2021
which implies
E(x,.y (L),s)))(((ww)))) - L(w.)]
G22
(C(w,) $ (1 2TH) E(xx- [C(wn) - -((w))] +
2
< (1 -211)" (C(w1)- ((w.)) L22
Σ
2111)².
1=0
Due to (1 - 22pp)) E (xsp(-2mun)) and Σi=o (1-211))* $
1 2np
then by using the setting of
1
(n -(Ww)))
log
2np
L
we have
(( (L(Wn+1) [(w.)]
G2
sexp (-2nm)) (C(w1) ((w.))+
44
G
GLL
(C(w)) [(ww))
log
8np2
8n42
G
L log(n))
50
n1u2
E PROOF OF PROPOSITION 1
Proof. By using the Assumption of Ž satisfying the property in Definition 3, we have
Σ(w.) - Z(W.)
VL(w.)II
24
Vi(w.) - -V.(W.)
(a)
2/1
67G
241
where (a) is due to the definition w. in (6) so that VC(w.) 00 (b) follows the same analysis
&²
of (33) Lemma Thus we know w. E A(7) when 7 2 7o == 2p definition of μu()) in (12) and the Assumption of L satisfying the property in Definition 3, we know u(>) 2 a when ? 8po.
On the other hand, by the
F ALGORITHM AUGDROP AND PROOF OF THEOREM 1
We present the details of update steps for AugDrop and its convergence analysis in this section.
17

---

Under review as a conference paper at ICLR 2021
Algorithm 2 AugDrop
Input: Ti
2: Initialize: W1 E DR nh 122 > 0
// Stage Train Augmented Data
3: for t = 1,2. 7i do
4: draw mi examples (Xe,1 11,1, (.l)............. ..., (x..i Vr.m.) x at random from augmented data
5: update Wx+1
31 mi8
2i=1 Vwl (Vt,i, (Xx,1 wt))
6: end for
// Stage II: Train Original Data
7: for t = Ti + 1, + .., .i + +n/m2 do
8: draw m2 examples (Xt,1 Vt.1). (xe, Ve,m2) without replacement at random from orig- inal data
9:
update Wi+1 = We 92 Σ m2
Vwl (Ye,s, (x),
10: end for
11: Output: Wo1+n/m2+1-
Proof. In the first stage of the proposed algorithm, we run a mini-batch SGD over the augmented data DD with my the size of mini-batch. Let (Kt,i, Ve,a), = 1..... be the my examples sampled in the fth iteration. Let ge be the average gradient for the iterationn i.e.
V ( wi))
ge
m1
We then update the solution by mini-batch SGD: w(+1 We We-mggr- By using Lemma 4 of (Ghadimi et al., 2016), with probability 1 8', we have
g+ - V s (1+
(1+ log 51
8G
(34)
mi
By the Assumption of 2 satisfying the property in Definition 2 and the update of Wr+1 W, W//1//
we have
ā(w(+1) -(((,)
np
S -- (V
+
g
2
|VV[(.:)
1B
171
[V[(w)))?
71 - L) ig 1
2
2
(0) m 2
SG2
+
3 log
-
8'
m
where (a) uses the facts that (34), 7/1 = 1/L and the Assumption of 2 satisfying the property in Definition 3. Thus, with a probability (1 8')s, using the recurrence relation, we have
[(w++1) Z(W.)
2
8G2 m1
5(1 - (1-mm)(((w.)) I(w.)) 71
1 3 log
(11 - mH) ([(wi) - [(W.)) + 2
1 1
3 log
1
8G
1
m
(35)
8
m1 1=0
Due to (1 < exp(-tmmppp and Σt1 (1
mp)² -
1 when
mp
1
2(((w,) Z(w.))
t t> Ti
log
17
83G²
18

---

Under review as a conference paper at ICLR 2021
have
[(we+1)= - [(w))
I <exp(-tmil) - -
+ V
3 log 1
8' 5)
4G2
Tmi
2
62
(1+ log 8r
4G2
(36)
2p
pmn
Let 8'
6. if we choose my such that 27.
271
mi 1 V 3 3log 8
then for any 2 Tí, then with a probability 1 - 5/2 we have
8 G2
[(We+1) - -...) S
(37)
In the second stage of the proposed algorithm, we run mini-batch SGD over the original data D with m2 as the size of mini-batch. Let Nt,i), 1,..., m2 be the m2 examples sampled in the lth iteration. Let g. the average gradient for the iteration, i.e.
m2
g:
V (yy,a f(x,,;
m2
We then update the solution Wx+1 72B: By using Lemma 4 of (Ghadimi et al., 2016), with a probability 8", we have
8G2
g VL(w:)ll
1 1 3log 8" V
(38)
m2
By the smoothness of ((w) and the update of Wt+1 = Ww 72g.. we have [(w(+1) [(w.)
+ - (VC((, ..))
|g
2
72 V[(w.) 8/11 122
v[(w:))1
V
72(1 772L)
1@:11
2
2
(a)
S
m2
VV[(w. VL(W;))I + 12
V[(w:)]1 mp(([(w)) [(w.))
2
1/2 26
1 /3ogg 8"
1
SC2
-na(([(w.)) [(W.)),
(39)
m2
where (a) uses the facts that Young's inequality, 72 S 1/L and the Assumption of Z satisfying the property in Definition 3; (b) uses inequality (38) and the same analysis of (33) in Lemma 1. It is easy to verify that for any & E (Ti +,,,, Ti + n m/ we have, with a probability (1 - /0mmm,mm,, have [(we+1) - I(ww)
802
<( - 122) (Z(w.)) - I(W.)) - 922 G2 + [1 y 3log 8"
m22
2 8G2
(1
m21)
([(WT++1)- - -Z(w.) + m/2
G2O
1
3
724)
m22
G2
1
8G2
G2
1 v 3log 8"
m2
19

---

Under review as a conference paper at ICLR 2021
where the last inequality is due to (1 72/13* 1 1, Ex
(1
n2t23'
1
and (37). Let 8"
Tay"
2
2n/mg" we choose my such that m2
1 + / log m28
for example.
69
2n
M2 = 1+ 3 log
8
8
then with a probability - 6, for any t E {Ii + 1, ......
Ti + m/m2) we have
[(w(+1)- - -((w) S 45
Therefore, We E A(8%)) for any E {Ti+22 +21....... ......T, Ti + n m2 + 1}. Following the standard analysis in Appendix D, we have
G Ampi?
G log 4mpy
AAn (0))-(((w)) ((W.)))
E (f/wwttn/matt) - [(w.)) S
+
G
where ((8m))
G OPTIMAL SOLUTIONS OF Ca(w) AND Lo(w) (w)
By the definition of la in (14) and Px Px. we know
((x; w))
min {(z, ((x
12-y Is6,
<((y; f(x;
(40)
since ly -l1 S 6y. Therefore, by (15), (40) and Px = Px we have
(w())
= -EylCo(w) (w)]
=Eix.. (fa wx)]
SE(x,5.)) [e f(; w])]
-E [e(y, fx; w))l
-(())
(41)
Since a non-negative loss function, then we know
0 S La (w.) S L(w.) =0, 0.
which implies that
Cw(w)) = O,
and thus
Ca(w.) s La(w), Vw.
Therefore, w. also minimizes La (w).
On the other hand, by (16) we know
Cc(w.) XC(w.) + (1 (w.) 0.
Therefore,
(w.)) S Le(w), Vw.
iee, w. also minimizes (w),, indicating that (.(w) shares the minimizer as L(w))
20

---

Under review as a conference paper at ICLR 2021
H ALGORITHM MIXLOSS AND PROOF OF THEOREM 2
We present the details of update steps for MixLoss and its convergence analysis in this section.
Algorithm 3 MixLoss
Input:
2: Initialize: WT E ER.,,/
3: for (= 1, 2...., do
4: draw example Yc) without replacement at random from original data
5: draw mo examples Vt.1), (x, Vt. at random from augmented data
6: compute g: = XV f (x::w.)) + (1- X) mg mm Vla (Vr (Kx,,) Wt))
7: update Wx+1 = We 7/8
8: end for
9: Output: Wn+1.
Proof. Recall that
C.(w) XC(w) + (1 (w),
(42)
E(2,, [(; w))
where Lo Lw( =E(R,) [(x F(R
min S6,
and
1 mg
g: =
(1 A)
Vla (Xxe,
(43)
mu
By the update of W++1 w. - and by the Assumption of La satisfying the property in Defini- tion 2, we have (Nesterov, 2004)
E, [Co(44+)1) Co
S < (w.)). 8:)] + r² E, [|]
(0)
3 1111 E E. VCc )1
17 LE, [[g. - Vc.(
(
9 22 LE,
S 11 E VL. (w.) |
V((w:)ll?
1 mo mo
+ 9(1 1)3/PF.F.
Vla (fr. (x.,,:.)) VLa (w:))
(c)
9 x2
LG + 36(1 2) LG
5 - 1111 - EL
VL 11)
2
me
( S 711 LL F.
VV. (w. |1
1
(44)
where E. [-] is taken over random variables yt), (xx, Vt.1) (3e Fo.m ); (a) uses the facts that Young's inequality ||a-b|l
b1l² < 2| + 2||b|| and Egg (W.):
(w.); (b) uses the
facts that (42) (43) and Young's 's inequality ||a + b1l < (1+ 1/c]laall + (1 + b)|l with 8; (c) use the same analysis in (31) from the proof of Lemma 1, the facts that the Assumption of L satisfying the property in Definition I and by Jensen's inequality, we also have [|V[(w)] < G,
implying that [| w)) VL(w)lI < 4G²; (d) holds by setting mg >
72(1-A)²
x² since
we have sufficiently large number of augmented examples. Thus, since 17 S and by using the Assumption of L. satisfying the property Definition 3, we have
E, [Co(We+1) Lc(w))] S MnE [c.(w)) + L
21

---

Under review as a conference paper at ICLR 2021
and therefore
En [Cc(
Sexp (-rim) Lo(w)) 52
1i
Sexp ((wi) + 5X2nLC2
(45)
1
where last incquality is due to the fact that (.(w) - L(w). In (45), by choosing
n4²L((w)
= um L2
log
we have
L2 LA2 nu²
np²[((w)
En (Co(wn+1)) v
+
log
(46)
nu²
LG
Since L(w)= (wW)
L La (w), then (46) becomes
En [c(wn+1)]
ALG2
5ALG2
np (((1)
1 - X [[(
log
np²
mdr
LL
ALGa
5XLG2
np² [(w1)
log
(47)
np²
np²
L2
where the last inequality is due (0, and La La 2 fa(w.) O.
I PROOFS IN APPENDIX A
We include the proof' for Appendix section "Main Results for label-preserving Augmentation".
1.1 PROOF OF LEMMA 2
The analysis is similar to that for Lemma 1. For completeness, we include it here.
Proof. Following the same analysis in Lemma 1, we can have the same result as in (32). That is to say, we have
" [IV[(w)) vū(w:)
4G
IV
(48)
mo
We have
VL(w.) vā(w:))l
dxy (x (x
(0)
&G / [xx[Px(x) (x)]
(8)
SG (2/22KL P&)
12) G.2ppp
(49)
where (a) is due to the Assumption of C satisfying the property in Definition 1; (b) uses Pinsker's inequality (Csiszar & Körner, 201 Tsybakov, 2008); (c) is due (18). With inequality (49).
22

---

Under review as a conference paper at ICLR 2021
by using the facts that 1/L and the Assumption of L satisfying the property in Definition 3, inequality (48) becomes
(c,.)))((((il) - [(w)]
SndpG² + 4G
2
ma
4G
Sn0pG² + 1110
mh (L(w.) L(w.))
S2n0pG² - nu (C(w.) f(w.)),
where the last inequality is due the selection of mo 2 4. vpp
Then we have
E E(x...))[(we+)) [(w.)]
$ (1 m11) E(s..1.1- - [(ww)] pn0
$ (1 - mun) (C(w1) - ((w.)) + 2
(1
1=0
Due to (1 - (1)) v exp( (n))) and Ei=0 -11)² - 1 when
mp
Σ log (C(w1) - L(w.) 1p
L
28p
know
((.,5))((W(e1)) (((+))))-(w.)))) 48pG2
1
1.2 PROOF OF PROPOSITION
Proof. By using the Assumption of 2 satisfying the property in Definition 3, we have
Ō(w.) - [(w.)
VC(w.))l 1112
24
V((w.) V.(w.)ll 2/1
(b) 8pG²
p
where (a) is due the definition w. in (6) so that VL(w.) 0: (b) follows the same analysis
of (49) in Lemma 2. Thus we know w. E A(r) when 2 71 := Sp On the other hand, by the definition of u()) in (12) and the Assumption of L satisfying the property in Definition 3, we know u(7) 2 μ when ? S 4441
I.3 PROOF OF THEOREM 3
This proof is similar to the proof of Theorem 1. For completeness, we include here.
Proof. In the first stage of the proposed algorithm, we run mini-batch SGD over the augmented data D with m, as the size of mini-batch. Using the similar analysis in (35) from Theorem we have, with probability (1 8
t-1 8G2
[(w++1) - -(W.) (1 mn)² (A(wa) I(W.)) "
3 33og
Σ mp)
2
m2 ==0
23

---

Under review as a conference paper at ICLR 2021
Due to 1m 2 exp(-ttt/t and Ei mp)y 1 mN
when
L log
2(((w1) L(w.) D)
12 fi =
åpG²
we have
SpG2
8G2
[(w(+1)- [(w,)
+ (1 33log
(50)
2p
2um
Let 8'
8 if we choose my such that 277
2
271 8
1 1/y33og log
m1
8p
then for any 2 Tı, we have
[(w++1) [(..) S 8pG2
(51)
In the second stage of the proposed algorithm, we run mini-batch SGD over the original data 'D with m2 the size mini-batch. Using the same analysis in (39) from Theorem we have + 722 g - vL((;; no)([(w,) - [(w.)).
[(we+1) - [w,,) Snal //l,) Vl((w)))l
Then by (38) in the proof of Theorem and (49) in the proof of Lemma 2, we have, with a probability 1 8",
[(we-1) - [(w,) <2nz0pG2 + 1+ 3
1
SG
n2u(C(we) Z(w.)).
8"
m2
easy to verify that for any E {Tr + +....., Ti + n n/m2), we have, with a probability (1 m//m//m, we have
1 <1 1211) (Z(w:) - -((w)) + 412 28p + 1+ 1/ 5"
SG2
12
n n/m² 11
8G2
<(1 - mamuamm (I(WT)+1)- - [(W.)) + +122
28p + 1 1 log
3log
Σ i=0
(1 nau)
8"
m2
8pG2
(1+133 3log 8"
/
8G2
+
P
12
where the last inequality is due to (1 7211²* $ Ex (1
72/4)*
1 and (51), Let 8"
2n/ma we choose m2 such that m2
1
/ log
8p for example,
m2d
2
2n
8
m2
1 v log
8p
then with a probability 1- 6, for any t E {Ti + L,..., Ti + n/ma) we have
48p
Σ(w(41) _(w.) <
Therefore, w, E A(47)) for any i (Ti + 2, Ti + n/ m2 + 1}. Following the similar analysis in Appendix D, we have
GLL
G log 4mp?
An4i [(w.)))
E [[[(TT4n/mm+)) [(w.)] Amp?
G2L
where Me (.441).
24


Reviews:
[{"review_id": "q-ZVmSIMow", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to improve the generalization of deep networks when the input is applied strong augmentations, i.e. mixup, resulting in large data bias. The authors proposes theoretical foundation behind their two methods, AugDrop and MixLoss and show their effectiveness on CIFAR10/CIFAR100 datasets. Although the paper has technical novelty and it only provides incremental gains on relatively small-size dataset. Below, you can find some of my comments/questions about the paper.\n\n1. The paper has some typos and grammar mistakes. Some of them are \"By contrast, we do not need original data is unlimited.\", \"it is sufficiently to optimize Lc(w)\"\n\n2. I feel like the notations used in the introduction is repeated in the beginning of the section 3. There is no need to repeat the same explanations and notations in section 3.\n\n3. I am not sure about the definition of data bias mentioned in the paper. The bias is defined as the difference between the label of the augmented instance and original instance. In reality, there might be strong bias in the label-preserving augmentations, i.e. Jigsaw. I would like to get a clarification on this.\n\n4. The paper only shows experiments only on CIFAR10 and CIFAR100. Potential positive results on ImageNet would have made the paper much convincing and stronger. This is because, in the past, many augmentation methods, i.e. CutOut, has been shown to perform well on CIFAR10/100 but not on ImageNet.\n\n5. The improvements on CIFAR10/CIFAR100 are very marginal. There exists better data augmentation techniques, i.e. auto-augment, yielding better results on CIFAR10/100. It would be nice to include the results from different augmentation techniques into the experiments sections.\n\n6. Similar to my point at 3, do the authors think that their method would work on an another augmentation method, i.e. jigsaw, that preserves the label? Or is it only constrained for the ones that changes the label.\n\n7. What are the disadvantages of the proposed methods? What kind of complexity do they add, i.e. increased training complexity?\n\n8. I understand that the authors try to show that in the case of very strong augmentations (mixup with 10 images), they can improve the baseline mixup augmentation. However, their performance on the mixup with 10 images is worse than the mixup with 3 images. Then, what exactly is the point of using very strong augmentations if we can not benefit from them?", "extra_scores": {}, "extra_text": {}}, {"review_id": "pc7AsWvjGMT", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper theoretically analyzes the effect of bias in augmented data on the efficacy of data augmentation approaches, and proposes three new approaches to leveraging data augmentation in ways that purport to limit the negative effects of data bias in the augmented data distribution, and both theoretically and experimentally analyze them.  The algorithmic approaches and analyses are interesting, but (A) the theoretical results don't deliver a lot of specific actionable insight beyond the high level intuition they start with, and (B) the experimental section is somewhat weak.\n\nIn the main analysis (Sec. 4), the authors establish a bound (Lemma 1) on the generalization error due to training on augmented data (nit: unclear why $w_{t+1}$ is used to denote the final (?) params learned...?) and compare with a standard bound training on normal data.  However, the bound in Lemma 1 is not overly helpful (quadratic dependence on max augmented label divergence- unclear how this helps / is actionable), and the comparison with the normal bound doesn't reveal much but the original intuition that this approach *could* easily introduce bias, as we started with.\n\nThe paper then proposes two algorithms- \"AugDrop\", which uses a constrained optimization approach, and a corresponding theoretical analysis that shows potentially less biased and better performance than training on normal augmented data- and \"MixLoss\".  They also propose combining the two into \"WeMix\".\n\nAt this point, the main issue is with the experiments.  The authors could potentially have tried to fulfill two main objectives here: (A) make this mostly a theory paper, and seek to explore the quantities and tradeoffs of interest in the theoretical analysis in some experiments; and/or (B) show the practical efficacy of the proposed algorithms by running them with real data augmentation approaches, compared against modern approaches.  However, the experiments section accomplishes neither.\n- Re (A): There is no detailed experimental analysis beyond comparing end performance of models trained with the proposed approaches.\n- Re (B): The authors only use very rudimentary data augmentation techniques- mixup and contrast- and only compare to simple and internal baselines, even though there is a rich literature of recent data augmentation approaches that perform much better (e.g. even on the leaderboards recently: https://paperswithcode.com/sota/image-classification-on-cifar-100).\n- Again to be clear: this reviewer is *not* saying that all papers need to compare with SOTA.  But if the goal is not to show strong real world performance / applicability as in (B), then the experiments section should at least accomplish something about elucidating core theoretical tradeoffs, insights, etc. as in (A).", "extra_scores": {}, "extra_text": {}}, {"review_id": "k3cYguqvlt", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper illustrated two algorithms that correct bias in data augmentation: AugDrop and MixLoss. The authors illustrate their algorithms from a theoretical perspective and have mathematical proof the correctness of their algorithms. \n\nThe paper tackled the common problem on how to train with augmented data. it also half-opened a black-box of the training schemes. Authors' initial motivation can help the field significantly.\n\nThe concern I have is that if their method could be applied to a large range of applications since their experiment only based on some typical benchmarks, which already shows a good result in this field.\n\nBut generally, this paper is interesting and acceptable.", "extra_scores": {}, "extra_text": {}}, {"review_id": "aSVsmokM1xK", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\n\nIn this work, the authors first prove a deep model can benefit from augmented data when the data bias is small. Then they propose two methods, namely \"AugDrop\" that corrects  and \"MixLoss\", that correct data bias by \"constrained optimisation\" and \"modified loss function\" respectively. Finally they show that these two methods can be combined and further improve the performance.\n\n##########################################################################\n\nReasons for score:\n\n\nI'm not an expert in the area of data augmentation, and I didn't check all the proofs and derivations in this paper. If I understand it correctly, the realisation of \"AugDrop\" is to omit data augmentation and train the model on original data, and the realisation of \"MixLoss\" is to train the model on both augmented and original data. The combined method, namely, \"WeMix\" is to run \"MixLoss\" first and then run \"AugDrop\". Overall, this paper is hard for me to follow. Yet another concern is that, the method was tested on CIFAR-10 and CIFAR-100, with \"mixup\" as the only competitor.\n\n##########################################################################\n\nPros:\n\n1. The paper tried to answer an important question \"when will the deep models benefit from data augmentation?\" They proved (though I didn't check the proof) that \"a deep model can benefit from augmented data when the data bias is small\". Based on this, they developed two methods \"AugDrop\" and \"MixLoss\".\n\n2. The proposed method seems to be very easy to implement.\n\n##########################################################################\n\nCons:\n\n\n1. I'm not convinced by the current experiments: (i) it runs on CIFAR-10 and CIFAR-100 only (ii) the only competitor is \"mix-up\".\n\n2. It wasn't quite clear (at least for me) why \"AugDrop\" and \"MixLoss\" can correct the data bias. Actually I have been lost at how the authors quantify the data bias.\n\n##########################################################################", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents a new theoretically motivated data augmentation technique. Reviewers agreed that the theory was interesting and has value, however raised concerns regarding the experimental evaluation which was limited to the Cifar datasets. There was some discussion over whether or not a comparison with AutoAugment would be fair, the proposed method is theoretically motivated whereas AutoAugment takes significant compute to train. I agree with the authors that if the method doesn't outperform AutoAugment on CIFAR, this would not necessarily invalidate their results. Nonetheless the work would be significantly strengthened if it included results. on additional datasets to stress test the theory. I recommend the authors add additional supporting evidence and resubmit.
Decision: Reject