MixUp as Directional Adversarial Training
Authors: Guillaume Perrault-Archambault
Yongyi Mao
Hongyu Guo
Richong Zhang
Venue:  (2020)
Keywords: MixUp
Adversarial Training
Untied MixUp

Abstract: MixUp is a data augmentation scheme in which pairs of training samples and their corresponding labels are mixed using linear coefficients. Without label mixing, MixUp becomes a more conventional scheme: input samples are moved but their original labels are retained. Because samples are preferentially moved in the direction of other classes \iffalse -- which are typically clustered in input space -- \fi we refer to this method as directional adversarial training, or DAT. We show that under two mild conditions, MixUp asymptotically convergences to a subset of DAT. We define untied MixUp (UMixUp), a superset of MixUp wherein training labels are mixed with different linear coefficients to those of their corresponding samples. We show that under the same mild conditions, untied MixUp converges to the entire class of DAT schemes. Motivated by the understanding that UMixUp is both a generalization of MixUp and a form of adversarial training, we experiment with different datasets and loss functions to show that UMixUp provides improved performance over MixUp. In short, we present a novel interpretation of MixUp as belonging to a class highly analogous to adversarial training, and on this basis we introduce a simple generalization which outperforms MixUp.

Under review as a conference paper at ICLR 2020
MIXUP AS DIRECTIONAL ADVERSARIAL TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
MixUp is a data augmentation scheme in which pairs of training samples and their corresponding labels are mixed using linear coefficients. Without label mix- ing, MixUp becomes a more conventional scheme: input samples are moved but their original labels are retained. Because samples are preferentially moved in the direction of other classes we refer to this method as directional adversarial training, or DAT. We show that under two mild conditions, MixUp asymptotically convergences to a subset of DAT. We define untied MixUp (UMixUp), a superset of MixUp wherein training labels are mixed with different linear coefficients those of their corresponding samples. We show that under the same mild con- ditions, untied MixUp converges to the entire class of DAT schemes. Motivated by the understanding that UMixUp is both generalization of MixUp and a form of adversarial training, we experiment with different datasets and loss functions to show that UMixUp provides improved performance over MixUp. In short, we present a novel interpretation of Mix Up as belonging to class highly analogous adversarial training, and on this basis we introduce a simple generalization which outperforms MixUp.
INTRODUCTION
Deep learning applications often require complex networks with large number of parameters (He
et al. 2016 Zagoruyko & Komodakis] 2016 Devlin et al. 2018). Although neural networks perform
so well that their ability generalize is an area of study in itself (Zhang et al., 2017a| Arpit al. their high complexity nevertheless causes them to overfit their training data (Kukacka et al. For this reason, effective regularization techniques are in high demand.
2017),
2017
There are two flavors of regularization: complexity curtailing and data augmentation | Complexity curtailing methods constrain models to learning in subset of parameter space which has higher probability of generalizing well. Notable examples are weight decay (Krogh & Hertz 1991) and dropout (Srivastava et al. 2014).
Data augmentation methods add transformed versions of training samples to the original training set. Conventionally, transformed samples retain their original label, so that models effectively see a larger set of data-label training pairs. Commonly applied transformations in image applications include flips, crops and rotations.
A recently devised family of augmentation schemes called adversarial training has attracted active research interest (Szegedy et al. 22033] Goodfellow et al. (0044) Miyato et al.) 2016 Athalye et al.) 2018 Shaham et al.] 2018T He et al. 2018). Adversarial training seeks reduce model's 's propen- sity to misclassify minimally perturbed training samples, or adversarials. While attack algorithms used for testing model robustness may search for adversarials in unbounded regions of input space, adversarial training schemes generally focus on perturbing training samples within a bounded re-
gion, while retaining the sample's original label (Goodfellow et al. 2015 Shaham et 2018).
Another recently proposed data augmentation scheme is MixUp (Zhng et al. 20176). in which new samples are generated by mixing pairs of training samples using linear coefficients. Despite its well established generalization performance (Zhang et al. 2017bb] et al.) 2018 /eerm et al.) 2018 the working mechanism of MixUp is not well understood. [uuo al. 2018) suggest viewing MixUp as imposing local linearity on the model using points outside of the data manifold. While this
Some authors describe these flavors as "data independent" and "ddaaa-dpeenden" (Guo al. 2018

---

Under review as a conference paper at ICLR 2020
U
Uatied MixUp
Expocted Losses
DAT
MixUp
B,
Figure 1: The relationship between MixUp, DAT and Untied Mix
perspective is insightful, we do not believe it paints full picture of how MixUp operates. A recent study (Lamb et al. 2209)) provides empirical evidence that MixUp improves adversarial robustness, but does not present MixUp form of adversarial training.
We build framework to understand MixUp in a broader context: we argue that adversarial training is a central working principle of MixUp. To support this contention, we connect MixUp to MixUp- like scheme which does not perform label mixing, and we relate this scheme to adversarial training.
Without label mixing, MixUp becomes conventional augmentation scheme: input samples moved, but their original labels are retained. Because samples are moved the direction other samples which typically clustered in input space we describe this method 'directional', Because this method primarily moves training samples the direction adversarial classes, this method is analogous to adversarial training. We thus refer MixUp without label mixing direc- tional adversarial training (DAT). We show that MixUp converges to a subset of DAT under mild conditions, and we thereby argue that adversarial training is a working principle of MixUp.
Inspired by this new understanding of MixUp as a form of adversarial training, and upon realizing that MixUp (asymptotically) subset of DAT, we introduce Untied MixUp (UMixUp), a simple enhancement of MixUp which converges to the entire family of DAT schemes, as depicted in Figure Untied Mixup mixes data-label training pairs similar way to Mix Up, with the distinction that the label mixing ratio is an arbitrary function of the sample mixing ratio. We perform experiments to show that UMixUp's s classification performance improves upon MixUp.
In short, this research is motivated by a curiosity to better understand the working of MixUp. In-so- doing we aim to:
Establish DAT as analogous adversarial training. This discussed section 4
2. Establish UMixUp as a superset of MixUp, and as converging to the entire family of DAT schemes. In-so-doing, establish MixUp's convergence a subset of DAT, and thereby that it operates analogously to adversarial training; and b) establish UMixUp as broader class of MixUp-like schemes that operate analogously to adversarial training. This is discussed in 5
3. Establish empirically that UMixUp's classification performance improves upon MixUp. This is discussed in section 6
Finally we note that this paper has another contribution. Conventionally, Mix is only applicable to baseline models that use cross entropy loss. All analytical results we develop in this paper applicable to a wider family of models using any loss function which we term target-linear. We define target-linearity and experiment with new loss function called negative cosine-loss show its potential.
Essential proofs of theoretical results are given in the Appendix.
2

---

Under review as a conference paper at ICLR 2020
2
PRELIMINARIES
Column vectors are denoted by bold letters such m, and sets are denoted by calligraphic uppercase letters such as M. The component of a vector denoted by bracketed index. For example, m[i] denotes the ith component of m.
Regular (non-calll -on-calligraphcc) capitalized letters such as X will denote random variables, and their lowercase counterparts, e.g., x, will denote realizations of random variable. Any sequence, (11, a2...... (n)
will be denoted by a²- Likewise (A1, Ag...., A,) will be denoted by A7, and sequence of sample pairs ((x,, x (X2, x2),... (Xn, x,)) denoted by x').
For any value a [0, 1], we will use a as short notation for 1 a.
Classification Setting Consider a standard classification problem, in which one wishes to learn classifier that predicts the class label for a sample.
Formally, let X' be vector space in which the samples of interest live and let y be the set of all possible labels associated with these samples. The set of training samples will be denoted by D, a subset of X. We will use t(x) to denote the true label of x. Let /' be neural network function, parameterized by 6, which maps X' to another vector space Z. Let yp y + Z be a function that maps a label in ) to an element in Z such that for any y. v E V. if y # y' then ppv) * ppy'). In the space Z, we refer to Fx) as the model's prediction. With slight abuse of language, we will occasionally refer both ((x) and (x( as the "label" of x. Let 2 Z R be loss function, using which one defines an overall loss function as
1 Σ ((F(()) D
C
FED
Here we have taken the notational convention that the first argument of represents the model's pre- diction and the second represents the target label. In this setting, the learning problem formulated as minimizing L with respect to its characterizing parameters 6.
Target-Linearr Loss Functions We say that loss function f(z, z') is target-linear if for any scalars and 3,
((z, az) + (22) = al(z, 21) + 3l 22)
Target-linear loss functions arise naturally in many settings, for which we now provide two exam- ples. For convenience, we define the vectors v = Fxx) and y = ((((x)).
Cross-Entropy Loss The conventional cross-entropy loss function, written in our notation, is defined as:
dim
tcE (x)xx) = loB (vE(v.)) = Σ y/i log
where v and y are constrained to being probability vectors. We note that in conventional applications, dim( Z) Y) and the target label one-hot vector where y(i] 1 if t(x) and y[i] otherwise. Constraining v to being probability vector is achieved using softmax output layer.
Negative-Cosine Loss The "negative-cosine loss", usually used in its negated version, i.c., as the cosine similarity, can be defined as follows.
Ex (F(x), (o(((x))) y := -vv
where and constrained being unit-length vectors. For v this can be achieved by simple division at the output layer, and for by limiting the range of sp to an orthonormal basis (making a conventional label embedding function).
It is clear that the cross-entropy loss fcE and the negative-cosine loss IgC are both target-linear, directly following from the definition of target-linearity.
Assumptions The theoretical development of this paper relies on two fundamental assumptions, which we call "axioms".
Axiom 1 (Target linearity) The loss function & used for the classification setting is target-linear.
3

---

Under review as a conference paper at ICLR 2020
That is, the study of MixUp in this paper is in fact goes beyond the standard MixUp, which uses the cross-entropy loss.
Much of the development in this paper concerns drawing sample pairs (x,x') from D D. Suppose that (x, x') is length-n sequence of sample pairs drawn from D D. A sequence xx) is said to be symmetric if for every (a. b) D x D. the number of occurrences of (a, b) in the sequence is equal to that of (b, a). A distribution Q on x 'D will be called exchangeable, or symmetric, if for any (x, x E D x D. (()xx')) - Q))
Axiom 2 (Symmetric pair-sampling distribution) Whenever a sample pair (x, x') drawn from distribution Q Q is assumed to be symmetric.
In the standard MixUp, samples are drawn independently from D form pair, making this condition satisfied.
3 MIXUP, DAT, UNTIED MIXUP
3.1 INFORMAL SUMMARY
We first provide a summary each scheme for the reader's convenience. We then describe each scheme more systematically. For concision of equations to follow, we define
y (t((x)
and
(t((x))
MixUp is data augmentation scheme in which samples are linearly combined using some mixing ratio À E [0, 1]:
X Ax (1 A)x
where À ~ pMix, A target label generated using the same mixing ratio A:
xy + (1 A)y'
(MixUp)
DAT and UMixUp use the same method 2 for generating samples, but use different distributions (PDAT and puMix respectively). DAT and UMixUp also differ from MixUp in their target labels. DAT retains the sample's 's original label:
yg y
(DAT)
whereas UMixUp's label mixing ratio is function of A:
y, >()y + (1 ~AA))
(UMixUp)
In Untied MixUp, the label mixing ratio is "untied" from the sample mixing ratio, and can be any (A). We will refer to the weighting function. An Untied MixUp scheme is specified both by its mixing policy puMix and a weighting function 7.
3.2 FORMAL DEFINITIONS
To draw comparisons between MixUp, DAT, and Untied MixUp schemes, we establish a framework for characterizing their optimization problems. To that end, we define each model's 's loss function fm in terms of its baseline target-lineat loss function l,, where the superscript m is replaced with a model identifier (i.e. e Miix @DRT, guMix). Each model's overall loss function, L is defined in terms of its loss function nd as per equation (where equation I's (*r). We denote the expected value of each scheme's overall loss, LT. with respect its mixing ratio A.
Let n positive integer. In every scheme, a sequence (x,x))" == ((x,, x1), X2)....... (X,, xx,)) of sample pairs drawn i.i.d. from Q, and sequence X := (A Xx) of values drawn i..d. from pm, where pm is a distribution over [0, 1] unique to each model.
MixUp For any x' E D and any À E [0, 1), denote
4

---

Under review as a conference paper at ICLR 2020
Mix x x = :0 (F(Ax+ Xx'), Ay Xyy)
Let pm be PMix; in other words sequence X; = (A1, ... Ax) ) of values are drawn i.i.d. from pMix We denote the overall loss LMx ((x, x x)),, 23) and the expected overall loss L ((x, x x));):
M
1
Σ
e X A1)
(3)
n
Chix ((x,x))) = Xy pamn
L ((x, xxx
In MixUp, we refer to PMix as the mixing policy. Directional Adversarial Training (DAT) For any x, E D and any À E [0, 1]. we denote
(DAT x X) 6. (F(Ax + Xx)),)
Let pm be PDAT, such that members of le are drawn i.i.d. from PDAT, We denote the overall loss CDAT ((x, x' )f) and the expected overall loss CDAT ((x, x')):
CAA ((x,x')I,
DAT
(Xk. Xe, As)
(4)
CAAT ((x,
x))x
E A PDAT
CAA
In DAT, we refer to PDAT as the adversarial policy. Untied MixUp (UMixUp)
Let be a function mapping [0, to [0, 1]. For any x, x' E D and any À E [0, 1]. we denote guMix
x X A, (((((xx Xx'), 7()y) +7(A)y))
Let pm be puMix, and denote the overall and expected overall loss functions LuMix ((x, x') (1:7) and LuMix ((x,x");,)) respectively:
L
Mux
X
puMix (x, xxx,,,,7)
CuMix (xx,''I
LuMix ((x, x' AT.7)
Aid pumix
At this end, apparent that MixUp special case of Untied MixUp, where the function 7(A) takes the simple form 7(A) X.
4 DAT AS ANALOGOUS TO ADVERSARIAL TRAINING
The main theoretical result of this paper is the relationship established between DAT and UMixUp. and by extension MixUp. Both MixUp and UMixUp will be shown to converge to DAT as the number of mixed sample pairs, n, tends to infinity. Prior to developing these results, we provide insight into DAT, terms of its similarity to adversarial training and its regularization mechanisms.
Conventional adversarial training schemes augment the original training dataset by searching for approximations of true adversarials within bounded regions around each training sample. For training sample bounded region U known Over this region, the loss function with respect to the true label of x is maximized. typical loss function for an adversarial scheme is
Lp ball is defined as U = fx + 7 ||llpp e).
((F(x),)) max (F((()))))
5

---

Under review as a conference paper at ICLR 2020
where l, is the baseline loss function. Simply put, baseline training serves to learn correct clas- sification over the training data, whereas adversarial training moves the classification boundary to improve generalization.
DAT, on the other hand, combines intra-class mixing (mixing two samples the same class) and inter-class mixing (mixing samples of different classes). Intra-class mixing serves to smooth classi- fication boundaries of inner-class regions, while inter-class mixing perturbs training samples in the direction of adversarial classes, which improves generalization. Inter-class mixing dwarves intra- class mixing by volume of generated samples seen by the learning model in most many-class learn- ing problems (by a 9-1 ratio balanced 10-class problems for instance). DAT, which primarily consists of inter-class mixing, can therefore be seen as analogous to adversarial training.
The key distinction between conventional adversarial training and inter-class mixing that MixUp movement is determined probabilistically within a bounded region, while adversarial movement is deterministic.
Figure 2 illustrates the connection between standard adversarial training and DAT. Consider the problem of classifying the blue points and the black points in Figure 22), where the dashed curve is ground-truth classifier and the black curve indicates the classification boundary of F(x), which overfits the training data. adversarial training, a training sample moved location within an Lp-ball around while keeping its label to further train the model; the location, denoted by & in Figure p)), chosen to maximize training loss.
(a) Baseline training
(b) Adversarial training
(c) Inter-class mixing
(d) Intra-class mixing
Figure DAT as a form of adversarial training
In DAT, a second sample x' governs the direction in which x is perturbed. If x' is chosen from different class as shown Figure (p), then the generated sample Ng is used further train the model. If x' is chosen from the same class as shown in Figure 20), then the sample X3 is used in further training. Note that the inter-class mixed sample X2 pushes the model's classification boundary closer to the ground-truth classifier, thus connecting adversarial training and DAT. The intra-class sample &3, on the other hand, mainly helps to smooth inner parts the class region. The latter behaviour is additional feature of DAT and MixUp, which distinguishes these schemes from adversarial training.
5
UNTIED MIXUP AS ASYMPTOTICALLY EQUIVALENT TO DAT
We now show that Untied MixUp and DAT are equivalent when tends to infinity. A consequence of this equivalence is that it infuses both MixUp and UM with the intuition of adversarial training. To that end, we relate the Untied MixUp loss function, guMix with the DAT loss function, (DAT
Lemma For any (x,x)) D and any X [0, 1].
MiMix
= ((A DAT (x.x,A)
+ PAA)eAAT(X
This result follows directly from the target-linearity of the loss function.
The next two Jemmas show that as n tends to infinity, the overall loss of both DAT and UMixUp converge in probability their respective overall expected losses.
Lemma 2 As n increases, CAT ((x, x') AT) converges to CAT ((x, x() in probability.

---

Under review as a conference paper at ICLR 2020
Lemma 3 As n. increases, LuMix ((x, AT; converges C ((x,x)) 7) in probability.
These two lemmas have similar proofs, thus only the proof of Lemma is given in section A.
Next show that n tends infinity, UMixUp converges in probability subset of DAT, and DAT converges in probability to subset of UMixUp. In other words, we show that as n increases, UMixUp converges to being equivalent to the entire class of DAT schemes.
For that purpose, let F. denote the space of all functions mapping [0, 1J to [0, 1]. Each configuration in P F defines an Untied MixUp scheme.
We now define U. which maps a DAT scheme to an Untied MixUp scheme. Specifically U is a map from P to F such that for any p E P. UUp) is a configuration (p'.g) P x F, where
1 (P(A) (p(A p'(A) = =
((A)
+ p(1 - A)) and g(A) = P(A) + p(1 X)
(5)
Lemma Let (x, x be sequence of sample pairs on which an Untied MixUp scheme specified by (puMix 7) 7) and DAT scheme with policy PDAT will apply independently. (x x') is symmetric
and (publix -7) = U(PDAT), ) then LuMix ((x,x)) xx))),77) CDAT ((x,x))).
We now define another map DD that maps an Untied MixUp scheme to a DAT scheme. Specifically D. is map from P x F to P such that for any (p.g) E P F, Du g) is configuration p' E P, where
P((A) = (o(A)p(A) + s(A)p(1 -
It is easy to verify that 1or P/ (A)dA= 1. Thus p' is indeed distribution in P and D. is well defined.
Lemma 5 Let (x, x') be sequence of sample pairs which an Untied MixUp scheme specified by (PuMix, 7) and a DAT scheme with policy PDAT will apply independently. (x, x') is symmetric and PDAT Du (puMix,, (7), then Lubiix
(xxx)) = CDAT ((x,x')T).
Lemmas 3 4 and 5 provide the building blocks for theorem T which state hereafter. As increases, both DAT and UMixUp converge in probability toward their respective expected loss (lem- mas 2 and
and 3. Since as n increases, the sequence (x, x becomes arbitrarily close to the symmetric sampling distribution Q, then by lemma 4 the family of DAT schemes converges in probability to subset of UMixUp schemes. Lemma proves the converse, i.e. that as n increases the family of UMixUp schemes converges in probability to subset of DAT schemes. As n increases, the family of UMixUp schemes therefore converges probability to the entire family of DAT schemes.
Theorem Let (X,X')r be drawn i...d. from Q. On this sample-pait data, Untied MixUp scheme specified by (PMix, 7) and a DAT scheme specified by PDAT will apply. In the Untied MixUp scheme, les AÇO be drawn i.i.d. from phlix, in the DAT scheme, let rp be drawn i.i.d. from PDAT, If PDAT Du (pMix, 7) or (phixx ,)) U(PDAT), then LMIx (X,X')P,AT,))) CDAT X"), TTH
P, 0, as n > XD
The equivalence between the two families of schemes also indicates that there DAT schemes that do not correspond to a Mp scheme. These DAT schemes correspond to Untied MixUp scheme beyond the standard Mix The relationship between MixUp. DAT and Untied MixUp is shown in Figure
6
UMIXUP AS A USEFUL
OF MIXUP: EXPERIMENTS
6.1 EXPERIMENT SETUP AND IMPLEMENTATION
We consider image classification task on the Cifar10, Cifar100, MNIST and Fashion-MNIST datasets. The baseline classifier chosen is PreActResNet18 (see Liu (2017), noting the same choice is made by the authors of MixurZhang al. (2017b).
Both MixUp and Untied MixUp are considered the experiments. The MixUp policies are chosen as Beta distribution B(a, 3). The Untied MixUp policy is taken as U(B(o,B))
7

---

Under review as a conference paper at ICLR 2020
Two target-linear loss functions essayed: cross-entropy (CE) loss and the negative-cosine (CE) loss as defined earlier. We implement CE loss similarly to previous works, which use CE loss to implement the baseline model. In our implementation of the NC loss model, for each label y, (o()) is mapped to a randomly selected unit-length vector of dimension d and fixed during training; the feature map of the original Net is linearly transformed to d-dimensional vector. The dimension d is chosen as 300 for Cifar I0, MNIST and Fashion-Mniss (which have one black-and- white channel) and 700 for C0 (which has 3 colored channels).
Our implementation of MixUp and Untied Mix improves upon the published implementation from the original authors of Mix Zhang et al. (2017b). For example, the original authors' imple- mentation samples only one À per mini- giving rise to unnecessarily higher stochasticity of the gradient signal. Our implementation samples X independently for each sample. Additionally, the original code combines inputs by mixing mini-batch of samples with shuffled version of itself. This approach introduces dependency between sampled pairs and again increases the stochasticity of training. Our implementation creates two shuffled copies of the entire training dataset prior to each epoch, pairs them up, and then splits them into mini-batches. This gives a closer approxima- tion to i..dd sampling and makes training smoother. While these implementation improvements have merit on their own, they do not provide a theoretical leap in understanding, and so we do not quantify their impact in our results analysis.
All models examined are trained using mini-batched backpropagation, for 200 epochs.
6.2 RESULTS
We sweep over the policy space of MixUp and Untied MixUp. For MixUp, is sufficient consider distribution PMix to be symmetric about 0.5. Thus we consider only consider pMix in the form of B(a, a), and scan through single parameter a systematically. Since the policy of Untied MixUp is in the form of ((( 3)), searching through (a,33) 3) becomes more difficult. Thus our policy search for Untied MixUp is restricted to an ad hoc heuristic search. For this reason, the found best policy for Untied Mix might be quite far from the true optimal.
The main results of our experiments are given in tables to As shown in the tables, each setting is run 100 times. For each run, we compute the eπгоr rate in a run as the average test error rate over the final 10 epochs. The estimated mean ("MEAN") performance of setting is computed as the average of the error rates over all runs for the same setting. The 95%f-confieence interval ("ConfInt") for the estimated mean performance is also computed and shown in the table.
From these results, we see that the Untied MixUp schemes each outperform their MixUp counter- parts. Specifically, in of the 8 cases (those printed in bold font), the confidence interval of Untied MixUp is completely disjoint from that of the corresponding MixUp scheme; and in some cases, the separation of confidence intervals is by a large margin. Note that the baseline model (PreActRes- Net18) has been designed with highly focused inductive bias for image classification tasks. Under such an inductive bias, one expects that the room for regularization (or the "amount of overfitting") isn't ian'tann abundant. As such, we consider the improvement of Untied MixUp over MixUp rather signifi- cant.
The results show empirically that MixUp and Untied MixUp both work on the NC loss models. This validates our generalization of MixUp (and Untied Mix to models built with target linear losses.
model
policy
runs
MEAN
ConfInt
baseline-Cie
100
3.476%
0.027%
mixUp-CO CE
B(0.9,
100
4.199%
0.023%
uMixUp-CE
((B(2.2)0.))
100
4.177%
0.025%
baseline-Ny
100
5.605%
0.030%
NCC
B(1.0, 1.0)
100
4.508%
0.022%
U(B(1.8, I.))
100
4.455 %
0.025%
Table I: Test error rate on CIFARIO.
8

---

Under review as a conference paper at ICLR 2020
model
policy
runs
MEAN
ConfInt
baseline-CE
100
24.848%
0.060%
mixUp-CE
B(0.9, 0.9)
100
22.020%
0.050%
UMixUp-CE
U(B(1.4,007) 0.7))
100
21.884
0.051
baseline-NC
100
25.270%
0.050%
mixUp-NC
B(0.9, 0.9)
24.298%
0.051
uMixUp-NC
(1.33 0.9))
100
23.819% %
0.054%
Table 2: Test error rate CIFARI00.
model
policy
runs
MEAN
ConfInt
baseline-Ce CE
100
0.816%
0.007%
mixUp-CE
B(1.0, 1.0)
100
0.632%
0.005%
uMixUp-CEB
U(B 1.0))
100
0.609
0.005 %
baseline-NC
100
0.720%
0.007%
mixUp-NC
B(1.0, 1.0)
100
0.607%
0.004%
MiMiUUpNNC
U 0.9))
100
0.592%
0.005%
Table 3: Test error
MNIST.
model
policy
runs
MEAN
ConfInt
baseline-Ch CE
100
5.060%
0.027%
mixUp-CE
B(1.0, 1.0)
100
4.585%
0.013%
uMixUp-CU
U(B(117,008)
100
4.570%
0.013%
baseline-No NC
100
5.083%
0.016%
-iXUp-NC
B(100, 1.0)
100
4.767%
0.013%
u-
(L.,, 0.9))
100
4.613
0.011% %
Table 4: Test error rate on Fashion-MNIST.
7 CONCLUDING REMARKS
This paper establishes connection between MixUp and adversarial training. This connection allows for better understanding of the working mechanism of MixUp as well as generalization of MixUp to wider family, namely Untied MixUp. Despite the development in this work, it is the authors' belief that the current designs of MixUp and Untied MixUp are far from optimal. particular, we believe better design should allow individualized policy for each training pair. How this can be done remains open at this time.
REFERENCES
Devansh Arpit, Stanislaw K. Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In Proceedings of the 34th In- lernational Conference on Machine Learning, ICMI 2017, Sydney, NSW, Australia, 6-11 August 2017, pp. 233 242. 2017. URL http::/proceeeeaiiee html
mlr www.ppressvvoio wttpppasss//.ppp.11111
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. arXiv preprint arXiv: arXiv:/802.0000 8822000200 2018.
Jacob Devlin, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/ bbs./880,04485, 2018. URL
http://arxiv.cm wtt.oorg//abbbbooeeeeee 04805
Jan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:/412.633 2014.

---

Under review as a conference paper at ICLR 2020
Ian J. Goodfellow. Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR 2015, 2015.
Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup as locally linear out-of-manifold regular- ization. arXiv preprini arXiv: 02499, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Warren He, Bo Li, and Dawn Song. Decision boundary analysis of adversarial examples. In 6th In- ternational Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018. URL forum? -d---kiiBBBb...
nett/
Anders Krogh and John Hertz. simple weight decay can improve generalization, In Advances in Neural Information Processing Systems 4, NIPS Conference, Denver, Colorado, USA, December 2-5, 1991, pp. 950-957, 1991. URL /tLp:ttp:///a /tattp::/paper..... nips..ips.................. cc/paper/
563 a a-3-a-simp--1 wwmpssimmeeeeeeeee weight decay can improve gww.rg
Jan Kukacka, Vladimir Golkov, and Daniel Cremers. Regularization for deep learning: A taxonomy. CoRR, abs/1710. 10686, 2017.
Alex Lamb, Vikas Verma, Juho Kannala, and Yoshua Bengio. Interpolated Adversarial Training: Achieving Robust Neural Networks without Sacrificing Too Much Accuracy. arXiv e-prints, art. arXiv:1906.06784, Jun 2019.
2017.
Colin McDiarmid. On the method of bounded differences. Surveys in combinatorics, 141(1):148- 188, 1989.
Takeru Miyato, Andrew M Dai, and Ian Goodfellow. Adversarial training methods for semi- supervised text classification. arXiv preprint arXiv:1605.07725, 2016.
Uri Shaham, Yutaro Yamada, and Sahand Negahban. Understanding adversarial training: Increasing local stability of supervised models through robust optimization. Neurocomputing, 307:195-220 204, 2018.
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learn- ing Research, 15(1):1929-195) 19588 1958, 2014. http:http///ll acm. org/ htt//iiaaiionn .onercffmm cfm? id=
2670313
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 13/2,6199, 2013.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Aaron Courville, Ioannis Mitliagkas, and Yoshua Bengio. Manifold mixup: Learning better representations by interpolating hidden states. 2018
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference 2016. BMVC 2016. York, UK. September 19-22, 2016, 2016. URL attpp:/wwit bmva.
http:////apppeepppppepe////eeeee html
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, -trt:e/ Sy8gdB9xx
2017a. URL https::////mm httt:tttt:::/ppeeteee http::///pppreeeee.eooomm htt.:nne//eeeeeee woreee///oummmm
Hongyi Zhang, Moustapha Cisse, Yann and
and David Lopez-Pazz Paz. mixup: Beyond empirical
risk minimization. arXiv preprint 1rX1v:1710.00412,, 2017b. URL https: / //gi thub. com/
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 PROOF OF LEMMA
For any fixed infinite sequence (x, x samples drawn i..d. from Q and any infinite sequence of i..d. random variables As drawn from PDAT, let [DAT ((x, ('), ,,))) be defined according to (4). with the first n elements of (x, x and the first n elements of A as imput. Define
ÖDAT max sup
IADAT (x, -DDAT (x, X"I
(x.')E (AAA) DxD [0.1]x[0.1]
For any given Xi [0, 1J"h and any of its modified version up [0, 1] which differs from Xp in exactly one location, it can be verified, following the definition of δDAT, that
CDAT ((x, x ((K,))),A)) _DAT ((x, x x'xx' u S ÓDAT/N
Since A1, A2....AA. are independent and by McDiarmid Incquality McDiarmid (1989). it follows that for any 0.
Pr DAT ((x,x)),,3)) CDAT ((x,x))) 20 2xxp (-
22 n. (ODAT/n))
which proves the lemma
A.2 PROOF OF LEMMA 4
CE LUMIX
(xx,x)),7)) =
Ex- ( (XE X/. () + 7(A)) (x X/. 8)}
/ (((A))MM(())DDAAA (KL. A) Paix (A) (X Xc, X)) da
n
PDAT (A)EDAT A) +2 PDAT (A)E X A) d7
PDAT (D)E ApdA Σ
pDAt (T)EDAT (X N1. I)dd
2K
(0)
PDAT (A) (NL. X A)dA
PDAT (X NA. A)dA
2K
(8)
PDAT (A)EDAT (XL. X A)dA +
PDAT (A)EDAA (x X A)da
2K
PDAT (DADDA)AAAT (xk. X AjdA
n
CE ((x, x'))
where (a) is due to a change of variable in the integration, (b) is due to the symmetry of (x,x') Note that in equation (A) is undefined at values of À for which the denominator is zero. But the lemma holds true because the denominator is only zero when p()) = 0, so those À for which g(A) undefined never get drawn in the DAT scheme.
11

---

Under review as a conference paper at ICLR 2020
A.3 PROOF OF LEMMA 5
C ((x,x))i,>)
-
1
Y PMix
(AA)DDAT (xe, X A) 7(JeeDAT (x XR. 2))
k=l
BAn
7() (X): X A) pMix
(X):
(A)EDAT (x NA: X)
n
1
Bx
(A) (Kk: Xxx A) E Ex~ PMix
{A) [K Na: 2)
7
(b)
1
BAN 71) (KL. Xxx A) T PMix
(X) (xis X/: 2)
72
(xe, xx: d) + (2) (M)x) 2) (XL: X 2) da (ANS) (XE,
(XL, A) (7()) (2) x(X) (1 -1) dd
(u(piiiz,))
(DAT(Xx E PDAT DDAT (xx. A)
CB ((x, x')x
where (a) is due the symmetry of (x, ). and (b) is by a change of variable the second term (renaming 1 - A as A).
12


Reviews:
[{"review_id": "SJl42viOtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "First of all, the concept of 'Directional Adversarial Training (DAT)' is not appropriate. Actually similar method has been proposed in Hiroshi Inoue (2018) as a data augmentation method.\n\nWhat surprised me the most is that after trying to connect UMixUp with adversarial training in the whole paper, there is no evaluation of adversarial robustness in the experiments? The main character of adversarial training is an improvement in robustness and degeneration on clean accuracy, which is different from the performance of UMixup or Mixup.  \n\nThe authors should do a major modification on their motivation and experiments before the paper is able to be published.\n\nReference:\n[1] Hiroshi Inoue. Data Augmentation by Pairing Samples for Images Classification. arXiv 1801.02929\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "Hkl9FN5Z5r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces directional adversarial training (DAT) and UMixUP, which are extension methods of MixUp. DAT and UMixUp use the same method of MixUp for generating samples but use different label mixing ratios where DAT retains the sample's original label. In contrast, UMixUp uses a function of the input mixing ratio. This paper shows that UMixUp and DAT are equivalent when the number of samples tends to infinity. In the experiments, UMixUp provides an improvement over MixUp.\n\nThis paper should be rejected because the originality of the proposed method over MixUp is marginal, and the improvement of classification accuracy is not surprising, although the explanation of the relationship among DAT, UMixUp, and the original MixUp is nice. The modification of label mixing ratios is not enough contribution. A more precise description of why DAT and UMixUp work better over the original MixUp is required.\n\nThe same idea of MixUp was proposed at the same conference (ICLR2018). It should be cited.\nTokozume et al., Learning from Between-class Examples for Deep Sound Recognition. ICLR, 2018.\nTokozume et al. explain why the mixture of examples works well from the different perspectives of adversarial examples. Also, BC learning uses a KL loss, not a cross-entropy loss. Therefore, the reviewer doubts the following statement: \"MixUp is only applicable to baseline models that use cross entropy loss\" on page 2.\n\nMinor comments\n1) In page 8, \"negative-cosine (CE)\" might be \"negative-cosine　(NC)\".", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJeCr7patr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel data augmentation method, untied MixUp (UMixUp), which is a general case of both MixUp and Directional Adversarial Traning (DAT). DAT is referred to in this paper as a scheme that only input feature vectors are mixed, while MixUp also incorporates their corresponding labels. The authors provide a theoretical discussion that both DAT and UMixUp converges to be equivalent to each other when the number of training samples becomes infinity. Experimental results on Cifar 10, Cifar 100, MNIST, and Fashion MNIST show quantitative comparisons among the baseline, MixUp, and UMixUp.\n\nAccording to the author guideline,\n> There will be a strict upper limit of 10 pages for the main text. Reviewers will be instructed to apply a higher standard to papers in excess of 8 pages.\nThe authors use nine pages. Therefore, the review should be more careful about its quality.\n\nCurrently, I have three major concerns that keep me from judging this paper acceptable in ICLR 2020.\n\nFirst, the authors failed to cite two closely related papers below:\n- Tokozume et al., LEARNING FROM BETWEEN-CLASS EXAMPLES FOR DEEP SOUND RECOGNITION. ICLR, 2018.\n- Tokozume et al., Between-class Learning for Image Classification. CVPR, 2018.\nThe first one is published in the previous ICLR and mixing two samples belonging to different classes. The second one is an application to image classification using ImageNet dataset, which is larger than the dataset used in this paper.  What's more important is that both papers propose that the mixing ratio of two samples is not linearly but depending on the strength of their signals. Since UMixUp is also focusing on the mixing ratio between two training samples, Between-Class Learning should have been compared to the proposed method.\n\nSecondly, the theoretical discussion is not so fascinating. Actually, both MixUp and UMixUp are shown to converge to DAT when the number of training samples tends to infinity. Data augmentation is, however, performed to remedy the lack of training samples in general. The discussion that the number of training samples is assumed to be large is the opposite situation. \n\nThirdly, the experimental results show that the performance gain by UMixUp is relatively small in comparison to that of the original MixUp. There are no ablation studies using different values for alpha and beta, which are parameters for the policy of UMixUp. The authors reported that these values are defined using a heuristic search. Thus, we cannot see if the performance is sensitive to the parameter selection.\n\nI lean to reject this paper because of these concerns. I'm looking forward to seeing the revised version in another conference.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper builds a connection between MixUp and adversarial training. It introduces untied MixUp (UMixUp), which generalizes the methods of MixUp. Then, it also shows that DAT and UMixUp use the same method of MixUp for generating samples but use different label mixing ratios. Though it has some valuable theoretical contributions, I agree with the reviewers that it’s important to include results on adversarial robustness, where both adversarial training and MixUp are playing an important role.
Decision: Reject