Title: MIXUP AS DIRECTIONAL ADVERSARIAL TRAINING

Abstract: MixUp is a data augmentation scheme in which pairs of training samples and their corresponding labels are mixed using linear coefficients. Without label mixing, MixUp becomes a more conventional scheme: input samples are moved but their original labels are retained. Because samples are preferentially moved in the direction of other classes we refer to this method as directional adversarial training, or DAT. We show that under two mild conditions, MixUp asymptotically convergences to a subset of DAT. We define untied MixUp (UMixUp), a superset of MixUp wherein training labels are mixed with different linear coefficients to those of their corresponding samples. We show that under the same mild conditions, untied MixUp converges to the entire class of DAT schemes. Motivated by the understanding that UMixUp is both a generalization of MixUp and a form of adversarial training, we experiment with different datasets and loss functions to show that UMixUp provides improved performance over MixUp. In short, we present a novel interpretation of MixUp as belonging to a class highly analogous to adversarial training, and on this basis we introduce a simple generalization which outperforms MixUp.

1. INTRODUCTION:

Deep learning applications often require complex networks with a large number of parameters (He et al., 2016; Zagoruyko & Komodakis, 2016; Devlin et al., 2018) . Although neural networks perform so well that their ability to generalize is an area of study in itself (Zhang et al., 2017a; Arpit et al., 2017) , their high complexity nevertheless causes them to overfit their training data (Kukacka et al., 2017) . For this reason, effective regularization techniques are in high demand.

There are two flavors of regularization: complexity curtailing and data augmentation1 . Complexity curtailing methods constrain models to learning in a subset of parameter space which has a higher probability of generalizing well. Notable examples are weight decay (Krogh & Hertz, 1991) and dropout (Srivastava et al., 2014) .

Data augmentation methods add transformed versions of training samples to the original training set. Conventionally, transformed samples retain their original label, so that models effectively see a larger set of data-label training pairs. Commonly applied transformations in image applications include flips, crops and rotations.

A recently devised family of augmentation schemes called adversarial training has attracted active research interest (Szegedy et al., 2013; Goodfellow et al., 2014; Miyato et al., 2016; Athalye et al., 2018; Shaham et al., 2018; He et al., 2018) . Adversarial training seeks to reduce a model's propensity to misclassify minimally perturbed training samples, or adversarials. While attack algorithms used for testing model robustness may search for adversarials in unbounded regions of input space, adversarial training schemes generally focus on perturbing training samples within a bounded region, while retaining the sample's original label (Goodfellow et al., 2015; Shaham et al., 2018) .

Another recently proposed data augmentation scheme is MixUp (Zhang et al., 2017b) , in which new samples are generated by mixing pairs of training samples using linear coefficients. Despite its well established generalization performance (Zhang et al., 2017b; Guo et al., 2018; Verma et al., 2018) , the working mechanism of MixUp is not well understood. Guo et al. (2018) perspective is insightful, we do not believe it paints a full picture of how MixUp operates. A recent study (Lamb et al., 2019) provides empirical evidence that MixUp improves adversarial robustness, but does not present MixUp as a form of adversarial training.

We build a framework to understand MixUp in a broader context: we argue that adversarial training is a central working principle of MixUp. To support this contention, we connect MixUp to a MixUplike scheme which does not perform label mixing, and we relate this scheme to adversarial training.

Without label mixing, MixUp becomes a conventional augmentation scheme: input samples are moved, but their original labels are retained. Because samples are moved in the direction of other samples -which are typically clustered in input space -we describe this method as 'directional'. Because this method primarily moves training samples in the direction of adversarial classes, this method is analogous to adversarial training. We thus refer to MixUp without label mixing as directional adversarial training (DAT). We show that MixUp converges to a subset of DAT under mild conditions, and we thereby argue that adversarial training is a working principle of MixUp.

Inspired by this new understanding of MixUp as a form of adversarial training, and upon realizing that MixUp is (asymptotically) a subset of DAT, we introduce Untied MixUp (UMixUp), a simple enhancement of MixUp which converges to the entire family of DAT schemes, as depicted in Figure 1 . Untied Mixup mixes data-label training pairs in a similar way to MixUp, with the distinction that the label mixing ratio is an arbitrary function of the sample mixing ratio. We perform experiments to show that UMixUp's classification performance improves upon MixUp.

In short, this research is motivated by a curiosity to better understand the working of MixUp. In-sodoing we aim to:

1. Establish DAT as analogous to adversarial training. This is discussed in section 4.

2. Establish UMixUp as a superset of MixUp, and as converging to the entire family of DAT schemes. In-so-doing, a) establish MixUp's convergence to a subset of DAT, and thereby that it operates analogously to adversarial training; and b) establish UMixUp as a broader class of MixUp-like schemes that operate analogously to adversarial training. This is discussed in 5.

3. Establish empirically that UMixUp's classification performance improves upon MixUp. This is discussed in section 6.

Finally we note that this paper has another contribution. Conventionally, MixUp is only applicable to baseline models that use cross entropy loss. All analytical results we develop in this paper are applicable to a wider family of models using any loss function which we term target-linear. We define target-linearity and experiment with a new loss function called negative cosine-loss to show its potential.

Essential proofs of theoretical results are given in the Appendix.

2. PRELIMINARIES:

Column vectors are denoted by bold letters such as m, and sets are denoted by calligraphic uppercase letters such as M. The component of a vector is denoted by a bracketed index. For example, m[i] denotes the ith component of m.

Regular (non-calligraphic) capitalized letters such as X will denote random variables, and their lowercase counterparts, e.g., x, will denote realizations of a random variable. Any sequence, (a 1 , a 2 , . . . , a n ) will be denoted by a n 1 . Likewise (A 1 , A 2 , . . . , A n ) will be denoted by A n 1 , and a sequence of sample pairs ((x 1 , x 1 ), (x 2 , x 2 ), . . . , (x n , x n )) denoted by (x, x ) n 1 . For any value a ∈ [0, 1], we will use a as a short notation for 1 -a.

Classification Setting Consider a standard classification problem, in which one wishes to learn a classifier that predicts the class label for a sample.

Formally, let X be a vector space in which the samples of interest live and let Y be the set of all possible labels associated with these samples. The set of training samples will be denoted by D, a subset of X . We will use t(x) to denote the true label of x. Let F be a neural network function, parameterized by θ, which maps X to another vector space Z. Let ϕ : Y → Z be a function that maps a label in Y to an element in Z such that for any y, y ∈ Y, if y = y , then ϕ(y) = ϕ(y ).

In the space Z, we refer to F (x) as the model's prediction. With slight abuse of language, we will occasionally refer to both t(x) and ϕ(t(x)) as the "label" of x. Let : Z ×Z → R be a loss function, using which one defines an overall loss function as

EQUATION

Here we have taken the notational convention that the first argument of represents the model's prediction and the second represents the target label. In this setting, the learning problem is formulated as minimizing L with respect to its characterizing parameters θ.

Target-Linear Loss Functions We say that a loss function (z, z ) is target-linear if for any scalars α and β, (z,

αz 1 + βz 2 ) = α (z, z 1 ) + β (z, z 2 )

Target-linear loss functions arise naturally in many settings, for which we now provide two examples. For convenience, we define the vectors v = F (x) and y = ϕ(t(x)).

Cross-Entropy Loss The conventional cross-entropy loss function, written in our notation, is defined as:

CE (F (x), ϕ(t(x))) = CE (v, y) := dim(Z) i=1 y[i] log v[i]

where v and y are constrained to being probability vectors. We note that in conventional applications, dim(Z) = |Y|, and the target label v is a one-hot vector where y

[i] = 1 if i = t(x)

and y[i] = 0 otherwise. Constraining v to being a probability vector is achieved using a softmax output layer.

Negative-Cosine Loss The "negative-cosine loss", usually used in its negated version, i.e., as the cosine similarity, can be defined as follows.

NC (F (x), ϕ(t(x))) = NC (v, y) := -v T y

where v and y are constrained to being unit-length vectors. For v this can be achieved by simple division at the output layer, and for y by limiting the range of ϕ to an orthonormal basis (making it a conventional label embedding function).

It is clear that the cross-entropy loss CE and the negative-cosine loss NC are both target-linear, directly following from the definition of target-linearity.

Assumptions The theoretical development of this paper relies on two fundamental assumptions, which we call "axioms".

Axiom 1 (Target linearity) The loss function used for the classification setting is target-linear.

That is, the study of MixUp in this paper is in fact goes beyond the standard MixUp, which uses the cross-entropy loss.

Much of the development in this paper concerns drawing sample pairs (x, x ) from D × D. Suppose that (x, x ) n 1 is a length-n sequence of sample pairs drawn from D × D. A sequence (x, x ) n 1 is said to be symmetric if for every (a, b) ∈ D × D, the number of occurrences of (a, b) in the sequence is equal to that of (b, a). A distribution Q on D × D will be called exchangeable, or symmetric, if for any

(x, x ) ∈ D × D, Q((x, x )) = Q((x , x)).

Axiom 2 (Symmetric pair-sampling distribution) Whenever a sample pair (x, x ) is drawn from a distribution Q, Q is assumed to be symmetric.

In the standard MixUp, two samples are drawn independently from D to form a pair, making this condition satisfied.

3. MIXUP, DAT, UNTIED MIXUP:



3.1. INFORMAL SUMMARY:

We first provide a summary of each scheme for the reader's convenience. We then describe each scheme more systematically. For concision of equations to follow, we define

y = ϕ(t(x)) and y = ϕ(t(x ))

MixUp is a data augmentation scheme in which samples are linearly combined using some mixing ratio λ ∈ [0, 1]:

EQUATION

where λ ∼ P Mix . A target label is generated using the same mixing ratio λ:

y g = λy + (1 -λ)y (MixUp)

DAT and UMixUp use the same method (2) for generating samples, but use different λ distributions (P DAT and P uMix respectively). DAT and UMixUp also differ from MixUp in their target labels. DAT retains the sample's original label:

y g = y (DAT)

whereas UMixUp's label mixing ratio is a function of λ:

y g = γ(λ)y + (1 -γ(λ))y (UMixUp)

In Untied MixUp, the label mixing ratio is "untied" from the sample mixing ratio, and can be any γ(λ). We will refer to γ as the weighting function. An Untied MixUp scheme is specified both by the its mixing policy P uMix and a weighting function γ.

3.2. FORMAL DEFINITIONS:

To draw comparisons between MixUp, DAT, and Untied MixUp schemes, we establish a framework for characterizing their optimization problems. To that end, we define each model's loss function m in terms of its baseline target-linear loss function b , where the superscript m is replaced with a model identifier (i.e. Mix , DAT , uMix ). Each model's overall loss function, L m is defined in terms of its loss function m as per equation 1 (where equation 1's is m ). We denote the expected value of each scheme's overall loss, L m E , with respect to its mixing ratio Λ. Let n be a positive integer. In every scheme, a sequence (x, x ) n 1 := ((x 1 , x 1 ), (x 2 , x 2 ), . . . , (x n , x n )) of sample pairs are drawn i.i.d. from Q, and a sequence λ n 1 := (λ 1 , λ 2 , . . . , λ n ) of values are drawn i.i.d. from P m , where P m is a distribution over [0, 1] unique to each model.

MixUp For any x, x ∈ D and any λ ∈ [0, 1], denote Mix (x, x , λ) := b F (λx + λx ), λy + λy Let P m be P Mix ; in other words a sequence λ n 1 := (λ 1 , λ 2 , . . . , λ K ) of values are drawn i.i.d. from P Mix . We denote the overall loss L Mix ((x, x ) n 1 , λ n 1 ) and the expected overall loss L Mix E ((x, x ) n 1 ):

L Mix ((x, x ) n 1 , λ n 1 ) := 1 n n k=1 Mix (x k , x k , λ k ) (3) L Mix E ((x, x ) n 1 ) := E λ n 1 iid ∼ P Mix L Mix ((x, x ) n 1 , λ n 1 )

In MixUp, we refer to P Mix as the mixing policy.

Directional Adversarial Training (DAT) For any x, x ∈ D and any λ ∈ [0, 1], we denote

DAT (x, x , λ) := b F (λx + λx ), y

Let P m be P DAT , such that members of λ n 1 are drawn i.i.d. from P DAT . We denote the overall loss L DAT ((x, x ) n 1 , λ n 1 }) and the expected overall loss L DAT E ((x, x ) n 1 ):

L DAT ((x, x ) n 1 , λ n 1 }) := 1 n n k=1 DAT (x k , x k , λ k ) (4) L DAT E ((x, x ) n 1 ) := E λ n 1 iid ∼ P DAT L DAT ((x, x ) n 1 , λ n 1 )

In DAT, we refer to P DAT as the adversarial policy.

Untied MixUp (UMixUp):

Let γ be a function mapping [0, 1] to [0, 1]. For any x, x ∈ D and any λ ∈ [0, 1], we denote uMix (x, x , λ, γ) := b (F (λx + λx ), γ(λ)y) + γ(λ)y )

Let P m be P uMix , and denote the overall and expected overall loss functions L uMix ((x, x ) n 1 , λ n 1 , γ) and L uMix E ((x, x ) n 1 , γ) respectively:

L uMix ((x, x ) n 1 , λ n 1 , γ) := 1 n n k=1 uMix (x, x , λ, γ) L uMix E ((x, x ) n 1 , γ) := E λ n 1 iid ∼ P uMix L uMix ((x, x ) n 1 , λ n 1 , γ)

At this end, it is apparent that MixUp is a special case of Untied MixUp, where the function γ(λ) takes the simple form γ(λ) = λ.

4. DAT AS ANALOGOUS TO ADVERSARIAL TRAINING:

The main theoretical result of this paper is the relationship established between DAT and UMixUp, and by extension MixUp. Both MixUp and UMixUp will be shown to converge to DAT as the number of mixed sample pairs, n, tends to infinity. Prior to developing these results, we provide insight into DAT, in terms of its similarity to adversarial training and its regularization mechanisms. The key distinction between conventional adversarial training and inter-class mixing is that MixUp movement is determined probabilistically within a bounded region, while adversarial movement is deterministic.

Figure 2 illustrates the connection between standard adversarial training and DAT. Consider the problem of classifying the blue points and the black points in Figure 2a ), where the dashed curve is a ground-truth classifier and the black curve indicates the classification boundary of F (x), which overfits the training data. In adversarial training, a training sample x is moved to a location within an L p -ball around x while keeping its label to further train the model; the location, denoted by x 1 in Figure 2b ), is chosen to maximize training loss. In DAT, a second sample x governs the direction in which x is perturbed. If x is chosen from a different class as shown in Figure 2c ), then the generated sample x 2 is used to further train the model. If x is chosen from the same class as shown in Figure 2d ), then the sample x 3 is used in further training. Note that the inter-class mixed sample x 2 pushes the model's classification boundary closer to the ground-truth classifier, thus connecting adversarial training and DAT. The intra-class sample x 3 , on the other hand, mainly helps to smooth inner parts of the class region. The latter behaviour is an additional feature of DAT and MixUp, which distinguishes these schemes from adversarial training.

5. UNTIED MIXUP AS ASYMPTOTICALLY EQUIVALENT TO DAT:

We now show that Untied MixUp and DAT are equivalent when n tends to infinity. A consequence of this equivalence is that it infuses both MixUp and UMixUp with the intuition of adversarial training.

To that end, we relate the Untied MixUp loss function, uMix , with the DAT loss function, DAT .

Lemma 1 For any (x, x ) ∈ D × D and any λ ∈ [0, 1],

uMix (x, x , λ, γ) = γ(λ) DAT (x, x , λ) + γ(λ) DAT (x , x, λ)

This result follows directly from the target-linearity of the loss function.

The next two lemmas show that as n tends to infinity, the overall loss of both DAT and UMixUp converge in probability to their respective overall expected losses.

Lemma 2 As n increases, L DAT ((x, x ) n 1 , Λ n 1 ) converges to L DAT E (x, x ) n 1 in probability.

Two target-linear loss functions are essayed: cross-entropy (CE) loss and the negative-cosine (CE) loss as defined earlier. We implement CE loss similarly to previous works, which use CE loss to implement the baseline model. In our implementation of the NC loss model, for each label y, ϕ(y) is mapped to a randomly selected unit-length vector of dimension d and fixed during training; the feature map of the original PreActResNet18 is linearly transformed to a d-dimensional vector. The dimension d is chosen as 300 for Cifar10, MNIST and Fashion-Mnist (which have one black-andwhite channel) and 700 for Cifar100 (which has 3 colored channels).

Our implementation of MixUp and Untied MixUp improves upon the published implementation from the original authors of MixUp Zhang et al. (2017b) . For example, the original authors' implementation samples only one λ per mini-batch, giving rise to unnecessarily higher stochasticity of the gradient signal. Our implementation samples λ independently for each sample. Additionally, the original code combines inputs by mixing a mini-batch of samples with a shuffled version of itself. This approach introduces a dependency between sampled pairs and again increases the stochasticity of training. Our implementation creates two shuffled copies of the entire training dataset prior to each epoch, pairs them up, and then splits them into mini-batches. This gives a closer approximation to i.i.d. sampling and makes training smoother. While these implementation improvements have merit on their own, they do not provide a theoretical leap in understanding, and so we do not quantify their impact in our results analysis.

All models examined are trained using mini-batched backpropagation, for 200 epochs.

6.2. RESULTS:

We sweep over the policy space of MixUp and Untied MixUp. For MixUp, it is sufficient to consider distribution P Mix to be symmetric about 0.5. Thus we consider only consider P Mix in the form of B(α, α), and scan through a single parameter α systematically. Since the policy of Untied MixUp is in the form of U(B(α, β)), searching through (α, β) becomes more difficult. Thus our policy search for Untied MixUp is restricted to an ad hoc heuristic search. For this reason, the found best policy for Untied MixUp might be quite far from the true optimal.

The main results of our experiments are given in tables 1 to 4. As shown in the tables, each setting is run 100 times. For each run, we compute the error rate in a run as the average test error rate over the final 10 epochs. The estimated mean ("MEAN") performance of a setting is computed as the average of the error rates over all runs for the same setting. The 95%-confidence interval ("ConfInt") for the estimated mean performance is also computed and shown in the table.

From these results, we see that the Untied MixUp schemes each outperform their MixUp counterparts. Specifically, in 6 of the 8 cases (those printed in bold font), the confidence interval of Untied MixUp is completely disjoint from that of the corresponding MixUp scheme; and in some cases, the separation of confidence intervals is by a large margin. Note that the baseline model (PreActRes-Net18) has been designed with highly focused inductive bias for image classification tasks. Under such an inductive bias, one expects that the room for regularization (or the "amount of overfitting") isn't abundant. As such, we consider the improvement of Untied MixUp over MixUp rather significant.

The results show empirically that MixUp and Untied MixUp both work on the NC loss models. This validates our generalization of MixUp (and Untied MixUp) to models built with target linear losses. model policy runs MEAN ConfInt baseline-CE -100 5.476% 0.027% mixUp-CE B(0.9, 0.9) 100 4.199% 0.023% uMixUp-CE U(B(2.2, 0.9)) 100 4.177% 0.025% baseline-NC -100 5.605% 0.030% mixUp-NC B(1.0, 1.0) 100 4.508% 0.022% uMixUp-NC U(B(1.8, 1.0)) 100 4.455% 0.025% B(0.9, 0.9) 100 24.298% 0.051% uMixUp-NC U(B(1.3, 0.9)) 100 23.819% 0.054% B(1.0, 1.0) 100 0.632% 0.005% uMixUp-CE U(B(1.7, 1.0)) 100 0.609% 0.005% baseline-NC -100 0.720% 0.007% mixUp-NC B(1.0, 1.0) 100 0.607% 0.004% uMixUp-NC U(B(1.3, 0.9)) 100 0.592% 0.005% 

7. CONCLUDING REMARKS:

This paper establishes a connection between MixUp and adversarial training. This connection allows for a better understanding of the working mechanism of MixUp as well as a generalization of MixUp to a wider family, namely Untied MixUp. Despite the development in this work, it is the authors' belief that the current designs of MixUp and Untied MixUp are far from optimal. In particular, we believe a better design should allow individualized policy for each training pair. How this can be done remains open at this time. ) . where (a) is due to the symmetry of (x, x ) n 1 , and (b) is by a change of variable in the second term (renaming 1 -λ as λ).

:

Some authors describe these flavors as "data independent" and "data-dependent"(Guo et al.,

2018).

Figures and Tables:

Figure 1: The relationship between MixUp, DAT and Untied MixUp.

Conventional adversarial training schemes augment the original training dataset by searching for approximations of true adversarials within bounded regions around each training sample. For a training sample x, a bounded region U known as an L p ball is defined as U = {x + η η η : ||η η η|| p < }. Over this region, the loss function with respect to the true label of x is maximized. A typical loss function for an adversarial scheme is(F (x), y) = max x∈U b (F (x), y)where b is the baseline loss function. Simply put, baseline training serves to learn correct classification over the training data, whereas adversarial training moves the classification boundary to improve generalization.DAT, on the other hand, combines intra-class mixing (mixing two samples of the same class) and inter-class mixing (mixing samples of different classes). Intra-class mixing serves to smooth classification boundaries of inner-class regions, while inter-class mixing perturbs training samples in the direction of adversarial classes, which improves generalization. Inter-class mixing dwarves intraclass mixing by volume of generated samples seen by the learning model in most many-class learning problems (by a 9-1 ratio in balanced 10-class problems for instance). DAT, which primarily consists of inter-class mixing, can therefore be seen as analogous to adversarial training.

Figure 2: DAT as a form of adversarial training

DAT (x k , x k , λ) + γ(λ) DAT (x k , x k , λ) DAT (x k , x k , λ) + E λ∼P Mix n k=1 γ(λ) DAT (x k , x k , λ) DAT (x k , x k , λ) + E λ∼P Mix n k=1 γ(λ) DAT (x k , x k , λ) DAT (x k , x k , λ) + E λ∼P Mix n k=1 γ(λ) DAT (x k , x k , λ) P Mix (λ) DAT (x k , x k , λ) + γ(λ)P Mix (1 -λ) DAT (x k , x k , λ) dλ = 1 n n k=1 DAT (x k , x k , λ) γ(λ)P Mix (λ) + γ(λ)P Mix (1 -λ) Du(P Mix ,γ) λ∼P DAT DAT (x k , x k , λ) = L DAT E ((x, x ) n 1

Test error rate on CIFAR10.

Table content: <table><tr><td>model</td><td>policy</td><td>runs</td><td>MEAN</td><td>ConfInt</td></tr><tr><td>baseline-CE</td><td>-</td><td colspan="3">100 24.848% 0.060%</td></tr><tr><td>mixUp-CE</td><td>B(0.9, 0.9)</td><td colspan="3">100 22.020% 0.050%</td></tr><tr><td colspan="5">uMixUp-CE U(B(1.4, 0.7)) 100 21.884% 0.051%</td></tr><tr><td>baseline-NC</td><td>-</td><td colspan="3">100 25.270% 0.050%</td></tr><tr><td>mixUp-NC</td><td/><td/><td/><td/></tr></table>

Test error rate on CIFAR100.

Table content: <table><tr><td>model</td><td>policy</td><td>runs MEAN ConfInt</td></tr><tr><td>baseline-CE</td><td>-</td><td>100 0.816% 0.007%</td></tr><tr><td>mixUp-CE</td><td/><td/></tr></table>

Test error rate on MNIST.

Table content: <table><tr><td>model</td><td>policy</td><td>runs MEAN ConfInt</td></tr><tr><td>baseline-CE</td><td>-</td><td>100 5.060% 0.027%</td></tr><tr><td>mixUp-CE</td><td>B(1.0, 1.0)</td><td>100 4.585% 0.013%</td></tr><tr><td colspan="3">uMixUp-CE U(B(1.7, 0.8)) 100 4.570% 0.013%</td></tr><tr><td>baseline-NC</td><td>-</td><td>100 5.083% 0.016%</td></tr><tr><td>mixUp-NC</td><td>B(1.0, 1.0)</td><td>100 4.767% 0.013%</td></tr><tr><td colspan="3">uMixUp-NC U(B(1.3, 0.9)) 100 4.613% 0.011%</td></tr></table>

Test error rate on Fashion-MNIST.

Table content: <table/>

Peer review (iclr2020):
First of all, the concept of *Directional Adversarial Training (DAT)* is not appropriate. Actually similar method has been proposed in Hiroshi Inoue (2018) as a data augmentation method. What surprised me the most is that after trying to connect UMixUp with adversarial training in the whole paper, there is no evaluation of adversarial robustness in the experiments? The main character of adversarial training is an improvement in robustness and degeneration on clean accuracy, which is different from the performance of UMixup or Mixup. The authors should do a major modification on their motivation and experiments before the paper is able to be published. Reference: [1] Hiroshi Inoue. Data Augmentation by Pairing Samples for Images Classification. arXiv 1801.02929