Year: 2019

Abstract:
MixUp is a data augmentation scheme in which pairs of training samples and their corresponding labels are mixed using linear coefficients. Without label mixing, MixUp becomes a more conventional scheme: input samples are moved but their original labels are retained. Because samples are preferentially moved in the direction of other classes we refer to this method as directional adversarial training, or DAT. We show that under two mild conditions, MixUp asymptotically convergences to a subset of DAT. We define untied MixUp (UMixUp), a superset of MixUp wherein training labels are mixed with different linear coefficients to those of their corresponding samples. We show that under the same mild conditions, untied MixUp converges to the entire class of DAT schemes. Motivated by the understanding that UMixUp is both a generalization of MixUp and a form of adversarial training, we experiment with different datasets and loss functions to show that UMixUp provides improved performance over MixUp. In short, we present a novel interpretation of MixUp as belonging to a class highly analogous to adversarial training, and on this basis we introduce a simple generalization which outperforms MixUp.

heading: 1 INTRODUCTION
text: Deep learning applications often require complex networks with a large number of parameters (He et al., 2016; Zagoruyko & Komodakis, 2016; Devlin et al., 2018). Although neural networks perform so well that their ability to generalize is an area of study in itself (Zhang et al., 2017a; Arpit et al., 2017), their high complexity nevertheless causes them to overfit their training data (Kukacka et al., 2017). For this reason, effective regularization techniques are in high demand.
There are two flavors of regularization: complexity curtailing and data augmentation 1. Complexity curtailing methods constrain models to learning in a subset of parameter space which has a higher probability of generalizing well. Notable examples are weight decay (Krogh & Hertz, 1991) and dropout (Srivastava et al., 2014).
Data augmentation methods add transformed versions of training samples to the original training set. Conventionally, transformed samples retain their original label, so that models effectively see a larger set of data-label training pairs. Commonly applied transformations in image applications include flips, crops and rotations.
A recently devised family of augmentation schemes called adversarial training has attracted active research interest (Szegedy et al., 2013; Goodfellow et al., 2014; Miyato et al., 2016; Athalye et al., 2018; Shaham et al., 2018; He et al., 2018). Adversarial training seeks to reduce a model’s propensity to misclassify minimally perturbed training samples, or adversarials. While attack algorithms used for testing model robustness may search for adversarials in unbounded regions of input space, adversarial training schemes generally focus on perturbing training samples within a bounded region, while retaining the sample’s original label (Goodfellow et al., 2015; Shaham et al., 2018).
Another recently proposed data augmentation scheme is MixUp (Zhang et al., 2017b), in which new samples are generated by mixing pairs of training samples using linear coefficients. Despite its well established generalization performance (Zhang et al., 2017b; Guo et al., 2018; Verma et al., 2018), the working mechanism of MixUp is not well understood. Guo et al. (2018) suggest viewing MixUp as imposing local linearity on the model using points outside of the data manifold. While this
1Some authors describe these flavors as “data independent” and “data-dependent” (Guo et al., 2018).
perspective is insightful, we do not believe it paints a full picture of how MixUp operates. A recent study (Lamb et al., 2019) provides empirical evidence that MixUp improves adversarial robustness, but does not present MixUp as a form of adversarial training.
We build a framework to understand MixUp in a broader context: we argue that adversarial training is a central working principle of MixUp. To support this contention, we connect MixUp to a MixUplike scheme which does not perform label mixing, and we relate this scheme to adversarial training.
Without label mixing, MixUp becomes a conventional augmentation scheme: input samples are moved, but their original labels are retained. Because samples are moved in the direction of other samples – which are typically clustered in input space – we describe this method as ‘directional’. Because this method primarily moves training samples in the direction of adversarial classes, this method is analogous to adversarial training. We thus refer to MixUp without label mixing as directional adversarial training (DAT). We show that MixUp converges to a subset of DAT under mild conditions, and we thereby argue that adversarial training is a working principle of MixUp.
Inspired by this new understanding of MixUp as a form of adversarial training, and upon realizing that MixUp is (asymptotically) a subset of DAT, we introduce Untied MixUp (UMixUp), a simple enhancement of MixUp which converges to the entire family of DAT schemes, as depicted in Figure 1. Untied Mixup mixes data-label training pairs in a similar way to MixUp, with the distinction that the label mixing ratio is an arbitrary function of the sample mixing ratio. We perform experiments to show that UMixUp’s classification performance improves upon MixUp.
In short, this research is motivated by a curiosity to better understand the working of MixUp. In-sodoing we aim to:
1. Establish DAT as analogous to adversarial training. This is discussed in section 4. 2. Establish UMixUp as a superset of MixUp, and as converging to the entire family of
DAT schemes. In-so-doing, a) establish MixUp’s convergence to a subset of DAT, and thereby that it operates analogously to adversarial training; and b) establish UMixUp as a broader class of MixUp-like schemes that operate analogously to adversarial training. This is discussed in 5.
3. Establish empirically that UMixUp’s classification performance improves upon MixUp. This is discussed in section 6.
Finally we note that this paper has another contribution. Conventionally, MixUp is only applicable to baseline models that use cross entropy loss. All analytical results we develop in this paper are applicable to a wider family of models using any loss function which we term target-linear. We define target-linearity and experiment with a new loss function called negative cosine-loss to show its potential.
Essential proofs of theoretical results are given in the Appendix.

heading: 2 PRELIMINARIES
text: Column vectors are denoted by bold letters such as m, and sets are denoted by calligraphic uppercase letters such asM. The component of a vector is denoted by a bracketed index. For example, m[i] denotes the ith component of m.
Regular (non-calligraphic) capitalized letters such as X will denote random variables, and their lowercase counterparts, e.g., x, will denote realizations of a random variable. Any sequence, (a1, a2, . . . , an) will be denoted by an1 . Likewise (A1, A2, . . . , An) will be denoted by A n 1 , and a sequence of sample pairs ((x1, x′1), (x2, x′2), . . . , (xn, x′n)) denoted by (x, x′)n1 .
For any value a ∈ [0, 1], we will use a as a short notation for 1− a. Classification Setting Consider a standard classification problem, in which one wishes to learn a classifier that predicts the class label for a sample.
Formally, let X be a vector space in which the samples of interest live and let Y be the set of all possible labels associated with these samples. The set of training samples will be denoted by D, a subset of X . We will use t(x) to denote the true label of x. Let F be a neural network function, parameterized by θ, which maps X to another vector space Z . Let ϕ : Y → Z be a function that maps a label in Y to an element in Z such that for any y, y′ ∈ Y , if y 6= y′, then ϕ(y) 6= ϕ(y′). In the space Z , we refer to F (x) as the model’s prediction. With slight abuse of language, we will occasionally refer to both t(x) and ϕ(t(x)) as the “label” of x. Let ` : Z×Z → R be a loss function, using which one defines an overall loss function as
L := 1 |D| ∑ x∈D ` (F (x), ϕ(t(x))) (1)
Here we have taken the notational convention that the first argument of ` represents the model’s prediction and the second represents the target label. In this setting, the learning problem is formulated as minimizing L with respect to its characterizing parameters θ. Target-Linear Loss Functions We say that a loss function `(z, z′) is target-linear if for any scalars α and β,
`(z, αz1 + βz2) = α`(z, z1) + β`(z, z2) Target-linear loss functions arise naturally in many settings, for which we now provide two examples. For convenience, we define the vectors v = F (x) and y = ϕ(t(x)).
Cross-Entropy Loss The conventional cross-entropy loss function, written in our notation, is defined as:
`CE (F (x), ϕ(t(x))) = `CE(v, y) := dim(Z)∑ i=1 y[i] log v[i]
where v and y are constrained to being probability vectors. We note that in conventional applications, dim(Z) = |Y|, and the target label v is a one-hot vector where y[i] = 1 if i = t(x) and y[i] = 0 otherwise. Constraining v to being a probability vector is achieved using a softmax output layer.
Negative-Cosine Loss The “negative-cosine loss”, usually used in its negated version, i.e., as the cosine similarity, can be defined as follows.
`NC (F (x), ϕ(t(x))) = `NC(v, y) := −vTy
where v and y are constrained to being unit-length vectors. For v this can be achieved by simple division at the output layer, and for y by limiting the range of ϕ to an orthonormal basis (making it a conventional label embedding function).
It is clear that the cross-entropy loss `CE and the negative-cosine loss `NC are both target-linear, directly following from the definition of target-linearity.
Assumptions The theoretical development of this paper relies on two fundamental assumptions, which we call “axioms”.
Axiom 1 (Target linearity) The loss function ` used for the classification setting is target-linear.
That is, the study of MixUp in this paper is in fact goes beyond the standard MixUp, which uses the cross-entropy loss.
Much of the development in this paper concerns drawing sample pairs (x, x′) from D×D. Suppose that (x, x′)n1 is a length-n sequence of sample pairs drawn from D ×D. A sequence (x, x′)n1 is said to be symmetric if for every (a,b) ∈ D ×D, the number of occurrences of (a,b) in the sequence is equal to that of (b, a). A distribution Q on D ×D will be called exchangeable, or symmetric, if for any (x, x′) ∈ D ×D, Q((x, x′)) = Q((x′, x)).
Axiom 2 (Symmetric pair-sampling distribution) Whenever a sample pair (x, x′) is drawn from a distribution Q, Q is assumed to be symmetric.
In the standard MixUp, two samples are drawn independently from D to form a pair, making this condition satisfied.

heading: 3 MIXUP, DAT, UNTIED MIXUP

heading: 3.1 INFORMAL SUMMARY
text: We first provide a summary of each scheme for the reader’s convenience. We then describe each scheme more systematically. For concision of equations to follow, we define
y = ϕ(t(x)) and y′ = ϕ(t(x′))
MixUp is a data augmentation scheme in which samples are linearly combined using some mixing ratio λ ∈ [0, 1]:
xg = λx + (1− λ)x′ (2)
where λ ∼ PMix. A target label is generated using the same mixing ratio λ:
yg = λy + (1− λ)y′ (MixUp)
DAT and UMixUp use the same method (2) for generating samples, but use different λ distributions (PDAT and P uMix respectively). DAT and UMixUp also differ from MixUp in their target labels. DAT retains the sample’s original label:
yg = y (DAT)
whereas UMixUp’s label mixing ratio is a function of λ:
yg = γ(λ)y + (1− γ(λ))y′ (UMixUp)
In Untied MixUp, the label mixing ratio is “untied” from the sample mixing ratio, and can be any γ(λ). We will refer to γ as the weighting function. An Untied MixUp scheme is specified both by the its mixing policy P uMix and a weighting function γ.

heading: 3.2 FORMAL DEFINITIONS
text: To draw comparisons between MixUp, DAT, and Untied MixUp schemes, we establish a framework for characterizing their optimization problems. To that end, we define each model’s loss function `m in terms of its baseline target-linear loss function `b, where the superscript m is replaced with a model identifier (i.e. `Mix, `DAT, `uMix). Each model’s overall loss function, Lm is defined in terms of its loss function `m as per equation 1 (where equation 1’s ` is `m). We denote the expected value of each scheme’s overall loss, LmE , with respect to its mixing ratio Λ. Let n be a positive integer. In every scheme, a sequence (x, x′)n1 := ((x1, x′1), (x2, x′2), . . . , (xn, x′n)) of sample pairs are drawn i.i.d. from Q, and a sequence λn1 := (λ1, λ2, . . . , λn) of values are drawn i.i.d. from Pm, where Pm is a distribution over [0, 1] unique to each model.
MixUp For any x, x′ ∈ D and any λ ∈ [0, 1], denote
`Mix(x, x′, λ) := `b ( F (λx + λx′), λy + λy′ ) Let Pm be PMix; in other words a sequence λn1 := (λ1, λ2, . . . , λK) of values are drawn i.i.d. from PMix. We denote the overall loss LMix ((x, x′)n1 , λn1 ) and the expected overall loss LMixE ((x, x′)n1 ):
LMix ((x, x′)n1 , λn1 ) := 1
n n∑ k=1 `Mix(xk, x′k, λk) (3)
LMixE ((x, x′)n1 ) := Eλn1 iid∼PMix LMix ((x, x′)n1 , λn1 )
In MixUp, we refer to PMix as the mixing policy.
Directional Adversarial Training (DAT) For any x, x′ ∈ D and any λ ∈ [0, 1], we denote `DAT(x, x′, λ) := `b ( F (λx + λx′), y ) Let Pm be PDAT, such that members of λn1 are drawn i.i.d. from P
DAT. We denote the overall loss LDAT ((x, x′)n1 , λn1}) and the expected overall loss LDATE ((x, x′)n1 ):
LDAT ((x, x′)n1 , λn1}) := 1
n n∑ k=1 `DAT (xk, x′k, λk) (4)
LDATE ((x, x′)n1 ) := Eλn1 iid∼PDAT LDAT ((x, x′)n1 , λn1 )
In DAT, we refer to PDAT as the adversarial policy.
Untied MixUp (UMixUp)
Let γ be a function mapping [0, 1] to [0, 1]. For any x, x′ ∈ D and any λ ∈ [0, 1], we denote
`uMix(x, x′, λ, γ) := `b(F (λx + λx′), γ(λ)y) + γ(λ)y′)
Let Pm be P uMix, and denote the overall and expected overall loss functions LuMix ((x, x′)n1 , λn1 , γ) and LuMixE ((x, x′)n1 , γ) respectively:
LuMix ((x, x′)n1 , λn1 , γ) := 1
n n∑ k=1 `uMix(x, x′, λ, γ)
LuMixE ((x, x′)n1 , γ) := Eλn1 iid∼PuMix LuMix ((x, x′)n1 , λn1 , γ)
At this end, it is apparent that MixUp is a special case of Untied MixUp, where the function γ(λ) takes the simple form γ(λ) = λ.

heading: 4 DAT AS ANALOGOUS TO ADVERSARIAL TRAINING
text: The main theoretical result of this paper is the relationship established between DAT and UMixUp, and by extension MixUp. Both MixUp and UMixUp will be shown to converge to DAT as the number of mixed sample pairs, n, tends to infinity. Prior to developing these results, we provide insight into DAT, in terms of its similarity to adversarial training and its regularization mechanisms.
Conventional adversarial training schemes augment the original training dataset by searching for approximations of true adversarials within bounded regions around each training sample. For a training sample x, a bounded region U known as an Lp ball is defined as U = {x + η : ||η||p < }. Over this region, the loss function with respect to the true label of x is maximized. A typical loss function for an adversarial scheme is
`(F (x), y) = max x̃∈U `b(F (x̃), y)
where `b is the baseline loss function. Simply put, baseline training serves to learn correct classification over the training data, whereas adversarial training moves the classification boundary to improve generalization.
DAT, on the other hand, combines intra-class mixing (mixing two samples of the same class) and inter-class mixing (mixing samples of different classes). Intra-class mixing serves to smooth classification boundaries of inner-class regions, while inter-class mixing perturbs training samples in the direction of adversarial classes, which improves generalization. Inter-class mixing dwarves intraclass mixing by volume of generated samples seen by the learning model in most many-class learning problems (by a 9-1 ratio in balanced 10-class problems for instance). DAT, which primarily consists of inter-class mixing, can therefore be seen as analogous to adversarial training.
The key distinction between conventional adversarial training and inter-class mixing is that MixUp movement is determined probabilistically within a bounded region, while adversarial movement is deterministic.
Figure 2 illustrates the connection between standard adversarial training and DAT. Consider the problem of classifying the blue points and the black points in Figure 2a), where the dashed curve is a ground-truth classifier and the black curve indicates the classification boundary of F (x), which overfits the training data. In adversarial training, a training sample x is moved to a location within an Lp-ball around x while keeping its label to further train the model; the location, denoted by x̂1 in Figure 2b), is chosen to maximize training loss.
In DAT, a second sample x′ governs the direction in which x is perturbed. If x′ is chosen from a different class as shown in Figure 2c), then the generated sample x̂2 is used to further train the model. If x′ is chosen from the same class as shown in Figure 2d), then the sample x̂3 is used in further training. Note that the inter-class mixed sample x̂2 pushes the model’s classification boundary closer to the ground-truth classifier, thus connecting adversarial training and DAT. The intra-class sample x̂3, on the other hand, mainly helps to smooth inner parts of the class region. The latter behaviour is an additional feature of DAT and MixUp, which distinguishes these schemes from adversarial training.

heading: 5 UNTIED MIXUP AS ASYMPTOTICALLY EQUIVALENT TO DAT
text: We now show that Untied MixUp and DAT are equivalent when n tends to infinity. A consequence of this equivalence is that it infuses both MixUp and UMixUp with the intuition of adversarial training. To that end, we relate the Untied MixUp loss function, `uMix, with the DAT loss function, `DAT.
Lemma 1 For any (x, x′) ∈ D ×D and any λ ∈ [0, 1],
`uMix(x, x′, λ, γ) = γ(λ)`DAT(x, x′, λ) + γ(λ)`DAT(x′, x, λ)
This result follows directly from the target-linearity of the loss function.
The next two lemmas show that as n tends to infinity, the overall loss of both DAT and UMixUp converge in probability to their respective overall expected losses.
Lemma 2 As n increases, LDAT ((x, x′)n1 ,Λn1 ) converges to LDATE ( (x, x′)n1 ) in probability.
Lemma 3 As n increases, LuMix ((x, x′)n1 ,Λn1 , γ) converges to LuMixE ( (x, x′)n1 , γ ) in probability.
These two lemmas have similar proofs, thus only the proof of Lemma 2 is given in section A.1.
Next we show that as n tends to infinity, UMixUp converges in probability to a subset of DAT, and DAT converges in probability to a subset of UMixUp. In other words, we show that as n increases, UMixUp converges to being equivalent to the entire class of DAT schemes.
For that purpose, let F denote the space of all functions mapping [0, 1] to [0, 1]. Each configuration in P × F defines an Untied MixUp scheme. We now define U, which maps a DAT scheme to an Untied MixUp scheme. Specifically U is a map from P to P × F such that for any p ∈ P , U(p) is a configuration (p′, g) ∈ P × F , where
p′(λ) := 1 2 (p(λ) + p(1− λ)) and g(λ) := p(λ) p(λ) + p(1− λ) (5)
Lemma 4 Let (x, x′)n1 be a sequence of sample pairs on which an Untied MixUp scheme specified by (P uMix, γ) and a DAT scheme with policy PDAT will apply independently. If (x, x′)n1 is symmetric and ( P uMix, γ ) = U(PDAT), then LuMix ((x, x′)n1 , γ) = LDAT ((x, x′)n1 ) .
We now define another map Du that maps an Untied MixUp scheme to a DAT scheme. Specifically Du is a map from P ×F to P such that for any (p, g) ∈ P ×F , Du(p, g) is a configuration p′ ∈ P , where
p′(λ) := ( g(λ)p(λ) + g(λ)p(1− λ) ) It is easy to verify that ∫ 1 0 p′(λ)dλ = 1. Thus p′ is indeed a distribution in P and Du is well defined.
Lemma 5 Let (x, x′)n1 be a sequence of sample pairs on which an Untied MixUp scheme specified by (P uMix, γ) and a DAT scheme with policy PDAT will apply independently. If (x, x′)n1 is symmetric and PDAT = Du ( P uMix, γ ) , then LuMix ((x, x′)n1 , γ) = LDAT ((x, x′)n1 ) .
Lemmas 2, 3, 4 and 5 provide the building blocks for theorem 1, which we state hereafter. As n increases, both DAT and UMixUp converge in probability toward their respective expected loss (lemmas 2 and 3). Since as n increases, the sequence (x, x′)n1 becomes arbitrarily close to the symmetric sampling distribution Q, then by lemma 4 the family of DAT schemes converges in probability to a subset of UMixUp schemes. Lemma 5 proves the converse, i.e. that as n increases the family of UMixUp schemes converges in probability to a subset of DAT schemes. As n increases, the family of UMixUp schemes therefore converges in probability to the entire family of DAT schemes.
Theorem 1 Let (X,X′)∞1 be drawn i.i.d. from Q. On this sample-pair data, an Untied MixUp scheme specified by (PMix, γ) and a DAT scheme specified by PDAT will apply. In the Untied MixUp scheme, let Λ∞1 be drawn i.i.d. from P
Mix; in the DAT scheme, let Υ∞1 be drawn i.i.d. from PDAT. If PDAT = Du ( PMix, γ ) or ( PMix, γ ) = U(PDAT), then∣∣LMix ((X,X ′)n1 ,Λn1 , γ)− LDAT ((X,X ′)n1 ,Υn1 )∣∣ p−→ 0, as n→∞
The equivalence between the two families of schemes also indicates that there are DAT schemes that do not correspond to a MixUp scheme. These DAT schemes correspond to Untied MixUp scheme beyond the standard MixUp. The relationship between MixUp, DAT and Untied MixUp is shown in Figure 1.

heading: 6 UMIXUP AS A USEFUL GENERALIZATION OF MIXUP: EXPERIMENTS

heading: 6.1 EXPERIMENT SETUP AND IMPLEMENTATION
text: We consider an image classification task on the Cifar10, Cifar100, MNIST and Fashion-MNIST datasets. The baseline classifier chosen is PreActResNet18 (see Liu (2017)), noting the same choice is made by the authors of MixupZhang et al. (2017b).
Both MixUp and Untied MixUp are considered in the experiments. The MixUp policies are chosen as Beta distribution B(α, β). The Untied MixUp policy is taken as U(B(α, β)).
Two target-linear loss functions are essayed: cross-entropy (CE) loss and the negative-cosine (CE) loss as defined earlier. We implement CE loss similarly to previous works, which use CE loss to implement the baseline model. In our implementation of the NC loss model, for each label y, ϕ(y) is mapped to a randomly selected unit-length vector of dimension d and fixed during training; the feature map of the original PreActResNet18 is linearly transformed to a d-dimensional vector. The dimension d is chosen as 300 for Cifar10, MNIST and Fashion-Mnist (which have one black-andwhite channel) and 700 for Cifar100 (which has 3 colored channels).
Our implementation of MixUp and Untied MixUp improves upon the published implementation from the original authors of MixUp Zhang et al. (2017b). For example, the original authors’ implementation samples only one λ per mini-batch, giving rise to unnecessarily higher stochasticity of the gradient signal. Our implementation samples λ independently for each sample. Additionally, the original code combines inputs by mixing a mini-batch of samples with a shuffled version of itself. This approach introduces a dependency between sampled pairs and again increases the stochasticity of training. Our implementation creates two shuffled copies of the entire training dataset prior to each epoch, pairs them up, and then splits them into mini-batches. This gives a closer approximation to i.i.d. sampling and makes training smoother. While these implementation improvements have merit on their own, they do not provide a theoretical leap in understanding, and so we do not quantify their impact in our results analysis.
All models examined are trained using mini-batched backpropagation, for 200 epochs.

heading: 6.2 RESULTS
text: We sweep over the policy space of MixUp and Untied MixUp. For MixUp, it is sufficient to consider distribution PMix to be symmetric about 0.5. Thus we consider only consider PMix in the form of B(α, α), and scan through a single parameter α systematically. Since the policy of Untied MixUp is in the form of U(B(α, β)), searching through (α, β) becomes more difficult. Thus our policy search for Untied MixUp is restricted to an ad hoc heuristic search. For this reason, the found best policy for Untied MixUp might be quite far from the true optimal.
The main results of our experiments are given in tables 1 to 4. As shown in the tables, each setting is run 100 times. For each run, we compute the error rate in a run as the average test error rate over the final 10 epochs. The estimated mean (“MEAN”) performance of a setting is computed as the average of the error rates over all runs for the same setting. The 95%-confidence interval (“ConfInt”) for the estimated mean performance is also computed and shown in the table.
From these results, we see that the Untied MixUp schemes each outperform their MixUp counterparts. Specifically, in 6 of the 8 cases (those printed in bold font), the confidence interval of Untied MixUp is completely disjoint from that of the corresponding MixUp scheme; and in some cases, the separation of confidence intervals is by a large margin. Note that the baseline model (PreActResNet18) has been designed with highly focused inductive bias for image classification tasks. Under such an inductive bias, one expects that the room for regularization (or the “amount of overfitting”) isn’t abundant. As such, we consider the improvement of Untied MixUp over MixUp rather significant.
The results show empirically that MixUp and Untied MixUp both work on the NC loss models. This validates our generalization of MixUp (and Untied MixUp) to models built with target linear losses.

heading: 7 CONCLUDING REMARKS
text: This paper establishes a connection between MixUp and adversarial training. This connection allows for a better understanding of the working mechanism of MixUp as well as a generalization of MixUp to a wider family, namely Untied MixUp. Despite the development in this work, it is the authors’ belief that the current designs of MixUp and Untied MixUp are far from optimal. In particular, we believe a better design should allow individualized policy for each training pair. How this can be done remains open at this time.

heading: A APPENDIX
text: A.1 PROOF OF LEMMA 2:
For any fixed infinite sequence (x, x′)∞1 of samples drawn i.i.d. from Q and any infinite sequence of i.i.d. random variables Λ∞1 drawn from P
DAT, let LDAT ((x, x′)n1 ,Λn1 ) be defined according to (4), with the first n elements of (x, x′)∞1 and the first n elements of Λ∞1 as input. Define
δDAT := max (x,x′)∈ D×D sup (λ,λ′)∈
[0,1]×[0,1]
|`DAT(x, x′, λ)− `DAT(x, x′, λ′)|
For any given λn1 ∈ [0, 1]n and any of its modified version un1 ∈ [0, 1]n which differs from λn1 in exactly one location, it can be verified, following the definition of δDAT, that∣∣LDAT ((x, x′)n1 , λn1 )− LDAT ((x, x′)n1 , un1 )∣∣ ≤ δDAT/n Since Λ1,Λ2, . . .ΛK are independent and by McDiarmid Inequality McDiarmid (1989), it follows that for any > 0,
Pr [ LDAT ((x, x′)n1 ,Λn1 )− LDATE ( (x, x′)n1 ) ≥ ] < 2 exp ( − 2 2
n · (δDAT/n)2 ) which proves the lemma
A.2 PROOF OF LEMMA 4:
LuMixE ((x, x′)n1 , γ) := 1
n n∑ k=1 Eλ∼PMix { γ(λ)`DAT(xk, x′k, λ) + γ(λ)` DAT(x′k, xk, λ) }
= 1
n n∑ k=1 ∫ ( γ(λ)PMix(λ)`DAT(xk, x′k, λ) +γ(λ)P Mix(λ)`DAT(x′k, xk, λ) ) dλ
= 1
n n∑ k=1 ∫ ( 1 2 PDAT(λ)`DAT(xk, x′k, λ) + 1 2 PDAT(λ)`DAT(x′k, xk, λ) ) dλ
= 1
2K ( n∑ k=1 ∫ PDAT(λ)`DAT(xk, x′k, λ)dλ+ n∑ k=1 ∫ PDAT(λ)`DAT(x′k, xk, λ)dλ ) (a) = 1
2K ( n∑ k=1 ∫ PDAT(λ)`DAT(xk, x′k, λ)dλ+ n∑ k=1 ∫ PDAT(λ)`DAT(x′k, xk, λ)dλ ) (b) = 1
2K ( n∑ k=1 ∫ PDAT(λ)`DAT(xk, x′k, λ)dλ+ n∑ k=1 ∫ PDAT(λ)`DAT(xk, x′k, λ)dλ )
= 1
n n∑ k=1 ∫ PDAT(λ)`DAT(xk, x′k, λ)dλ
=LDATE ((x, x′)n1 )
where (a) is due to a change of variable in the integration, (b) is due to the symmetry of (x, x′)n1 . Note that in equation 5 g(λ) is undefined at values of λ for which the denominator is zero. But the lemma holds true because the denominator is only zero when p(λ) = 0, so those λ for which g(λ) is undefined never get drawn in the DAT scheme.
A.3 PROOF OF LEMMA 5:
LuMixE ((x, x′)n1 , γ) = 1
n Eλ∼PMix n∑ k=1 ( γ(λ)`DAT(xk, x′k, λ) + γ(λ)` DAT(x′k, xk, λ) )
= 1
n
( Eλ∼PMix
n∑ k=1 γ(λ)`DAT(xk, x′k, λ) + Eλ∼PMix n∑ k=1 γ(λ)`DAT(x′k, xk, λ) ) (a) = 1
n
( Eλ∼PMix
n∑ k=1 γ(λ)`DAT(xk, x′k, λ) + Eλ∼PMix n∑ k=1 γ(λ)`DAT(xk, x′k, λ) ) (b) = 1
n
( Eλ∼PMix
n∑ k=1 γ(λ)`DAT(xk, x′k, λ) + Eλ∼PMix n∑ k=1 γ(λ)`DAT(xk, x′k, λ)
)
= 1
n n∑ k=1 ∫ ( γ(λ)PMix(λ)`DAT(xk, x′k, λ) + γ(λ)P Mix(1− λ)`DAT(xk, x′k, λ) ) dλ
= 1
n n∑ k=1 ∫ `DAT(xk, x′k, λ) ( γ(λ)PMix(λ) + γ(λ)PMix(1− λ) ) ︸ ︷︷ ︸
Du(PMix,γ)
dλ
= 1
n n∑ k=1 Eλ∼PDAT`DAT(xk, x′k, λ)
= LDATE ((x, x′)n1 ) .
where (a) is due to the symmetry of (x, x′)n1 , and (b) is by a change of variable in the second term (renaming 1− λ as λ).

Review summaries:
This paper proposes a novel data augmentation method, untied MixUp (UMixUp), which is a general case of both MixUp and Directional Adversarial Traning (DAT). DAT is referred to in this paper as a scheme that only input feature vectors are mixed, while MixUp also incorporates their corresponding labels. The authors provide a theoretical discussion that both DAT and UMixUp converges to be equivalent to each other when the number of training samples becomes infinity. Experimental results on Cifar 10, Cifar 100, MNIST, and Fashion MNIST show quantitative comparisons among the baseline, MixUp, and UMixUp.

This paper introduces directional adversarial training (DAT) and UMixUP, which are extension methods of MixUp. DAT and UMixUp use the same method of MixUp for generating samples but use different label mixing ratios where DAT retains the sample's original label. In contrast, UMixUp uses a function of the input mixing ratio. This paper shows that UMixUp and DAT are equivalent when the number of samples tends to infinity. In the experiments, UMixUp provides an improvement over MixUp.