MixUp as Directional Adversarial Training

1 INTRODUCTION . Deep learning applications often require complex networks with a large number of parameters ( He et al. , 2016 ; Zagoruyko & Komodakis , 2016 ; Devlin et al. , 2018 ) . Although neural networks perform so well that their ability to generalize is an area of study in itself ( Zhang et al. , 2017a ; Arpit et al. , 2017 ) , their high complexity nevertheless causes them to overfit their training data ( Kukacka et al. , 2017 ) . For this reason , effective regularization techniques are in high demand . There are two flavors of regularization : complexity curtailing and data augmentation 1 . Complexity curtailing methods constrain models to learning in a subset of parameter space which has a higher probability of generalizing well . Notable examples are weight decay ( Krogh & Hertz , 1991 ) and dropout ( Srivastava et al. , 2014 ) . Data augmentation methods add transformed versions of training samples to the original training set . Conventionally , transformed samples retain their original label , so that models effectively see a larger set of data-label training pairs . Commonly applied transformations in image applications include flips , crops and rotations . A recently devised family of augmentation schemes called adversarial training has attracted active research interest ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ; Miyato et al. , 2016 ; Athalye et al. , 2018 ; Shaham et al. , 2018 ; He et al. , 2018 ) . Adversarial training seeks to reduce a model ’ s propensity to misclassify minimally perturbed training samples , or adversarials . While attack algorithms used for testing model robustness may search for adversarials in unbounded regions of input space , adversarial training schemes generally focus on perturbing training samples within a bounded region , while retaining the sample ’ s original label ( Goodfellow et al. , 2015 ; Shaham et al. , 2018 ) . Another recently proposed data augmentation scheme is MixUp ( Zhang et al. , 2017b ) , in which new samples are generated by mixing pairs of training samples using linear coefficients . Despite its well established generalization performance ( Zhang et al. , 2017b ; Guo et al. , 2018 ; Verma et al. , 2018 ) , the working mechanism of MixUp is not well understood . Guo et al . ( 2018 ) suggest viewing MixUp as imposing local linearity on the model using points outside of the data manifold . While this 1Some authors describe these flavors as “ data independent ” and “ data-dependent ” ( Guo et al. , 2018 ) . perspective is insightful , we do not believe it paints a full picture of how MixUp operates . A recent study ( Lamb et al. , 2019 ) provides empirical evidence that MixUp improves adversarial robustness , but does not present MixUp as a form of adversarial training . We build a framework to understand MixUp in a broader context : we argue that adversarial training is a central working principle of MixUp . To support this contention , we connect MixUp to a MixUplike scheme which does not perform label mixing , and we relate this scheme to adversarial training . Without label mixing , MixUp becomes a conventional augmentation scheme : input samples are moved , but their original labels are retained . Because samples are moved in the direction of other samples – which are typically clustered in input space – we describe this method as ‘ directional ’ . Because this method primarily moves training samples in the direction of adversarial classes , this method is analogous to adversarial training . We thus refer to MixUp without label mixing as directional adversarial training ( DAT ) . We show that MixUp converges to a subset of DAT under mild conditions , and we thereby argue that adversarial training is a working principle of MixUp . Inspired by this new understanding of MixUp as a form of adversarial training , and upon realizing that MixUp is ( asymptotically ) a subset of DAT , we introduce Untied MixUp ( UMixUp ) , a simple enhancement of MixUp which converges to the entire family of DAT schemes , as depicted in Figure 1 . Untied Mixup mixes data-label training pairs in a similar way to MixUp , with the distinction that the label mixing ratio is an arbitrary function of the sample mixing ratio . We perform experiments to show that UMixUp ’ s classification performance improves upon MixUp . In short , this research is motivated by a curiosity to better understand the working of MixUp . In-sodoing we aim to : 1 . Establish DAT as analogous to adversarial training . This is discussed in section 4 . 2 . Establish UMixUp as a superset of MixUp , and as converging to the entire family of DAT schemes . In-so-doing , a ) establish MixUp ’ s convergence to a subset of DAT , and thereby that it operates analogously to adversarial training ; and b ) establish UMixUp as a broader class of MixUp-like schemes that operate analogously to adversarial training . This is discussed in 5 . 3 . Establish empirically that UMixUp ’ s classification performance improves upon MixUp . This is discussed in section 6 . Finally we note that this paper has another contribution . Conventionally , MixUp is only applicable to baseline models that use cross entropy loss . All analytical results we develop in this paper are applicable to a wider family of models using any loss function which we term target-linear . We define target-linearity and experiment with a new loss function called negative cosine-loss to show its potential . Essential proofs of theoretical results are given in the Appendix . 2 PRELIMINARIES . Column vectors are denoted by bold letters such as m , and sets are denoted by calligraphic uppercase letters such asM . The component of a vector is denoted by a bracketed index . For example , m [ i ] denotes the ith component of m. Regular ( non-calligraphic ) capitalized letters such as X will denote random variables , and their lowercase counterparts , e.g. , x , will denote realizations of a random variable . Any sequence , ( a1 , a2 , . . . , an ) will be denoted by an1 . Likewise ( A1 , A2 , . . . , An ) will be denoted by A n 1 , and a sequence of sample pairs ( ( x1 , x′1 ) , ( x2 , x′2 ) , . . . , ( xn , x′n ) ) denoted by ( x , x′ ) n1 . For any value a ∈ [ 0 , 1 ] , we will use a as a short notation for 1− a . Classification Setting Consider a standard classification problem , in which one wishes to learn a classifier that predicts the class label for a sample . Formally , let X be a vector space in which the samples of interest live and let Y be the set of all possible labels associated with these samples . The set of training samples will be denoted by D , a subset of X . We will use t ( x ) to denote the true label of x . Let F be a neural network function , parameterized by θ , which maps X to another vector space Z . Let ϕ : Y → Z be a function that maps a label in Y to an element in Z such that for any y , y′ ∈ Y , if y 6= y′ , then ϕ ( y ) 6= ϕ ( y′ ) . In the space Z , we refer to F ( x ) as the model ’ s prediction . With slight abuse of language , we will occasionally refer to both t ( x ) and ϕ ( t ( x ) ) as the “ label ” of x . Let ` : Z×Z → R be a loss function , using which one defines an overall loss function as L : = 1 |D| ∑ x∈D ` ( F ( x ) , ϕ ( t ( x ) ) ) ( 1 ) Here we have taken the notational convention that the first argument of ` represents the model ’ s prediction and the second represents the target label . In this setting , the learning problem is formulated as minimizing L with respect to its characterizing parameters θ. Target-Linear Loss Functions We say that a loss function ` ( z , z′ ) is target-linear if for any scalars α and β , ` ( z , αz1 + βz2 ) = α ` ( z , z1 ) + β ` ( z , z2 ) Target-linear loss functions arise naturally in many settings , for which we now provide two examples . For convenience , we define the vectors v = F ( x ) and y = ϕ ( t ( x ) ) . Cross-Entropy Loss The conventional cross-entropy loss function , written in our notation , is defined as : ` CE ( F ( x ) , ϕ ( t ( x ) ) ) = ` CE ( v , y ) : = dim ( Z ) ∑ i=1 y [ i ] log v [ i ] where v and y are constrained to being probability vectors . We note that in conventional applications , dim ( Z ) = |Y| , and the target label v is a one-hot vector where y [ i ] = 1 if i = t ( x ) and y [ i ] = 0 otherwise . Constraining v to being a probability vector is achieved using a softmax output layer . Negative-Cosine Loss The “ negative-cosine loss ” , usually used in its negated version , i.e. , as the cosine similarity , can be defined as follows . ` NC ( F ( x ) , ϕ ( t ( x ) ) ) = ` NC ( v , y ) : = −vTy where v and y are constrained to being unit-length vectors . For v this can be achieved by simple division at the output layer , and for y by limiting the range of ϕ to an orthonormal basis ( making it a conventional label embedding function ) . It is clear that the cross-entropy loss ` CE and the negative-cosine loss ` NC are both target-linear , directly following from the definition of target-linearity . Assumptions The theoretical development of this paper relies on two fundamental assumptions , which we call “ axioms ” . Axiom 1 ( Target linearity ) The loss function ` used for the classification setting is target-linear . That is , the study of MixUp in this paper is in fact goes beyond the standard MixUp , which uses the cross-entropy loss . Much of the development in this paper concerns drawing sample pairs ( x , x′ ) from D×D . Suppose that ( x , x′ ) n1 is a length-n sequence of sample pairs drawn from D ×D . A sequence ( x , x′ ) n1 is said to be symmetric if for every ( a , b ) ∈ D ×D , the number of occurrences of ( a , b ) in the sequence is equal to that of ( b , a ) . A distribution Q on D ×D will be called exchangeable , or symmetric , if for any ( x , x′ ) ∈ D ×D , Q ( ( x , x′ ) ) = Q ( ( x′ , x ) ) . Axiom 2 ( Symmetric pair-sampling distribution ) Whenever a sample pair ( x , x′ ) is drawn from a distribution Q , Q is assumed to be symmetric . In the standard MixUp , two samples are drawn independently from D to form a pair , making this condition satisfied .

Summary: This paper introduces directional adversarial training (DAT) and UMixUP, which are extension methods of MixUp. DAT and UMixUp use the same method of MixUp for generating samples but use different label mixing ratios where DAT retains the sample's original label. In contrast, UMixUp uses a function of the input mixing ratio. This paper shows that UMixUp and DAT are equivalent when the number of samples tends to infinity. In the experiments, UMixUp provides an improvement over MixUp.