Bypassing the Random Input Mixing in Mixup

Mixup and its variants have promoted a surge of interest due to their capability of boosting the accuracy of deep models . For a random sample pair , such approaches generate a set of synthetic samples through interpolating both the inputs and their corresponding one-hot labels . Current methods either interpolate random features from an input pair or learn to mix salient features from the pair . Nevertheless , the former methods can create misleading synthetic samples or remove important features from the given inputs , and the latter strategies incur significant computation cost for selecting descriptive input regions . In this paper , we show that the effort needed for the input mixing can be bypassed . For a given sample pair , averaging the features from the two inputs and then assigning it with a set of soft labels can effectively regularize the training . We empirically show that the proposed approach performs on par with state-of-the-art strategies in terms of predictive accuracy . 1 INTRODUCTION . Deep neural networks have demonstrated their profound successes in many challenging real-world applications , including image classification ( Krizhevsky et al. , 2012 ) , speech recognition ( Graves et al. , 2013 ) , and machine translation ( Bahdanau et al. , 2015 ; Sutskever et al. , 2014 ) . One key factor attributing to such successes is the deployment of effective model regularization techniques , which empower the learning to avoid overfitting the training data and to generalize well to unseen samples . This is because current deep models typically embrace high modeling freedom with a very large number of parameters . To this end , many regularizers for deep models have been introduced , including weight decay ( Hanson & Pratt , 1988 ) , dropout ( Srivastava et al. , 2014 ) , stochastic depth ( Huang et al. , 2016 ) , batch normalization ( Ioffe & Szegedy , 2015 ) , and data augmentation schemes ( Cubuk et al. , 2019 ; Hendrycks et al. , 2020 ; Inoue , 2018 ; Lecun et al. , 1998 ; Simard et al. , 1998 ) . Among those effective regularizers , Mixup ( Zhang et al. , 2018 ) is a simple and yet effective , dataaugmentation based regularizer for enhancing the deep classification models . Through linearly interpolating random input pairs and their training targets in one-hot representation , Mixup generates a set of synthetic examples with soft labels to regularize the training . Such pairwise , label-variant data augmentation techniques ( Guo , 2020 ; Guo et al. , 2019 ; Kim et al. , 2020 ; Li et al. , 2020a ; Tokozume et al. , 2018a ; b ; Verma et al. , 2019 ; Yun et al. , 2019 ; Zhang et al. , 2018 ) have attracted a surge of interest and shown their effectiveness on boosting the accuracy of deep networks . Nevertheless , unlike label-preserving data augmentation such as rotation , flip , and crop , there is still limited knowledge about how to design better mixing policies for sample pairs for effective label-variant regularization . Current Mixup-based approaches either mix a pair of inputs using random mixing coefficients ( Guo , 2020 ; Guo et al. , 2019 ; Summers & Dinneen , 2019 ; Tokozume et al. , 2018b ; Verma et al. , 2019 ; Yun et al. , 2019 ; Zhang et al. , 2018 ) or learn to mix salient features from the given pair ( Dabouei et al. , 2020 ; Kim et al. , 2020 ; Li et al. , 2020b ; Walawalkar et al. , 2020 ) to create a set of synthetic samples . Nonetheless , the former methods may create misleading synthetic samples ( Guo et al. , 2019 ) or remove important features from the given inputs ( Kim et al. , 2020 ) and the latter strategies can incur significant computation cost for identifying and selecting the most descriptive input regions ( Kim et al. , 2020 ; Walawalkar et al. , 2020 ) . In this paper , we show that the effort needed for the input mixing with a range of mixing ratios can be bypassed . For a given input pair , one can average the features from the two inputs , and then assign it with a set of soft labels . These soft labels are adaptively learned during training to incorporate class information beyond the provided label pair as well as the evolving states of the training . The method is illustrated in Figure 1 , where the mixed input is the pixel-wise average of features from the two inputs and its training target is the combination of the local soft label , which is the average of the two one-hot targets , and the global soft label , which is generated by the networks during training . We empirically show that the proposed approach performs on par with state-of-the-art methods with random input mixing policy or learning to mixing strategy , in terms of predictive accuracy . We also demonstrate that the synthetic samples created by our method keep tuning the networks long after the training error on the original training set is minimal , encouraging the learning to generate , for each class of the training samples , tight representation clusters . Also , the Class Activation Mapping ( CAM ) ( Zhou et al. , 2016 ) visualization suggests that our method tends to focus on narrower regions of an image for classification . 2 BYPASSING THE INPUT MIXING IN MIXUP . 2.1 MIXUP-BASED DATA AUGMENTATION . For a standard classification setting with a training data set ( X ; Y ) , the objective of the task is to develop a classifier which assigns every input x ∈ X a label y ∈ Y . Instead of using the provided training set ( X ; Y ) , Mixup ( Zhang et al. , 2018 ) generates synthetic samples with soft labels for training . For a pair of random training samples ( xi ; yi ) and ( xj ; yj ) , where x is the input and y the one-hot encoding of the corresponding class , Mixup creates a synthetic sample as follows . x̃ijλ = λx i + ( 1− λ ) xj , ( 1 ) ỹijλ = λy i + ( 1− λ ) yj , ( 2 ) where λ is a scalar mixing policy for mixing both the inputs and the modeling targets of the sample pair . λ is sampled from a Beta ( α , α ) distribution with a hyper-parameter α . The generated samples ( x̃ijλ , ỹ ij λ ) are then fed into the model for training to minimize the cross-entropy loss function . Current variants of Mixup focus on the introduction of a representation function ψ for input mixing : x̃ijλ = ψ ( x i|xj , λ ) + ψ ( xj |xi , 1− λ ) . ( 3 ) The state-of-the-art random input mixing variant CutMix ( Yun et al. , 2019 ) defines the ψ function to form a binary rectangular mask applying to a randomly chosen rectangle covering λ proportion of the input image . PuzzleMix ( Kim et al. , 2020 ) is the state-of-the-art learning to mixing variant . This method defines the ψ function to compute the saliency map of the input pair , find the optimal mask , and optimize the transport plans for generating the mixed example . This is to ensure that the mixed image contain sufficient target class information corresponding to the mixing ratio λ while preserving the local statistics of each input . 2.2 FORMING THE TRAINING INPUT . Similar to Mixup and its variants , our method also creates mixed samples from a random sample pair , but for a given sample pair our method does not generate a set of inputs with different features . Instead , for a range of mixing ratios λs , our approach uses the same mixed input , invariant to the mixing ratios provided . We term our method Label driven Mixup ( denoted as LaMix ) . To this end , the mixed input x̃ij in LaMix has two forms . For the random input mixing methods such as Mixup and CutMix , it is formed by pixel-wise average of the raw features of the input pair ( xi , xj ) : x̃ij = 0.5xi + 0.5xj . ( 4 ) For the learning to mixing methods such as PuzzleMix , the mixed input is formed by pixel-wise average of the saliency features computed with λ value as 0.5 : x̃ij = ψ ( xi|xj , λ = 0.5 ) + ψ ( xj |xi , λ = 0.5 ) . ( 5 ) The newly resulting input x̃ij is then used for training , by passing through a neural network model fϕ ( parameterized with ϕ ) to generate the m-dimensional input embedding Sij ∈ Rm : Sij = fϕ ( x̃ ij ) . ( 6 ) Next , the resulting Sij is then fed into a linear fullyconnected layer Wl ∈ Rc×m to produce the predicted classification distribution over the c classification target classes : y ij = Softmax ( WlSij ) . ( 7 ) The error of the training is computed by comparing the prediction yij and the training target of the input x̃ij , which is discussed next . 2.3 FORMING THE TRAINING TARGET . For the same mixed input x̃ij created from a random input pair ( xi , xj ) , LaMix associates it with various soft target label ÿij , which is learned by the networks with two steps : 1 ) obtaining global soft label and 2 ) integrating with local soft label . In detail , the dynamically assigned soft target label ÿij is a function of ( xi , xj , yi , yj , λ ) , as follows . ÿijλ = τ ( x i , xj , λ , yi , yj ) . ( 8 ) As illustrated in Figure 1 , LaMix implements the τ ( · ) function through adding an additional fullyconnected layer Wt ∈ Rc×m to the original networks . That is , for the given input x̃ij with λ , LaMix first computes its probabilities over the c classification targets , denoted as pijλ : pijλ = Softmax ( σ ( WtS ij ) ) , ( 9 ) where σ denotes the Sigmoid function , and Sij is the same input embedding as that in Equation 7 . In other words , the two predictions ( i.e. , Equations 7 and 9 ) share the same networks except for the last layer . Because the predictions pijλ are about all the c target classes of the given data , such predictions thus reflect the class information beyond the provided local label pair , namely ( yi , yj ) . For description purpose , we term pijλ global soft label . The Sigmoid function here will provide the probability of associating the input to a particular label . After having the global soft label pijλ , we then integrate it with the local label of the given sample pair , i.e. , ỹijλ as described in Equation 2 , as follows : ÿijλ = βỹ ij λ + ( 1− β ) p ij λ , ( 10 ) where β is a scalar coefficient between [ 0,1 ] . Doing so , we enable the synthetic sample ( x̃ij , ÿijλ ) to have information from both the local labels of the provided sample pair ( i.e. , ỹijλ ) and the class information beyond the labels of the given pair ( i.e. , pijλ ) . This process is also illustrated in Figure 1 .

Summary: This paper simply combines mixup and self-distillation to achieve more adaptive soft label, which effectively regularize the training. In the manuscript, authors argue that the existed mixup-based approaches has two mainly efforts, may create misleading training samples or meet computation cost issue on creating samples. Motivated by this, they propose "LaMix", which can leverage the information of self-distillation, to solve those two efforts and achieve competitive performance with SOTA "Puzzle-mixup".