Co-Mixup: Saliency Guided Joint Mixup with Supermodular Diversity

1 Introduction . Deep neural networks have been applied to a wide range of artificial intelligence tasks such as computer vision , natural language processing , and signal processing with remarkable performance ( Ren et al. , 2015 ; Devlin et al. , 2018 ; Oord et al. , 2016 ) . However , it has been shown that neural networks have excessive representation capability and can even fit random data ( Zhang et al. , 2016 ) . Due to these characteristics , the neural networks can easily overfit to training data and show a large generalization gap when tested on previously unseen data . To improve the generalization performance of the neural networks , a body of research has been proposed to develop regularizers based on priors or to augment the training data with task-dependent transforms ( Bishop , 2006 ; Cubuk et al. , 2019 ) . Recently , a new taskindependent data augmentation technique , called mixup , has been proposed ( Zhang et al. , 2018 ) . The original mixup , called Input Mixup , linearly interpolates a given pair of input data and can be easily applied to various data and tasks , improving the generalization performance and robustness of neural networks . Other mixup methods , such as manifold mixup ( Verma et al. , 2019 ) or CutMix ( Yun et al. , 2019 ) , have also been proposed addressing different ways to mix a given pair of input data . Puzzle Mix ( Kim et al. , 2020 ) utilizes saliency information and local statistics to ensure mixup data to have rich supervisory signals . However , these approaches only consider mixing a given random pair of input data and do not fully utilize the rich informative supervisory signal in training data including collection of object saliency , relative arrangement , etc . In this work , we simultaneously consider mixmatching different salient regions among all input data so that each generated mixup example accumulates as many salient regions from multiple input data as possible while ensuring Correspondence to : Hyun Oh Song . diversity among the generated mixup examples . To this end , we propose a novel optimization problem that maximizes the saliency measure of each individual mixup example while encouraging diversity among them collectively . This formulation results in a novel discrete submodular-supermodular objective . We also propose a practical modular approximation method for the supermodular term and present an efficient iterative submodular minimization algorithm suitable for minibatch-based mixup for neural network training . As illustrated in the Figure 1 , while the proposed method , Co-Mixup , mix-matches the collection of salient regions utilizing inter-arrangements among input data , the existing methods do not consider the saliency information ( Input Mixup & CutMix ) or disassemble salient parts ( Puzzle Mix ) . We verify the performance of the proposed method by training classifiers on CIFAR-100 , Tiny-ImageNet , ImageNet , and the Google commands dataset ( Krizhevsky et al. , 2009 ; Chrabaszcz et al. , 2017 ; Deng et al. , 2009 ; Warden , 2017 ) . Our experiments show the models trained with Co-Mixup achieve the state of the performance compared to other mixup baselines . In addition to the generalization experiment , we conduct weakly-supervised object localization and robustness tasks and confirm Co-Mixup outperforms other mixup baselines . 2 Related works . Mixup Data augmentation has been widely used to prevent deep neural networks from over-fitting to the training data ( Bishop , 1995 ) . The majority of conventional augmentation methods generate new data by applying transformations depending on the data type or the target task ( Cubuk et al. , 2019 ) . Zhang et al . ( 2018 ) proposed mixup , which can be independently applied to various data types and tasks , and improves generalization and robustness of deep neural networks . Input mixup ( Zhang et al. , 2018 ) linearly interpolates between two input data and utilizes the mixed data with the corresponding soft label for training . Following this work , manifold mixup ( Verma et al. , 2019 ) applies the mixup in the hidden feature space , and CutMix ( Yun et al. , 2019 ) suggests a spatial copy and paste based mixup strategy on images . Guo et al . ( 2019 ) trains an additional neural network to optimize a mixing ratio . Puzzle Mix ( Kim et al. , 2020 ) proposes a mixup method based on saliency and local statistics of the given data . In this paper , we propose a discrete optimization-based mixup method simultaneously finding the best combination of collections of salient regions among all input data while encouraging diversity among the generated mixup examples . Saliency The seminal work from Simonyan et al . ( 2013 ) generates a saliency map using a pre-trained neural network classifier without any additional training of the network . Following the work , measuring the saliency of data using neural networks has been studied to obtain a more precise saliency map ( Zhao et al. , 2015 ; Wang et al. , 2015 ) or to reduce the saliency computation cost ( Zhou et al. , 2016 ; Selvaraju et al. , 2017 ) . The saliency information is widely applied to the tasks in various domains , such as object segmentation or speech recognition ( Jung and Kim , 2011 ; Kalinli and Narayanan , 2007 ) . Submodular-Supermodular optimization A submodular ( supermodular ) function is a set function with diminishing ( increasing ) returns property ( Narasimhan and Bilmes , 2005 ) . It is known that any set function can be expressed as the sum of a submodular and supermodular function ( Lovász , 1983 ) , called BP function . Various problems in machine learning can be naturally formulated as BP functions ( Fujishige , 2005 ) , but it is known to be NP-hard ( Lovász , 1983 ) . Therefore , approximate algorithms based on modular approximations of submodular or supermodular terms have been developed ( Iyer and Bilmes , 2012 ) . Our formulation falls into a category of BP function consisting of smoothness function within a mixed output ( submodular ) and a diversity function among the mixup outputs ( supermodular ) . 3 Preliminary . Existing mixup methods return { h ( x1 , xi ( 1 ) ) , . . . , h ( xm , xi ( m ) ) } for given input data { x1 , . . . , xm } , where h : X × X → X is a mixup function and ( i ( 1 ) , . . . , i ( m ) ) is a random permutation of the data indices . In the case of input mixup , h ( x , x′ ) is λx+ ( 1− λ ) x′ , where λ ∈ [ 0 , 1 ] is a random mixing ratio . Manifold mixup applies input mixup in the hidden feature space , and CutMix uses h ( x , x′ ) = 1B x + ( 1 − 1B ) x′ , where 1B is a binary rectangular-shape mask for an image x and represents the element-wise product . Puzzle Mix defines h ( x , x′ ) as z Πᵀx + ( 1 − z ) Π′ᵀx′ , where Π is a transport plan and z is a discrete mask . In detail , for x ∈ Rn , Π ∈ { 0 , 1 } n and z ∈ Ln for L = { lL | l = 0 , 1 , . . . , L } . In this work , we extend the existing mixup functions as h : Xm → Xm′ which performs mixup on a collection of input data and returns another collection . Let xB ∈ Rm×n denote the batch of input data in matrix form . Then , our proposed mixup function is h ( xB ) = ( g ( z1 xB ) , . . . , g ( zm′ xB ) ) , where zj ∈ Lm×n for j = 1 , . . . , m′ with L = { lL | l = 0 , 1 , . . . , L } and g : R m×n → Rn returns a column-wise sum of a given matrix . Note that , the kth column of zj , denoted as zj , k ∈ Lm , can be interpreted as the mixing ratio among m inputs at the kth location . Also , we enforce ‖zj , k‖1 = 1 to maintain the overall statistics of the given input batch . Given the one-hot target labels yB ∈ { 0 , 1 } m×C of the input data with C classes , we generate soft target labels for mixup data as yᵀB õj for j = 1 , . . . , m ′ , where õj = 1n ∑n k=1 zj , k ∈ [ 0 , 1 ] m represents the input source ratio of the jth mixup data . We train models to estimate the soft target labels by minimizing the cross-entropy loss . 4 Method . 4.1 Objective . Saliency Our main objective is to maximize the saliency measure of mixup data while maintaining the local smoothness of data , i.e. , spatially nearby patches in a natural image look similar , temporally adjacent signals have similar spectrum in speech , etc . ( Kim et al. , 2020 ) . As we can see from CutMix in Figure 1 , disregarding saliency can give a misleading supervisory signal by generating mixup data that does not match with the target soft label . While the existing mixup methods only consider the mixup between two inputs , we generalize the number of inputs m to any positive integer . Note , each kth location of outputs has m candidate sources from the inputs . We model the unary labeling cost as the negative value of the saliency , and denote the cost vector at the kth location as ck ∈ Rm . For the saliency measure , we calculate the gradient values of training loss with respect to the input and measure ` 2 norm of the gradient values across input channels ( Simonyan et al. , 2013 ; Kim et al. , 2020 ) . Note that this method does not require any additional architecture dependent modules for saliency calculation . In addition to the unary cost , we encourage adjacent locations to have similar labels for the smoothness of each mixup data . In summary , the objective can be formulated as follows : m′∑ j=1 n∑ k=1 cᵀkzj , k + β m′∑ j=1 ∑ ( k , k′ ) ∈N ( 1− zᵀj , kzj , k′ ) − η m′∑ j=1 n∑ k=1 log p ( zj , k ) , where the prior p is given by zj , k ∼ 1LMulti ( L , λ ) with λ = ( λ1 , . . . , λm ) ∼ Dirichlet ( α , . . . , α ) , which is a generalization of the mixing ratio distribution of Zhang et al . ( 2018 ) , and N denotes a set of adjacent locations ( i.e. , neighboring image patches in vision , subsequent spectrums in speech , etc. ) . Diversity Note that the naive generalization above leads to the identical outputs because the objective is separable and identical for each output . In order to obtain diverse mixup outputs , we model a similarity penalty between outputs . First , we represent the input source information of the jth output by aggregating assigned labels as ∑n k=1 zj , k . For simplicity , let us denote ∑n k=1 zj , k as oj . Then , we measure the similarity between oj ’ s by using the inner-product on Rm . In addition to the input source similarity between outputs , we model the compatibility between input sources , represented as a symmetric matrix Ac ∈ Rm×m+ . Specifically , Ac [ i1 , i2 ] quantifies the degree to which input i1 and i2 are suitable to be mixed together . In summary , we use inner-product on A = ( 1 − ω ) I + ωAc for ω ∈ [ 0 , 1 ] , resulting in a supermodular penalty term . Note that , by minimizing 〈oj , oj′〉A = oᵀjAoj′ , ∀j 6= j′ , we penalize output mixup examples with similar input sources and encourage each individual mixup examples to have high compatibility within . In this work , we measure the distance between locations of salient objects in each input and use the distance matrix Ac [ i , j ] = ‖argmaxksi [ k ] − argmaxksj [ k ] ‖1 , where si is the saliency map of the ith input and k is a location index ( e.g. , k is a 2-D index for image data ) . From now on , we denote this inner-product term as the compatibility term . Over-penalization The conventional mixup methods perform mixup as many as the number of examples in a given mini-batch . In our setting , this is the case when m = m′ . However , the compatibility penalty between outputs is influenced by the pigeonhole principle . For example , suppose the first output consists of two inputs . Then , the inputs must be used again for the remaining m′ − 1 outputs , or only m− 2 inputs can be used . In the latter case , the number of available inputs ( m− 2 ) is less than the outputs ( m′ − 1 ) , and thus , the same input must be used more than twice . Empirically , we found that the remaining compatibility term above over-penalizes the optimization so that a substantial portion of outputs are returned as singletons without any mixup . To mitigate the over-penalization issue , we apply clipping to the compatibility penalty term . Specifically , we model the objective so that no extra penalty would occur when the compatibility among outputs is below a certain level . Now we present our main objective as following : z∗ = argmin zj , k∈Lm , ‖zj , k‖1=1 f ( z ) , where f ( z ) : = m′∑ j=1 n∑ k=1 cᵀkzj , k + β m′∑ j=1 ∑ ( k , k′ ) ∈N ( 1− zᵀj , kzj , k′ ) ( 1 ) + γmax τ , m′∑ j=1 m′∑ j′ 6=j ( n∑ k=1 zj , k ) ᵀ A ( n∑ k=1 zj′ , k ) ︸ ︷︷ ︸ =fc ( z ) −η m′∑ j=1 n∑ k=1 log p ( zj , k ) . In Figure 2 , we describe the properties of the BP optimization problem of Equation ( 1 ) and statistics of the resulting mixup data . Next , we verify the supermodularity of the compatibility term . We first extend the definition of the submodularity of a multi-label function as follows ( Windheuser et al. , 2012 ) . Definition 1 . For a given label set L , a function s : Lm × Lm → R is pairwise submodular , if ∀x , x′ ∈ Lm , s ( x , x ) +s ( x′ , x′ ) ≤ s ( x , x′ ) +s ( x′ , x ) . A function s is pairwise supermodular , if −s is pairwise submodular . Proposition 1 . The compatibility term fc in Equation ( 1 ) is pairwise supermodular for every pair of ( zj1 , k , zj2 , k ) if A is positive semi-definite . Proof . See Appendix B.1 . Finally note that , A = ( 1 − ω ) I + ωAc , where Ac is a symmetric matrix . By using spectral decomposition , Ac can be represented as UDUᵀ , where D is a diagonal matrix and UᵀU = UUᵀ = I . Then , A = U ( ( 1− ω ) I + ωD ) Uᵀ , and thus for small ω > 0 , we can guarantee A to be positive semi-definite .

Summary: This paper proposes a new mixup method that encourages diversity among the samples mixed from a minibatch of data in addition to saliency of each mixed sample. The authors formulate two objectives: 1. a BP set function (submodular + supermodular), and 2. a submodular relaxation obtained by modularizing the supermodular component. Then they solve this problem approximately with coordinate descent by modularizing wrt the update coordinate at each step. This approach outperforms mixup baselines on image classification and several other tasks (calibration, object localization, and robustness).