MixFeat: Mix Feature in Latent Space Learns Discriminative Space

Deep learning methods perform well in various tasks . However , the over-fitting problem , which causes the performance to decrease for unknown data , remains . We hence propose a method named MixFeat that directly creates latent spaces in a network that can distinguish classes . MixFeat mixes two feature maps in each latent space in the network and uses unmixed labels for learning . We discuss the difference between a method that mixes only features ( MixFeat ) and a method that mixes both features and labels ( mixup and its family ) . Mixing features repeatedly is effective in expanding feature diversity , but mixing labels repeatedly makes learning difficult . MixFeat makes it possible to obtain the advantages of repeated mixing by mixing only features . We report improved results obtained using existing network models with MixFeat on CIFAR-10/100 datasets . In addition , we show that MixFeat effectively reduces the over-fitting problem even when the training dataset is small or contains errors . MixFeat is easy to implement and can be added to various network models without additional computational cost in the inference phase . 1 INTRODUCTION . Deep neural networks ( LeCun et al. , 1998 ) have performed well for various tasks , such as image recognition ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2015 ; He et al. , 2016a ; b ; Han et al. , 2017 ; Huang et al. , 2017 ) , object detection ( Ren et al. , 2015 ; Redmon et al. , 2016 ) , and semantic segmentation ( Chen et al. , 2018 ; Badrinarayanan et al. , 2017 ) . One remaining problem with training deep neural networks is the over-fitting of training data , despite many methods having been proposed to solve this problem ; e.g. , the dropout method ( Srivastava et al. , 2014 ) drops randomly selected elements of feature maps , the mixup method ( Zhang et al. , 2018a ) and between-class learning ( Tokozume et al. , 2018 ) mix pairs of training images and labels , and the manifold mixup method ( Verma et al. , 2018 ) mixes pairs of training feature maps on a randomly selected latent space and labels . We propose a method named MixFeat to make each latent space in the network better distinguish each class . The main contributions of this paper are as follows . • We propose the MixFeat scheme to reduce the over-fitting problem by mixing two feature maps in each latent space in the network so that it can distinguish each class without any additional computational cost in the inference phase . • We conduct extensive experiments to demonstrate the effectiveness of the generalization of MixFeat . MixFeat is easy to implement and can be added to various neural network models . It has the potential to be applied for various tasks , such as object detection , semantic segmentation , and anomaly detection . MixFeat is described in the next section . 2 MIXFEAT . 2.1 OVERVIEW . To avoid overfitting , we consider a method that creates latent spaces in a neural network such that each space can distinguish all classes . Training a network fed with a perturbed sample to output the same inference as when it is fed the pure sample enlarges Fisher ’ s criterion ( Fisher , 1936 ) ( i.e. , the ratio of the between-class distance to the within-class variance ) . It is therefore conceivable for a network to learn the perturbed samples in order to make each latent space able to distinguish the classes . However , a perturbation that is independent of the given examples is inefficient because the latent space is extremely high-dimensional and dynamically changes during learning . We consider that the perturbation should be determined according to the subspace spanned by several samples in the latent space . For simplification , we adopt a partial plane spanned by the origin , the base example , and another example , as shown in Fig 6 ( a ) . In this study , we propose the MixFeat method , which mixes two feature maps in the latent spaces in the network to create latent spaces that can distinguish each class . 2.2 MIXING ONLY FEATURES OR BOTH FEATURES AND LABELS . In related studies , methods that mix two images and labels have recently been proposed . The mixup method ( Zhang et al. , 2018a ) , between-class learning ( BCL ) ( Tokozume et al. , 2018 ) , and manifold mixup method ( Verma et al. , 2018 ) mix two training examples both with respect to features and labels with random weights , and they have been reported to appreciably reduce over-fitting . We argue that mixing methods are able to constrain the feature distribution , which can not be achieved by training without mixing , and the feature space obtained by MixFeat differs greatly from those obtained by the mixup family of methods . Figure 1 shows the assumed class distributions in learned feature spaces . Without mixing , the feature distribution of the mixed features increases and overlaps the feature distributions of classes A and B ( Fig . 1 ( left ) ) . Mixing methods reduce the feature distribution of each class and make it easy to distinguish between classes using the following schemes : • Regression learning : a learning method that associates mixed features with mixed labels ; • Learning against perturbation : a learning method that associates mixed features with unmixed labels . Moreover , we consider repeatedly mixing features in various layers to obtain more diverse features . Learning highly diverse features makes it easier for a latent space to distinguish each class and better suppress over-fitting problems . However , learning to associate repeatedly mixed features with repeatedly mixed labels is intuitively difficult . This behavior was also reported in Tokozume et al . ( 2018 ) . The authors stated that “ the performance when we used only the mixtures of three different classes ( N = 3 ) was worse than that of N = 2 despite the larger variation in training data , ” which indicates that we can not mix repeatedly in the mixup family of methods . In contrast , learning repeatedly perturbed features is easy because the perturbation parameter is not a learning target . Hence , we adopt a method that mixes only features so that repeatedly mixed features can be effectively learned . MixFeat repeatedly mixes features in one training pass , so MixFeat achieves a latent space that can distinguish each class more easily ( Fig . 1 ( right ) ) . 2.3 MIXFEAT ALGORITHM The process of MixFeat is shown in Fig . 2 . Let ⊕ denote the addition operator and ⊙ denote the sample-wise product . The forward training pass , as shown in Fig . 2 ( a ) , is described as Y = X + ( a⊙X + b⊙ F ( X ) ) , ( 1 ) where a = r cos θ , b = r sinθ , r ∼ N ( 0 , σ2 ) , θ ∼ U ( −π , π ) , where the first term X is the input mini-batch tensor , the second term a ⊙ X + b ⊙ F ( X ) is the perturbation , Y denotes the output mini-batch tensors of MixFeat , r is a Gaussian random vector with N ( 0 , σ2 ) , θ is a uniform random vector with U ( −π , π ) , and π is the circular constant . Each element of r or θ is associated with the feature map of each sample in the mini-batch . Furthermore , F ( X ) denotes X randomly sorted along the example axis . An appropriate value for σ is discussed later . The backward training pass , shown in Fig.2 ( b ) , is calculated as GX = GY + ( a⊙GY + F −1 ( b⊙GY ) ) , ( 2 ) where GX and GY respectively denote the partial derivatives of the final output loss function with respect to X and Y , F−1 ( · ) denotes the inverse operation of F ( · ) , which restores the order of examples before the random sorting , and vectors a and b are copies of a and b in the forward training pass ( copy in Fig . 2 ) . During inference , as shown in Fig . 2 ( c ) , the perturbation branches are not necessary for inference and can be removed as follows : Y = X . ( 3 ) Equation ( 3 ) indicates that the MixFeat layer in the inference phase returns the input as is ; i.e. , the MixFeat layer can be simply removed from the inference phase and it thus does not lead to any additional computational cost at this point . 3 EXPERIMENTS . 3.1 CIFAR-10 AND 100 DATASETS . The following experiments were conducted on the CIFAR-10 and -100 datasets ( Krizhevsky & Hinton , 2009 ) . The two CIFAR datasets consist of RGB natural images comprising 32 × 32 pixels . CIFAR-10 consists of images drawn from 10 classes while CIFAR-100 is drawn from 100 classes . The CIFAR-10 and -100 datasets respectively contain 50,000 training images and 10,000 test images . In our experiments , the input images of the CIFAR-10 and -100 datasets were processed adopting the following conventional augmentation process ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2015 ) . The original image of 32× 32 pixels was color-normalized and then horizontally flipped with 50 % probability . It was then zero-padded to a size of 40× 40 pixels and randomly cropped to an image of 32× 32 pixels . All models were trained employing back-propagation and a stochastic gradient descent with Nesterov momentum ( Sutskever et al. , 2013 ) . We adopted the weight initialization introduced by He et al . ( 2015 ) . A single graphics processing unit ( GeForce GTX Titan X or GeForce GTX 1080 Ti ) was used for each training . The initial learning rate was set to 0.05 and decayed by a factor of 0.1 at the half and three-quarter points of the overall training process ( 300 epochs ) , following Huang et al . ( 2017 ) . In addition , we used a weight decay ( Krogh & Hertz , 1992 ) of 5× 10−4 , momentum of 0.9 , and batch size of 128 . When using the proposed MixFeat method , MixFeat is placed directly after each convolution and σ = 0.2 is used unless otherwise specified . We compared the performance of the proposed MixFeat method with two other mixing methods : the mixup method ( Zhang et al. , 2018a ) and manifold mixup method ( Verma et al. , 2018 ) . We trained a ResNet ( pre-activation version ) ( He et al. , 2016b ) , DenseNet , DenseNetBC , ( Huang et al. , 2017 ) , and PyramidNet ( Han et al. , 2017 ) . Following the settings of the original papers , the distribution parameter α = 1.0 was used for the mixup method and α = 2.0 was used for the manifold mixup method . For PyramidNet , the initial learning rate was set to 0.01 and a batch size of 32 was used depending on the memory limitations of the graphics processing unit . We implemented the methods using Chainer v4.4.0 ( Tokui et al. , 2015 ) . Results are given in Table 1 . The results obtained with the mixup method were consistently better than those obtained with the vanilla model , which is a model that does not avoid over-fitting . In addition , the manifold mixup method sometimes achieved better result than original mixup method . Ultimately , the best performance was obtained when MixFeat was adopted for almost all network models . The best results on the CIFAR-10 and -100 datasets were 2.92 % and 16.03 % for the 272- layer PyramidNet . These results demonstrate that MixFeat improves the performance of various network models . The training and test error curves obtained with and without MixFeat are shown in Fig . 3 . The test error rate training with MixFeat shows better convergence than the rate training without MixFeat . In addition , the training and testing error curves for training with MixFeat are closer together than those for training without MixFeat , which demonstrates that MixFeat reduces over-fitting .

Summary: The paper proposes a method MixFeat for regularizing deep neural networks models, aiming at avoiding overfitting in training. The MixFeat interpolates, based on a careful selected mixing ratio, the hidden states (feature maps) of two randomly selected examples. Unlike MixUp, the MixFeat does not interpolate the labels of the two selected examples and the feature interpolation processes are conducted in the hidden space. Experiments on both Cifar10 and Cifar100 show that the networks with MixFeat improve their predictive accuracy as well as outperform networks with Mixup as regularizer.