Fast adversarial training for semi-supervised learning

In semi-supervised learning , Bad GAN approach is one of the most attractive method due to the intuitional simplicity and powerful performances . Bad GAN learns a classifier with bad samples distributed on complement of the support of the input data . But Bad GAN needs additional architectures , a generator and a density estimation model , which involves huge computation and memory consumption cost . VAT is another good semi-supervised learning algorithm , which utilizes unlabeled data to improve the invariance of the classifier with respect to perturbation of inputs . In this study , we propose a new method by combining the ideas of Bad GAN and VAT . The proposed method generates bad samples of high-quality by use of the adversarial training used in VAT . We give theoretical explanations why the adversarial training is good at both generating bad samples and semi-supervised learning . An advantage of the proposed method is to achieve the competitive performances with much fewer computations . We demonstrate advantages our method by various experiments with well known benchmark image datasets . 1 INTRODUCTION . Deep learning has accomplished unprecedented success due to the development of deep architectures , learning techniques and hardwares ( Krizhevsky et al. , 2012 ; Ioffe & Szegedy , 2015 ; Szegedy et al. , 2015 ; Hinton et al. , 2012 ; Kingma & Ba , 2014 ) . However , deep learning has also suffered from collecting large amount of labeled data which requires both cost and time . Thus it becomes important to develop semi-supervised methodologies that learn a classifier ( or discriminator ) by using small labeled data and large unlabeled data . Various semi-supervised learning methods have been proposed for deep learning . Weston et al . ( 2012 ) employs a manifold embedding technique using the pre-constructed graph of unlabeled data and Rasmus et al . ( 2015 ) uses a specially designed auto-encoder to extract essential features for classification . Variational auto encoder ( Kingma & Welling , 2013 ) is also used in the context of semi-supervised learning by maximizing the variational lower bound of both labeled and unlabeled data ( Kingma et al. , 2014 ; Maaløe et al. , 2016 ) . Recently , semi-supervised learning based on generative adversarial networks ( GAN , Goodfellow et al . ( 2014a ) ) has received much attention . For K-class classification problems , Salimans et al . ( 2016 ) solves the ( K + 1 ) -class classification problem where the additional ( K + 1 ) th class consists of synthetic images made by a generator of the GAN learned by unlabeled data . Dai et al . ( 2017 ) notices that not a good generator but a bad generator which generates synthetic images much different from observed images is crucial for the success of semi-supervised learning . Dai et al . ( 2017 ) gives theoretical justifications of using a bad generator and develops a semi-supervised learning algorithm called Bad GAN which achieves the state-of-the-art performances over multiple benchmark datasets . However , Bad GAN has several limitations . It needs two additional deep architectures - bad generator and pre-trained density estimation model besides the one for the classifier . Learning these multiple deep architectures requires huge computation and memory consumption . In particular , the PixelCNN++ ( Salimans et al. , 2017 ) is used for the pre-trained density estimation model which needs very large computational resources . Another difficulty in Bad GAN is that it requires a two-step learning procedure - the first step is to learn the PixelCNN++ model and the second step is to learn the classifier and the bad generator . The optimal learning of the first step may not be optimal for the second step and hence the regularization of the both steps would need special techniques . In this study , we propose a new semi-supervised learning method which competes well with other stateof-the-art semi-supervised learning algorithms and yet needs much smaller amount of computational resources . In particular , the proposed method employs only one deep architecture and hence the corresponding learning phase is much easier and faster . Our proposed method is motivated by close investigation of VAT ( Virtual Adversarial Training ) method ( Miyato et al. , 2015 ; 2017 ) . VAT tries to find a deep classifier which has a good prediction accuracy on training data and at the same time is less sensitive to data perturbation toward the adversarial direction . Here , the adversarial direction for a given datum is the direction to which the probabilities of each class change most . In Section 3 , we prove that the perturbed data toward their adversarial directions can serve as ‘ good ’ bad samples . By using the adversarial directions for both measuring the invariance and generating the bad samples , the proposed method combines the advantages of Bad GAN and VAT together . Note that only a deep architecture for classification is needed to calculate the adversarial directions and thus the corresponding learning procedure is cheaper , easier and faster . We call our proposed method FAT ( Fast Adversarial Training ) . Dai et al . ( 2017 ) proves that bad samples play a role to pull the decision boundary toward the low density regions of data . In Section 5 , we give a theoretical explanation that VAT pushes the decision boundary away from the high density regions of data . That is , FAT accelerates the learning procedure by using both pushing and pulling operations simultaneously . In section 6 , we show that FAT achieves almost the state-of-the-art performances with much fewer training epochs . Especially , for the MNIST dataset , FAT achieves similar test accuracies to those of Bad GAN and VAT with 5 times and 7 times fewer training epochs , respectively . This paper is organized as follows . In Section 2 , we review the Bad GAN and VAT methods briefly . In Section 3 , the technique to generate bad samples using the adversarial directions is described , and our proposed semi-supervised learning method is presented in Section 4 . Theoretical analysis of VAT is given in Section 5 . Results of various experiments are presented in Section 6 and conclusions follow in Section 7 . 2 Bad GAN AND VAT 2.1 Bad GAN APPROACH Bad GAN is a method that trains a good discriminator with a bad generator . This procedure trains a generator as well as a discriminator simultaneously . Let DG ( φ ) be generated bad samples with a bad generator pG ( · ; φ ) parametrized by φ . Here , the ‘ bad generator ’ is a deep architecture to generate samples different from observed data . Let ppt ( · ) be a pre-trained density estimation model . For a given discriminator with a feature vector v ( x ; θ ) of a given input x parameterized by θ , Bad GAN learns the bad generator by minimizing the following : Ex∼DG ( φ ) [ log ppt ( x ) I ( ppt ( x ) > τ ) ] + ||Ex∼Utrv ( x ; θ̂ ) − Ex∼DG ( φ ) v ( x ; θ̂ ) || 2 with respect to φ , where τ > 0 is a tuning parameter , U tr is the unlabeled data , and θ̂ is the current estimate of θ and ‖ · ‖ is the Euclidean norm . In turn , to train the discriminator , we consider the K-class classification problem as the ( K+1 ) -class classification problem where the ( K + 1 ) -th class is an artificial label of the bad samples generated by the bad generator . We estimate the parameter θ in the discriminator by minimizing the following : −Ex , y∼Ltr [ log p ( y|x , y ≤ K ; θ ) ] − Ex∼Utr [ log { K∑ k=1 p ( k|x ; θ ) } ] −Ex∼DG ( φ ) [ log p ( K + 1|x ; θ ) ] − Ex∼Utr [ K∑ k=1 p ( k|x ; θ ) log p ( k|x ; θ ) ] ( 1 ) for given φ , where Ltr is the labeled set . The second and the third terms in ( 1 ) are the cross-entropies between the unlabeled and the bad samples . The fourth term is similar the entropy of the unlabeled data which is usually helpful for semi-supervised learning ( Grandvalet & Bengio , 2005 ) . See Dai et al . ( 2017 ) for details of the objective function ( 1 ) . 2.2 VAT APPROACH VAT is a regularization method which is inspired by the adversarial training ( Goodfellow et al. , 2014b ) . The regularization term of VAT is given as : LVAT ( θ ; θ̂ , x , ) = DKL ( p ( ·|x ; θ̂ ) ||p ( ·|x+ radvr ( x , ) ; θ ) ) = − K∑ k=1 p ( k|x ; θ̂ ) log p ( k|x+ radvr ( x , ) ; θ ) + C , where radvr ( x , ) = argmax r ; ||r||≤ DKL ( p ( ·|x ; θ̂ ) ||p ( ·|x+ r ; θ̂ ) ) , ( 2 ) > 0 is a tuning parameter , θ is the parameter in the discriminator to train , θ̂ is the current estimate of θ and C is a constant . Combining with the cross-entropy term of the labeled data , we get the final objective function of VAT : −Ex , y∼Ltr [ log p ( y|x ; θ ) ] + Ex∼Utr [ LVAT ( θ ; θ̂ , x , ) ] . ( 3 ) 3 GENERATION OF BAD SAMPLES BY ADVERSARIAL TRAINING . The key role of bad samples in Bad GAN is to enforce the decision boundary to be pulled toward the low density regions of the unlabeled data . To do this , bad samples must be located at the valleys of the distribution of the unlabeled data . In this section , we propose a novel technique to generate ‘ good ’ bad samples by use of only a given classifier . 3.1 MOTIVATION . In this subsection , we explain why the adversarial direction is toward the decision boundary . For simplicity , we only consider the linear decision boundary . For the decision boundary made by the DNN model with ReLU activation function , see Appendix A.2 . Let us consider the 2-class linear logistic regression model parametrized by η = { w , b } , that is , p ( y = 1|x ; η ) = ( 1 + exp ( −b− w′x ) ) −1 . Note that the decision boundary is { x : b+ w′x = 0 } , and for any given x , the distance between x and the decision boundary is |b+ w′x|/||w|| . The key result is that moving x toward the adversarial direction radvr ( x , ) is equivalent to moving x toward the decision boundary which is stated rigorously in the following proposition . The proof is in the appendix . Proposition 1 For a sufficiently small > 0 , we have sign ( w ′ x+ b ) · sign ( w ′ radvr ( x , ) ) = −1 . ( 4 ) Proposition 1 implies that |b + w′x|/||w|| > |b + w′ ( x + radvr ( x , ) |/||w|| unless |b + w′x| = 0 . Hence , we can treat x+ δradvr ( x , ) /||radvr ( x , ) || for appropriately choosing δ > 0 as a bad sample ( a sample closer to the decision boundary ) .

Summary: The paper proposes to use the technique in VAT to generate adversarial complementary examples in the (K+1)-class semi-supervised learning framework described by the Bad GAN paper. This leads to a formulation that combines the VAT loss and the (K+1)-class classification loss. The paper also provides analysis regarding why VAT is useful for semi-supervised learning.