Fast adversarial training for semi-supervised learning

In semi-supervised learning , Bad GAN approach is one of the most attractive method due to the intuitional simplicity and powerful performances . Bad GAN learns a classifier with bad samples distributed on complement of the support of the input data . But Bad GAN needs additional architectures , a generator and a density estimation model , which involves huge computation and memory consumption cost . VAT is another good semi-supervised learning algorithm , which utilizes unlabeled data to improve the invariance of the classifier with respect to perturbation of inputs . In this study , we propose a new method by combining the ideas of Bad GAN and VAT . The proposed method generates bad samples of high-quality by use of the adversarial training used in VAT . We give theoretical explanations why the adversarial training is good at both generating bad samples and semi-supervised learning . An advantage of the proposed method is to achieve the competitive performances with much fewer computations . We demonstrate advantages our method by various experiments with well known benchmark image datasets . 1 INTRODUCTION . Deep learning has accomplished unprecedented success due to the development of deep architectures , learning techniques and hardwares ( Krizhevsky et al. , 2012 ; Ioffe & Szegedy , 2015 ; Szegedy et al. , 2015 ; Hinton et al. , 2012 ; Kingma & Ba , 2014 ) . However , deep learning has also suffered from collecting large amount of labeled data which requires both cost and time . Thus it becomes important to develop semi-supervised methodologies that learn a classifier ( or discriminator ) by using small labeled data and large unlabeled data . Various semi-supervised learning methods have been proposed for deep learning . Weston et al . ( 2012 ) employs a manifold embedding technique using the pre-constructed graph of unlabeled data and Rasmus et al . ( 2015 ) uses a specially designed auto-encoder to extract essential features for classification . Variational auto encoder ( Kingma & Welling , 2013 ) is also used in the context of semi-supervised learning by maximizing the variational lower bound of both labeled and unlabeled data ( Kingma et al. , 2014 ; Maaløe et al. , 2016 ) . Recently , semi-supervised learning based on generative adversarial networks ( GAN , Goodfellow et al . ( 2014a ) ) has received much attention . For K-class classification problems , Salimans et al . ( 2016 ) solves the ( K + 1 ) -class classification problem where the additional ( K + 1 ) th class consists of synthetic images made by a generator of the GAN learned by unlabeled data . Dai et al . ( 2017 ) notices that not a good generator but a bad generator which generates synthetic images much different from observed images is crucial for the success of semi-supervised learning . Dai et al . ( 2017 ) gives theoretical justifications of using a bad generator and develops a semi-supervised learning algorithm called Bad GAN which achieves the state-of-the-art performances over multiple benchmark datasets . However , Bad GAN has several limitations . It needs two additional deep architectures - bad generator and pre-trained density estimation model besides the one for the classifier . Learning these multiple deep architectures requires huge computation and memory consumption . In particular , the PixelCNN++ ( Salimans et al. , 2017 ) is used for the pre-trained density estimation model which needs very large computational resources . Another difficulty in Bad GAN is that it requires a two-step learning procedure - the first step is to learn the PixelCNN++ model and the second step is to learn the classifier and the bad generator . The optimal learning of the first step may not be optimal for the second step and hence the regularization of the both steps would need special techniques . In this study , we propose a new semi-supervised learning method which competes well with other stateof-the-art semi-supervised learning algorithms and yet needs much smaller amount of computational resources . In particular , the proposed method employs only one deep architecture and hence the corresponding learning phase is much easier and faster . Our proposed method is motivated by close investigation of VAT ( Virtual Adversarial Training ) method ( Miyato et al. , 2015 ; 2017 ) . VAT tries to find a deep classifier which has a good prediction accuracy on training data and at the same time is less sensitive to data perturbation toward the adversarial direction . Here , the adversarial direction for a given datum is the direction to which the probabilities of each class change most . In Section 3 , we prove that the perturbed data toward their adversarial directions can serve as ‘ good ’ bad samples . By using the adversarial directions for both measuring the invariance and generating the bad samples , the proposed method combines the advantages of Bad GAN and VAT together . Note that only a deep architecture for classification is needed to calculate the adversarial directions and thus the corresponding learning procedure is cheaper , easier and faster . We call our proposed method FAT ( Fast Adversarial Training ) . Dai et al . ( 2017 ) proves that bad samples play a role to pull the decision boundary toward the low density regions of data . In Section 5 , we give a theoretical explanation that VAT pushes the decision boundary away from the high density regions of data . That is , FAT accelerates the learning procedure by using both pushing and pulling operations simultaneously . In section 6 , we show that FAT achieves almost the state-of-the-art performances with much fewer training epochs . Especially , for the MNIST dataset , FAT achieves similar test accuracies to those of Bad GAN and VAT with 5 times and 7 times fewer training epochs , respectively . This paper is organized as follows . In Section 2 , we review the Bad GAN and VAT methods briefly . In Section 3 , the technique to generate bad samples using the adversarial directions is described , and our proposed semi-supervised learning method is presented in Section 4 . Theoretical analysis of VAT is given in Section 5 . Results of various experiments are presented in Section 6 and conclusions follow in Section 7 . 2 Bad GAN AND VAT 2.1 Bad GAN APPROACH Bad GAN is a method that trains a good discriminator with a bad generator . This procedure trains a generator as well as a discriminator simultaneously . Let DG ( φ ) be generated bad samples with a bad generator pG ( · ; φ ) parametrized by φ . Here , the ‘ bad generator ’ is a deep architecture to generate samples different from observed data . Let ppt ( · ) be a pre-trained density estimation model . For a given discriminator with a feature vector v ( x ; θ ) of a given input x parameterized by θ , Bad GAN learns the bad generator by minimizing the following : Ex∼DG ( φ ) [ log ppt ( x ) I ( ppt ( x ) > τ ) ] + ||Ex∼Utrv ( x ; θ̂ ) − Ex∼DG ( φ ) v ( x ; θ̂ ) || 2 with respect to φ , where τ > 0 is a tuning parameter , U tr is the unlabeled data , and θ̂ is the current estimate of θ and ‖ · ‖ is the Euclidean norm . In turn , to train the discriminator , we consider the K-class classification problem as the ( K+1 ) -class classification problem where the ( K + 1 ) -th class is an artificial label of the bad samples generated by the bad generator . We estimate the parameter θ in the discriminator by minimizing the following : −Ex , y∼Ltr [ log p ( y|x , y ≤ K ; θ ) ] − Ex∼Utr [ log { K∑ k=1 p ( k|x ; θ ) } ] −Ex∼DG ( φ ) [ log p ( K + 1|x ; θ ) ] − Ex∼Utr [ K∑ k=1 p ( k|x ; θ ) log p ( k|x ; θ ) ] ( 1 ) for given φ , where Ltr is the labeled set . The second and the third terms in ( 1 ) are the cross-entropies between the unlabeled and the bad samples . The fourth term is similar the entropy of the unlabeled data which is usually helpful for semi-supervised learning ( Grandvalet & Bengio , 2005 ) . See Dai et al . ( 2017 ) for details of the objective function ( 1 ) . 2.2 VAT APPROACH VAT is a regularization method which is inspired by the adversarial training ( Goodfellow et al. , 2014b ) . The regularization term of VAT is given as : LVAT ( θ ; θ̂ , x , ) = DKL ( p ( ·|x ; θ̂ ) ||p ( ·|x+ radvr ( x , ) ; θ ) ) = − K∑ k=1 p ( k|x ; θ̂ ) log p ( k|x+ radvr ( x , ) ; θ ) + C , where radvr ( x , ) = argmax r ; ||r||≤ DKL ( p ( ·|x ; θ̂ ) ||p ( ·|x+ r ; θ̂ ) ) , ( 2 ) > 0 is a tuning parameter , θ is the parameter in the discriminator to train , θ̂ is the current estimate of θ and C is a constant . Combining with the cross-entropy term of the labeled data , we get the final objective function of VAT : −Ex , y∼Ltr [ log p ( y|x ; θ ) ] + Ex∼Utr [ LVAT ( θ ; θ̂ , x , ) ] . ( 3 ) 3 GENERATION OF BAD SAMPLES BY ADVERSARIAL TRAINING . The key role of bad samples in Bad GAN is to enforce the decision boundary to be pulled toward the low density regions of the unlabeled data . To do this , bad samples must be located at the valleys of the distribution of the unlabeled data . In this section , we propose a novel technique to generate ‘ good ’ bad samples by use of only a given classifier . 3.1 MOTIVATION . In this subsection , we explain why the adversarial direction is toward the decision boundary . For simplicity , we only consider the linear decision boundary . For the decision boundary made by the DNN model with ReLU activation function , see Appendix A.2 . Let us consider the 2-class linear logistic regression model parametrized by η = { w , b } , that is , p ( y = 1|x ; η ) = ( 1 + exp ( −b− w′x ) ) −1 . Note that the decision boundary is { x : b+ w′x = 0 } , and for any given x , the distance between x and the decision boundary is |b+ w′x|/||w|| . The key result is that moving x toward the adversarial direction radvr ( x , ) is equivalent to moving x toward the decision boundary which is stated rigorously in the following proposition . The proof is in the appendix . Proposition 1 For a sufficiently small > 0 , we have sign ( w ′ x+ b ) · sign ( w ′ radvr ( x , ) ) = −1 . ( 4 ) Proposition 1 implies that |b + w′x|/||w|| > |b + w′ ( x + radvr ( x , ) |/||w|| unless |b + w′x| = 0 . Hence , we can treat x+ δradvr ( x , ) /||radvr ( x , ) || for appropriately choosing δ > 0 as a bad sample ( a sample closer to the decision boundary ) .

Summary: The authors propose to combine BadGAN framework and VAT to accelerate learning in the semi-supervised setting. The paper shows that the VAT approach is actively pushing the decision boundary away from the high-density regions. While the BadGAN approach pulls the decision boundary to low-density regions. This simultaneous push and pull lead to after convergence in testing accuracy. The authors also report competitive results on standard datasets used for SSL such as SVHN and CIFAR10.