Manifold regularization with GANs for semi-supervised learning

1 INTRODUCTION . Deep neural network classifiers typically require large labeled datasets to obtain high predictive performance . Obtaining such a dataset could be time and cost prohibitive especially for applications where careful expert labeling is required , for instance , in healthcare and medicine . Semi-supervised learning algorithms that enable models to be learned from a small amount of labeled data augmented with ( large amounts of ) unlabeled data have the potential to vastly reduce this labeling burden . Fundamentally , semi-supervised learning requires assumptions relating the distribution of the data Px ( which can be derived from the unlabeled data ) to the classification task ( Chapelle et al. , 2010 ) . For instance , the classic manifold regularization framework ( Belkin et al. , 2006 ) for semi-supervised learning makes the assumption that that the data lie on a low-dimensional manifoldM and moreover that a classifier f is smooth on this manifold , so nearby points on the manifold are assigned similar labels . Algorithms based on this framework enforce a classifier ’ s invariance to local perturbations on the manifold by penalizing its Laplacian norm ‖f‖2L = ∫ x∈M ‖∇Mf ( x ) ‖ 2 dPX ( x ) . More generally , regularization terms penalizing classifier gradients in regions of high data density have also been proposed ( Bousquet et al. , 2003 ) . Recently , generative adversarial networks ( GANs ) have been used for semi-supervised learning , where they are competitive with state-of-the-art methods for semi-supervised image classification ( Dai et al. , 2017 ; Kumar et al. , 2017 ; Qi et al. , 2018 ) . GAN-based semi-supervised learning methods typically build upon the formulation in ( Salimans et al. , 2016 ) , where the discriminator is extended to determine the specific class of an image or whether it is generated ; by contrast , the original GAN ’ s discriminator is only expected to determine whether an image is real or generated . Another key application for GANs is image synthesis , where they have been shown to model the image manifold well ( Zhu et al. , 2016 ) . Recent work ( Kumar et al. , 2017 ; Qi et al. , 2018 ) has used this property of GANs to enforce discriminator invariance on the image manifold , resulting in improved accuracy on semi-supervised image classification . In this work , we leverage the ability of GANs to model the image manifold to efficiently approximate the Laplacian norm and related regularization terms through Monte-Carlo integration . We show that classifiers ( with varying network architectures ) regularized with our method outperform baselines on the SVHN and CIFAR-10 benchmark datasets with a method that is significantly simpler to implement than competing methods . In particular , when applied to the semi-supervised featurematching GAN ( Salimans et al. , 2016 ) , our method achieves state-of-the-art performance amongst GAN-based methods , and is highly competitive with other non-GAN approaches especially when the number of labeled examples is small . We show that manifold regularization improves the quality of generated images as measured by Inception and FID scores when applied to the semi-supervised feature-matching GAN , thus linking manifold regularization to recent work on gradient penalties for stabilizing GAN training ( Roth et al. , 2017 ; Gulrajani et al. , 2017 ; Mescheder et al. , 2017 ) . We also found that generator quality ( as measured by the quality of generated images ) influences the benefit provided by our manifold regularization strategy in that using a better quality generator results in larger improvements in classification performance over a supervised baseline . 2 RELATED WORK . There have been several works adapting GANs for semi-supervised learning . One approach is to change the standard binary discriminator of a standard GAN to predict class labels of labeled examples , while enforcing the constraint that generated data should result in uncertain classifier predictions ( Springenberg , 2016 ) . The related approach of Salimans et al . ( 2016 ) also uses the discriminator of the GAN as the final classifier , but instead modifies it to predict K + 1 probabilities ( K real classes and the generated class ) . This approach performs well when combined with a feature matching loss for the generator . The work of Li et al . ( 2017 ) introduces an additional classifier as well as uses a conditional generator instead of adapting the discriminator to overcome limitations with the two-player formulation of standard GANs in the context of semi-supervised learning . The idea of encouraging local invariances dates back to the TangentProp algorithm ( Simard et al. , 1991 ) where manifold gradients at input data points are estimated using explicit transformations of the data that keep it on the manifold , for example small rotations and translations . Since then other approaches have tried to estimate these tangent directions in different ways . High order contractive autoencoders were used by Rifai et al . ( 2011b ) to capture the structure of the manifold ; this representation learning algorithm was then used to encourage a classifier to be insensitive to local direction changes along the manifold ( Rifai et al. , 2011a ) . This approach was recently revisited in the context of GANs ( Kumar et al. , 2017 ) , where the tangent space to the data manifold is estimated using GANs with an encoder in order to inject invariance into the classifier . In addition , this work also explored the use of an additional ambient regularization term which promotes invariance of the discriminator to perturbations on training images along all directions in the data space . The proposed method is competitive with the state-of-the-art GAN method of ( Dai et al. , 2017 ) , which argues that a generator that generates images that are in the complement of the training data distribution is necessary for good semi-supervised learning performance . Most recently ( Qi et al. , 2018 ) proposed the use of a local GAN which attempts to model the local manifold geometry around data points without the need for an encoder . The local GAN is then used to approximate the Laplacian norm for semi-supervised learning , and is shown to enable state-of-the-art classification results . Aside from GAN-based approaches , Virtual Adversarial Training ( VAT ) ( Miyato et al. , 2017 ) , which is based on constraining model predictions to be consistent to local perturbation , has also achieved state-of-the-art performance on benchmarks . Specifically , VAT smooths predictions of the classifier over adversarial examples centered around labeled and unlabeled examples . ( Park et al. , 2018 ) similarly introduces adversarial pruning of the neural network to regularize the training . This method gives state-of-the-art results , when combined with VAT . Other recent works are based on the self-training paradigm ( Chapelle et al. , 2010 ) . Such methods label the unlabeled data using classifiers trained on the labeled data , and then use this expanded labeled dataset to train a final classifier . Recent progress has been due to clever use of ensembling to produce better predictions on the unlabeled data : instead of simply using predictions of a model trained on the labeled data , ( Laine & Aila , 2017 ) ensembled model predictions under various perturbations or at different time steps . In follow-up work , the Mean Teacher method ( Tarvainen & Valpola , 2017 ) averages model weights instead at different time steps using exponential moving averages , and achieves state-of-the-art performance on benchmark image datasets . Most recently , the work of ( Luo et al. , 2018 ) proposes a consistency loss that considers perturbations not only on a datapoint but also on its neighboors . 3 MANIFOLD REGULARIZATION . We present an approach to approximate any density-based regularization term of the form Ω ( f ) =∫ x∈M L ( f ) dχ ( PX ) ( Bousquet et al. , 2003 ) with L denoting a measure of smoothness of the clas- sifier function f and χ a strictly-increasing function . This class of regularizers enforce classifier smoothness in regions of high data density and includes the Laplacian norm with L ( f ) = ‖∇Mf‖2 and χ the identity function . We focus on the following variant of the Laplacian norm Ω ( f ) = ∫ x∈M ‖∇Mf‖F dPX with L ( f ) = ‖∇Mf‖F in this work and show how it can be approximated efficiently 1 . Our approach relies on two commonly held assumptions about GANs : 1 . GANs can model the distribution over images ( Radford et al. , 2016 ) , such that samples from the GAN are distributed approximately as PX ( x ) , the marginal distribution over images x . 2 . GANs learn the image manifold ( Radford et al. , 2016 ; Zhu et al. , 2016 ) . Specifically , we assume that the generator g learns a mapping from the low-dimensional latent space with coordinates z to the image manifold embedded in a higher-dimensional space , enabling us to compute gradients on the manifold by taking derivatives with respect to z ( Kumar et al. , 2017 ; Shao et al. , 2017 ) . With these assumptions , we may approximate Ω ( f ) as follows , where we list the relevant assumption above each approximation step Ω ( f ) = ∫ x∈M ‖∇Mf‖F dP ( 1 ) ≈ 1 n n∑ i=1 ∥∥∥∇Mf ( g ( z ( i ) ) ) ∥∥∥ F ( 2 ) ≈ 1 n n∑ i=1 ∥∥∥Jzf ( g ( z ( i ) ) ) ∥∥∥ F . Here , Jz denotes the Jacobian matrix of partial derivatives of classifier outputs f with respect to latent generator variables z 2 . Computing gradients of Ω ( f ) during model learning is computationally prohibitive for deep neural networks as it requires computing the Hessian of a model with large numbers of parameters 3 . We hence used stochastic finite differences to approximate the gradient term for computational efficiency . Issues with manifold gradients : To motivate the specific approximation we used , we first illustrate several issues with manifold gradients as estimated with a GAN when considering the obvious candidate approximation ∥∥Jzf ( g ( z ( i ) ) ) ∥∥F ≈ ∥∥f ( g ( z ( i ) ) ) − f ( g ( z ( i ) + δ ) ) ∥∥F , δ ∼ N ( 0 , σ2I ) ( Figure 1 ) , using the Two Circles dataset and MNIST . The Two Circles dataset is an example of data lying on disjoint manifolds . In this case , even though the GAN is able to accurately model the data distribution ( Figure 1a , left ) , we see several instances where the manifold gradients as per the GAN are extremely noisy ( Figure 1a , center ) with large magnitudes . If data from the inner and outer circle were to belong to different classes , enforcing classifier smoothness at those points in the manifold with these large noisy gradients could result in the classifier predicting similar values for both circles ( as g ( z ( i ) ) and g ( z ( i ) + δ ) ≈ g ( z ( i ) ) +Jz ( g ( z ( i ) ) ) δ lie on different circles ) , causing erroneous classifications . At the other extreme , there are points on the manifold where the estimated gradient has such small magnitude such that the regularizer has minimal smoothing effect . These issues are also evident on the MNIST dataset ( Figure 1b ) , where we directly show how g ( z ( i ) ) and g ( z ( i ) + δ ) can lie on different manifolds ( red rectangle ) or be virtually identical ( blue rectangle ) , resulting in oversmoothing and under-smoothing respectively of the classifier function . We note that the analysis of Chen et al . ( 2018 ) suggests that the issue of large noisy gradients that we described may be inherent to generative latent variable models , as such models may place points from disjoint manifolds in the data space nearby in the latent space ( see Figure A4 for an example ) . Our improved gradient approximation : In light of these issues arising from the magnitude of manifold gradients , we used the following approximation that takes a step of tunable size in the direction of the manifold gradient , thus ignoring the magnitude of the gradient while enforcing smoothness in its direction Ω ( f ) ≈ 1 n n∑ i=1 ∥∥∥f ( g ( z ( i ) ) ) − f ( g ( z ( i ) ) + r̄ ( z ( i ) ) ) ∥∥∥ F 1In our early experiments we used the regular Laplacian norm but found that this variant worked better in a wider range of settings ; results using the Laplacian norm are included in the Appendix . 2In our experiments , we defined f to be the logits of the softmax output layer instead of the resultant normalized probabilities as we found it gave better performance . 3In fact , for multi-class classifiers f , we need to compute a Hessian tensor – one matrix for each component ( class output ) of f , which quickly becomes impractical even with moderate numbers of classes . Here r ( z ) = g ( z + η δ̄ ) − g ( z ( i ) ) , δ ∼ N ( 0 , I ) is an approximation of the manifold gradient at z with tunable step size η , and v̄ = v‖v‖ denotes a unit vector . Note that the stochastic finite difference approximates a directional derivative in the randomly chosen direction δ : f ( g ( z ) + r̄ ) − f ( g ( z ) ) ≈ Jf .Jg δ̄||Jg δ̄|| . To gain further intuition , we can rewrite the expression g ( z ) + r̄ ( z ) as g ( z ) ( 1− ||r ( z ) || ) + g ( z + η δ̄ ) ||r ( z ) || to see that a large manifold gradient ( large ||r ( z ) || ) will cause f to be evaluated closer to g ( z ) instead of g ( z + η δ̄ ) , which is the desired behavior . Advantages of our approach : We wish to highlight that our approach only relies on training a standard GAN . In contrast to the approach of Kumar et al . ( 2017 ) , we do not explicitly enforce classifier smoothness on input data points , allowing us to avoid the added complexity of learning an encoder network to determine the underlying manifold coordinates for a data sample , as well as other tricks required to estimate tangent directions . Our approach leverages the GAN ’ s ability to interpolate smoothly between input images to enforce classifier smoothness on the entirety of the manifold and not just at the provided data points . We note that Qi et al . ( 2018 ) proposed an alternate and elegant solution to the issues we identified by learning a local GAN instead at input data points , but at the price of training a GAN with a more complex local generator .

Summary: The paper tackles the problem of semi-supervised classification using GAN-based models. They proposed a manifold regularization by approximating the Laplacian norm using the stochastic finite difference. The motivation is that making the classifier invariant to perturbations along the manifold is more reasonable than random perturbations. The idea is to use GAN to learn the manifold. The difficulty is that (the gradient of) Laplacian norm is impractical to compute for DNNs. They stated that another approximation of the manifold gradient, i.e. adding Gaussian noise \delta to z directly (||f(z) - f(g(z+\delta))||_F) has some drawbacks when the magnitude of noise is too large or too small. The authors proposed another improved gradient approximation by first computing the normalized manifold gradient \bar r(z) and then adding a tunable magnitude of \bar r(z) to g(z), i.e., ||f(z) - f(g(z) +\epsilon \bar r(z) )||_F. Since several previous works Kumar et al. (2017) and Qi et al. (2018) also applied the idea of manifold regularization into GAN, the authors pointed out several advantages of their new regularization.