Year: 2018

Abstract:
Generative Adversarial Networks are powerful generative models that can model the manifold of natural images. We leverage this property to perform manifold regularization by approximating a variant of the Laplacian norm using a Monte Carlo approximation that is easily computed with the GAN. When incorporated into the semi-supervised feature-matching GAN we achieve state-of-the-art results for semi-supervised learning on CIFAR-10 benchmarks when few labels are used, with a method that is significantly easier to implement than competing methods. We find that manifold regularization improves the quality of generated images, and is affected by the quality of the GAN used to approximate the regularizer.

heading: 1 INTRODUCTION
text: Deep neural network classifiers typically require large labeled datasets to obtain high predictive performance. Obtaining such a dataset could be time and cost prohibitive especially for applications where careful expert labeling is required, for instance, in healthcare and medicine. Semi-supervised learning algorithms that enable models to be learned from a small amount of labeled data augmented with (large amounts of) unlabeled data have the potential to vastly reduce this labeling burden.
Fundamentally, semi-supervised learning requires assumptions relating the distribution of the data Px (which can be derived from the unlabeled data) to the classification task (Chapelle et al., 2010). For instance, the classic manifold regularization framework (Belkin et al., 2006) for semi-supervised learning makes the assumption that that the data lie on a low-dimensional manifoldM and moreover that a classifier f is smooth on this manifold, so nearby points on the manifold are assigned similar labels. Algorithms based on this framework enforce a classifier’s invariance to local perturbations on the manifold by penalizing its Laplacian norm ‖f‖2L = ∫ x∈M ‖∇Mf(x)‖ 2 dPX(x). More generally, regularization terms penalizing classifier gradients in regions of high data density have also been proposed (Bousquet et al., 2003).
Recently, generative adversarial networks (GANs) have been used for semi-supervised learning, where they are competitive with state-of-the-art methods for semi-supervised image classification (Dai et al., 2017; Kumar et al., 2017; Qi et al., 2018). GAN-based semi-supervised learning methods typically build upon the formulation in (Salimans et al., 2016), where the discriminator is extended to determine the specific class of an image or whether it is generated; by contrast, the original GAN’s discriminator is only expected to determine whether an image is real or generated. Another key application for GANs is image synthesis, where they have been shown to model the image manifold well (Zhu et al., 2016). Recent work (Kumar et al., 2017; Qi et al., 2018) has used this property of GANs to enforce discriminator invariance on the image manifold, resulting in improved accuracy on semi-supervised image classification.
In this work, we leverage the ability of GANs to model the image manifold to efficiently approximate the Laplacian norm and related regularization terms through Monte-Carlo integration. We show that classifiers (with varying network architectures) regularized with our method outperform baselines on the SVHN and CIFAR-10 benchmark datasets with a method that is significantly simpler to implement than competing methods. In particular, when applied to the semi-supervised featurematching GAN (Salimans et al., 2016), our method achieves state-of-the-art performance amongst GAN-based methods, and is highly competitive with other non-GAN approaches especially when the number of labeled examples is small. We show that manifold regularization improves the quality
of generated images as measured by Inception and FID scores when applied to the semi-supervised feature-matching GAN, thus linking manifold regularization to recent work on gradient penalties for stabilizing GAN training (Roth et al., 2017; Gulrajani et al., 2017; Mescheder et al., 2017). We also found that generator quality (as measured by the quality of generated images) influences the benefit provided by our manifold regularization strategy in that using a better quality generator results in larger improvements in classification performance over a supervised baseline.

heading: 2 RELATED WORK
text: There have been several works adapting GANs for semi-supervised learning. One approach is to change the standard binary discriminator of a standard GAN to predict class labels of labeled examples, while enforcing the constraint that generated data should result in uncertain classifier predictions (Springenberg, 2016). The related approach of Salimans et al. (2016) also uses the discriminator of the GAN as the final classifier, but instead modifies it to predict K + 1 probabilities (K real classes and the generated class). This approach performs well when combined with a feature matching loss for the generator. The work of Li et al. (2017) introduces an additional classifier as well as uses a conditional generator instead of adapting the discriminator to overcome limitations with the two-player formulation of standard GANs in the context of semi-supervised learning.
The idea of encouraging local invariances dates back to the TangentProp algorithm (Simard et al., 1991) where manifold gradients at input data points are estimated using explicit transformations of the data that keep it on the manifold, for example small rotations and translations. Since then other approaches have tried to estimate these tangent directions in different ways. High order contractive autoencoders were used by Rifai et al. (2011b) to capture the structure of the manifold; this representation learning algorithm was then used to encourage a classifier to be insensitive to local direction changes along the manifold (Rifai et al., 2011a). This approach was recently revisited in the context of GANs (Kumar et al., 2017), where the tangent space to the data manifold is estimated using GANs with an encoder in order to inject invariance into the classifier. In addition, this work also explored the use of an additional ambient regularization term which promotes invariance of the discriminator to perturbations on training images along all directions in the data space. The proposed method is competitive with the state-of-the-art GAN method of (Dai et al., 2017), which argues that a generator that generates images that are in the complement of the training data distribution is necessary for good semi-supervised learning performance. Most recently (Qi et al., 2018) proposed the use of a local GAN which attempts to model the local manifold geometry around data points without the need for an encoder. The local GAN is then used to approximate the Laplacian norm for semi-supervised learning, and is shown to enable state-of-the-art classification results.
Aside from GAN-based approaches, Virtual Adversarial Training (VAT) (Miyato et al., 2017), which is based on constraining model predictions to be consistent to local perturbation, has also achieved state-of-the-art performance on benchmarks. Specifically, VAT smooths predictions of the classifier over adversarial examples centered around labeled and unlabeled examples. (Park et al., 2018) similarly introduces adversarial pruning of the neural network to regularize the training. This method gives state-of-the-art results, when combined with VAT.
Other recent works are based on the self-training paradigm (Chapelle et al., 2010). Such methods label the unlabeled data using classifiers trained on the labeled data, and then use this expanded labeled dataset to train a final classifier. Recent progress has been due to clever use of ensembling to produce better predictions on the unlabeled data: instead of simply using predictions of a model trained on the labeled data, (Laine & Aila, 2017) ensembled model predictions under various perturbations or at different time steps. In follow-up work, the Mean Teacher method (Tarvainen & Valpola, 2017) averages model weights instead at different time steps using exponential moving averages, and achieves state-of-the-art performance on benchmark image datasets. Most recently, the work of (Luo et al., 2018) proposes a consistency loss that considers perturbations not only on a datapoint but also on its neighboors.

heading: 3 MANIFOLD REGULARIZATION
text: We present an approach to approximate any density-based regularization term of the form Ω(f) =∫ x∈M L(f)dχ(PX) (Bousquet et al., 2003) with L denoting a measure of smoothness of the clas-
sifier function f and χ a strictly-increasing function. This class of regularizers enforce classifier smoothness in regions of high data density and includes the Laplacian norm with L(f) = ‖∇Mf‖2 and χ the identity function. We focus on the following variant of the Laplacian norm
Ω(f) = ∫ x∈M ‖∇Mf‖F dPX
with L(f) = ‖∇Mf‖F in this work and show how it can be approximated efficiently 1. Our approach relies on two commonly held assumptions about GANs:
1. GANs can model the distribution over images (Radford et al., 2016), such that samples from the GAN are distributed approximately as PX(x), the marginal distribution over images x.
2. GANs learn the image manifold (Radford et al., 2016; Zhu et al., 2016). Specifically, we assume that the generator g learns a mapping from the low-dimensional latent space with coordinates z to the image manifold embedded in a higher-dimensional space, enabling us to compute gradients on the manifold by taking derivatives with respect to z (Kumar et al., 2017; Shao et al., 2017) .
With these assumptions, we may approximate Ω(f) as follows, where we list the relevant assumption above each approximation step
Ω(f) = ∫ x∈M ‖∇Mf‖F dP (1) ≈ 1 n n∑ i=1 ∥∥∥∇Mf(g(z(i)))∥∥∥ F (2) ≈ 1 n n∑ i=1 ∥∥∥Jzf(g(z(i)))∥∥∥ F .
Here, Jz denotes the Jacobian matrix of partial derivatives of classifier outputs f with respect to latent generator variables z 2. Computing gradients of Ω(f) during model learning is computationally prohibitive for deep neural networks as it requires computing the Hessian of a model with large numbers of parameters 3. We hence used stochastic finite differences to approximate the gradient term for computational efficiency.
Issues with manifold gradients: To motivate the specific approximation we used, we first illustrate several issues with manifold gradients as estimated with a GAN when considering the obvious candidate approximation ∥∥Jzf(g(z(i)))∥∥F ≈ ∥∥f(g(z(i)))− f(g(z(i) + δ))∥∥F , δ ∼ N (0, σ2I) (Figure 1), using the Two Circles dataset and MNIST. The Two Circles dataset is an example of data lying on disjoint manifolds. In this case, even though the GAN is able to accurately model the data distribution (Figure 1a, left), we see several instances where the manifold gradients as per the GAN are extremely noisy (Figure 1a, center) with large magnitudes. If data from the inner and outer circle were to belong to different classes, enforcing classifier smoothness at those points in the manifold with these large noisy gradients could result in the classifier predicting similar values for both circles (as g(z(i)) and g(z(i) + δ) ≈ g(z(i)) +Jz(g(z(i)))δ lie on different circles), causing erroneous classifications. At the other extreme, there are points on the manifold where the estimated gradient has such small magnitude such that the regularizer has minimal smoothing effect. These issues are also evident on the MNIST dataset (Figure 1b), where we directly show how g(z(i)) and g(z(i) + δ) can lie on different manifolds (red rectangle) or be virtually identical (blue rectangle), resulting in oversmoothing and under-smoothing respectively of the classifier function. We note that the analysis of Chen et al. (2018) suggests that the issue of large noisy gradients that we described may be inherent to generative latent variable models, as such models may place points from disjoint manifolds in the data space nearby in the latent space (see Figure A4 for an example).
Our improved gradient approximation: In light of these issues arising from the magnitude of manifold gradients, we used the following approximation that takes a step of tunable size in the direction of the manifold gradient, thus ignoring the magnitude of the gradient while enforcing smoothness in its direction
Ω(f) ≈ 1 n n∑ i=1 ∥∥∥f (g (z(i)))− f (g (z(i))+ r̄ (z(i)))∥∥∥ F
1In our early experiments we used the regular Laplacian norm but found that this variant worked better in a wider range of settings; results using the Laplacian norm are included in the Appendix.
2In our experiments, we defined f to be the logits of the softmax output layer instead of the resultant normalized probabilities as we found it gave better performance.
3In fact, for multi-class classifiers f , we need to compute a Hessian tensor – one matrix for each component (class output) of f , which quickly becomes impractical even with moderate numbers of classes.
Here r(z) = g(z + η δ̄) − g(z(i)), δ ∼ N (0, I) is an approximation of the manifold gradient at z with tunable step size η, and v̄ = v‖v‖ denotes a unit vector. Note that the stochastic finite difference approximates a directional derivative in the randomly chosen direction δ: f(g(z) + r̄)− f(g(z)) ≈ Jf .Jg δ̄||Jg δ̄|| . To gain further intuition, we can rewrite the expression g(z) + r̄(z) as g(z)(1− ||r(z)|| ) + g(z + η δ̄) ||r(z)|| to see that a large manifold gradient (large ||r(z)||) will cause f to be evaluated closer to g(z) instead of g(z + η δ̄), which is the desired behavior.
Advantages of our approach: We wish to highlight that our approach only relies on training a standard GAN. In contrast to the approach of Kumar et al. (2017), we do not explicitly enforce classifier smoothness on input data points, allowing us to avoid the added complexity of learning an encoder network to determine the underlying manifold coordinates for a data sample, as well as other tricks required to estimate tangent directions. Our approach leverages the GAN’s ability to interpolate smoothly between input images to enforce classifier smoothness on the entirety of the manifold and not just at the provided data points. We note that Qi et al. (2018) proposed an alternate and elegant solution to the issues we identified by learning a local GAN instead at input data points, but at the price of training a GAN with a more complex local generator.

heading: 4 EXPERIMENTS

heading: 4.1 EXPERIMENTAL SETUP
text: Our experimental setup follows (Tarvainen & Valpola, 2017) – we separated 10% of the training data on CIFAR-10 into a validation set, and did similarly for SVHN. We designed our algorithms and chose our hyperparameters based on this validation set. We perform semi-supervised training using a small fraction of the labeled training data containing an equal number of examples from each class; the remaining training images are used as unlabeled data. Note that the classifier we used for evaluation had weights that were the exponential moving average of classifier weights obtained during training (this technique was also used in (Salimans et al., 2016)). We report the error rates on the test set for models which performed best on the validation set. Details of hyperparameters and network architectures we used in our experiments can be found in the Appendix.

heading: 4.2 INCORPORATING MANIFOLD REGULARIZATION INTO SEMI-SUPERVISED GANS
text: We first evaluated our regularization method when incorporated into the semi-supervised GAN framework. We reproduced the semi-supervised feature-matching GAN of (Salimans et al., 2016) and added our manifold regularizer to the model. The final loss function of the discriminator is:
L = Lsupervised + Lunsupervised + γm Ωmanifold, where
Lunsupervised = −Ex∼pdata(x) [log [1− pf (y = K + 1|x)]]− Ex∼g [log [pf (y = K + 1|x)]] Lsupervised = −Ex,y∼pdata(x,y) [log pf (y|x, y < K + 1)] Ωmanifold = Ez∼U(z),δ∼N(δ)) ‖f (g (z))− f (g (z) + r̄)‖2
and we used the feature matching loss for our generator, ∥∥Ex∼pdatah(x)− Ez∼pz(z)h(g(z))∥∥. Here, h(x) denotes activations on an intermediate layer of the discriminator. We also checked if the anisotropic regularization along manifold directions in the data space that we used provides additional benefits over simple ambient regularization in the data space. Specifically, we evaluated the ambient regularizer λ Ex∼pd(x) ‖Jxf‖, as proposed in (Kumar et al., 2017) that we similarly approximate using a stochastic finite difference as shown below: L = Lsupervised + Lunsupervised + γaΩambient , where
Ωambient = Eδ∼N(δ)) ∥∥f (x)− f (x+ δ̄)∥∥
2 .
We present results on CIFAR-10 (Krizhevsky, 2009) in in Tables 1 and 2, and results on SVHN (Netzer et al., 2011) in Table 3. We first note that our implementation of the feature-matching GAN with weight normalization (Salimans & Kingma, 2016) (Improved GAN) (Salimans et al., 2016) significantly outperforms the original (and many other recent methods) after we tuned training hyperparameters, illustrating the sensitivity of semi-supervised GANs to hyperparameter settings. Adding manifold regularization to the feature-matching GAN further improves performance, achieving state-of-the-art results amongst all GAN-based methods, as well as being highly competitive with other non-GAN-based methods, especially when less labeled data is used. We also observe
that while simple (isotropic) ambient regularization provides some benefit, our (anisotropic) manifold regularization term provides additional performance gains.
Our results are consistent with recent work in semi-supervised learning and more generally regularization of neural networks. While studies have shown that promoting classifier robustness against local perturbations is effective for semi-supervised learning (Laine & Aila, 2017; Tarvainen & Valpola, 2017), other recent work suggests that it is difficult to achieve local isotropy by enforcing invariance to random perturbations independent of the inputs (which is what the simple ambient regularizer does) in highly non-linear models (Szegedy et al., 2014), so data-dependent perturbations should be used instead. One possibility is to enforce invariance to perturbations along adversarial directions of the classifier (Miyato et al., 2017). Our approach instead enforces invariance to perturbations on the data manifold as modelled by the GAN.

heading: 4.3 INTERACTION OF OUR REGULARIZER WITH THE GENERATOR
text: In the semi-supervised GAN framework, applying manifold regularization to the discriminator has the potential to affect the generator through the adversarial training procedure. We explored the effects of our regularization on the generator by evaluating the quality of images generated using the Inception (Salimans et al., 2016) and FID scores (Heusel et al., 2017) as shown in Table 4. We observe that adding manifold regularization yields significant improvements in image quality across both CIFAR-10 and SVHN datasets and with varying amounts of labeled data. Why might manifold regularization help? Here we briefly discuss two possible explanations for this improvement – gradient penalties and improved generator conditioning.
Gradient penalties: Our results are consistent with recent work suggesting that gradient penalties on the discriminator may be used to stabilize GAN training (Gulrajani et al., 2017; Roth et al., 2017; Mescheder et al., 2018b). Our regularization term is closely related to the proposed penalties in (Mescheder et al., 2018b). In particular, the proposed R2 regularizer γ2Ex∼Pg [ ||∇Dx(x)||
2] that penalizes the gradient of the discriminator D with respect to the data space, over samples drawn from the generator distribution Pg , is almost identical to our proposed regularizer, except that we take gradients with respect to the generator’s latent variables z, and we do not square the norm.
Generator conditioning: Recent work suggests that the condition number of the Jacobian of the generator (generator conditioning) is causally related to GAN performance as measured by Inception and FID scores in that well-conditioned generators perform better (Odena et al., 2018); we note that a similar stochastic finite difference regularizer was used in this work to encourage the spectrum of the generator Jacobian to lie in a specific range. If our manifold regularizer influenced generator conditioning, this could explain the improvements in GAN quality we observed. To test this hypothesis, we computed the mean log condition number (MLCN) of the generator with and without manifold regularization. Consistent with this hypothesis, we observed improved conditioning
with regularization (MLCN: 7.97 with regularization, 8.45 without); in this experiment GANs were trained with 1000 labeled training examples on the CIFAR-10.

heading: 4.4 INCORPORATING MANIFOLD REGULARIZATION INTO CONVOLUTIONAL NEURAL NETS
text: We also explored the potential for our manifold regularization framework to improve performance of classifiers outside the semi-supervised GAN framework. We performed a series of experiments where we first trained a GAN to learn the marginal distribution of the data PX and the data manifold M, and subsequently used the trained GAN to regularize a separate neural network classifier. In this setup, we minimize the following loss where V is the cross entropy loss on the labeled examples:
L = 1
n ∑ n V (x(i), y(i), f) + γmΩ(f).
Here unlabeled examples are implicitly used to regularize the classifier f since they are used to train the GAN. This “decoupled” setting enables us to understand how the quality of the generator used to approximate the manifold regularizer affects classification accuracy as there is no interaction between the generator and the classifier being regularized.
We first verified that our method works on synthetic 2D datasets when the manifold is known exactly (Figure A2) and also when a GAN is used to learn both the data manifold and the data distribution on this manifold (Figure A3). In these experiments, we penalized a neural network classifier with our manifold regularizer as approximated with a consensus GAN (Mescheder et al., 2017). The classifier we used consists of 6 fully-connected layers with 384 neurons each. We observed that perfect classification could be achieved with only one labeled example per class even when the GAN does not perfectly learn the data distribution or manifold.
We then evaluated our manifold regularization method on a real image dataset (CIFAR-10), using a standard DCGAN to approximate the manifold regularizer and a 13 layer CNN as the classifier (see the Appendix for more details). We observe that manifold regularization is able to reduce classification error by 2-3% over the purely supervised baseline when different amounts of labeled data are used (Table 5 bottom; first two rows).

heading: 4.5 EFFECT OF GAN QUALITY
text: To quantify the importance of the first step of manifold learning, we also compared the performance of classifiers when regularized using GANs of differing quality (as assessed by the quality of gen-
erated images). We compared the following three generators: 1) a DCGAN which produces decent looking images (inception score of 6.68 and FID of 32.32 on CIFAR-10), 2) a DCGAN with a much lower inception score (inception score of 3.57 on CIFAR-10), and 3) a noise image generator where pixels are generated independently from uniform distributions in the range [0, 255].
We observe that using GANs with better generators for manifold regularization resulted in lower classification errors (Table 5; last 3 rows). Our results also suggest that even GANs that generate lower quality images but nonetheless have captured some aspects of the image manifold are able to provide some performance benefit when used for manifold regularization. As a negative control, we observe that performance degrades relative to the supervised baseline when we compute the regularizer using randomly generated images.

heading: 4.6 UNDERSTANDING OUR MANIFOLD REGULARIZATION APPROXIMATION METHOD
text: Finally, we attempt to provide some intuition of how our manifold regularization approximation method works. Our method promotes classifier invariance between generated samples g(z) and their corresponding perturbations g(z) + r̄ obtained by perturbing their latent code. In other words, the regularizer promotes invariance of the classifier specifically along directions on the data manifold. We show in Figure A8 some examples of generated samples and their corresponding perturbations. We see that even if some features such as the background color may change, there exists a global class consistency between the images as is varied. Hence, enforcing invariance in predictions along these directions will result in correct classifications provided is not too large (which would result in larger distortions and unrecognizable images). An intuitive explanation is that manifold regularization performs label propagation across “semantically similar” images by minimizing the manifold consistency cost Ω(f), so that an image of a red car gets the same label as an orange car.

heading: 5 CONCLUSION
text: GANs are powerful generative models that are able to model the distribution and manifold over natural images. We leverage these properties to perform manifold regularization by approximating a variant of the Laplacian norm using a Monte Carlo approximation that is easily computed with the GAN. We show that our regularization strategy consistently improves classification performance using unlabeled data on the CIFAR-10 and SVHN benchmarks, on several neural network architectures, and with varying amounts of labeled data. In particular, when incorporated into the featurematching GAN of (Salimans et al., 2016), we achieve state-of-the-art results for semi-supervised image classification when few labels are used with a method that is significantly easier to implement than competing methods. We explored the interaction between our regularization and the generator in this framework and reveal a connection with gradient penalties for stabilizing GAN training. Using an experimental setup where we decoupled the GAN used for estimating the regularizer and the classifier, we further observed a positive correlation between generator image quality and prediction accuracy. Our work uses GANs in a novel way for semi-supervised classification, and we expect that our approach will be applicable to semi-supervised regression (Belkin & Niyogi, 2004; Lafferty & Wasserman, 2007; Moscovich et al., 2017) as well as unsupervised learning (Belkin et al., 2006).

heading: A ADDITIONAL RESULTS ON CIFAR-10
text: Comparison of the finite difference methods: We provide in this section additional results for the semi-supervised learning experiments with the semi-supervised GAN model. We compare the two different manifold regularization methods: the normalized and unnormalized stochastic finite difference. We observe that the normalization provides an additional performance boost that grows as the number of labeled examples is reduced (Table A1).
VAT experimental details: We used the original VAT implementation made available online by the authors. We kept the same hyperparameters that the authors used for their experiments with 4000 labels.
Figure A1: Results including data augmentation (4x4 random translations and random horizontal flips similar to previous works).
Table A1: Error rate on CIFAR-10 averaged over 4 runs with different random seeds. Results were obtained without data augmentation.
1000 labels (2%) 2000 labels (4%) 4000 labels (8%) CIFAR-10 50000 images 50000 images 50000 images Π model 32.18±1.33 23.92±1.03 16.55 ± 0.29 Mean Teacher 30.62 ±1.13 23.14 ± 0.46 17.74 ± 0.30 VAT (large) 22.18* 18.32* 14.18 VAT+EntMin (large) 20.52* 15.52* 13.15 Π+SNTG 13.62±0.17 VAT+Ent+SNTG 12.49±0.36 Improved GAN 21.83 ± 2.01 19.61 ± 2.09 18.63 ± 2.32 Improved GAN + SNTG 14.93 Improved Semi-GAN 19.52 ±1.5 16.20 ± 1.6 ALI 19.98 ± 0.89 19.09 ± 0.44 17.99 ± 1.62 Bad GAN 14.41 ± 0.30 Local GAN 17.44 ± 0.25 14.23 ± 0.27 Improved GAN (ours) 17.50 ± 0.34 16.80 ± 0.07 15.5 ± 0.35 Ambient (ours) 16.98 ± 0.36 15.99 ± 0.14 14.75 ± 0.37 Manifold reg. unnormalized (ours) 18.09 15.8 14.40 ± 0.21 Manifold (ours) 16.37 ± 0.42 15.25 ± 0.35 14.34 ± 0.17
B 2D DECOUPLED EXPERIMENTS
Known manifold: In Figure A2 we show an example of decoupled training with a perfectly known manifold. We can compute the tangent to each data point (using its polar coordinates:
θ = arctan( yx )). Instead of the finite difference method, we can plug in the exact tangent direction in our regularizer. As shown in Figure A2 the classifier achieves perfect classification thanks to the tangential directions incorporated to the model.
Unknown manifold: In Figure A3 the distribution of the data is unknown. A GAN makes use of the unlabeled examples to learn the data distribution. The data manifold modeled by the GAN is used with by regularizer. It effectively improves generalization power of the model. As shown in Figure A3, all the models achieves perfect classification with only one labeled example per class. An example in the case of a noisy dataset is also given in Figure A11.
Figure A2: Manifold regularization with known tangent directions on a synthetic dataset. Left: Semi-supervised classification of the disjoint manifold. Middle: Magnitude of the regularization term for a batch of generated samples. Darker fill color reflects larger magnitude. Generated datapoints near the decision boundary are highly penalized. Right: Known tangent directions.
Figure A3: Behavior of manifold regularization with a separate classifier on toy examples. Left: Classification boundaries of trained classifier. Labeled examples are shown as points, unlabeled examples were drawn from distributions over the curves shown. The classifier achieves perfect accuracy on the two datasets. Middle: Magnitude of the regularization term for a batch of generated samples. Darker fill color reflects larger magnitude. Generated data-points near the decision boundary are highly penalized. Right: Direction of invariance promoted by our norm. The trained GANs are able to approximate the data distribution and manifold gradients. In this example γ = 6 and = 0.15.

heading: C GENERATOR CONDITIONNING
text: Log condition number (MLCN): Following Odena et al. (2018), G defines a mapping from a low dimensional space Z ∈ Rnz to the image space X ∈ Rnx , G : Z −→ X . For any z ∈ Z we can define the Jacobian matrix Jz ∈ Rnz×nx . We can write the metric tensor Mz = JTz .Jz . The condition number of the generator is defined for Mz as λminλmax . Large eigenvalues of M correspond to the direction in Z which result in high changes in G(z). A generator with a low mean condition number, should provide smooth changes from Z space to X space.
CIFAR10 Mean log condition number Improved GAN 8.48 Manifold reg (unnormalized) 7.78 Manifold reg (normalized) 8.03
Table A2: Log condition number of the generator taken at the end of the training. The models are trained with 1000 labels
Figure A4: GANs trained on a disjoint manifold. Visualization of the non continuous mapping.
We computed the MLCN by first computing the Jacobian for a set of random samples in the Z space and then we extracted the eigenvalues of the resulted tensors by doing SVD. We report in Table A2 the results of our analysis concerning the conditioning of the generators. It shows that the manifold regularization has a positive effect on the conditioning of the generator.

heading: D DISJOINT MANIFOLD LEARNING
text: We provide an example of two disjoint manifold learnt by GANs in Figure A4. The latent space is a 2D Gaussian vector. This example reveals the issue with generative latent variable models: such models may place points from disjoint manifolds in the data space nearby in the latent space (see left plot).

heading: E GENERATED IMAGES
text: We provide in Figure A6 samples of generated images of the trained generator at the end of the training with and without regularization. Mode collapse is visible for the Improved GAN images with no regularization. While both ambient and manifold regularizations seem to increase images quality.
(a) (b) (c)
Figure A5: Generated images from the coupled experiments trained on CIFAR-10. (a) Improved GAN (b) Ambient regularization (c) Manifold regularization. Mode collapse is visible for the Improved GAN images with no regularization.
(a) (b) (c)
Figure A6: Generated images from the coupled experiments trained on SVHN. (a) Improved GAN (b) Ambient regularization (c) Manifold regularization. Mode collapse is visible for the Improved GAN images with no regularization.
Figure A7: Generated images for decoupled experiments

heading: F TANGENT IMAGES
text: We show samples from the decoupled generator trained on the unlabeled examples on CIFAR10, in Figure A7. The model has an inception score of 6.68 and a FID of 32.32
In this section, for a better understanding of our method, we provide examples of tangent images, approximated by our trained GAN. We shows the influence of the different hyperparameters η, in Figure A9 and A8 on the visual aspect of the images.
Figure A8: Effect of hyperparameter for approximating manifold regularization. Generated images with varying perturbations as per the gradient approximation (g(z) + r̄) are shown for η = 1. We used ( = 20, η = 1) in our experiments.
Figure A9: Effect of hyperparameter η for approximating manifold regularization. Moving η in the range [10−4, 10−3, 10−2, 10−1, 1, 10] with = 20 fixed
Figure A10: Unsupervised learning of the two moon dataset. No labels are used for the training. The objective function is made of the Laplacian norm, a l2 norm and the entropy of the output prediction Ω(f) = γL ‖f‖L + γK ‖f‖ 2 K − γhH(f). We added the entropy to avoid degenerated solutions. In this example = 0.15 , γ = 3 ,γK = 1 , γh = 0.1

heading: G UNSUPERVISED LEARNING OF SIMPLE MANIFOLDS BY MINIMIZING THE LAPLACIAN NORM
text: Similarly to Belkin et al. (2006) we show that this is possible to go even further with the manifold regularizer. Under certain conditions, we noticed that it was possible to cluster without any supervision (Figure A10).

heading: H SEMI SUPERVISED LEARNING ON A NOISY DATASET
text: Finally, Figure A11 proves the robustness of the method to the noise. The model successfully performs perfect classification with only one labeled datapoint per class on a noisy Two Moons dataset.

heading: I ARCHITECTURES & HYPER PARAMETERS
text: I.1 TOY EXAMPLE
For the GAN, we used an simple setup. The generator and discriminator both have 6 ReLU layers of 384 neurons. The latent space of the generator has a latent space of two dimensions. The generator has two outputs, the discriminator has one. The networks are trained under RMSProp. The networks were trained using consensus optimization, introduced in this method is very effective to stabilize the training of the GAN and to successfully capture highly multi modal distributions. The neural network we used for the classification has a similar architecture than the discriminator with 6 ReLU layers of 384 neurons.
I.2 SEMI-SUPERVISED GANS
For reproducibility sake, we provide in this section the exact architectures and hyper-parameters that we used for our different models. We provide in Tables A3 and A4 the architectures of the coupled models. The corresponding hyper-parameters are given in Table A5.
I.3 DECOUPLED CONVNET
We provide in this section the architectures and hyper-parameters of the models used for the decoupled experiments. In Table A6 we give the architecture of the GAN we trained to learn the distribution of the training examples. Table A7 describes the architecture of the convolutional neural network we used to perform semi supervised classification. Finally, Table A8 shows hyperperameters we found, with a grid-search on the validation set for the training of our models.
Table A4: Discriminator architecture we used in our semi supervised GANs experiments taken from
conv-large CIFAR-10 conv-small SVHN 32×32×3 RGB images
dropout, p = 0.2 3×3 conv. weightnorm 96 lReLU 3×3 conv. weightnorm 64 lReLU 3×3 conv. weightnorm 96 lReLU 3×3 conv. weightnorm 64 lReLU
3×3 conv. weightnorm 96 lReLU stride=2 3×3 conv. weightnorm 64 lReLU stride=2 dropout, p = 0.5
3×3 conv. weightnorm 192 lReLU 3×3 conv. weightnorm 128 lReLU 3×3 conv. weightnorm 192 lReLU 3×3 conv. weightnorm 128 lReLU
3×3 conv. weightnorm 192 lReLU stride=2 3×3 conv. weightnorm 128 lReLU stride=2 dropout, p = 0.5
3×3 conv. weightnorm 192 lReLU pad=0 3×3 conv. weightnorm 128 lReLU pad=0 NiN weightnorm 192 lReLU NiN weightnorm 128 lReLU NiN weightnorm 192 lReLU NiN weightnorm 128 lReLU
global-pool dense weightnorm 10
Table A5: Hyperparameters of the models based on the validation set used to report our semisupervised GAN experiments
Hyperparameters CIFAR SVHN γ 10−3 10−3
20 20 η 1 1 Epoch 1400 400 Batch size 25 50 Optimizer ADAM(α = 3 ∗ 10−4, beta1 = 0.5) Learning rate linearly decayed to 0 after 1200 epochs no decay Leaky ReLU slope 0.2 Weight initialization Isotropic gaussian (µ = 0, σ = 0.05) Biais initialization Constant(0)
Table A6: GAN architecture we used for our experiments
Discriminator Generator 32×32×3 RGB images latent space 100 (gaussian noise)
4×4 64 conv stride=2 batchnorm lReLU 4×4 1024 batchnorm conv.T pad=0 ReLU 4×4 256 conv stride=2 batchnorm lReLU 4×4 256 stride=2 batchnorm conv.T ReLU
4×4 1024 conv stride=2 batchnorm lReLU 4×4 64 stride=2 batchnorm conv.T ReLU 4×4 1 conv pad=0 4×4 3 stride=2 batchnorm conv.T tanh
Table A7: The convolutional network architecture we used in our decoupled experiments.
CIFAR-10 & SVHN convnet 32 × 32 × 3
3 × 3, 96 conv. batchnorm lReLU 3 × 3, 96 conv. batchnorm lReLU 3 × 3, 96 conv. batchnorm lReLU
2 × 2 maxpool dropout = 0.5
3 × 3, 128 conv. batchnorm lReLU 3 × 3, 128 conv. batchnorm lReLU 3 × 3, 128 conv. batchnorm lReLU
2 × 2 maxpool dropout=0.5
3 × 3, 256 conv. batchnorm lReLU pad=0 1 × 1, 128 conv. batchnorm lReLU 1 × 1, 128 conv. batchnorm lReLU
average pooling 10 dense
Table A8: Hyperparameters of the models based on the validation set used to report our decoupled convnet experiments
Hyperparameters CIFAR γ 10−4
20 η 1 Epoch 200 Batch size 100 Optimizer ADAM(α = 3 ∗ 10−4, beta1 = 0.9) Learning rate no decay Leaky ReLU slope 0.2 Weight initialization Isotropic gaussian (µ = 0, σ = 0.05) Biais initialization Constant(0)

Review summaries:
The paper tackles the problem of semi-supervised classification using GAN-based models. They proposed a manifold regularization by approximating the Laplacian norm using the stochastic finite difference. The motivation is that making the classifier invariant to perturbations along the manifold is more reasonable than random perturbations. The idea is to use GAN to learn the manifold. The difficulty is that (the gradient of) Laplacian norm is impractical to compute for DNNs. They stated that another approximation of the manifold gradient, i.e. adding Gaussian noise \delta to z directly (||f(z) - f(g(z+\delta))||_F) has some drawbacks when the magnitude of noise is too large or too small. The authors proposed another improved gradient approximation by first computing the normalized manifold gradient \bar r(z) and then adding a tunable magnitude of \bar r(z) to g(z), i.e., ||f(z) - f(g(z) +\epsilon \bar r(z) )||_F. Since several previous works Kumar et al. (2017) and Qi et al. (2018) also applied the idea of manifold regularization into GAN, the authors pointed out several advantages of their new regularization.