Title: FAST ADVERSARIAL TRAINING FOR SEMI-SUPERVISED LEARNING

Abstract: In semi-supervised learning, Bad GAN approach is one of the most attractive method due to the intuitional simplicity and powerful performances. Bad GAN learns a classifier with bad samples distributed on complement of the support of the input data. But Bad GAN needs additional architectures, a generator and a density estimation model, which involves huge computation and memory consumption cost.
VAT is another good semi-supervised learning algorithm, which utilizes unlabeled data to improve the invariance of the classifier with respect to perturbation of inputs. In this study, we propose a new method by combining the ideas of Bad GAN and VAT. The proposed method generates bad samples of high-quality by use of the adversarial training used in VAT. We give theoretical explanations why the adversarial training is good at both generating bad samples and semi-supervised learning. An advantage of the proposed method is to achieve the competitive performances with much fewer computations. We demonstrate advantages our method by various experiments with well known benchmark image datasets.

1. INTRODUCTION:

Deep learning has accomplished unprecedented success due to the development of deep architectures, learning techniques and hardwares (Krizhevsky et al., 2012; Ioffe & Szegedy, 2015; Szegedy et al., 2015; Hinton et al., 2012; Kingma & Ba, 2014) . However, deep learning has also suffered from collecting large amount of labeled data which requires both cost and time. Thus it becomes important to develop semi-supervised methodologies that learn a classifier (or discriminator) by using small labeled data and large unlabeled data.

Various semi-supervised learning methods have been proposed for deep learning. Weston et al. (2012) employs a manifold embedding technique using the pre-constructed graph of unlabeled data and Rasmus et al. (2015) uses a specially designed auto-encoder to extract essential features for classification. Variational auto encoder (Kingma & Welling, 2013) is also used in the context of semi-supervised learning by maximizing the variational lower bound of both labeled and unlabeled data (Kingma et al., 2014; Maaløe et al., 2016) .

Recently, semi-supervised learning based on generative adversarial networks (GAN, Goodfellow et al. (2014a) ) has received much attention. For K-class classification problems, Salimans et al. (2016) solves the (K + 1)-class classification problem where the additional (K + 1)th class consists of synthetic images made by a generator of the GAN learned by unlabeled data. Dai et al. (2017) notices that not a good generator but a bad generator which generates synthetic images much different from observed images is crucial for the success of semi-supervised learning. Dai et al. (2017) gives theoretical justifications of using a bad generator and develops a semi-supervised learning algorithm called Bad GAN which achieves the state-of-the-art performances over multiple benchmark datasets.

However, Bad GAN has several limitations. It needs two additional deep architectures -bad generator and pre-trained density estimation model besides the one for the classifier. Learning these multiple deep architectures requires huge computation and memory consumption. In particular, the Pixel-CNN++ (Salimans et al., 2017) is used for the pre-trained density estimation model which needs very large computational resources.

Another difficulty in Bad GAN is that it requires a two-step learning procedure -the first step is to learn the PixelCNN++ model and the second step is to learn the classifier and the bad generator. The optimal learning of the first step may not be optimal for the second step and hence the regularization of the both steps would need special techniques.

In this study, we propose a new semi-supervised learning method which competes well with other stateof-the-art semi-supervised learning algorithms and yet needs much smaller amount of computational resources. In particular, the proposed method employs only one deep architecture and hence the corresponding learning phase is much easier and faster.

Our proposed method is motivated by close investigation of VAT (Virtual Adversarial Training) method (Miyato et al., 2015; 2017) . VAT tries to find a deep classifier which has a good prediction accuracy on training data and at the same time is less sensitive to data perturbation toward the adversarial direction. Here, the adversarial direction for a given datum is the direction to which the probabilities of each class change most. In Section 3, we prove that the perturbed data toward their adversarial directions can serve as 'good' bad samples. By using the adversarial directions for both measuring the invariance and generating the bad samples, the proposed method combines the advantages of Bad GAN and VAT together. Note that only a deep architecture for classification is needed to calculate the adversarial directions and thus the corresponding learning procedure is cheaper, easier and faster. We call our proposed method FAT (Fast Adversarial Training). Dai et al. (2017) proves that bad samples play a role to pull the decision boundary toward the low density regions of data. In Section 5, we give a theoretical explanation that VAT pushes the decision boundary away from the high density regions of data. That is, FAT accelerates the learning procedure by using both pushing and pulling operations simultaneously. In section 6, we show that FAT achieves almost the state-of-the-art performances with much fewer training epochs. Especially, for the MNIST dataset, FAT achieves similar test accuracies to those of Bad GAN and VAT with 5 times and 7 times fewer training epochs, respectively. This paper is organized as follows. In Section 2, we review the Bad GAN and VAT methods briefly. In Section 3, the technique to generate bad samples using the adversarial directions is described, and our proposed semi-supervised learning method is presented in Section 4. Theoretical analysis of VAT is given in Section 5. Results of various experiments are presented in Section 6 and conclusions follow in Section 7.

2 Bad GAN AND VAT

2.1. Bad GAN APPROACH:

Bad GAN is a method that trains a good discriminator with a bad generator. This procedure trains a generator as well as a discriminator simultaneously. Let D G (φ) be generated bad samples with a bad generator p G (•; φ) parametrized by φ. Here, the 'bad generator' is a deep architecture to generate samples different from observed data. Let p pt (•) be a pre-trained density estimation model. For a given discriminator with a feature vector v(x; θ) of a given input x parameterized by θ, Bad GAN learns the bad generator by minimizing the following:

E x∼D G (φ) log p pt (x)I(p pt (x) > τ ) + ||E x∼U tr v(x; θ) -E x∼D G (φ) v(x; θ)|| 2

with respect to φ, where τ > 0 is a tuning parameter, U tr is the unlabeled data, and θ is the current estimate of θ and • is the Euclidean norm.

In turn, to train the discriminator, we consider the K-class classification problem as the (K + 1)-class classification problem where the (K + 1)-th class is an artificial label of the bad samples generated by the bad generator. We estimate the parameter θ in the discriminator by minimizing the following:

EQUATION

for given φ, where L tr is the labeled set. The second and the third terms in (1) are the cross-entropies between the unlabeled and the bad samples. The fourth term is similar the entropy of the unlabeled data which is usually helpful for semi-supervised learning (Grandvalet & Bengio, 2005) . See Dai et al. (2017) for details of the objective function (1).

2.2. VAT APPROACH:

VAT is a regularization method which is inspired by the adversarial training (Goodfellow et al., 2014b) . The regularization term of VAT is given as:

L VAT (θ; θ, x, ) = D KL p(•|x; θ)||p(•|x + r advr (x, ); θ) = - K k=1

p(k|x; θ) log p(k|x + r advr (x, ); θ) + C, where r advr (x, ) = argmax r;||r||≤

EQUATION

> 0 is a tuning parameter, θ is the parameter in the discriminator to train, θ is the current estimate of θ and C is a constant. Combining with the cross-entropy term of the labeled data, we get the final objective function of VAT:

-E x,y∼L tr [log p(y|x; θ)] + E x∼U tr L VAT (θ; θ, x, ) .

(3)

3. GENERATION OF BAD SAMPLES BY ADVERSARIAL TRAINING:

The key role of bad samples in Bad GAN is to enforce the decision boundary to be pulled toward the low density regions of the unlabeled data. To do this, bad samples must be located at the valleys of the distribution of the unlabeled data. In this section, we propose a novel technique to generate 'good' bad samples by use of only a given classifier.

3.1. MOTIVATION:

In this subsection, we explain why the adversarial direction is toward the decision boundary. For simplicity, we only consider the linear decision boundary. For the decision boundary made by the DNN model with ReLU activation function, see Appendix A.2.

Let us consider the 2-class linear logistic regression model parametrized by η = {w, b}, that is,

p(y = 1|x; η) = 1 + exp(-b -w x) -1

. Note that the decision boundary is {x : b + w x = 0}, and for any given x, the distance between x and the decision boundary is |b + w x|/||w||. The key result is that moving x toward the adversarial direction r advr (x, ) is equivalent to moving x toward the decision boundary which is stated rigorously in the following proposition. The proof is in the appendix.

Proposition 1 For a sufficiently small > 0, we have

sign(w x + b) • sign w r advr (x, ) = -1. (4) Proposition 1 implies that |b + w x|/||w|| > |b + w (x + r advr (x, )|/||w|| unless |b + w x| = 0.

Hence, we can treat x + δr advr (x, )/||r advr (x, )|| for appropriately choosing δ > 0 as a bad sample (a sample closer to the decision boundary).

3.2. BAD SAMPLE GENERATION WITH GENERAL CLASSIFIER:

Motivated by Proposition 1, we propose a bad sample generator as follows. Let δ > 0 be fixed and θ be the current estimate of θ. For a given input x and a classifier p(•|x; θ), we calculate the adversarial direction r advr (x, ) for given by (2). Then, we consider x bad = x + δr advr (x, )/ r advr (x. ) as a bad sample (a sample closer to the decision boundary). We generate bad samples for all unlabeled data. In practice, we apply the same δ to all unlabeled data and choose δ based on the validation data accuracy.

In Figure 1 , we illustrate how the bad samples generated by the proposed adversarial training are distributed for multi-class problem. With a good classifier, we can clearly see that most bad samples are located well in the low density regions of the data.

It may happen that a generated bad sample is not sufficiently close to the decision boundary to be a 'good' bad sample, in particular when δ is too large or too small. To avoid such a situation, we exclude x bad which satisfies the following condition:

max k p(k|x bad ; θ) > 1 -τ for a prespecified τ > 0.

In our experiments, we set the optimal τ with validation data.

4. FAST ADVERSARIAL TRAINING WITH BAD SAMPLES:

Once we generate bad samples by the adversarial training, FAT updates θ by minimizing the following objective function:

EQUATION

where D bad ( θ, , δ) is the set of generated bad samples with θ, and δ,

L true (θ; x) = - K k=1 exp(g k (x; θ)) 1 + K k =1 exp(g k (x; θ)) log exp(g k (x; θ)) 1 + K k =1 exp(g k (x; θ)) , L fake (θ; x) = -log 1 1 + K k=1 exp(g k (x; θ))

, g(x; θ) ∈ R K is a pre-softmax vector of a given deep architecture and λ > 0. We treat and δ as tuning parameters to be selected based on the validation data accuracy. We minimize the objective function with one of the standard optimization algorithms such as Adam (Kingma & Ba, 2014) or RMSProp (Tieleman & Hinton, 2012) .

The objective function ( 5) differs from the objective function (1) of Bad GAN in a way that the second term of (1), the cross-entropy of not being a bad sample, is deleted and the regularization term of VAT is added. We delete the second term of (1) because it can be easily shown that a perfect classifier of unlabeled data can be obtained from the minimizer of the objective function (1) without the second term under the conditions in Dai et al. (2017) . The regularization term of VAT is added to improve the bad sample generator based on adversarial training. See Section 5.1 for detailed discussions. This modification helps to improve convergence speed of the test accuracy, which will be demonstrated in the ablation experiments.

5. ROLE OF VAT FOR SEMI-SUPERVISED LEARNING:

In this section, we investigate the roles of the regularization term of VAT in our method in detail. First, we verify that VAT regularization term does help to generate 'better' bad samples with a simple experiment. Furthermore, we give a theoretical insight for the role of the regularization term of VAT in semi-supervised learning. We will show that the regularization term of VAT pushes the decision boundary from the high density regions of unlabeled data. As a result, FAT uses pushing operations by VAT term as well as pulling operations by bad samples simultaneously, which makes it possible to accelerate the training process with improved performances.

5.1. IMPROVEMENT OF BAD SAMPLES WITH VAT:

For generated samples by adversarial training to be 'good' bad samples, the adversarial directions should be toward the decision boundary. While this always happens for the linear model by Proposition 1, adversarial directions could be opposite to the decision boundary for deep model. To avoid such undesirable cases as much as possible, it would be helpful to smoothen the classifier with a regularization term. In this section, we explain that the regularization term of VAT plays such a role.

The adversarial direction obtained by maximizing the KL divergence is sensitive to local fluctuations of the class probabilities which is examplified in Figure 2 . The regularization term of VAT is helpful 

5.2. THEORETICAL ANALYSIS OF VAT:

Suppose that X is partitioned by (K + 1) mutually disjoint subsets X k for k = 1, . . . , K + 1 such that y(x) = k for all x ∈ X k , k ≤ K and p * (x) = 0 for x ∈ X K+1 , where y(x) is the ground-truth label of x and p * (x) is the true density of x. For a given feature map u : X → R m and weight vectors w 1 , . . . , w k , let p(y = k|x) ∝ exp(w k u(x)) for k = 1, . . . , K and p(y = K + 1|x) ∝ 1. Let L tr satisfy X k ∩ L tr = ∅ for k = 1, ...K. Suppose that (i) argmax h p(y = h|x) = y for x ∈ L tr , (ii) argmax h p(y = h|x) ≤ K for x ∈ ∪ K k=1 X k and (iii) argmax h p(y = h|x) = K + 1 for x ∈ X K+1 . Under these three conditions, Dai et al. (2017) proves that argmax h p(y = h|x) = y(x) for all x ∈ ∪ K k=1 X k . That is, generating large 'good' bad samples is helpful for classifying unlabeled data correctly only with a small amount of labeled data.

A similar result can be obtained for VAT objective function itself under mild additional conditions. For given two subsets A 1 and A 2 of X , we define d(A 1 , A 2 ) = min x1∈A1,x2∈A2 d(x 1 , x 2 ) for a given metric d. We define a tuple (x, x ) is -connected if d(x, x ) < . A finite subset A of X is called -connected iff for all x, x ∈ A, there exists a finite path (x, x 1 , ..., x q , x ) such that x j ∈ A for j = 1, . . . , q and(x, x 1 ), (x 1 , x 2 ), ..., (x q-1 , x q ), (x q , x ) are all -connected. Proposition 2 Assume that there exists > 0 such that 1)  and 3 ) for each k ≤ K, there exists at least one (x, k) ∈ L tr such that d({x}, U tr k ) < . Suppose that there exists a classifier f : X → {1, ..., K} such that f (x) = y for all (x, y) ∈ L tr and

U tr k = U tr ∩ X k , k = 1, . . . , K are -connected, 2) d(X k , X k ) ≥ 2 for all k = k ≤ K,

EQUATION

for all x ∈ U tr , where B(x, ) = {x : d(x, x ) ≤ }. Then, the function f classifies the unlabeled set perfectly, that is:

f (x) = y(x) for all x ∈ U tr .

Condition 1) and 2) mean that the unlabeled data are dense enough and the supports of each class are separated sufficiently, respectively. Condition 3) assumes that at least one labeled instance exists near the supports of each class. The main condition of Proposition 2 is (6) which essentially assumes that the classifier f does not change much locally. That is, f is invariant with respect to all small perturbations of an input x. Note that the regularization term of VAT is devised to improve the invariancy of the classifier and Proposition 2 explains why improving the invariancy is helpful for semi-supervised learning.

5.3. INTERPRETATION OF VAT:

Let f (x; θ) = argmax h p(y = h|x; θ). Proposition 2 implies that it would be good to pursue a classifier which predicts the labeled data correctly and at the same time is invariant with respect to all local perturbations on the unlabeled data. For this purpose, a plausible candidate of the objective function is

EQUATION

Note that a classifier f which achieves 0 value of the objective function ( 7) satisfies the conditions of Proposition 2 and thus classifies all unlabeled data correctly.

The objective function ( 7) is not practically usable since neither optimizing the indicator function nor checking f (x; θ) = f (x ; θ) for all x in B(x, ) is possible. To resolve these problems, we replace the indicator functions in (7) with the cross-entropies, and the neighborhood B(x, ) in the second term with the adversarial direction. By doing so, we have the following alternative objective function: The condition (6) in Proposition 2 means that the decision boundary is not located inside the support X k of each class. That is, the regularization term of VAT prevents the decision boundary from being located at the high density regions of data or equivalently pushes the decision boundary from the high density regions of data.

-E x,

6. EXPERIMENTS:



6.1. PREDICTION PERFORMANCES IN SEMI-SUPERVISED LEARNING:

We compare prediction performances of FAT over the benchmark datasets with other semi-supervised learning algorithms. We consider the most widely used datasets: MNIST (LeCun et al., 1998) , SVHN (Marlin et al., 2010) and CIFAR10 (Krizhevsky & Hinton, 2009) . As in done by other works, we randomly sample 100, 1000 and 4000 labeled data from the MNIST, SVHN and CIFAR10 datasets, respectively and use them as the labeled data and the rest as the unlabeled data. For fair comparison, we use the same architectures as those used in Miyato et al. (2017) for MNIST, SVHN and CIFAR10.

The optimal tuning parameters ( , τ, δ) in FAT are chosen based on the validation data accuracy. We use Adam algorithm (Kingma & Ba, 2014) to update the parameters and do not use any data augmentation techniques. The results are summarized in Table 1 , which shows that FAT achieves the state-of-the-art accuracy for SVHN dataset and competitive accuracies with the state-of-the-art method (i.e. Bad GAN) for MNIST and CIFAR10.

We conduct another experiments where the numbers of labeled data are much smaller and consider a more complex dataset CIFAR100 (Krizhevsky & Hinton, 2009) , whose results are summarized in Table 2 . Note that FAT still dominates mod. VAT for the all datasets and the margins become larger. While FAT achieves the state-of-the-art performances for MNIST and SVHN, its performance degrades much for CIFAR10 compared to the accuracy for the case of 4000 labeled data. Note that the quality of bad samples depends on the quality of the estimated classifier. For CIFAR10 which is relatively more complex than MNIST and SVHN, the quality of the estimated classifier is influenced much to the amount of labeled data and thus the accuracy of FAT is more sensitive to the amount of labeled data. However, it is interesting to see that FAT is superior to mod. VAT for CIFAR10 and CIFAR100, which suggests that bad samples are helpful for complex problems where the quality of bad samples might not be sufficiently good.

The other advantage of FAT is its stability with respect to learning phase. With small labeled data, the test accuracies of each epoch tends to fluctuate much for VAT and mod VAT, while FAT provides much more stable results. See Figure 7 in Appendix. This may be partly because 'good' bad samples keep the classifier from fluctuation. FAT introduces two additional tuning parameters τ and δ compared to VAT, where τ is used to determine whether a bad sample is 'good' and δ is the radius to generate bad samples. We investigate the sensitivities of prediction performances with respect to the changes of the values of τ and δ with being fixed at 1.5 that is the optimal value. The results are reported in Table 3 . Unless τ is too small or too large, the prediction performances are not changed much. For δ, care should be done. Too small δ, smaller than , hampers the prediction performance much. Apparently, a similar value of δ to that of (i.e. δ = 2) gives the best result.

6.3. OBJECTIVE FUNCTION ANALYSIS:

In this subsection, we analyze the effects of L true and L VAT in this section. We do not consider the cross-entropy of the labeled data and L fake since they are necessary to reflect the idea utilizing bad data. Table 4 compares the performances without one or both of these terms. Note that L VAT is necessary for superior performances, which is because L VAT is indispensable to generate 'good' bad samples as explained in Section 5.1. On the other hand, the term L true , which is devised to separate unlabeled data from bad samples, is not always helpful. A possible explanation is that sometimes the term L true would give an undesirable effect on the misclassified data.

6.4. COMPUTATIONAL EFFICIENCY:

We investigate the computational efficiency of our method in view of learning speed and computation time per training epoch. For Bad GAN, we did not use PixelCNN++ on SVHN and CIFAR10 datasets since the pre-trained PixelCNN++ models are not publicly available. Figure 4 draws the bar plots about the numbers of epochs needed to achieve the prespecified test accuracies. We can clearly see that FAT requires much less epochs We also calculate the ratios of the computing time of each semi-supervised learning algorithm over the computing time of the corresponding supervised learning algorithm for CIFAR10 dataset, whose results are summarized in Table 5 . These ratios are almost same for different datasets. The computation time of FAT is competitive to VAT and mod. VAT, and hence we can conclude that FAT arrives at the prespecified performances much efficiently. Note that PixelCNN++ is not used for this experiment, and so comparison of computing time of FAT and Bad GAN with PixelCNN++ is meaningless We investigate how 'good' bad samples generated by the adversarial training are. The upper panel of Figure 5 shows the scatter plot of the synthetic data and the trace plot of prediction accuracies of FAT and VAT. And the lower panel of Figure 5 draws the scatter plots with generated bad samples at various epochs. We can clearly see that bad samples move to lower density regions as the epoch increases, which amply demonstrates that the adversarial training is good at generating bad samples. We also compare bad images of the MNIST data generated by FAT and Bad GAN in Figure 6 . The bad images by FAT do not look like real images and do not seem to be collapsed, which indicates that FAT consistently generates diverse and good bad samples. Bad GAN also generates diverse bad samples but some 'realistic' images can be found.

7. CONCLUSION:

In this paper, we propose a new method called FAT for semi-supervised learning which generates bad samples only with a given classifier. The objective function of FAT is devised to compromise the advantages of Bad GAN and VAT together, which makes FAT be faster and more accurate. In numerical experiments, we show that FAT achieves almost the state-of-the-art performances with much fewer epochs. Unlike Bad GAN , FAT only needs to learn a discriminator. Hence, it could be extended without much effort to other learning problems. For example, FAT can be modified easily for recurrent neural networks and hence can be applied to sequential data. We will leave this extension as a future work.

It would be useful to combine FAT with a generative approach such as Bad GAN, in particular when the initial classifier is bad. Since FAT uses only a classifier to generate bad samples, the initial estimate of the classifier would be important. Using the generator model learned by large unlabeled data would be helpful to find a good initial estimate of the classifier. For given x, suppose g(x; θ) > 0. If g(x : θ) is estimated reasonably, we expect that g(x; θ) is decreasing if x moves toward the decision boundary. A formal statement of this expectation would be that x -r∇ x g(x; θ) can arrive at the decision boundary for a finite value of r > 0, where ∇ x is the gradient with respect to x. Of course, for x with g(x; θ) < 0, we expect that x + r∇ x g(x; θ) can arrive at the decision boundary for a finite value of r > 0. We say that x is normal if there is r > 0 such that x -r∇ x g(x : θ)sign{g(x : θ)} locates at the decision boundary. We say that a linear region A j is normal if all x in A j are normal. We expect that most of A j are normal if g(x; θ) is reasonably estimated so that the probability decreases or increases depending on sign{g(x : θ)} if x is getting closer to the decision boundary.

The following proposition proves that the adversarial direction is toward the decision boundary for all xs in normal linear regions.

Proposition 3 If a linear region A j is normal. Then for any x ∈ int(A j ), there exists > 0 and δ > 0 such that x bad = x + δr advr (x, )/||r advr (x, )|| is on the decision boundary.

Proof. Take ˜ > 0 such that x + r ∈ A j for ∀r ∈ B(x, ˜ ). Then by Proposition 1, there exists 0 < * < ˜ such that for ∀0 < < * , r advr (x, ) =

• sign(-b j -w j x)) • w j ||w j || ∝ -∇ x g(x; θ)sign{g(x : θ)}.

x is normal, thus there exists δ > 0 such that x bad = x + δr advr (x, )/||r advr (x, )|| belongs to the decision boundary.

A.3 PROOF OF PROPOSITION 2:

It suffices to show that ∀x, x ∈ U tr k , f (x) = f (x ) for all k = 1, ..., K. For given U tr k , there exists (x, ỹ) ∈ L tr such that d({x}, U tr k ) < . So U tr k ∪ {x} is -connected. That is, for any x ∈ U tr k , there exists a path (x, x 1 , ..., x q , x) such that (x, x 1 ), ..., (x q , x) are all -connected. Therefore y(x) = y(x) = ỹ for ∀x ∈ U tr k , and the proof is done.

A.4 FIGURE 

:



Figures and Tables:

Figure 1: Demonstration of how the bad samples generated by the adversarial training are distributed. We consider two cases: 3-class classification problem (Left) and 4-class classification problem (Right). True data and bad data are coloured by blue and orange, respectively.

Miyato et al. (2017) proposes the fast approximation method to calculate the adversarial direction r advr (x, ) by using the second-order Taylor expansion. Let us defineH(x, θ) = ∇∇D KL p(•|x; θ)||p(•|x + r; θ) | r=0 .They claim that r advr emerges as the first dominant eigenvector u(x, θ) of H(x, θ) with magnitude . But there always exist two dominant eigenvectors, ±u(x, θ), and the sign should be selected carefully. So, we slightly modify the approximation method of Miyato et al. (2017) by r advr (x, ) = argmax r∈{u(x, θ),-u(x, θ)} D KL p(•|x; θ)||p(•|x + r; θ) .

Figure 2: Examples of P (y = 1|x) of smooth (Left) and wiggle (Right) cases. We plot 3 points and their adversarial directions on each case.

Figure 3: (Upper) 10 randomly sampled original MNIST dataset. (Middle and Lower) Bad samples obtained by the classifier learned with and without the regularization term of VAT.

y∼L tr [log p(y|x; θ)] -E x∼U tr K k=1 p(k|x; θ) log p(k|x + r advr (x, ); θ) . (8) Finally, we replace p(•|x; θ) in the second term of (8) by p(•|x; θ) to have the objective function of VAT (3).

Figure 4: The number of epochs to achieve the prespecified test accuracies (98%, 90% and 80%) with the four methods for (Left) MNIST, (Middle) SVHN and (Right) CIFAR10 datasets. Bad GAN is operated without PixelCNN++ for SVHN and CIFAR10 datasets.

Figure 5: (Upper Left) The scatter plot of synthetic data which consist of 1000 unlabeled data (gray) and 4 labeled data for each class (red and blue with black edge). (Upper Right) Accuracies of unlabeled data for each epochs for VAT and FAT. We use 2-layered NN with 100 hidden units each. (Lower) Bad samples and classified unlabeled data by colors at the different training epochs of FAT

Figure 7: Trace plot of the test accuracies with the four methods for MNIST dataset. 20 randomly sampled data are used as the labeled data and the rest are used as the unlabeled data.

Comparison of prediction accuracies of various semi-supervised learning algorithms for the three benchmark datasets. mod. VAT is the modified version of VAT stated in Section 4.

Table content: <table><tr><td>Method</td><td/><td>Test acc.(%)</td><td/></tr><tr><td/><td colspan="3">MNIST(100) SVHN(1000) CIFAR10(4000)</td></tr><tr><td>DGN (Kingma et al., 2014)</td><td>96.67</td><td>63.98</td><td>-</td></tr><tr><td>Ladder (Rasmus et al., 2015)</td><td>98.94</td><td>-</td><td>79.6</td></tr><tr><td>ALI (Donahue et al., 2016)</td><td>-</td><td>92.58</td><td>82.01</td></tr><tr><td>FM-GAN (Salimans et al., 2016)</td><td>99.07</td><td>91.89</td><td>81.37</td></tr><tr><td>FM-GAN-Tan (Kumar et al., 2017)</td><td>-</td><td>95.61</td><td>83.80</td></tr><tr><td>Bad GAN (Dai et al., 2017)</td><td>99.20</td><td>95.75</td><td>85.59</td></tr><tr><td>VAT (Miyato et al., 2017)</td><td>98.64</td><td>93.17</td><td>85.13</td></tr><tr><td>CrossEnt (use all data)</td><td>98.82</td><td>96.74</td><td>90.31</td></tr><tr><td>CrossEnt (use lab. data only)</td><td>79.16</td><td>88.91</td><td>67.45</td></tr><tr><td>mod. VAT</td><td>98.70</td><td>94.69</td><td>85.18</td></tr><tr><td>FAT</td><td>98.89</td><td>95.94</td><td>85.31</td></tr></table>

Comparison of prediction accuracies with small labeled data and more complex dataset.

Table content: <table><tr><td>Method</td></tr></table>

Test accuracies of MNIST for various values of τ and δ. The other two parameters on each case are fixed to the optimal values.

Table content: <table><tr><td>τ</td><td>0.001 0.01</td><td>0.1</td><td>0.2</td><td>δ</td><td>1</td><td>2.</td><td>4.</td><td>6.</td></tr><tr><td colspan="4">Test acc. 98.65 98.89 98.77 98.71</td><td colspan="5">Test acc. 89.54 98.89 98.79 98.55</td></tr></table>

Test accuracies for diverse objective loss functions.

Table content: <table><tr><td>Data</td><td colspan="3">MNIST SVHN CIFAR10</td></tr><tr><td>Setting</td><td/><td>Test acc.(%)</td><td/></tr><tr><td>L true + L fake + L VAT</td><td>98.89</td><td>95.94</td><td>85.15</td></tr><tr><td>L true + L fake</td><td>83.6</td><td>90.21</td><td>68.32</td></tr><tr><td>L fake + L VAT</td><td>98.77</td><td>95.71</td><td>85.31</td></tr><tr><td>6.2 EFFECTS OF TUNING PARAMETERS</td><td/><td/><td/></tr></table>

Learning time per training epoch ratios compared to supervised learning with cross-entropy for CIFAR10. Bad GAN is operated without PixelCNN++.

Table content: <table><tr><td>Method</td><td colspan="4">VAT mod. VAT FAT Bad GAN</td></tr><tr><td colspan="2">Time ratio 1.37</td><td>1.62</td><td>2.09</td><td>3.20</td></tr><tr><td colspan="5">6.5 QUALITY OF BAD SAMPLES GENERATED BY ADVERSARIAL TRAINING</td></tr></table>

Peer review (iclr2019):
 This paper makes the interesting observation that the generative procedure proposed by Bad GAN paper can be replaced by a slightly modified VAT procedure. The reasoning is sound and leverages the intuition that adversarial examples (subject to a sufficiently small perturbation radius) are likely to be closer to a decision boundary than the original sample. The paper is generally easy to follow but the presentation could be improved. In particular more could be done to describe the terms in Equation 5. I’m also curious about the behavior of L^true, which is equivalently the fourth term in Eq 1. Even when reading Bad GAN paper, I did not quite understand their claim that this can be correctly interpreted as a conditional entropy term (if they really wanted conditional entropy, they should probably have either done H(p(k|x)) or H(p(k|x, k <= K))). I agree with the authors that the roles of the second and fourth terms overlapped, and I think this is sufficiently interesting to warrant some further elaboration in the paper. I also liked the reminder that power iteration selects a non-unique sign for the first eigenvector (subject to the random vector initialization); I encourage the authors to do an ablation test to convince the reader that “this modification helps to improve convergence speed of the test accuracy.” The propositions in this paper were, in my opinion, not particularly insightful. While I think it is nice that the authors went through the effort of providing some formalism to the intuition that VAT has a “push decision boundary away from high-density regions”, I’m less sure if propositions 1 and 2 really provides any additional insight the behavior of VAT. Proposition 1 is pretty weak in that it only covers a 2-class logistic regression; it seems obvious that the adversarial perturbation points in the direction toward the decision hyperplane. If the authors could extend this to more general non-linear classifiers (perhaps subject to some assumptions), that would be more interesting. I don’t think Proposition 2 has any real value and recommend its relegation to the appendix. I think the biggest weakness of this paper is the experiments. Taking Table 1 at face value, the conclusion that FAT is simply competitive with existing approaches suggests that the additional machinery isn’t particularly useful, providing little more than a vanilla VAT. I also think MNIST/SVHN has run its course as good semi-supervised learning benchmarks and would prefer to see such algorithms being scaled to more complex data. The main argument for why FAT should be prefered over VAT comes from Section 6.2. Figure 4 is more interesting, but is complicated by the fact that FAT checks both possible eigenvectors (+/- u) during training, which requires two forward passes in the classifier; did the authors give a similar treatment to VAT? Please show wall-clock time too. Unfortunately the computational efficiency gain seems to only hold true for MNIST/SVHN, but not for CIFAR. I worry that the observed gains will not sustain once we move to more complicated datasets. Pros: + Simple and clean proposal + Easy to read Cons: - Limited insight - Weak experiments