Year: 2018

Abstract:
In semi-supervised learning, Bad GAN approach is one of the most attractive method due to the intuitional simplicity and powerful performances. Bad GAN learns a classifier with bad samples distributed on complement of the support of the input data. But Bad GAN needs additional architectures, a generator and a density estimation model, which involves huge computation and memory consumption cost. VAT is another good semi-supervised learning algorithm, which utilizes unlabeled data to improve the invariance of the classifier with respect to perturbation of inputs. In this study, we propose a new method by combining the ideas of Bad GAN and VAT. The proposed method generates bad samples of high-quality by use of the adversarial training used in VAT. We give theoretical explanations why the adversarial training is good at both generating bad samples and semi-supervised learning. An advantage of the proposed method is to achieve the competitive performances with much fewer computations. We demonstrate advantages our method by various experiments with well known benchmark image datasets.

text: In semi-supervised learning, Bad GAN approach is one of the most attractive method due to the intuitional simplicity and powerful performances. Bad GAN learns a classifier with bad samples distributed on complement of the support of the input data. But Bad GAN needs additional architectures, a generator and a density estimation model, which involves huge computation and memory consumption cost. VAT is another good semi-supervised learning algorithm, which utilizes unlabeled data to improve the invariance of the classifier with respect to perturbation of inputs. In this study, we propose a new method by combining the ideas of Bad GAN and VAT. The proposed method generates bad samples of high-quality by use of the adversarial training used in VAT. We give theoretical explanations why the adversarial training is good at both generating bad samples and semi-supervised learning. An advantage of the proposed method is to achieve the competitive performances with much fewer computations. We demonstrate advantages our method by various experiments with well known benchmark image datasets.

heading: 1 INTRODUCTION
text: Deep learning has accomplished unprecedented success due to the development of deep architectures, learning techniques and hardwares (Krizhevsky et al., 2012; Ioffe & Szegedy, 2015; Szegedy et al., 2015; Hinton et al., 2012; Kingma & Ba, 2014). However, deep learning has also suffered from collecting large amount of labeled data which requires both cost and time. Thus it becomes important to develop semi-supervised methodologies that learn a classifier (or discriminator) by using small labeled data and large unlabeled data.
Various semi-supervised learning methods have been proposed for deep learning. Weston et al. (2012) employs a manifold embedding technique using the pre-constructed graph of unlabeled data and Rasmus et al. (2015) uses a specially designed auto-encoder to extract essential features for classification. Variational auto encoder (Kingma & Welling, 2013) is also used in the context of semi-supervised learning by maximizing the variational lower bound of both labeled and unlabeled data (Kingma et al., 2014; Maaløe et al., 2016).
Recently, semi-supervised learning based on generative adversarial networks (GAN, Goodfellow et al. (2014a)) has received much attention. For K-class classification problems, Salimans et al. (2016) solves the (K + 1)-class classification problem where the additional (K + 1)th class consists of synthetic images made by a generator of the GAN learned by unlabeled data. Dai et al. (2017) notices that not a good generator but a bad generator which generates synthetic images much different from observed images is crucial for the success of semi-supervised learning. Dai et al. (2017) gives theoretical justifications of using a bad generator and develops a semi-supervised learning algorithm called Bad GAN which achieves the state-of-the-art performances over multiple benchmark datasets.
However, Bad GAN has several limitations. It needs two additional deep architectures - bad generator and pre-trained density estimation model besides the one for the classifier. Learning these multiple deep architectures requires huge computation and memory consumption. In particular, the PixelCNN++ (Salimans et al., 2017) is used for the pre-trained density estimation model which needs very large computational resources.
Another difficulty in Bad GAN is that it requires a two-step learning procedure - the first step is to learn the PixelCNN++ model and the second step is to learn the classifier and the bad generator. The optimal learning of the first step may not be optimal for the second step and hence the regularization of the both steps would need special techniques.
In this study, we propose a new semi-supervised learning method which competes well with other stateof-the-art semi-supervised learning algorithms and yet needs much smaller amount of computational resources. In particular, the proposed method employs only one deep architecture and hence the corresponding learning phase is much easier and faster.
Our proposed method is motivated by close investigation of VAT (Virtual Adversarial Training) method (Miyato et al., 2015; 2017). VAT tries to find a deep classifier which has a good prediction accuracy on training data and at the same time is less sensitive to data perturbation toward the adversarial direction. Here, the adversarial direction for a given datum is the direction to which the probabilities of each class change most. In Section 3, we prove that the perturbed data toward their adversarial directions can serve as ‘good’ bad samples. By using the adversarial directions for both measuring the invariance and generating the bad samples, the proposed method combines the advantages of Bad GAN and VAT together. Note that only a deep architecture for classification is needed to calculate the adversarial directions and thus the corresponding learning procedure is cheaper, easier and faster. We call our proposed method FAT (Fast Adversarial Training).
Dai et al. (2017) proves that bad samples play a role to pull the decision boundary toward the low density regions of data. In Section 5, we give a theoretical explanation that VAT pushes the decision boundary away from the high density regions of data. That is, FAT accelerates the learning procedure by using both pushing and pulling operations simultaneously. In section 6, we show that FAT achieves almost the state-of-the-art performances with much fewer training epochs. Especially, for the MNIST dataset, FAT achieves similar test accuracies to those of Bad GAN and VAT with 5 times and 7 times fewer training epochs, respectively.
This paper is organized as follows. In Section 2, we review the Bad GAN and VAT methods briefly. In Section 3, the technique to generate bad samples using the adversarial directions is described, and our proposed semi-supervised learning method is presented in Section 4. Theoretical analysis of VAT is given in Section 5. Results of various experiments are presented in Section 6 and conclusions follow in Section 7.
2 Bad GAN AND VAT
2.1 Bad GAN APPROACH
Bad GAN is a method that trains a good discriminator with a bad generator. This procedure trains a generator as well as a discriminator simultaneously. Let DG(φ) be generated bad samples with a bad generator pG(·;φ) parametrized by φ. Here, the ‘bad generator’ is a deep architecture to generate samples different from observed data. Let ppt(·) be a pre-trained density estimation model. For a given discriminator with a feature vector v(x; θ) of a given input x parameterized by θ, Bad GAN learns the bad generator by minimizing the following:
Ex∼DG(φ) [ log ppt(x)I(ppt(x) > τ) ] + ||Ex∼Utrv(x; θ̂)− Ex∼DG(φ)v(x; θ̂)|| 2
with respect to φ, where τ > 0 is a tuning parameter, U tr is the unlabeled data, and θ̂ is the current estimate of θ and ‖ · ‖ is the Euclidean norm.
In turn, to train the discriminator, we consider the K-class classification problem as the (K+1)-class classification problem where the (K + 1)-th class is an artificial label of the bad samples generated by the bad generator. We estimate the parameter θ in the discriminator by minimizing the following:
−Ex,y∼Ltr [log p(y|x, y ≤ K; θ)]− Ex∼Utr [ log { K∑ k=1 p(k|x; θ) }]
−Ex∼DG(φ) [log p(K + 1|x; θ)]− Ex∼Utr [ K∑ k=1 p(k|x; θ) log p(k|x; θ) ] (1)
for given φ, where Ltr is the labeled set. The second and the third terms in (1) are the cross-entropies between the unlabeled and the bad samples. The fourth term is similar the entropy of the unlabeled data which is usually helpful for semi-supervised learning (Grandvalet & Bengio, 2005). See Dai et al. (2017) for details of the objective function (1).
2.2 VAT APPROACH
VAT is a regularization method which is inspired by the adversarial training (Goodfellow et al., 2014b). The regularization term of VAT is given as:
LVAT(θ; θ̂, x, ) = DKL ( p(·|x; θ̂)||p(·|x+ radvr(x, ); θ) ) = −
K∑ k=1 p(k|x; θ̂) log p(k|x+ radvr(x, ); θ) + C,
where radvr(x, ) = argmax
r;||r||≤ DKL
( p(·|x; θ̂)||p(·|x+ r; θ̂) ) , (2)
> 0 is a tuning parameter, θ is the parameter in the discriminator to train, θ̂ is the current estimate of θ and C is a constant. Combining with the cross-entropy term of the labeled data, we get the final objective function of VAT:
−Ex,y∼Ltr [log p(y|x; θ)] + Ex∼Utr [ LVAT(θ; θ̂, x, ) ] . (3)

heading: 3 GENERATION OF BAD SAMPLES BY ADVERSARIAL TRAINING
text: The key role of bad samples in Bad GAN is to enforce the decision boundary to be pulled toward the low density regions of the unlabeled data. To do this, bad samples must be located at the valleys of the distribution of the unlabeled data. In this section, we propose a novel technique to generate ‘good’ bad samples by use of only a given classifier.

heading: 3.1 MOTIVATION
text: In this subsection, we explain why the adversarial direction is toward the decision boundary. For simplicity, we only consider the linear decision boundary. For the decision boundary made by the DNN model with ReLU activation function, see Appendix A.2.
Let us consider the 2-class linear logistic regression model parametrized by η = {w, b}, that is, p(y = 1|x; η) = ( 1 + exp(−b− w′x) )−1 . Note that the decision boundary is {x : b+ w′x = 0}, and for any given x, the distance between x and the decision boundary is |b+ w′x|/||w||. The key
result is that moving x toward the adversarial direction radvr(x, ) is equivalent to moving x toward the decision boundary which is stated rigorously in the following proposition. The proof is in the appendix.
Proposition 1 For a sufficiently small > 0, we have
sign(w ′ x+ b) · sign ( w ′ radvr(x, ) ) = −1. (4)
Proposition 1 implies that |b + w′x|/||w|| > |b + w′(x + radvr(x, )|/||w|| unless |b + w′x| = 0. Hence, we can treat x+ δradvr(x, )/||radvr(x, )|| for appropriately choosing δ > 0 as a bad sample (a sample closer to the decision boundary).

heading: 3.2 BAD SAMPLE GENERATION WITH GENERAL CLASSIFIER
text: Motivated by Proposition 1, we propose a bad sample generator as follows. Let δ > 0 be fixed and θ̂ be the current estimate of θ. For a given input x and a classifier p(·|x; θ̂), we calculate the adversarial direction radvr(x, ) for given by (2). Then, we consider xbad = x+ δradvr(x, )/‖radvr(x. )‖ as a bad sample (a sample closer to the decision boundary). We generate bad samples for all unlabeled data. In practice, we apply the same δ to all unlabeled data and choose δ based on the validation data accuracy.
In Figure 1, we illustrate how the bad samples generated by the proposed adversarial training are distributed for multi-class problem. With a good classifier, we can clearly see that most bad samples are located well in the low density regions of the data.
It may happen that a generated bad sample is not sufficiently close to the decision boundary to be a ’good’ bad sample, in particular when δ is too large or too small. To avoid such a situation, we exclude xbad which satisfies the following condition:
max k
p(k|xbad; θ̂) > 1− τ
for a prespecified τ > 0. In our experiments, we set the optimal τ with validation data.

heading: 4 FAST ADVERSARIAL TRAINING WITH BAD SAMPLES
text: Once we generate bad samples by the adversarial training, FAT updates θ by minimizing the following objective function:
−Ex,y∼Ltr [log p(y|x; θ)] + Ex∼Utr [ Ltrue(θ;x) ] + Ex∼Dbad(θ̂, ,δ) [ Lfake(θ, x) ] +Ex∼Utr [ LVAT(θ; θ̂, x, ) ] (5)
where Dbad(θ̂, , δ) is the set of generated bad samples with θ̂, and δ,
Ltrue(θ;x) = − K∑ k=1
[ exp(gk(x; θ))
1 + ∑K k′=1 exp(gk′ (x; θ)) log exp(gk(x; θ)) 1 + ∑K k′=1 exp(gk′ (x; θ))
] ,
Lfake(θ;x) = − log 1 1 + ∑K k=1 exp(gk(x; θ)) ,
g(x; θ) ∈ RK is a pre-softmax vector of a given deep architecture and λ > 0. We treat and δ as tuning parameters to be selected based on the validation data accuracy. We minimize the objective function with one of the standard optimization algorithms such as Adam (Kingma & Ba, 2014) or RMSProp (Tieleman & Hinton, 2012).
The objective function (5) differs from the objective function (1) of Bad GAN in a way that the second term of (1), the cross-entropy of not being a bad sample, is deleted and the regularization term of VAT is added. We delete the second term of (1) because it can be easily shown that a perfect classifier of unlabeled data can be obtained from the minimizer of the objective function (1) without the second term under the conditions in Dai et al. (2017). The regularization term of VAT is added to improve the bad sample generator based on adversarial training. See Section 5.1 for detailed discussions.
Miyato et al. (2017) proposes the fast approximation method to calculate the adversarial direction radvr(x, ) by using the second-order Taylor expansion. Let us define H(x, θ̂) = ∇∇DKL ( p(·|x; θ̂)||p(·|x+ r; θ̂) ) |r=0. They claim that radvr emerges as the first dominant eigenvec-
tor u(x, θ̂) of H(x, θ̂) with magnitude . But there always exist two dominant eigenvectors,±u(x, θ̂), and the sign should be selected carefully. So, we slightly modify the approximation method of Miyato et al. (2017) by
radvr(x, ) = argmax r∈{u(x,θ̂),−u(x,θ̂)} DKL
( p(·|x; θ̂)||p(·|x+ r; θ̂) ) .
This modification helps to improve convergence speed of the test accuracy, which will be demonstrated in the ablation experiments.
5 ROLE OF VAT FOR SEMI-SUPERVISED LEARNING
In this section, we investigate the roles of the regularization term of VAT in our method in detail. First, we verify that VAT regularization term does help to generate ’better’ bad samples with a simple experiment. Furthermore, we give a theoretical insight for the role of the regularization term of VAT in semi-supervised learning. We will show that the regularization term of VAT pushes the decision boundary from the high density regions of unlabeled data. As a result, FAT uses pushing operations by VAT term as well as pulling operations by bad samples simultaneously, which makes it possible to accelerate the training process with improved performances.
5.1 IMPROVEMENT OF BAD SAMPLES WITH VAT
For generated samples by adversarial training to be ‘good’ bad samples, the adversarial directions should be toward the decision boundary. While this always happens for the linear model by Proposition 1, adversarial directions could be opposite to the decision boundary for deep model. To avoid such undesirable cases as much as possible, it would be helpful to smoothen the classifier with a regularization term. In this section, we explain that the regularization term of VAT plays such a role.
The adversarial direction obtained by maximizing the KL divergence is sensitive to local fluctuations of the class probabilities which is examplified in Figure 2. The regularization term of VAT is helpful
to find a right adversarial direction which is toward the decision boundary by eliminating unnecessary local fluctuations of the class probabilities. In Figure 3, we compare bad samples generated by the adversarial training with and without the regularization term of VAT for the MNIST dataset. While the bad samples generated without the regularization term of VAT are visually similar to the given input vectors, the bad samples generated with the regularization term of VAT look like mixtures of two different digits and thus serve as ‘better’ bad samples.
5.2 THEORETICAL ANALYSIS OF VAT
Suppose that X is partitioned by (K + 1) mutually disjoint subsets Xk for k = 1, . . . ,K + 1 such that y(x) = k for all x ∈ Xk, k ≤ K and p∗(x) = 0 for x ∈ XK+1, where y(x) is the ground-truth label of x and p∗(x) is the true density of x. For a given feature map u : X → Rm and weight vectors w1, . . . , wk, let p(y = k|x) ∝ exp(w ′
ku(x)) for k = 1, . . . ,K and p(y = K + 1|x) ∝ 1. Let Ltr satisfy Xk ∩ Ltr 6= ∅ for k = 1, ...K. Suppose that (i) argmaxhp(y = h|x) = y for x ∈ Ltr, (ii) argmaxhp(y = h|x) ≤ K for x ∈ ∪Kk=1Xk and (iii) argmaxhp(y = h|x) = K + 1 for x ∈ XK+1. Under these three conditions, Dai et al. (2017) proves that argmaxhp(y = h|x) = y(x) for all x ∈ ∪Kk=1Xk. That is, generating large ‘good’ bad samples is helpful for classifying unlabeled data correctly only with a small amount of labeled data.
A similar result can be obtained for VAT objective function itself under mild additional conditions. For given two subsets A1 and A2 of X , we define d(A1,A2) = minx1∈A1,x2∈A2 d(x1, x2) for a given metric d. We define a tuple (x, x ′ ) is -connected if d(x, x ′ ) < . A finite subset A of X is called -connected iff for all x, x ′ ∈ A, there exists a finite path (x, x1, ..., xq, x ′ ) such that xj ∈ A for j = 1, . . . , q and(x, x1), (x1, x2), ..., (xq−1, xq), (xq, x ′ ) are all -connected.
Proposition 2 Assume that there exists > 0 such that 1) U trk = U tr ∩ Xk, k = 1, . . . ,K are -connected, 2) d(Xk,Xk′ ) ≥ 2 for all k 6= k
′ ≤ K, and 3) for each k ≤ K, there exists at least one (x, k) ∈ Ltr such that d({x},U trk ) < . Suppose that there exists a classifier f : X → {1, ...,K} such that f(x) = y for all (x, y) ∈ Ltr and
f(x) = f(x ′ ) for all x ′ ∈ B(x, ) (6)
for all x ∈ U tr, where B(x, ) = {x′ : d(x, x′) ≤ }. Then, the function f classifies the unlabeled set perfectly, that is:
f(x) = y(x) for all x ∈ U tr.
Condition 1) and 2) mean that the unlabeled data are dense enough and the supports of each class are separated sufficiently, respectively. Condition 3) assumes that at least one labeled instance exists near the supports of each class. The main condition of Proposition 2 is (6) which essentially assumes that the classifier f does not change much locally. That is, f is invariant with respect to all small perturbations of an input x. Note that the regularization term of VAT is devised to improve the invariancy of the classifier and Proposition 2 explains why improving the invariancy is helpful for semi-supervised learning.
5.3 INTERPRETATION OF VAT
Let f(x; θ) = argmaxhp(y = h|x; θ). Proposition 2 implies that it would be good to pursue a classifier which predicts the labeled data correctly and at the same time is invariant with respect to all local perturbations on the unlabeled data. For this purpose, a plausible candidate of the objective function is
E(x,y)∼Ltr [I(y 6= f(x; θ)] + Ex∼Utr [ I ( f(x; θ) 6= f(x ′ ; θ) for ∀x ′ ∈ B(x, ) )] . (7)
Note that a classifier f which achieves 0 value of the objective function (7) satisfies the conditions of Proposition 2 and thus classifies all unlabeled data correctly.
The objective function (7) is not practically usable since neither optimizing the indicator function nor checking f(x; θ) 6= f(x′ ; θ) for all x′ in B(x, ) is possible. To resolve these problems, we replace the indicator functions in (7) with the cross-entropies, and the neighborhood B(x, ) in the second term with the adversarial direction. By doing so, we have the following alternative objective function:
− Ex,y∼Ltr [log p(y|x; θ)]− Ex∼Utr [ K∑ k=1 p(k|x; θ) log p(k|x+ radvr(x, ); θ) ] . (8)
Finally, we replace p(·|x; θ) in the second term of (8) by p(·|x; θ̂) to have the objective function of VAT (3).
The condition (6) in Proposition 2 means that the decision boundary is not located inside the support Xk of each class. That is, the regularization term of VAT prevents the decision boundary from being located at the high density regions of data or equivalently pushes the decision boundary from the high density regions of data.

heading: 6 EXPERIMENTS

heading: 6.1 PREDICTION PERFORMANCES IN SEMI-SUPERVISED LEARNING
text: We compare prediction performances of FAT over the benchmark datasets with other semi-supervised learning algorithms. We consider the most widely used datasets: MNIST (LeCun et al., 1998), SVHN (Marlin et al., 2010) and CIFAR10 (Krizhevsky & Hinton, 2009). As in done by other works, we randomly sample 100, 1000 and 4000 labeled data from the MNIST, SVHN and CIFAR10 datasets, respectively and use them as the labeled data and the rest as the unlabeled data. For fair comparison, we use the same architectures as those used in Miyato et al. (2017) for MNIST, SVHN and CIFAR10. The optimal tuning parameters ( , τ, δ) in FAT are chosen based on the validation data accuracy.
We use Adam algorithm (Kingma & Ba, 2014) to update the parameters and do not use any data augmentation techniques. The results are summarized in Table 1, which shows that FAT achieves the state-of-the-art accuracy for SVHN dataset and competitive accuracies with the state-of-the-art method (i.e. Bad GAN) for MNIST and CIFAR10.
We conduct another experiments where the numbers of labeled data are much smaller and consider a more complex dataset CIFAR100 (Krizhevsky & Hinton, 2009), whose results are summarized in Table 2. Note that FAT still dominates mod. VAT for the all datasets and the margins become larger. While FAT achieves the state-of-the-art performances for MNIST and SVHN, its performance degrades much for CIFAR10 compared to the accuracy for the case of 4000 labeled data. Note that the quality of bad samples depends on the quality of the estimated classifier. For CIFAR10 which is relatively more complex than MNIST and SVHN, the quality of the estimated classifier is influenced much to the amount of labeled data and thus the accuracy of FAT is more sensitive to the amount of labeled data. However, it is interesting to see that FAT is superior to mod. VAT for CIFAR10 and CIFAR100, which suggests that bad samples are helpful for complex problems where the quality of bad samples might not be sufficiently good.
The other advantage of FAT is its stability with respect to learning phase. With small labeled data, the test accuracies of each epoch tends to fluctuate much for VAT and mod VAT, while FAT provides much more stable results. See Figure 7 in Appendix. This may be partly because ’good’ bad samples keep the classifier from fluctuation.

heading: 6.2 EFFECTS OF TUNING PARAMETERS
text: FAT introduces two additional tuning parameters τ and δ compared to VAT, where τ is used to determine whether a bad sample is ’good’ and δ is the radius to generate bad samples. We investigate the sensitivities of prediction performances with respect to the changes of the values of τ and δ with being fixed at 1.5 that is the optimal value. The results are reported in Table 3. Unless τ is too small or too large, the prediction performances are not changed much. For δ, care should be done. Too small δ, smaller than , hampers the prediction performance much. Apparently, a similar value of δ to that of (i.e. δ = 2) gives the best result.

heading: 6.3 OBJECTIVE FUNCTION ANALYSIS
text: In this subsection, we analyze the effects of Ltrue and LVAT in this section. We do not consider the cross-entropy of the labeled data and Lfake since they are necessary to reflect the idea utilizing bad data. Table 4 compares the performances without one or both of these terms. Note that LVAT is necessary for superior performances, which is because LVAT is indispensable to generate ‘good’ bad samples as explained in Section 5.1. On the other hand, the term Ltrue, which is devised to separate unlabeled data from bad samples, is not always helpful. A possible explanation is that sometimes the term Ltrue would give an undesirable effect on the misclassified data.

heading: 6.4 COMPUTATIONAL EFFICIENCY
text: We investigate the computational efficiency of our method in view of learning speed and computation time per training epoch. For Bad GAN, we did not use PixelCNN++ on SVHN and CIFAR10 datasets since the pre-trained PixelCNN++ models are not publicly available. Figure 4 draws the bar plots about the numbers of epochs needed to achieve the prespecified test accuracies. We can clearly see that FAT requires much less epochs
We also calculate the ratios of the computing time of each semi-supervised learning algorithm over the computing time of the corresponding supervised learning algorithm for CIFAR10 dataset, whose results are summarized in Table 5. These ratios are almost same for different datasets. The computation time of FAT is competitive to VAT and mod. VAT, and hence we can conclude that FAT arrives at the prespecified performances much efficiently. Note that PixelCNN++ is not used for this experiment, and so comparison of computing time of FAT and Bad GAN with PixelCNN++ is meaningless

heading: 6.5 QUALITY OF BAD SAMPLES GENERATED BY ADVERSARIAL TRAINING
text: We investigate how ‘good’ bad samples generated by the adversarial training are. The upper panel of Figure 5 shows the scatter plot of the synthetic data and the trace plot of prediction accuracies of FAT and VAT. And the lower panel of Figure 5 draws the scatter plots with generated bad samples at various epochs. We can clearly see that bad samples move to lower density regions as the epoch increases, which amply demonstrates that the adversarial training is good at generating bad samples. We also compare bad images of the MNIST data generated by FAT and Bad GAN in Figure 6. The bad images by FAT do not look like real images and do not seem to be collapsed, which indicates that FAT consistently generates diverse and good bad samples. Bad GAN also generates diverse bad samples but some ‘realistic’ images can be found.

heading: 7 CONCLUSION
text: In this paper, we propose a new method called FAT for semi-supervised learning which generates bad samples only with a given classifier. The objective function of FAT is devised to compromise the advantages of Bad GAN and VAT together, which makes FAT be faster and more accurate. In numerical experiments, we show that FAT achieves almost the state-of-the-art performances with much fewer epochs. Unlike Bad GAN , FAT only needs to learn a discriminator. Hence, it could be extended without much effort to other learning problems. For example, FAT can be modified easily for recurrent neural networks and hence can be applied to sequential data. We will leave this extension as a future work.
It would be useful to combine FAT with a generative approach such as Bad GAN, in particular when the initial classifier is bad. Since FAT uses only a classifier to generate bad samples, the initial estimate of the classifier would be important. Using the generator model learned by large unlabeled data would be helpful to find a good initial estimate of the classifier.

heading: ACKNOWLEDGMENTS
text: This work is supported by Samsung Electronics Co., Ltd.

heading: A APPENDIX
text: A.1 PROOF OF PROPOSITION 1
Without loss of generality, we assume that w ′ x+ b > 0, that is, p(y = 1|x; η) > p(y = 0|x; η). We will show that there exists > 0 such that w ′ r∗(x, ) < 0. Note that
argmax r,||r||≤ ,w′r>0
KL(x, r; η) = w
||w|| (=: r∗1) and
argmax r,||r||≤ ,w′r<0
KL(x, r; η) = − w ||w|| (=: r∗2).
So all we have to do is to show
KL(x, r∗2 ; η) > KL(x, r ∗ 1 ; η).
By simple calculation we can get the following:
KL(x, r∗2 ; η)−KL(x, r∗1 ; η) = − p(y = 1|x; θ)w′(r∗2 − r∗1)− log exp ( w ′ (x+ r∗2) + b ) + 1
exp (w′(x+ r∗1) + b) + 1  . Using the Taylor’s expansion up to the third-order, we obtain the following:
log [ exp ( w ′ (x+ r) + b ) + 1 ] = log [ exp ( w ′ x+ b ) + 1 ] + p(y = 1|x; η)w ′ r
+ 1
2 p(y = 1|x; η)p(y = 0|x; η)r
′ ww ′ r
−1 6 p(y = 1|x; η)p(y = 0|x; η) {p(y = 1|x; η)− p(y = 0|x; η)} p∑ i,j,k=1 wiwjwkrirjrk +o(||r||3).
So,
log exp
( w ′ (x+ r∗2) + b ) + 1
exp (w′(x+ r∗1) + b) + 1 = p(y = 1|x; η)w
′ (r∗2 − r∗1)
+ 1
3 p(y = 1|x; η)p(y = 0|x; η) {p(y = 1|x; η)− p(y = 0|x; η)} 3||w||3 + o( 3).
Thus, we have the following equations:
KL(x, r∗2 ; η)−KL(x, r∗1 ; η) = 1
3 p(y = 1|x; η)p(y = 0|x; η) {p(y = 1|x; η)− p(y = 0|x; η)} 3||w||3 + o( 3)
= C · 3 + o( 3).
Therefore, there exists ∗ > 0 such that KL(x, r∗2 ; η) > KL(x, r ∗ 1 ; η) for ∀0 < < ∗.
A.2 EXTENSION OF PROPOSITION 1 FOR THE DNN CLASSIFIER
Consider a binary classification DNN model with ReLU activation function p(y = 1|x; θ) = (1 + exp(−g(x; θ)))−1 parameterized by θ. Since g(x; θ) is piecewise linear, we can write g(·; θ) as
g(·; θ) = N∑ j=1 I(· ∈ Aj) · (wjx+ bj),
where Aj is a linear region and N is the number of linear regions.
For given x, suppose g(x; θ) > 0. If g(x : θ) is estimated reasonably, we expect that g(x; θ) is decreasing if x moves toward the decision boundary. A formal statement of this expectation would be that x − r∇xg(x; θ) can arrive at the decision boundary for a finite value of r > 0, where ∇x is the gradient with respect to x. Of course, for x with g(x; θ) < 0, we expect that x+ r∇xg(x; θ) can arrive at the decision boundary for a finite value of r > 0. We say that x is normal if there is r > 0 such that x− r∇xg(x : θ)sign{g(x : θ)} locates at the decision boundary. We say that a linear region Aj is normal if all x in Aj are normal. We expect that most of Aj are normal if g(x; θ) is reasonably estimated so that the probability decreases or increases depending on sign{g(x : θ)} if x is getting closer to the decision boundary.
The following proposition proves that the adversarial direction is toward the decision boundary for all xs in normal linear regions.
Proposition 3 If a linear region Aj is normal. Then for any x ∈ int(Aj), there exists > 0 and δ > 0 such that xbad = x+ δradvr(x, )/||radvr(x, )|| is on the decision boundary.
Proof. Take ̃ > 0 such that x + r ∈ Aj̃ for ∀r ∈ B(x, ̃). Then by Proposition 1, there exists 0 < ∗ < ̃ such that for ∀0 < < ∗,
radvr(x, ) = · sign(−bj̃ − w ′ j̃ x)) · wj̃ ||wj̃ ||
∝ −∇xg(x; θ)sign{g(x : θ)}.
x is normal, thus there exists δ > 0 such that xbad = x + δradvr(x, )/||radvr(x, )|| belongs to the decision boundary.
A.3 PROOF OF PROPOSITION 2
It suffices to show that ∀x, x′ ∈ U trk , f(x) = f(x ′ ) for all k = 1, ...,K. For given U trk , there exists (x̃, ỹ) ∈ Ltr such that d({x̃},U trk ) < . So U trk ∪ {x̃} is -connected. That is, for any x ∈ U trk , there exists a path (x̃, x1, ..., xq, x) such that (x̃, x1), ..., (xq, x) are all -connected. Therefore y(x) = y(x̃) = ỹ for ∀x ∈ U trk , and the proof is done.
A.4 FIGURE

Review summaries:
The authors propose to combine BadGAN framework and VAT to accelerate learning in the semi-supervised setting. The paper shows that the VAT approach is actively pushing the decision boundary away from the high-density regions. While the BadGAN approach pulls the decision boundary to low-density regions. This simultaneous push and pull lead to after convergence in testing accuracy. The authors also report competitive results on standard datasets used for SSL such as SVHN and CIFAR10.

The paper proposes to use the technique in VAT to generate adversarial complementary examples in the (K+1)-class semi-supervised learning framework described by the Bad GAN paper. This leads to a formulation that combines the VAT loss and the (K+1)-class classification loss. The paper also provides analysis regarding why VAT is useful for semi-supervised learning.