Title: TANGENT-NORMAL ADVERSARIAL REGULARIZATION FOR SEMI-SUPERVISED LEARNING

Abstract: The ever-increasing size of modern datasets combined with the difficulty of obtaining label information has made semi-supervised learning of significant practical importance in modern machine learning applications. In comparison to supervised learning, the key difficulty in semi-supervised learning is how to make full use of the unlabeled data. In order to utilize manifold information provided by unlabeled data, we propose a novel regularization called the tangent-normal adversarial regularization, which is composed by two parts. The two parts complement with each other and jointly enforce the smoothness along two different directions that are crucial for semi-supervised learning. One is applied along the tangent space of the data manifold, aiming to enforce local invariance of the classifier on the manifold, while the other is performed on the normal space orthogonal to the tangent space, intending to impose robustness on the classifier against the noise causing the observed data deviating from the underlying data manifold. Both of the two regularizers are achieved by the strategy of virtual adversarial training. Our method has achieved state-of-the-art performance on semi-supervised learning tasks on both artificial dataset and practical datasets.
Under review as a conference paper at ICLR 2019 Rifai et al., 2011). Thus directly performing VAT in input space might overly regularize and does potential harm to the classifier. Another kind of regularization called manifold regularization aims to encourage invariance of classifier on manifold (Simard et al., 

1. INTRODUCTION:

The recent success of supervised learning (SL) models, like deep convolutional neural networks, highly relies on the huge amount of labeled data. However, though obtaining data itself might be relatively effortless in various circumstances, to acquire the annotated labels is still costly, limiting the further applications of SL methods in practical problems. Semi-supervised learning (SSL) models, which requires only a small part of data to be labeled, does not suffer from such restrictions. The advantage that SSL depends less on well-annotated datasets makes it of crucial practical importance and draws lots of research interests. The common setting in SSL is that we have access to a relatively small amount of labeled data and much larger amount of unlabeled data. And we need to train a classifier utilizing those data. Comparing to SL, the main challenge of SSL is how to make full use of the huge amount of unlabeled data, i.e., how to utilize the marginalized input distribution p(x) to improve the prediction model i.e., the conditional distribution of supervised target p(y|x).

To solve this problem, there are mainly three streams of research.

The first approach, based on probabilistic models, recognizes the SSL problem as a specialized missing data imputation task for classification problem. The common scheme of this method is to establish a hidden variable model capturing the relationship between the input and label, and then applies Bayesian inference techniques to optimize the model (Kingma et al., 2014; Zhu et al., 2003; Rasmus et al., 2015) . Suffering from the estimation of posterior being either inaccurate or computationally inefficient, this approach performs less well especially in high-dimensional dataset (Kingma et al., 2014) .

The second line tries to construct proper regularization using the unlabeled data, to impose the desired smoothness on the classifier. One kind of useful regularization is achieved by adversarial training (Goodfellow et al., 2014b) , or virtual adversarial training (VAT) when applied to unlabeled data (Miyato et al., 2016; 2017) . Such regularization leads to robustness of classifier to adversarial examples, thus inducing smoothness of classifier in input space where the observed data is presented. The input space being high dimensional, though, the data itself is concentrated on a underlying manifold of much lower dimensionality (Cayton, 2005; Narayanan & Mitter, 2010; Chapelle et al., 2009;  Figure 1 : Illustration for tangent-normal adversarial regularization. x = x0 + n is the observed data, where x0 is exactly supported on the underlying manifold M and n is the noise independent of x0. r is the adversarial perturbation along the tangent space to induce invariance of the classifier on manifold; r ⊥ is the adversarial perturbation along the normal space to impose robustness on the classifier against noise n.

Inspired by the three assumptions, we introduce a novel regularization called the tangent-normal adversarial regularization (TNAR), which is composed by two parts. The tangent adversarial regularization (TAR) induces the smoothness of the classifier along the tangent space of the underlying manifold, to enforce the invariance of the classifier along manifold. And the normal adversarial regularization (NAR) penalizes the deviation of the classifier along directions orthogonal to the tangent space, to impose robustness on the classifier against the noise carried in observed data. The two regularization terms enforce different aspects of the classifier's smoothness and jointly improve the generalization performance, as demonstrated in Section 4.

To realize our idea, we have two challenges to conquer: how to estimate the underlying manifold and how to efficiently perform TNAR.

For the first issue, we take advantage of the generative models equipped with an extra encoder, to characterize coordinate chart of manifold (Kumar et al., 2017; Lecouat et al., 2018; Qi et al., 2018) . More specifically, in this work we choose variational autoendoer (VAE) (Kingma & Welling, 2013) and localized GAN (Qi et al., 2018) to estimate the underlying manifold from data.

For the second problem, we develop an adversarial regularization approach based on virtual adversarial training (VAT) (Miyato et al., 2017) . Different from VAT, we perform virtual adversarial training in tangent space and normal space separately as illustrated in Figure 1 , which leads to a number of new technical difficulties and we will elaborate the corresponding solutions later. Compared with the traditional manifold regularization methods based on tangent propagation (Simard et al., 1998; Kumar et al., 2017) or manifold Laplacian norm (Belkin et al., 2006; Lecouat et al., 2018) , our realization does not require explicitly evaluating the Jacobian of classifier. All we need is to calculate the derivative of matrix vector product, which only costs a few times of back or forward propagation of network.

2. BACKGROUND:



2.1. NOTATIONS:

We denote the labeled and unlabeled dataset as D l = {(x l , y l )} and D ul = {x ul } respectively, thus D := D l ∪ D ul is the full dataset. The output of classification model is written as p(y|x, θ), where θ is the model parameters to be trained. We use (•, •) to represent supervised loss function. And the regularization term is denoted as R with specific subscript for distinction. The observed space of x is written as R D . And the underlying manifold of the observed data x is written as M ∼ = R d , d D. We use z for the manifold representation of data x. We denote the decoder, or the generator, as x = g(z) and the encoder as z = h(x), which form the coordinate chart of manifold together. If not stated otherwise, we always assume x and z correspond to the coordinate of the same data point in observed space R D and on manifold M, i.e., g(z) = x and h(x) = z. The tangent space of M at point x is

T x M = J z g(R d ) ∼ = R d ,

where J z g is the Jacobian of g at point z. The tangent space T x M is also the span of the columns of J z g. For convenience, we define J := J z g.

The perturbation in the observed space R D is denoted as r ∈ R D , while the perturbation on the manifold representation is denoted as η ∈ R d . Hence the perturbation on manifold is g(z + η)g(z) ∈ R D . When the perturbation η is small enough for the holding of the first order Taylor's expansion, the perturbation on manifold is approximately equal to the perturbation on its tangent space, g(z + η)g(z) ≈ J • η ∈ T x M. Therefore we say a perturbation r ∈ R D is actually on manifold, if there is a perturbation η ∈ R d , such that r = J • η.

2.2. VIRTUAL ADVERSARIAL TRAINING:

VAT (Miyato et al., 2017) is an effective regularization method for SSL. The virtual adversarial loss introduced in VAT is defined by the robustness of the classifier against local perturbation in the input space R D . Hence VAT imposes a kind of smoothness condition on the classifier. Mathematically, the virtual adversarial loss in VAT for SSL is L(D l , D ul , θ) := E (x l ,y l )∈D l (y l , p(y|x l , θ)) + αE x∈D R vat (x, θ), where the VAT regularization R vat is defined as R vat (x; θ) := max r 2 ≤ dist(p(y|x, θ), p(y|x + r, θ)), where dist(•, •) is some distribution distance measure and controls the magnitude of the adversarial example. For simplicity, define F (x, r, θ) := dist(p(y|x, θ), p(y + r, θ)).

(1)

Then R vat = max r 2 ≤ F (x, r, θ). The so called virtual adversarial example is r * := arg max r ≤ F (x, r, θ). Once we have r * , the VAT loss can be optimized with the objective as

L(D l , D ul , θ) = E (x l ,y l )∈D l (y l , p(y|x l , θ)) + αE x∈D F (x, r * , θ).

To obtain the virtual adversarial example r * , Miyato et al. (2017) suggested to apply second order Taylor's expansion to F (x, r, θ) around r = 0 as

EQUATION

where H := ∇ 2 r F (x, r, θ)| r=0 denotes the Hessian of F with respect to r. The vanishing of the first two terms in Taylor's expansion occurs because that dist(•, •) is a distance measure with minimum zero and r = 0 is the corresponding optimal value, indicating that at r = 0, both the value and the gradient of F (x, r, θ) are zero. Therefore for small enough , r * ≈ arg max r 2 ≤ 1 2 r T Hr, which is an eigenvalue problem and the direction of r * can be solved by power iteration.

2.3. GENERATIVE MODELS FOR DATA MANIFOLD:

We take advantage of generative model with both encoder h and decoder g to estimate the underlying data manifold M and its tangent space T x M. As assumed by previous works (Kumar et al., 2017; Lecouat et al., 2018) , perfect generative models with both decoder and encoder can describe the data manifold, where the decoder g(z) and the encoder h(x) together serve as the coordinate chart of manifold M. Note that the encoder is indispensable for it helps to identify the manifold coordinate z = h(x) for point x ∈ M. With the trained generative model, the tangent space is given by T x M = J z g(R d ), or the span of the columns of J = J z g.

In this work, we adopt VAE (Kingma & Welling, 2013) and localized GAN (Qi et al., 2018) to learn the targeted underlying data manifold M as summarized below.

VAE VAE (Kingma & Welling, 2013) is a well known generative model consisting of both encoder and decoder. The training of VAE is by optimizing the variational lower bound of log likelihood,

log p(x, θ) ≥ E z∼q(z|x,θ) log p(x|z, θ) -KL(q(z|x, θ) p(z)).

(3)

Here p(z) is the prior of hidden variable z, and q(z|x, θ), p(x|z, θ) models the encoder and decoder in VAE, respectively. The derivation of the lower bound with respect to θ is well defined thanks to the reparameterization trick, thus it could be optimized by gradient based method. The lower bound could also be interpreted as a reconstruction term plus a regularization term (Kingma & Welling, 2013) . With a trained VAE, the encoder and decoder are given as h(x) = arg max z q(z|x) and g(z) = arg max x q(x|z) accordingly.

Localized GAN Localized GAN (Qi et al., 2018) suggests to use a localized generator G(x, z) to replace the global generator g(z) in vanilla GAN Goodfellow et al. (2014a) . The key difference between localized GAN and previous generative model for manifold is that, localized GAN learns a distinguishing local coordinate chart for each point x ∈ M, which is given by G(x, z), rather than one global coordinate chart. To model the local coordinate chart in data manifold, localized GAN requires the localized generator to satisfy two more regularity conditions: 1) locality:

G(x, 0) = x, so that G(x, z) is localized around x; 2) orthogonmality: ∂ G(x,z) ∂z T ∂ G(x,z) ∂z = I, to ensure G(x, z)

is non-degenerated. The two conditions are achieved by the following penalty during training of localized GAN:

R localized GAN := µ 1 G(x, 0) -x 2 + µ 2 ∂G(x, z) ∂z T ∂G(x, z) ∂z -I 2 .

Since G(x, z) defines a local coordinate chart for each x separately, in which the latent encode of x is z = 0, there is no need for the extra encoder to provide the manifold representation of x.

3. METHOD:

In this section we elaborate our proposed tangent-normal adversarial regularization (TNAR) strategy. The TNAR loss to be minimized for SSL is

L(D l , D ul , θ) := E (x l ,y l )∈D l y l , p(y|x l , θ) +α 1 E x∈D R tangent (x, θ)+α 2 E x∈D R normal (x, θ). (4)

The first term in Eq. ( 4) is a common used supervised loss. R tangent and R normal is the so called tangent adversarial regularization (TAR) and normal adversarial regularization (NAR) accordingly, jointly forming the proposed TNAR. We assume that we already have a well trained generative model for the underlying data manifold M, with encoder h and decoder g, which can be obtained as described in Section 2.3.

3.1. TANGENT ADVERSARIAL REGULARIZATION:

Vanilla VAT penalizes the variety of the classifier against local perturbation in the input space R D (Miyato et al., 2017) , which might overly regularize the classifier, since the semi-supervised learning assumption only indicates that the true conditional distribution varies smoothly along the underlying manifold M, but not the whole input space R D (Belkin et al., 2006; Rifai et al., 2011; Niyogi, 2013) . To avoid this shortcoming of vanilla VAT, we propose the tangent adversarial regularization (TAR), which restricts virtual adversarial training to the tangent space of the underlying manifold T x M, to enforce manifold invariance property of the classifier.

EQUATION

where F (x, r, θ) is defined as in Eq. (1). To optimize Eq. ( 5), we first apply Taylor's expansion to F (x, r, θ) so that R tangent (x; θ) ≈ max r 2 ≤ ,r∈TxM=Jzg(R d ) 1 2 r T Hr, where the notations and the derivation are as in Eq. ( 2). We further reformulate R tangent as maximize

EQUATION

)

This is a classic generalized eigenvalue problem, the optimal solution η * of which could be obtained by power iteration and conjugate gradient (and scaling). The iteration framework is as

EQUATION

)

Now we elaborate the detailed implementation of each step in Eq. ( 8).

Computing J T HJη. Note that z = h(x), x = g(z). Define r(η)

:= g(z + η) - g(z). For F x, r(η), θ = dist(p(y|x, θ) p(y|x + r(η), θ)), we have ∇ 2 η F (x, r(η), θ) = (J z+η g) T ∇ 2 r F (x, r(η), θ)(J z+η g) + ∇ 2 η g(z + η) • ∇ r F (x, r(η), θ).

While on the other hand, since dist(•, •) is some distance measure with minimum zero and r(0) = 0 is the corresponding optimal value, we have F (x, r(0), θ) = 0, ∇ r F (x, r(0), θ) = 0. Therefore, ∇ 2 η F (x, r(0), θ) = (J z g) T ∇ 2 r F (x, r(0), θ)J z g = J T HJ. Thus the targeted matrix vector product could be efficiently computed as

J T HJη = ∇ 2 η F (x, r(0), θ) • η = ∇ η ∇ η F (x, r(0), θ) • η . Note that ∇ η F (x, r(0), θ)

• η is a scalar, hence the gradient of which could be obtained by back propagating the network for once. And it only costs twice back propagating for the computation of J T HJη.

Solving J T Jµ = v. Similarly, define K(η) := g(z + η) -g(z)

T g(z + η)g(z) . We have

∇ 2 η K(η) = (J z+η g) T J z+η g + ∇ 2 η g(z + η) • K(η).

Since K(0) = 0, we have ∇ 2 η K(0) = (J z g) T J z g = J T J. Thus the matrix vector product J T Jµ could be evaluated similarly as

J T Jµ = ∇ η ∇ η K(0) • µ .

The extra cost for evaluating J T Jµ is still back propagating the network for twice. Due to J T J being positive definite (g is non-degenerated), we can apply several steps of conjugate gradient to solve J T Jµ = v efficiently. By iterating Eq. ( 8), we obtain the optimal solution η of Eq. ( 7). The desired optimal solution is then r = Jη / Jη , using which we obtain R tangent (x; θ) = F (x, r , θ).

Compared with manifold regularization based on tangent propagation (Simard et al., 1998; Kumar et al., 2017) or manifold Laplacian norm (Belkin et al., 2006; Lecouat et al., 2018) , which is computationally inefficient due to the evaluation of Jacobian, our proposed TAR could be efficiently implemented, thanks to the low computational cost of virtual adversarial training.

3.2. NORMAL ADVERSARIAL REGULARIZATION:

Motivated by the noisy observation assumption indicating that the observed data contains noise driving them off the underlying manifold, we come up with the normal adversarial regularization (NAR) to enforce the robustness of the classifier against such noise, by performing virtual adversarial training in the normal space. The mathematical description is

EQUATION

Note that T x M is spanned by the coloums of J = J z g, thus r⊥T x M ⇔ J T • r = 0. Therefore we could reformulate Eq. ( 9) as maximize

EQUATION

However, Eq. ( 10) is not easy to optimize since J T • r cannot be efficiently computed. To overcome this, instead of requiring r being orthogonal to the whole tangent space T x M, we take a step back to demand r being orthogonal to only one specific tangent direction, i.e., the tangent space adversarial perturbation r . Thus the constraint J T • r = 0 is relaxed to (r ) T • r = 0. And we further replace the constraint by a regularization term, maximize

EQUATION

)

where λ is a hyperparameter introduced to control the orthogonality of r.

Since Eq. ( 11) is again an eigenvalue problem, and we can apply power iteration to solve it. Note that a small identity matrix λ r I is needed to be added to keep 1 2 Hλr r T + λ r I semipositive definite, which does not change the optimal solution of the eigenvalue problem. The power iteration is as

EQUATION

)

And the evaluation of Hr is by Hr = ∇ r ∇ r F (x, 0, θ) • r , which could be computed efficiently.

After finding the optimal solution of Eq. ( 11) as r ⊥ , the NAR becomes R normal (x, θ) = F (x, r ⊥ , θ).

Finally, as in (Miyato et al., 2017) , we add entropy regularization to our loss function. It ensures neural networks to output a more determinate prediction and has implicit benefits for performing virtual adversarial training, R entropy (x, θ) :=y p(y|x, θ) log p(y|x, θ). Our final loss for SSL is

EQUATION

)

The TAR inherits the computational efficiency from VAT and the manifold invariance property from traditional manifold regularization. The NAR causes the classifier for SSL being robust against the off manifold noise contained in the observed data. These advantages make our proposed TNAR, the combination of TAR and NAR, a reasonable regularization method for SSL, the superiority of which will be shown in the experiment part in Section 4.

4. EXPERIMENTS:

To demonstrate the advantages of our proposed TNAR for SSL, we conduct a series of experiments on both artificial and real dataset. The compared methods for SSL include: 1) SL: supervised learning using only the labeled data; 2) VAT: vanilla VAT (Miyato et al., 2017) ; 3) TNAR-VAE: the proposed TNAR method, with the underlying manifold estimated by VAE; 4) TNAR-LGAN: the proposed TNAR method, with the underlying manifold estimated by localized GAN; 5) TNAR-Manifold: the proposed TNAR method with oracle underlying manifold for the observed data, only used for artificial dataset; 6) TNAR-AE: the proposed TNAR method, with the underlying manifold estimated roughly by autoendoer, only used for artificial dataset. 7) TAR: the tangent adversarial regularization, used in ablation study. 8) NAR: the normal adversarial regularization, used in ablation study. If not stated otherwise, all the above methods contain entropy regularization term.

4.1. TWO-RINGS ARTIFICIAL DATASET:

We introduce experiments on a two-rings artificial dataset to show the effectiveness of our proposed methods intuitively. In this experiments, there is 3, 000 unlabeled data (gray dots) and 6 labeled data (blue dots), 3 for each class. The detailed construction could be found in Appendix.

The performance of each compared methods is shown in Table 1 , and the corresponding classification boundary is demonstrated in Figure 2 . The TNAR under true underlying manifold (TNAR-Manifold) perfectly classifies the two-rings dataset with merely 6 labeled data, while the other methods fail to predict the correct decision boundary. Even with the underlying manifold roughly approximated by an autoendoer, our approach (TNAR-AE) outperforms VAT in this artificial dataset. However, the performance of TNAR-AE is worse than TNAR-Manifold, indicating that the effectiveness of TNAR relies on the quality of estimating the underlying manifold. We also conduct experiments on FashionMNIST dataset1 . There are three sets of experiments with the number of labeled data being 100, 200 and 1, 000, respectively. The details about the networks are in Appendix.

The corresponding results are shown in Table 2 , from which we observe at least two phenomena. The first is that our proposed TANR methods (TNAR-VAE, TNAR-LGAN) achieve lower classification errors than VAT in all circumstances with different number of labeled data. The second is that the performance of our method depends on the estimation of the underlying manifold of the observed data. In this case, TNAR-VAE brings larger improvement than TNAR-LGAN, since VAE produces better diverse examples according to our observation. As the development of generative model capturing more accurate underlying manifold, it is expected that our proposed regularization strategy benefits more for SSL. 

4.3. ABLATION STUDY:

We conduct ablation study on FashionMNIST datasets to demonstrate that both of the two regularization terms in TNAR are crucial for SSL. The results are reported in Table 2 . Removing either tangent adversarial regularization (NAR) or normal adversarial regularization (TAR) will harm the final performance, since they fail to enforce the manifold invariance or the robustness against the off-manifold noise. Furthermore, the adversarial perturbations and adversarial examples are shown in Figure 3 . We can easily observe that the tangent adversarial perturbation focuses on the edges of foreground objects, while the normal space perturbation mostly appears as certain noise over the whole image. This is consistent with our understanding on the role of perturbation along the two directions that capture the different aspects of smoothness.

4.4. CIFAR-10 AND SVHN:

There are two classes of experiments for demonstrating the effectiveness of TNAR in SSL, SVHN with 1, 000 labeled data, and CIFAR-10 with 4, 000 labeled data. The experiment setups are identical with Miyato et al. (2017) . We test two kinds of convolutional neural networks as classifier (denoted as "small" and "large") as in Miyato et al. (2017) . Since it is difficult to obtain satisfying VAE on CIFAR-10, we only conduct the proposed TNAR with the underlying manifold identified by Localized GAN (TNAR-LGAN) for CIFAR-10. Note that in Miyato et al. (2017) , the authors applied ZCA as pre-processing procedure, while other compared methods do not use this trick. For fair comparison, we only report the performance of VAT without ZCA. More detailed experimental settings are included in Appendix. GAN (Salimans et al., 2016) 8.11 18.63 Tripple GAN (Li et al., 2017) 5.77 16.99 FM GAN (Kumar et al., 2017) 4.39 16.20 LGAN (Qi et al., 2018) 4.73 14.23 TNAR-VAE (small) 3.93 -TNAR-VAE (large)

3.84 -TNAR-LGAN (small) 4.10 13.63 TNAR-LGAN (large) 3.93 13.53

In Table 3 we report the experiments results on CIFAR-10 and SVHN, showing that our proposed TNAR outperforms other state-of-the-art SSL methods on both SVHN and CIFAR-10, demonstrating the superiority of our proposed TNAR.

5. CONCLUSION:

We present the tangent-normal adversarial regularization, a novel regularization strategy for semisupervised learning, composing of regularization on the tangent and normal space separately. The tangent adversarial regularization enforces manifold invariance of the classifier, while the normal adversarial regularization imposes robustness of the classifier against the noise contained in the observed data. Experiments on artificial dataset and multiple practical datasets demonstrate that our approach outperforms other state-of-the-art methods for semi-supervised learning. The performance of our method relies on the quality of the estimation of the underlying manifold, hence the breakthroughs on modeling data manifold could also benefit our strategy for semi-supervised learning, which we leave as future work. The encoder of the VAE for identify the underlying manifold is a LeNet-like one, with two convolutional layers and one fully connected layer. And the decoder is symmetric with the encoder, except using deconvolutional layers to replace convolutional layer. The latent dimensionality is 128. The localized GAN for identify the underlying manifold is similar as stated in Qi et al. (2018) . And the implementation is modified from https://github.com/z331565360/Localized-GAN.

We change the latent dimensionality into 128.

We tried both joint training the LGAN with the classifier, and training them separately, observing no difference.

C EXPERIMENTS DETAILS ON SVHN AND CIFAR-10

In SVHN3 and CIFAR-104 experiments, we preserve 1, 000 data for validation from the original training set. That is, we use 1, 000/4, 000 labeled data for training and the other 1, 000 labeled data for validation. The only pre-processing on data is to scale the pixels value into 0 ∼ 1. We do not use data augmentation. The structure of classification neural network is shown in Table 4 , which is identical as in Miyato et al. (2017) .

For the labeled data, the batch size is 32, and for the unlabeled data, the batch size is 128. For SVHN, all networks are trained for 48, 000 updates. And for CIFAR-10, all networks are trained for 200, 000 updates. The optimizer is ADAM with initial learning rate 0.001, and linearly decay over the last 16, 000 updates. The hyperparameters tuned is the magnitude of the tangent adversarial perturbation ( 1 ), the magnitude of the normal adversarial perturbation ( 2 ) and the hyperparameter λ in Eq. ( 11). Other hyperparameters are all set to 1. We tune λ from {1, 0.1, 0.01, 0.001}, and 1 , 2 randomly from [0.05, 20].

The VAE for identify the underlying manifold for SVHN is implemented as in https:// github.com/axium/VAE-SVHN. The only modification is we change the coefficient of the regularization term from 0.01 to 1. The localized GAN for identify the underlying manifold for SVHN and CIFAR-10 is similar as stated in Qi et al. (2018) . And the implementation is modified from https://github.com/z331565360/Localized-GAN. We change the latent dimensionality into 512 for both SVHN and CIFAR-10.

D MORE ADVERSARIAL EXAMPLES:

More adversarial perturbations and adversarial examples in tangent space and normal space are shown in Figure 4 and Figure 5 . Note that the perturbations is actually too small to distinguish easily, thus we show the scaled perturbations.

From left to right: original example, tangent adversarial perturbation, normal adversarial perturbation, tangent adversarial example, normal adversarial example.

:

https://github.com/zalandoresearch/fashion-mnist

https://github.com/zalandoresearch/fashion-mnist

http://ufldl.stanford.edu/housenumbers/

https://www.cs.toronto.edu/ ˜kriz/cifar.html

Figures and Tables:

Figure 2: decision boundaries of compared methods on two-rings artificial dataset. Gray dots distributed on two rings: the unlabeled data. Blue dots (3 in each ring): the labeled data. Colored curves: the decision boundaries found by compared methods.

Figure3: The perturbations and adversarial examples in the tangent space and the normal space. Note that the perturbations is actually too small to distinguish easily, thus we show the scaled perturbations. First row: Fash-ionMNIST dataset; Second row: CIFAR-10 dataset. From left to right: original example, tangent adversarial perturbation, normal adversarial perturbation, tangent adversarial example, normal adversarial example.

Figure 4: The perturbations and adversarial examples in tangent space and normal space for FashionMNISTdataset. Note that the perturbations is actually too small to distinguish easily, thus we show the scaled perturbations. From left to right: original example, tangent adversarial perturbation, normal adversarial perturbation, tangent adversarial example, normal adversarial example.

Figure 5: The perturbations and adversarial examples in tangent space and normal space for CIFAR-10 dataset.

Classification errors (%) of com-

Table content: <table><tr><td colspan="2">pared methods on two-ring artificial dataset.</td></tr><tr><td colspan="2">We test with and without entropy regulariza-</td></tr><tr><td colspan="2">tion in each method and report the best one.</td></tr><tr><td colspan="2">In VAT and TNAR-AE, without entropy reg-</td></tr><tr><td colspan="2">ularization is better; For TNAR-Manifold,</td></tr><tr><td colspan="2">adding entropy regularization is better.</td></tr><tr><td>Model</td><td>Error (%)</td></tr><tr><td>SL</td><td>32.95</td></tr><tr><td>VAT</td><td>23.80</td></tr><tr><td>TNAR-AE</td><td>12.45</td></tr><tr><td>TNAR-Manifold</td><td>9.90</td></tr><tr><td colspan="2">TNAR-Manifold (ent) 0</td></tr></table>

Classification errors (%) of compared methods on FashionMNIST dataset. TAR/NAR-LGAN 23.65/24.87/28.73 18.32/19.16/24.49 13.52/14.09/15.94

Table content: <table><tr><td>Method</td><td>100 labels</td><td>200 labels</td><td>1000 labels</td></tr><tr><td>VAT</td><td>27.69</td><td>20.85</td><td>14.51</td></tr><tr><td>TNAR/</td><td/><td/><td/></tr></table>

Classification errors (%) of compared methods on SVHN / CIFAR-10 dataset.

Table content: <table><tr><td>Method</td><td colspan="2">SVHN 1,000 labels CIFAR-10 4,000 labels</td></tr><tr><td>VAT (small)</td><td>4.37</td><td>15.67</td></tr><tr><td>VAT (large)</td><td>4.23</td><td>15.29</td></tr><tr><td>ALI (Dumoulin et al., 2016)</td><td>7.41</td><td>17.99</td></tr><tr><td>Improved</td><td/><td/></tr></table>

The structure of convolutional neural networks for experiments on CIFAR-10 and SVHN, based on Springenberg et al. (2014); Salimans et al. (2016);Laine & Aila (2016). All the convolutional layers and fully connected layers are followed by batch normalization except the fully connected layer on CIFAR-10. The slopes of all lReLU functions in the networks are 0.1.

Table content: <table><tr><td colspan="3">Conv-Small on SVHN Conv-Small on CIFAR-10</td><td>Conv-Large</td></tr><tr><td/><td/><td>32×32 RGB image</td></tr><tr><td colspan="2">3×3 conv. 64 lReLU</td><td>3×3 conv. 96 lReLU</td><td>3×3 conv. 128 lReLU</td></tr><tr><td colspan="2">3×3 conv. 64 lReLU</td><td>3×3 conv. 96 lReLU</td><td>3×3 conv. 128 lReLU</td></tr><tr><td colspan="2">3×3 conv. 64 lReLU</td><td>3×3 conv. 96 lReLU</td><td>3×3 conv. 128 lReLU</td></tr><tr><td/><td/><td>2×2 max-pool, stride 2</td></tr><tr><td/><td/><td>dropout, p = 0.5</td></tr><tr><td colspan="2">3×3 conv. 128 lReLU</td><td>3×3 conv. 192 lReLU</td><td>3×3 conv. 256 lReLU</td></tr><tr><td colspan="2">3×3 conv. 128 lReLU</td><td>3×3 conv. 192 lReLU</td><td>3×3 conv. 256 lReLU</td></tr><tr><td colspan="2">3×3 conv. 128 lReLU</td><td>3×3 conv. 192 lReLU</td><td>3×3 conv. 256 lReLU</td></tr><tr><td/><td/><td>2×2 max-pool, stride 2</td></tr><tr><td/><td/><td>dropout, p = 0.5</td></tr><tr><td colspan="2">3×3 conv. 128 lReLU</td><td>3×3 conv. 192 lReLU</td><td>3×3 conv. 512 lReLU</td></tr><tr><td colspan="2">1×1 conv. 128 lReLU</td><td>1×1 conv. 192 lReLU</td><td>1×1 conv. 256 lReLU</td></tr><tr><td colspan="2">1×1 conv. 128 lReLU</td><td>1×1 conv. 192 lReLU</td><td>1×1 conv. 128 lReLU</td></tr><tr><td>dense 128 → 10</td><td colspan="3">global average pool, 6×6 → 1×1 dense 192→ 10 10-way softmax</td><td>dense 128→ 10</td></tr></table>

Peer review (iclr2019):
 The paper proposes to extend VAT by decomposing the regularization term into two spaces, the tangent space and the normal space. The tangent space is spanned over the columns of the Jacobian matrix of a function that maps latent (manifold) variables to data, and the normal space is defined as orthogonal to the tangent space. Power iteration was employed to optimize the inner-loop function. Pros 1. Positive signals have been shown on multiple datasets. 2. The proposed regularization terms seem to be novel and technically sound. Cons and questions 1. How did you obtain the results of VAT in the experiments? It seems the results in Table 3 could not be found in Miyato et al. It is important to obtain a proper baseline to substantiate the main claim of the paper. Moreover, it might not be appropriate to claim the proposed method is state-of-the-art (SoTA) on SVHN and CIFAR-10 given that the numbers in Table 3 are different from Miyato et al. To claim SoTA, you would need to at least implement your model in a setting that is comparable to the numbers in Miyato et al (e.g. using ZCA), rather than running your model in a setting that was not considered in Miyato et al (e.g. removing ZCA). 2. It seems that the proposed method introduces a bunch of additional hyper-parameters, including lambda, two epsilons, and the alphas that weigh different terms. Is it difficult to tune these hyper-parameters in practice? What values did you actually use for these hyper-parameters? 3. The paper would be further improved if an ablation study could be performed on SVHN and/or CIFAR-10, since these are more popular datasets for semi-supervised learning. It would become clearer how the proposed method is better than the previous ones. 4. The paper claimed that "TAR inherits the computational efficiency from VAT". Does power iteration bring additional computation costs? What is the actual computational time, compared to VAT?