Title: MIXING UP REAL SAMPLES AND ADVERSARIAL SAMPLES FOR SEMI-SUPERVISED LEARNING

Abstract: Consistency regularization methods have shown great success in semi-supervised learning tasks. Most existing methods focus on either the local neighborhood or in-between neighborhood of training samples to enforce the consistency constraint. In this paper, we propose a novel generalized framework called Adversarial Mixup (AdvMixup), which unifies the local and in-between neighborhood approaches by defining a virtual data distribution along the paths between the training samples and adversarial samples. Experimental results on both synthetic data and benchmark datasets exhibit the benefits of AdvMixup on semi-supervised learning.

1. INTRODUCTION:

Deep neural networks have achieved remarkable performance in various areas thanks to their excellent capability on data representation learning. However, successful training of deep learning models usually requires a large amount of labeled data. Such property poses a challenge to many practical tasks where labeling a larget amount of data is not feasible due to the high cost in time and finances. To address this problem, semi-supervised learning leverages the unlabeled data to improve the generalization performance of the model over a small amount of labeled data.

Cluster assumption Chapelle & Zien (2005) has been a basis for many successful semi-supervised learning models, which states that the data distribution forms discrete clusters and samples in the same cluster tend to share the same class label. This assumption has motivated many traditional semi-supervised learning approaches such as transductive support vector machines Joachims (1999) , entropy minimization Grandvalet & Bengio (2005) , and pseudo-labeling Lee (2013) . Recently, the consistency regularization based methods Sajjadi et al. (2016) ; Laine & Aila (2017) ; Tarvainen & Valpola (2017) ; Miyato et al. (2018) ; Verma et al. (2019) have renewed the state-of-the-art results across many semi-supervised learning tasks. Basically, consistency regularization enforces the predictions of an unlabeled sample x and its neighborhood sample x to be the same, which encourages the decision boundary to lie in the low-density regions. Different methods concentrate on different types of the neighborhood samples x.

One branch of the consistency regularization methods focuses on the local neighborhood around the training samples. The Π model Laine & Aila (2017) obtained x by adding a random noise to x. However, Szegedy et al. (2014) ; Goodfellow et al. (2015) have shown that models regularized with such isotropic noise can still be vulnerable to the perturbations in the adversarial direction Goodfellow et al. (2015) . Inspired by this, Miyato et al. (2018) proposed the Virtual Adversarial Training (VAT) model, where x is selected as the adversarial example of x, thus regularizing the model in the most non-smooth regions. These perturbation-based methods can be visualized as in Figure 1a , where the possible areas for the selection of x are centred around the training samples.

Another branch of the consistency regularization methods considers the in-between neighborhood of two training samples. The mixup model Zhang et al. (2018) picked x along the interpolation path between pairs of training samples x i and x j , i.e., x = λx i +(1-λ)x j , and enforced a linear transition along this path by requiring f (x) to approximate λy i + (1 -λ)y j , which is originally proposed for supervised learning. Verma et al. (2019) generalized the mixup model to semi-supervised learning by replacing the real labels with the predicted labels by a teacher model Tarvainen & Valpola (2017) . These interpolation-based methods can be visualized as in Figure 1b , where the possible areas for the selection of x are along the path between pairs of training samples. In this paper, we propose a novel consistency regularization technique, called Adversarial Mixup (AdvMixup), by unifying both the local neighborhood and in-between neighborhood. In particular, we define the neighborhood as the samples lying along the paths between the real samples and adversarial samples. For two random training samples x i and x j , we sample x along the interpolation path between x i and the adversarial sample x (adv) j

, i.e., x = λx i + (1 -λ)x

(adv) j

. Then we enforce the consistency between f (x) and λf (x i ) + (1 -λ)f (x j ). Note that we use the predicted label f (x j ) for x (adv) j , which implicitly incorporates the local neighborhood regularization method. The regularization area for our AdvMixup can be visualized as in Figure 1c .

We evaluate our AdvMixup on one synthetic dataset and several commonly used benchmark datasets, and the experimental results demonstrate AdvMixup outperforms the baseline methods which consider only local neighborhood or in-between neighborhood, especially when few labeled data is given.

2. ADVMIXUP:



2.1. PROBLEM DEFINITION:

In this paper, we focus on the standard semi-supervised learning task. Formally, Let X denote the input feature space and Y denote target label space. Given a labeled dataset S l = {(x i , y i )|i = 1, . . . , N l } and an unlabeled dataset S u = {x i |i = 1, . . . , N u } with x i ∈ X and y i ∈ Y, our aim is to learn a mapping function f : X → Y which can generalize to the unseen (x i , y i ) data pairs sampled from the joint probability distribution P (X , Y).

Standing on the cluster assumption Chapelle & Zien (2005) , we propose Adversarial Mixup (Ad-vMixup), a new consistency regularization approach for semi-supervised learning. AdvMixup implicitly defines a virtual data distribution P sampling along the interpolation paths between pairs of points from the real samples and the adversarial samples.

Formally, given a random pair of unlabeled training samples x i and x j , we first craft an adversarial sample x (adv) j for x j , then construct a virtual data sample (x i,j , ŷi,j ) using the interpolation between x i and x (adv) j as the virtual input and the interpolation between the soft labels of x i and x j as the virtual target:

EQUATION

where λ ∈ [0, 1] is sampled from the distribution P λ = Beta(α, α) with α ∈ [0, ∞]. Following the ICT model Verma et al. (2019) , we employ the predictions from the exponential moving average (EMA) model f t as the soft labels for better target quality Tarvainen & Valpola (2017) . The goal of AdvMixup is to fit the contructed virtual data samples by minimizing the divergence between the model prediction on the virtual input f (x i,j ) and the virtual target ŷi,j , which can be formulated as

L reg = E xi,xj ∼Su D Y [f (x i,j ), ŷi,j ] ,

(2) where D Y is a divergence metric defined on the Y space. The overview of our AdvMixup regularization is shown in Figure 2 .

Finally, we arrive at the full objective function for AdvMixup, i.e., minimizing

EQUATION

where

L nll = E (xi,yi)∼S l -y i ln f (x i )

is the typical negative log-likelihood loss for the labeded data, and β is a hyper-parameter controlling the importance of regularization term L reg . The training procedure of our model is illustrated in Algorithm 1.

Adversarial Sample Generation. An adversarial sample Szegedy et al. (2014) ; Goodfellow et al. (2015) is an slightly and carefully perturbed variant of a real data sample, with the aim of misleading a given classifier to make different predictions from the original real data sample. In this paper, we adopt the virtual adversarial example generation method from Miyato et al. (2018) , where the "virtual" means no ground-truth target labels are used to cater for the semi-supervised setting. Specifically, we craft an adversarial sample x

EQUATION

where > 0 is the norm constraint for the adversarial perturbation. The maximization problem can be approximated by the power iteration method. In practice, one step of iteration is enough to achieve strong performance Miyato et al. (2018) , which requires a low additional computational cost to the basic mixup model.

Generality. The proposed AdvMixup can generalize to both the perturbation-based regularization (e.g., VAT Miyato et al. (2018) ) and the mixup-based regularization (e.g., ICT Verma et al. (2019) ).

If λ → 0, the constructed virtual data sample is (x 1, reducing to the VAT model. If the adversarial perturbation degenerate to zero, i.e., x (adv) j = x j , Equation 1 reduces to the ICT model.

(adv) j , f t (x j )) in Equation

3. WHY ADVMIXUP?:

The AdvMixup model regularizes the classifier f along the interpolation paths between training samples and adversarial samples. In the following, we elaborate the reasonableness and advantages of this regularization scheme, and validate the effectiveness of AdvMixup via a case study on synthetic data. boundary to low-density areas. Our AdvMixup takes one additional step by creating an adversarial sample for one of the real sample pair. The created adversarial sample is supposed to share the same class label with its corresponding real sample. Therefore, given a random real sample pair x i and x j as well as the adversarial sample x (adv) j for x j , it is reasonable to enforce the classifier's predictions to linearly change from the (soft) target label f (x i ) of x i to the (soft) target label f (x j ) of x (adv) j along the path from x i to x (adv) j .

Reasonableness:

l : Lnll = 1 n l (x i ,y i )∈B l -y i ln f (xi) Associate the samples in Bu with soft labels B u + = {(xi, ft(xi))} nu i=1 Craft an adversarial batch B (adv) u + = {(x (adv) i = xi + r (adv) i , ft(xi))|xi = Bu[i]} nu i=1 using Eq. 4 Shuffle B (adv) u + as B (adv) u + ,s Sample λ ∼ Beta(α, α) as the interpolation parameter Construct a virtual data batch Bu + = {(xi, ŷi)} nu i=1 with xi = λx 1 i + (1 -λ)x 2 i , ŷi = λy 1 i + (1 -λ)y 2 i where (x 1 i , y 1 i ) = B u + [i], (x 2 i , y 2 i ) = B (adv) u + ,s [i] Compute the consistency regularization term Lreg = 1 nu (x i ,ŷ i )∈ Bu + DY [f (xi),

Advantages. Consistency regularization approaches are actually fixing possible flaws of the classifier which violates the cluster assumption. An effective approach is expected to detect the flaws violating the cluster assumption 1) more significantly and 2) more comprehensively. Compared with the methods seeking for the flaws in-between neighborhood of training samples like ICT Verma et al. (2019) , our AdvMixup considers the adversarial samples that violate the cluster assumption more 2019) and also illustrated in Figure 3 where the error rate reaches the maximum around λ = 0.5.

3.1. CASE STUDY ON SYNTHETIC DATA:

We evaluate the proposed AdvMixup against VAT and ICT on a synthetic dataset with two classes. As shown in Figure 4a , the training points forms two concentric circles with different radiuses, and the Gaussian noise (µ = 0, σ = 0.01) is applied to these points. The task is to classify these two classes of points, and each class contains 5 labeled samples and 100 unlabeled samples. Note that sometimes the distance between neighbor points within the same class can be comparable, if not larger than, the distance between neighbor points from different classes, making the problem a non-trivial task.

We utilize a neural network model as the classifier, which includes two hidden layers with 100 and 50 hidden units and ReLU activation functions. We fix the weight of the regularization terms as 10 for different methods, and search the optimal hyper-parameters specific to different methods (i.e., for VAT, α for ICT, and and α for AdvMixup) via a validation set.

The learned decision boundaries are shown in Figure 4 , and we have the following major observations. First, VAT can not successfully classify the two classes, since it mistakenly predicts a proportion of blue points as red points. Since these blue points have a relatively larger distance to other surrounding blue points and no labeled data lie in this region, it is possible for VAT to fail as the result has almost no objections to their local neighborhood based constraint. However, with the local neighborhood based constraint, the decision boundary of VAT desirably keeps a safe distance with the training samples. Second, ICT basically achieves to distinguish the two classes. However, the decision boundary stays too close to some data points, making the model vulnerate to even small noises. This is possible for ICT, because the points next to the decision boundary lie in a lighter area of the contour and thus have lower confidence scores, making the result still comply with the constraint for in-between training samples. Third, the proposed AdvMixup, considering both local and in-between neighborhood, is capable of learning a decision boundary which can both differentiate points from the two classes and stay a certain distance with the training samples.

4. EXPERIMENTS:

In this section, we evaluate the proposed AdvMixup against various strong baselines for semisupervised learning on benchmark datasets. We also conduct an ablation study to validate the effectiveness of our model.

4.1. DATASETS:

We conduct experiments on the widely-used CIFAR-10 and SVHN datasets. Hyper-parameters. We directly use the norm constraint in the code1 of the VAT model Miyato et al. (2018) , 8.0 for the CIFAR-10 dataset and 3.5 for the SVHN dataset. The update ratio γ of the EMA model is set to 0.001 following Verma et al. (2019) .We search the optimal beta distribution parameter α and the weight β of the regularization term in Equation 3 via the validation performance. As a result, α is set as 2.0 on CIFAR-10 and 0.1 on SVHN; β is set as 50, 100, and 100 for CIFAR-10 with 1000, 2000, and 4000 labeled samples, and as 50, 100, and 100 for SVHN with 250, 500, and 1000 labeled samples.

Model training. We adopt the mean squared error as the divergence metric in Equation 2 and 4 as in Verma et al. (2019) . The batch size is 32 for labeled data and 128 for unlabeled data. We follow the settings of Verma et al. (2019) for other details: The model is trained for 400 epochs, and optimized using the SGD algorithm with a momentum factor 0.9 and weight decay factor 1×10 -4 ; the learning rate is set to 0.1 initially and then decayed using the cosine annealing strategyLoshchilov & Hutter (2017); a sigmoid warm-up schedule is utilized to increase the regularization weight β from 0 to its maximum value within the first 100 epochs. Our code will be made publicly available soon.

4.3. RESULTS:

The evaluation results of the proposed AdvMixup against several state-of-the-art methods on CIFAR-10 and SVHN are shown in Table 1 and Table 2, 2018)) , in-between neighborhoods (Verma et al. (2019) ), as well as those combining them (Luo et al. (2018) ). From Table 1 and Table 2 , we have the following observations. (Luo et al., 2018) 18.41 ± 0.52 13.64 ± 0.32 10.93 ± 0.14 VAT+EntMin+SNTG (Luo et al., 2018) --9.89 ± 0.34 CT-GAN (Wei et al., 2018) --9.98 ± 0.21 CVT (Clark et al., 2018) --10.11 ± 0.15 MT+ fast-SWA (Athiwaratkun et al., 2019) 15.58 ± 0.12 11.02 ± 0.23 9.05 ± 0.21 ICT (Verma et al., 2019) 15.48 ± 0.78 9.26 ± 0.09 7.29 ± 0.02 AdvMixup 9.67 ± 0.08 8.04 ± 0.12 7.13 ± 0.08 (Park et al., 2018) --3.55 ± 0.05 Π+SNTG (Luo et al., 2018) 5.07 ± 0.25 4.52 ± 0.30 3.82 ± 0.25 MT+SNTG (Luo et al., 2018) 4.29 ± 0.23 3.99 ± 0.24 3.86 ± 0.27 ICT (Verma et al., 2019) 4.78 ± 0.68 4.23 ± 0.15 3.89 ± 0.04 AdvMixup 3.95 ± 0.70 3.37 ± 0.09 3.07 ± 0.18

Firstly, for CIFAR-10, AdvMixup outperforms all the baselines across different numbers of labeled data. In particular, AdvMixup improves the second-best method ICT by nearly 6% when only 1000 labeled samples are given.

Secondly, for SVHN, it is much easier than the task on CIFAR-10 as the house number images of SVHN has smaller variance compared to the natural images of CIFAR-10, and the baselines already achieve a high accuracy. Nevertheless, AdvMixup still demonstrates a clear improvement over all Thirdly, following Verma et al. (2019) , we also compare with the supervised methods (methods in the first block of Table 1 and Table 2 ), where only the labeled samples are used. For both CIFAR-10 SVHN, AdvMixup exhibits significant improvement over the supervised baselines across different numbers of labeled data.

4.4. ABLATION STUDY:

To provide more insights, we present the performance of the following three variants of our model on the CIFAR-10 dataset:

• ICT+VAT, which is an alternative of integrating the local neighborhood and in-between neighborhood approaches by simply combining the losses of the ICT model and the VAT model. • Adv-Adv Mixup, which is another alternative of integrating the local neighborhood and in-between neighborhood approaches by defining the interpolation paths between two adversarial samples, i.e., replacing x i with x (adv) i

in Equation 1. • AdvMixup w/o teacher model, which use the prediction of current model instead of the EMA model to compute the soft labels for the samples, i.e., replacing f t (x i ) and f t (x j ) with f (x i ) and f (x j ) in Equation 1.

The results of these ablated variants are shown in Table 3 . Firstly, ICT+VAT shows a major improvement over ICT with 4000 labels, but underperforms AdvMixup across different numbers of labeled samples. We conjecture the reason is: by enforcing consistency constraint in local neighborhood and the interpolation paths between training samples, ICT+VAT stills have no guarantee on the more difficult interpolation paths between adversarial samples and training samples. Secondly, interpolating between adversarial examples clearly degrade the performance across different numbers of labeled samples. Our explanation is that there can be a gap between the true data distribution and the virtual data distribution defined by this interpolation scheme where real samples are not utilized, thus increasing the prediction errors on the test samples lying in the true data distribution. Thirdly, eliminating the teacher model clearly degrade the performance by 1%-2% across different numbers of labeled samples. However, this difference resulted from the teacher model is smaller than the difference between ICT and ICT w/o teacher model, which is about 4% as reported in Verma et al. (2019) .

5. RELATED WORK:

Semi-supervised learning has been a hot topic for a long time to address the problem of limited labeled data which hinders the learning based models. By leveraging unlabeled data, semi-supervised learning methods are dedicated to designing a regularization term to encourage the model to comply with the cluster assumption Chapelle & Zien (2005) , which favors decision boundaries lying in lowdensity regions and stable model behaviors without abrupt changes. In the following, we mainly concentrate on the consistency regularization methods which represent the state-of-the-art and are mostly related to our work. 2018) proposed the mixup model encouraging the prediction on the linear combination of two samples to approach the linear combination of their labels. The mixup model has been extended from different perspectives owing to its efficiency and strong regularization ability. Verma et al. (2018) extended the mixup operation to the hidden layers for flattening representations. Guo et al. (2019) proposed to adaptively generate the mixing parameter for a specific pair, so as to avoid overlapping between the mixed samples and the real ones. Verma et al. (2019) generalized the mixup model to the semi-supervised setting where the labels are substituted by the soft labels from a teacher model. The concurrent work MixMatch Berthelot et al. (2019) generalized the mixup mechanism with several techniques such as multiple data augmentation and label sharpening, obtaining strong empirical results on semisupervised learning.

6. CONCLUSION:

In this paper, we propose a new consistency regularization method, AdvMixup, for semi-supervised learning. AdvMixup enforces the model to fit the virtual data points sampled from the interpolation paths between adversarial samples and real samples. Such an interpolation scheme integrates the local neighborhood around training samples and the neighborhood in-between the training samples.

Our experiments demonstrate the proposed AdvMixup constantly outperforms the baselines, especially when the labeled data are scarce. In future work, we plan to explore AdvMixup with different adversarial sample generation approaches. Also, it is promising to fit AdvMixup into the recent MixMatch framework Berthelot et al. (2019) for further performance improvement.

:

https://github.com/takerum/vat_tf

Figures and Tables:

Figure 1: Visualization of the consistency regularization areas for different methods. (a) Perturbation based methods. (b) Interpolation based methods. (c) Our AdvMixup.

Figure 2: Overview of the proposed AdvMixup framework.

The mixup model Zhang et al. (2018) and ICT model Verma et al. (2019) encourage the classifier to have linear transition in-between real samples, thus pushing the decision Algorithm 1 Minibatch training of AdvMixup for semi-supervised learning Require: labeled training set S l ; unlabeled training set Su classification model f with parameters θ; f 's EMA version ft with parameters θt Beta distribution parameter α;weight of regularization term β; ft's update ratio γ for k = 1, . . . , num iterations do Sample a labeled batch B l = {(xi, yi)} n l i=1 ∼ S l Sample an unlabeled batch Bu = {xi} nu i=1 ∼ Su Compute the negative log-likelihood loss using B

Figure 3: Prediction error rates of the model (trained with only labeled data) on the virtual samples along the interpolation paths (blue line) defined by the ICT model Verma et al. (2019) and the interpolation paths (orange line) defined by the proposed AdvMixup. (a) Results on the CIFAR-10 dataset where 4000 labeled data samples are used. (b) Results on the SVHN dataset where 1000 labeled data samples are used. Best viewed in color.

Figure 4: Comparison between VAT, ICT, and proposed AdvMixup on the concentric circles dataset. Red and blue points denote unlabeled samples from the two classes, i.e., inner circle and outter circle, and labeled data are marked with black triangles. (a) The concentric circles dataset. (bd) The contour plot and the decision boundaries (green curve) learned by VAT (b), ICT (c), and AdvMixup (d). Best viewed in color.

IMPLEMENTATION DETAILS Data preprocessing. Following our baselines, we adopt standard data augmentation and data normalization in the preprocessing phase. On the CIFAR-10 dataset, we first augment the training data by random horizontal flipping and random translation (in the range of [-2,2] pixels), and then apply global contrast normalization and ZCA normalization based on statistics of all training samples. On the SVHN dataset, we first augment the training data by random translation (in the range of [-2,2] pixels), and then apply zero-mean and unit-variance normalization. Model architecture. We adopt the exactly same 13-layer convoluation neural network architecture as in the ICT model Verma et al. (2019), which eliminates the dropout layers compared to the variant in Sajjadi et al. (2016); Laine & Aila (2017); Tarvainen & Valpola (2017); Miyato et al. (2018); Luo et al. (2018).

respectively. The baseline semi-supervised learning methods encompass consistency regularization methods based on local neighborhoods (Laine & Aila (2017); Tarvainen & Valpola (2017); Miyato et al. (2018); Park et al. (2018); Athiwaratkun et al. (2019); Clark et al. (

Test error rates (%) of different methods on CIFAR-10. Random horizontal flipping and random translation are used to augment training data. Results for methods in the first block (i.e., Supervised, Supervised (Mixup), and Supervised (Manifold Mixup)) are duplicated fromVerma et al. (2019). Results of AdvMixup are averaged over 3 runs.

Table content: <table><tr><td>Method</td></tr></table>

Test error rates (%) of different methods on SVHN. Random translation is used to augment training data. Results for methods in the first block (i.e., Supervised, Supervised (Mixup), and Supervised (Manifold Mixup)) are duplicated fromVerma et al. (2019). Results for AdvMixup are averaged over 3 runs.

Table content: <table><tr><td>Method</td><td colspan="2">Test error rates (%) 250 labels 500 labels</td><td>1000 labels</td></tr><tr><td>Supervised</td><td colspan="3">40.62 ± 0.95 22.93 ± 0.67 15.54 ± 0.61</td></tr><tr><td>Supervised (Mixup)</td><td colspan="3">33.73 ± 1.79 21.08 ± 0.61 13.70 ± 0.47</td></tr><tr><td>Supervised (Manifold Mixup)</td><td colspan="3">31.75 ± 1.39 20.57 ± 0.63 13.07 ± 0.53</td></tr><tr><td>Π model (Laine &amp; Aila, 2017)</td><td>9.93 ± 1.15</td><td>6.65 ± 0.53</td><td>4.82 ± 0.17</td></tr><tr><td>TempEns (Laine &amp; Aila, 2017)</td><td>12.62 ± 2.91</td><td>5.12 ± 0.13</td><td>4.42 ± 0.16</td></tr><tr><td>MT (Tarvainen &amp; Valpola, 2017)</td><td>4.35 ± 0.50</td><td>4.18 ± 0.27</td><td>3.95 ± 0.19</td></tr><tr><td>VAT (Miyato et al., 2018)</td><td>-</td><td>-</td><td>5.42 ± 0.22</td></tr><tr><td>VAT+EntMin (Miyato et al., 2018)</td><td>-</td><td>-</td><td>3.86 ± 0.11</td></tr><tr><td>VAdD (Park et al., 2018)</td><td>-</td><td>-</td><td>4.16 ± 0.08</td></tr><tr><td>VAdD + VAT</td><td/><td/><td/></tr></table>

Test error rates (%) of different ablated versions on CIFAR-10. Random horizontal flipping and random translation are used to augment training data. Results are averaged over 3 runs.

Table content: <table><tr><td>Method</td><td colspan="2">Test error rates (%) 1000 labels 2000 labels</td><td>4000 labels</td></tr><tr><td>AdvMixup</td><td colspan="3">9.67 ± 0.08 8.04 ± 0.12 7.13 ± 0.08</td></tr><tr><td>ICT + VAT</td><td>9.91 ± 0.25</td><td>8.90 ± 0.20</td><td>7.97 ± 0.03</td></tr><tr><td>Adv-Adv Mixup</td><td>10.90 ± 0.11</td><td>8.99 ± 0.14</td><td>8.22 ± 0.09</td></tr><tr><td colspan="2">AdvMixup w/o teacher model 11.64 ± 0.41</td><td>9.78 ± 0.11</td><td>8.20 ± 0.17</td></tr><tr><td colspan="4">the baselines across different numbers of labeled data. In particular, AdvMixup achieves an error</td></tr><tr><td colspan="4">rate of 3.95% for 250 labels, which beats the results of 500 labels of all baselines.</td></tr></table>

An important research line in consistency regularization constrains the model to have consistent predictions around local neighborhood around training inputs, where the local neighborhood is usually represented as variants of the input or model parameters. The Π model fromSajjadi et al. (2016) andLaine & Aila (2017) constructed different input variants with stochastic image transformation and additive Gaussian noise, as well as different model variants with dropout layers.Wei et al. (2018) integrated the Π model with the generative adversarial networks (GAN) based semisupervised learning approachesSalimans et al. (2016), where the classifier was forced to correctly classify labeled samples and distinguish real unlabeled samples and fake samples from the generator.Clark et al. (2018) proposed cross-view learning which formed input variants by randomly masking partial inputs.Laine & Aila (2017) also proposed a Temporal Ensembling approach to apply the consistency constraint between current model prediction and the EMA of all historical predictions for a specific input.Tarvainen & Valpola (2017) further improved Temporal Ensembling by considering the consistency between the variants of model parameters, i.e., predictions from current parameters and from the EMA of parameters. Considering the insufficient power of the isotropic perturbations,Miyato et al. (2018) proposed the VAT model by using adversarial perturbations, which point out the model's most vulnerable direction, to better represent the local neighborhood. Another promising research line in consistency regularization considers the consistency between pairs of training samples. Luo et al. (2018) enhanced the local neighborhood based methods by narrowing down the distance between similar sample pairs while pushing the dissimilar pairs away in the low-dimensional feature space. Under supervised setting, Zhang et al. (

Table content: <table/>

Peer review (iclr2020):
Summary This paper looks at the semi-supervised learning problem, where a combination of labeled and unlabeled data is available. It builds off of two major ideas which follow the general trend of perturbing unlabeled data and applying a consistency loss between the original and perturbed outputs: 1) virtual adversarial training (VAT) which picks an adversarial direction for perturbation, and 2) mixup which interpolates between images or features of real data. The method specifically pertubs unlabeled examples by interpolating between itself and the adversarial example. Results are shown on synthetic and standard semi-supervised image datasets. Overall, my decision is to reject this paper. The idea is an extremely simple combination of two prior methods, with little justification for why it is the correct thing to do. Indeed the ablation shows that other methods work similarly (ICT+VAT) and newer papers such as MixMatch (which was out in May and accepted to NeurIPS) perform better than this method with normal augmentations and mixup (with multi-view pseudolabeling), showing that the underlying thesis of the work is not necessarily the most important part of the semi-supervised learning problem. Argument: - The set of justifications put forth for the method are not rigorous. While I agree that combining multiple methods such that a larger space of augmentations is used (*our AdvMixup explores a more comprehensive searching area*), it is not clear to me why *this* particular combination is more justified than any others one can come up with (mixup between PI perturbation, random augmentations, etc.). In the end, various things are tried and some work better than others. Indeed ICT+VAT already does very well. It*s not clear to me that this paper therefore discovers anything novel in terms of generalizable information that can be used for other problems (for example). - MixMatch performs better than this method, and has been out since May and accepted at NeurIPS 2019. I would not say it is concurrent given that it has been out for a while now. Further, it shows that the underlying thesis of the method in this paper is not necessarily the crux of the semi-supervised learning problem. - VAT is known to be extremely sensitive in terms of its hyperparameters (e.g. epsilon), which has to be tuned per-dataset (you use the value from the paper, which did this tuning to a larger validation set). Does this method offer any decreased sensitivity to that? - I would not say VAT has a low computational overhead; you need an additional forward and backward pass per batch, which is not insignificant. Some specific questions: - Why do you use 5k validation examples for cifar-10 and what size labeled data does that correspond to (1k, 2k, etc)? This seems to go against the Oliver et al. recommendations (which you cite).