Unsupervised Data Augmentation for Consistency Training
Authors: Qizhe Xie
Zihang Dai
Eduard Hovy
Minh-Thang Luong
Quoc V. Le
Venue:  (2020)
Keywords: Semi-supervised learning
computer vision
natural language processing

Abstract: Semi-supervised learning lately has shown much promise in improving deep learning models when labeled data is scarce. Common among recent approaches is the use of consistency training on a large amount of unlabeled data to constrain model predictions to be invariant to input noise. In this work, we present a new perspective on how to effectively noise unlabeled examples and argue that the quality of noising, specifically those produced by advanced data augmentation methods, plays a crucial role in semi-supervised learning. By substituting simple noising operations with advanced data augmentation methods, our method brings substantial improvements across six language and three vision tasks under the same consistency training framework. On the IMDb text classification dataset, with only 20 labeled examples, our method achieves an error rate of 4.20, outperforming the state-of-the-art model trained on 25,000 labeled examples. On a standard semi-supervised learning benchmark, CIFAR-10, our method outperforms all previous approaches and achieves an error rate of 2.7% with only 4,000 examples, nearly matching the performance of models trained on 50,000 labeled examples. Our method also combines well with transfer learning, e.g., when finetuning from BERT, and yields improvements in high-data regime, such as ImageNet, whether when there is only 10% labeled data or when a full labeled set with 1.3M extra unlabeled examples is used.

Under review as a conference paper at ICLR 2020
UNSUPERVISED DATA AUGMENTATION FOR CONSISTENCY TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Semi-supervised learning lately has shown much promise in improving deep learning models when labeled data is scarce. Common among recent approaches is the use of consistency training on a large amount unlabeled data to con- strain model predictions to be invariant to input noise. In this work, we present a new perspective on how to effectively noise unlabeled examples and argue that the quality of noising, specifically those produced by advanced data augmenta- tion methods, plays a crucial role in semi-supervised learning. By substituting simple noising operations with advanced data augmentation methods, our method brings substantial improvements across six language and three vision tasks under the same consistency training framework. On the IMDb text classification dataset, with only 20 labeled examples, our method achieves an error rate of 4.20, outper- forming the state-of-the-ar art model trained on 25, 5,,000 labeled examples. On a stan- dard semi-supervisee learning benchmark, CIFAR-10 10, our method outperforms all previous approaches and achieves an error rate of 2.7% with only 4,000 examples, nearly matching the performance of models trained on 50,000 labeled examples. Our method also combines well with transfer learning, e.g.. when finetuning from BERT, and yields improvements in high-data regime, such as ImageNet, whether when there only 10% labeled data or when a full labeled set with 1.3M extra unlabeled examples is used.
INTRODUCTION
A fundamental weakness of deep learning is that it typically requires lot of labeled data work
well. Semi-supervised learning (SSL) (Chapelle et al. 2009) is one of the most promising methods of leveraging unlabeled data address this weakness. The recent works in SSL are diverse but
those that are based on consistency training (Bachman et al. 2014; Rasmus et al., 2015: Laine &
Aila 2016: Tarvainen Valpola 2017 have shown work well on many benchmarks.
In a nutshell, consistency training methods simply regularize model predictions to be invariant to small noise applied to either input examples (Miyato al. [2018] Sajjadi et al. 2016 Clark al., 20188) or hidden states (Bachman et al.) 2014 Laine & Aila (016)) This framework makes sense intuitively because a good model should be robust to any small change in an input example hidden states. Under this framework, different methods in this category differ mostly in how and where the noise injection is applied. Typical noise injection methods are additive Gaussian noise, dropout noise or adversarial noise.
In this work, we investigate the role of noise injection in consistency training and observe that advanced data augmentation methods, specifically those work best in supervised learning (Simard et al. 1998 Krizhevsky et al. 2012: Cubuk et al. 2018 Yu et al.) 2018), also perform well in semi- supervised learning. There is indeed strong correlation between the performance of data augmen- tation operations in supervised learning and their performance in consistency training. We, hence, propose to substitute the traditional noise injection methods with high quality data augmentation methods in order to improve consistency training. To emphasize the use of better data augmentation in consistency training, we name method Unsupervised Data Augmentation or UDA.
Cooe available anonymous link.

---

Under review as a conference paper at ICLR 2020
We evaluate UDA wide variety of language and vision tasks. On six text classification tasks, method achieves significant improvements over state-of-the-art models. Notably, on IMDb, UDA with 20 labeled examples outperforms the state-of-the-ee tr-tte-rt model trained on 1250x more labeled data. We also evaluate UDA on standard semi-supervised learning benchmarks for vision such as CIFAR-10 and SVHN. UDA outperforms all existing semi-supervised learning methods by signif- icant margins. On CIFAR-10 with 4,000 labeled examples, UDA achieves an eπrоr rate of 5.29, nearly matching the performance of the fully supervised model that uses 50,000 labeled examples. UDA achieves a new state-of-
Furthermore, with better architecture, PyramidNet+Shakee the-art eтror rate of 2.7. On SVHN, UDA achieves an error rate of 2.55 with only 1,000 labeled examples. Finally, we also find UDA to be beneficial when there is a large amount of supervised data. For instance, on ImageNet, UDA leads to improvements of top-1 accuracy from 58.84 to 68.78 with 10% of the labeled set and from 78.43 to 79.05 when we use the full labeled set and an external dataset with 1.3M unlabeled examples.
Our key contributions and findings can be summarized as follows:
First, we show that state-of-the-ar data augmentations found in supervised learning can also serve as a superior source of perturbation under the consistency enforcing semi-supervised framework. See results in Table and Table 2
Second, we show that UDA match and even outperform purely supervised learning that uses orders of magnitude more labeled data.
State-of-the-art results for both vision and language tasks reported in Table 3 and 4 The effectiveness of UDA across different training data sizes are highlighted in Figure andd
Finally, show that UDA combines well with transfer learning, c.g., when fine-tuning from BERT (see Table 4. and is effective at high-data regime, e.g. on ImageNet (see T).
UNSUPERVISED DATA AUGMENTATION (UDA)
In this section, first formulate our task and then present the key method and insights behind UDA. Throughout this paper, we focus on classification problems and will use x to denote the input and y" to denote its ground-truth prediction target. We are interested in learning a model pa x) to predict y² based on the input x, where 0 denotes the model parameters. Finally, we will use L and U to denote the sets of labeled and unlabeled examples respectively.
2.1 BACKGROUND: SUPERVISED DATA AUGMENTATION
Data augmentation aims creating novel and realistic-looking training data by applying trans- formation to an example, without changing its label. Formally, let q( x) be the augmentation transformation from which one draw augmented examples å based on original example x. For an augmentation transformation to be valid, it is required that any example è ~ g( x) drawn from the distribution shares the same ground truth label as x. Given a valid augmentation transfor- mation, we can simply minimize the negative log-likelihodd on augmented examples.
Supervised data augmentation can be equivalently seen constructing augmented labeled set from the original supervised set and then training the model on the augmented set. Therefore, the augmented set needs to provide additional inductive biases be more effective. How to design the augmentation transformation has, thus, become critical.
In recent years, there have been significant advancements on the design of data augmentations for NLP Yu et al. 2018)) vision (Krizhevsky et 2012) [Cbuuk et al. (2018) and speech (Hannun et al.) 2014 Park et al. 2019) in supervised settings. Despite the promising results, data augmentation mostly regarded as the "cherry on the cake" which provides steady but limited performance boost because these augmentations has so far only been applied to a set of labeled examples which is usually of small size. Motivated by this limitation, via the consistency training framework, we extend the advancement in supervised data augmentation to semi-supervised learning where abundant unlabeled data is available.
2

---

Under review as a conference paper at ICLR 2020
Finel Loss
8
Ps Or ())
M
Back translation
Supervised
Unsupervisdd Conrisleacy Loes
RandAngment
Craesventzopy Loss
Augnectations
TF-IDF
replaceeet
O (x)
M
6 x)
M
Labeled Data
Unlabbeldd Data
Figure 1: Training objective for UDA, where M is model that predicts distribution of y given
2.2 UNSUPERVISED DATA AUGMENTATION
As discussed in the introduction, recent line of work in semi-supervised learning has been utilizing unlabeled examples to enforce smoothness of the model. The general form of these works can be summarized as follows:
Given an input v, compute the output distribution poly 2) given æ and a noised version po
I, t) by injecting small noise e. The noise can be applied to x or hidden states.
Minimize divergence metric between the two distributions D (pa(y) x)) pe(y x)))).
This procedure enforces the model be insensitive to the noise and hence smoother with respect to changes in the input (or hidden) space. From another perspective, minimizing the consistency loss gradually propagates label information from labeled examples to unlabeled ones.
In this work, = q(x, E), as considered by prior works (Sajjadi et al., 2016 Laine & Aila 2016 Miyato et al. 2018). But different from existing work, focus on the unattended question of how the form or "quality" of the noising operation can influence the performance of this consistency training frame- work. Specifically,
interested in particular setting where the noise is injected to the input i.c.,
enforce consistency, prior methods generally employ simple noise injection methods such as adding Gaussian noise, simple input augmentations to noise unlabeled examples. In contrast, we hypothesize that stronger data augmentations supervised learning can also lead to superior performance when used to noise unlabeled examples in the semi cumi-suereiseed consistency training framework, since has been shown that more advanced data augmentations that are more diverse and natural can lead to significant performance gain in the supervised setting.
Following this idea, we propose to use a rich set of state-of oftee-of-theaat data augmentations verified in various supervised settings to inject noise and optimize the same consistency training objective on unlabeled examples. When jointly trained with labeled examples, we utilize a weighting factor to balance the supervised cross entropy and the unsupervised consistency training loss, which is illustrated in Figure|
[ Formally, the full objective can be written as follows:
min J(0) = EEz log (yy" =)] + XEzeu Ezrg(H(z) [DKL (pa(y) ()) po +)))] 0
where q(t x)) is a data augmentation transformation and Ø is a fixed copy of the current parameters 0 indicating that the gradient is not propagated through 6, as suggested by Miyato et al. 20118). We also follow VAT (Miyato et al.) 2018) to use the KL divergence. We set À to for most of our experiments and use different batch sizes for the supervised data and the unsupervised data. In the vision domain, simple augmentations including cropping and flipping are applied to labeled examples. To minimize the discrepancy between supervised training and prediction unlabeled examples, we apply the same simple augmentations to unlabeled examples for computing ( x))
Discussion. Before detailing the augmentation operations used in this work, first provide some intuitions on how more advanced data augmentations can provide extra advantages over simple ones used in earlier works from three aspects:
Valid noise: Advanced data augmentation methods that achieve great performance supervised learning usually generate realistic augmented examples that share the same ground-truth labels
3

---

Under review as a conference paper at ICLR 2020
with the original example. Thus, it is safe to encourage the consistency between predictions the original unlabeled example and the augmented unlabeled examples.
Diverse noise: Advanced data augmentation can generate diverse set of examples since it can make large modifications to the input example without changing its label, while simple Gaussian noise only make local changes. Encouraging consistency on diverse of augmented examples can significantly improve the sample efficiency.
Targeted inductive biases: Different tasks require different inductive biases. Data augmentation operations that work well in supervised training essentially provides the missing or most wanted inductive biases in an original labeled set.
2.3 AUGMENTATION STRATEGIES FOR DIFFERENT TASKS
We now detail the augmentation methods, tailored for different tasks, that we use in this work.
RandAugment for Image Classification, We make use of data augmentation method called RandAugment, which is inspired by AutoAugment (Cubuk et al. (0188). AutoAugment uses search method to combine all image processing transformations in the Python Image Library (PIL) to find a good augmentation strategy. In RandAugment, we do not use search, but instead uniformly sample from the same set of augmentation transformations in PIL. other words, RandAngment is simpler and requires no labeled data there is need to search for optimal policies.
Back-translation for Text Classification. translation (Sennrich et al.) 2015 Edunov et al. 2018) refers to the procedure of translating an existing example x in language A into another language B and then translating it back into A to ob- tain augmented example át As observed by Yu al. (2018), back-translation can generate diverse paraphrases while preserving the semantics of the original sentences, leading to significant perfor- mance improvements in question answering, our case, we use sack-translation paraphrase the training data of our text classification tasks
When used as an augmentation method, back-
We find that the diversity of the paraphrases more important than the quality the validity. Hence, we employ random sampling with a tunable temperature instead of beam search for the generation. As shown Figure 2 the paraphrases generated by back translatiioon sentence are diverse and have similar semantic meanings. More specifically, we use WMT' 14 English-French translation models (in both directions) to perform back-translation on each sentence. To facilitate future research, we have open- spen-sourced our sack-rranslation system together with the translation checkpoints.
Since was highly limited in terms budget, the production restrictions, the film was choerful.
Given the low buigee and
Buelstranslation
There are few budget iteras and production linitations to make this film a really good one.
production limitations, this movie is very good.
Dioe to the small dollar amount and production limitations the onest, film very beantiful.
RoudAngment
Figure 2: Augmented examples using
and RandAugment.
Word replacing with TF-IDF for Text Classification. While back-translation is good at maintain- ing the global semantics of sentence, there is little control over which words will be retained. This requirement is important for topic classification tasks, such as DBPedia, in which some keywords are more informative than other words in determining the topic. We, therefore, propose an augmen-
We also note that while translation uses labeled dataset, the translation task itself is quite distinctive from a text classification task and does not make use of any text classificationn label. In addition, hack-translation is general data augmentation method that can be applied to many tasks with the same model checkpoints.
4

---

Under review as a conference paper at ICLR 2020
tation method that replaces uninformative words with low TF-IDF scores while keeping those with high TF-IDF values. We refer readers to Appendix for a detailed description.
2.4 TRAINING SIGNAL ANNEALING FOR LOW-DATA REGIME
In semi-supervised learning, we often encounter a situation where there huge gap between the amount of unlabeled data and that of labeled data. Hence, the model often quickly overfits the limited amount of labeled data while still underfitting the unlabeled data. To tackle this difficulty, we introduce new training technique, called Training Signal Annealing (TSA), which gradually releases the "training signals" of the labeled examples as training progresses. Intuitively, we only utilize a labeled example if the model's confidence on that example is lower than predefined thresh- old which increases according to a schedule. Specifically, at training step t, if the model's s predicted probability for the correct category py x)) is higher than threshold T). we remove that example from the loss function. Suppose K is the number of categories, by gradually increase from к to the threshold Y/c serves ceiling to prevent over-training easy labeled examples.
We consider three increasing schedules of f with different application scenarios. Let 7 be the total number of training steps, the three schedules shown in Figure 3 Intuitively, when the model is prone to overfit, e.g., when the problem is relatively easy or the number of labeled examples is very limited, the exp-schedule is most suitable as the supervised signal mostly released at the end of training. In contrast, when the model is less likely to overfit (e.g., when we have abundant labeled examples or when the model employs effective regularization), the log- schedulee can serve well.
02
02
0.8
64
1.8
60
66
Figure 3: Three schedules TSA. We set т)t = a; * (1- +) + +. a is set to exp(-f * 5),
L and exp(( 1) 5) for the log, linear and exp schedules.
T
3 EXPERIMENTS
In this section, we evaluate UDA on variety of language and vision tasks. For language, we rely on six text classification benchmark datasets, including IMDb, Yelp-2, Yelp-5, Amazon-2 and Amazon- 5 sentiment classification and DBPedia topic classification (Maas al. 2011 Zhang et al. 2015). For vision, we employ two smaller datasets CIFAR-10 10 (Krizhevsky & Hinton 2009). SVHN (Net-) zer et al.] (201I). which are often used to compare semi semi-supervised algorithms, as well Ima- geNet (Deng et al.] 2209) of a larger scale to test the scalability of UDA. For details of the laheled and unlabeled data and experiment details, we refer readers to Appendix E
3.1 CORRELATION BETWEEN SUPERVISED AND SEMI-SUPERVISED PERFORMANCES
As the first step, we try verify the fundamental idea of UDA, i.e., there is positive correlation of data augmentation's 's effectiveness in supervised learning and semi-supervised learning. Based on Yelp-5 (a language task) and CIFAR-10 (a vision task), compare the performance of different data augmentation methods in either fully supervised or semi-supervised settings. For Yclp-5, apart from back-translation we include simpler method Switchout (Wang al. 2018) which replaces a token with a random token uniformly sampled from the vocabulary. For CIFAR-10, we compare RandAugment with two simpler methods: (1) cropping & flipping augmentation and (2) Cutout.
Based on this setting, Table and Table 2 exhibit strong correlation of an augmentation's 's ef- fectiveness between supervised and semi-supervised settings. This validates idea of stronger
5

---

Under review as a conference paper at ICLR 2020
data augmentations found in supervised learning can always lead to more gains when applied to the semi-supervised learning settings.
Augmentation
Sup (50k)
Semi-Sup (4k)
Augmentation
Sup (650k)
Semi-sup (2.5k)
(# Sup examples)
(# Sup examples)
Crop & flip
5.36
16.17
X
38.36
50.80
Cutout
4.42
6.42
Switchout
37.24
43.38
RandAugment
4.23
5.29
36.71
41.35
Table I: Error rates on CIFAR-10 100.
Table 2: Error rate on Yelp-5.
3.2 ALGORITHM COMPARISON ON VISION SEMI-SUPERVISED LEARNING BENCHMARKS
With the correlation established above, the next question we ask is how well UDA performs com- pared existing semi-supervised learning algorithms. To answer the question, focus on the most commonly used semi-supervised learning benchmarks CIFAR-10 and SVHN.
Vary the size of labeled data. Firstly, follow the settings in (liver al. 2018) and employ -idt-Res (Zagoruyko & Komodakis 2016 He et al. 2016) as the backbone model and evaluate UDA with varied supervised data sizes. Specifically, we compare UDA with two highly competitive baselines: (1) Virtual adversarial training (VAT) (Miyato et al. 2018), an algorithm that generates adversarial Gaussian noise on input, and (2) MixMatch (Berthelot et al. 2019). a parallel work that combines previous advancements in semi-supervised learning. The comparison is shown in Figure 4 with two key observations F
First, UDA consistently outperforms the two baselines with a clear margin given different sizes of labeled data.
Moreover, the performance difference between UDA and VAT shows the superiority of data aug- mentation based noise. The difference of UDA and VAT is essentially the noise process. While the noise produced by VAT often contain high-frequency artifacts that do not exist in real images, data augmentation mostly generates diverse and realistic images.
46.0
VAT
USA Bledkggnerr
MoVacc
7.0
6.0
28.0
5.0
15.0
4.3
18.0
3.3
5.
2.0
299
.90
10:00
2006
4392
150
300
1300
2000
4000
#Labded euamples
If Labeldd moompls
(a) CIFAR-10
(b) SVHN
Figure 4: Comparison with two semi-supervised learning methods on CIFAR-10 and SVHN with varied number of labeled examples.
Comparisons with published results Next, directly compare UDA with previously published results under different model architectures. Following previous work, 4k and 1 labeled examples are used for CIFAR-I0 and SVHN respectively. As shown in Table 3 given the same architecture, UDA outperforms all published results by significant margins. This shows the huge potential of state-of -hef-thhetttt data augmentations under the consistency training framework in the vision domain.
for the case of 250 examples on -1. The hyperparameters for other data sizes are the same. For the case with 250 examples on CIFAR-10, 10, applying hyperparameters used in other data sizes leads to an error rate of 16.84 + 4.19, which might be resulted from stability issue.
3 Please refer to Appendix E.2 for detailed hyper-parametee We only use a different

---

Under review as a conference paper at ICLR 2020
Method
Model
# Param
CIFAR-I0 (4k)
SVHN (1k)
II-Mode (Laine & Aila 2016)
Conv-Large
3.1M
12.36 0.31
4.82
Mcan Teacher Tarvamen Valpola 2017)
Conv-Large
3.1M
12.31 0
3,95 0.19
VAT + EntMin MMyato ot 2018
Conv- Large
3.1M
10.55 0.05
0.05
3.86 = 0.11
SNTG Luo ct al.
2018)
Conv- Large
3.1M
10,93 0.14
3,86 = 0.27
VAdD Park et al.
2018
Conv-Large
3.IM
11.32 1 0.11
4.16 - 0.08
Fast-SWA (Athrwaratkun al. 2018
Conv-Large
3.1M
9.05
ICT VVrma al. (0199)
Conv-Large
3.1M
7.29
0.02
+ 0.04
Pseudo-Labee (Lee) 2013
WRN-28-2
1.5M
16.21
0.11
7.62 + 0.29
LGA VAT (Jackson & Schulman
2019
WRN-28-2
1.5M
0.19 12.06
6.58 + 0.36
mixmixup (Hataya Nakayama 2019
WRN-28-2
L5M
10
ICT (Verma al. (2019)
WRN-28-2
1,5M
7.66
= 0.07
MixMatch (Berthelot
2019
WRN-28-2
1.5M
6,24 0.06
2.89 + 0.06
Mcan Teacher (Tarvainnn & Valpola
2017 77
Shake-Shake
26M
6.28 + 0.15
Fast-SWA (Athwaratkknn et 2018
Shake-Shake
26M
5.0
MixMatch [Berthelot
WRN
26M
4,95 + 0.08
UDA (RandAugment)
WRN-28-2
1.5M
5.29 = 0.25
2.55 + 0.09
UDA (RandAugment) UDA (RandAugment)
Shake-Shake
26M
3.7
PyramidNet 26M
2.7
Table 3: Comparison between methods using different models where PyramidNet is used with ShakeDrop regularization, Fully supervised Wide-ResNett28. and PyamiddNe++ShakeDDop have an emor rate of 5.4 and 2.7 when trained on 50,000 examples without RandAAugmett, On CIFAR- 10, with only 4,000 labeled examples, UDA matches the performance models. On SVHN, UDA also matches the performance of our fully supervised model trained on 73,257 examples without RandAugment, which has an error rate of 2.84.
the two fully supervised
3.3 EVALUATION ON TEXT CLASSIFICATION DATASETS
Next, we further evaluate UDA in the language domain. Moreover, in order to test whether UDA can be combined with the success of unsupervised representation learning, such as BERT (Devlin et al. 2018). we further consider four initialization schemes: (a) random Transformer; (b) BERT BERTRASES BaSE (c) BERTLARGE: (d) BERT FINETUNE: BERT.ARGE fine-tuned on in-domain unlabeled data Under each of these four initialization schemes, we compare the performances with and without UDA.
The results are presented Table where we would like to emphasize three observations:
competitive perfor- mances compared to the SOTA model trained with full supervised data. Particularly, on binary sentiment analysis tasks, with only 20 supervised examples, UDA outperforms the previous SOTA trained with full supervised data on IMDb and is competitive on Yelp-2 and Amazon-2.
First, with very few labeled examples, UDA offer decent
Second, UDA is complementary to transfer learning / representation learning. As see, when initialized with BERT and further finetuned on in-domain data, UDA can still significantly reduce the error rate from 6.50 4.20 IMDb.
Finally, we also note that for five-category sentiment classification tasks, there still exists clear gap between UDA with 500 labeled examples per class and BERT trained on the entire supervised set. Intuitively, five-category sentiment classifications are much more difficult than their binary counterparts. This suggests room for further improvement in the future.
Results with different labeled set sizes. We also show in Figure that UDA leads to consistent improvements across all labeled data sizes IMDb and Yelp-2.
"One exception is that we do not pursue BERTINETUUEE on DBPedia as fine-tuning BERT on DBPedia does not yield further performance gain. This is probably due to the fact that DBPedia is based on Wikipedia while BERT is already trained the whole Wikipedia corpus.

---

Under review as a conference paper at ICLR 2020
Fully supervised baseline
Datasets
IMDb
Yelp-2 (560k)
Yelp-5 (650k)
Amazon-2 (3.6m)
Amazon-5 (3m)
DBpedia (560k)
(# Sup examples)
(25k)
Pre-BERT
SOTA
4.32
2.16
29.98
3.32
34.81
0.70
BERTLARG
4.51
1.89
29.32
2.63
34.17
0.64
Semi-supervised setting
IMDb (20)
Yelp-2
Yelp-5
Amazon-2 (20)
Amazon-5 (2.5k)
DBpedia
Initialization
UDA
(20)
(2.5k)
(140)
X
43.27
40.25
50.80
45.39
55.70
41.14
Random
25.23
8.33
41.35
16.16
44.19
7.24
X
18.40
13.60
41.00
26.75
44.09
2.58
BERTRASE BASE
5.45
2.61
33.80
3.96
38.40
1.33
BERTLARGG ARGE
/
X
11.72
10.55
38.90
15.54
42.30
1.68
4.78
2.50
33.54
3.93
37.80
1.09
X
6.50
2.94
32.39
12.17
37.32
BERT FINETUNE
4.20
2.05
32.08
3.50
37.12
Table 4: Error rates text classification datasets. In the fully supervised settings, the pre-BERT SO- TAs include ULMFiT (Howard & Ruder 2018) for Yelp-2 and Yelp-5, DPCNN (Johnson & Zhang 2017) for Amazon-2 and Amazon-5, Mixed VAT (Sachan et al. 2018) for IMDb and DBPedia. All of our experiments use a sequence length of 512.
98.8
96.0
99.3
95.5
Ascunay
Ascmncy
99.0
95.0
97.8
96.5
94.0
Superived (BERT-Ginct
Semi- Superciedd (MA)
Semi-Supervieed (1 (LA))
Superiveal (BERT-find
Previsux
Previous SOTA
93.5
97.0
2000
25000
200
2000
20000)
exumples
examples
(a) IMDb
(b) Yelp-2
Figure 5: Accuracy on IMDb and Yelp-2 with different number of labeled examples. In the large- data regime, with the full training set of IMDb, UDA also provides robust gains.
3.4 SCALABILITY TEST ON THE IMAGENET DATASET
Then, to evaluate whether UDA can scale to problems with large scale and a higher difficulty, we now turn the ImageNet dataset with ResNet-50 being the underlying architecture. Specifically, we consider two experiment settings with different natures:
We use 10% of the supervised data of ImageNet while using all other data as unlabeled data. As result, the unlabeled exmaples are entirely in-domain.
In the second setting, we keep all images in ImageNet as supervised data. Then, we use the domain-relevane data filtering method (See Appendix B for details) filter out 1.3M images from an anonymous dataset. Hence, the unlabeled set is not necessarily in-domain.
The results are summarized in Table 5 In both 10% and the full data settings, UDA consistently brings significant gains compared to the supervised baseline. This shows UDA is not only able to scale but also able utilize out-of-domain unlabcled examples to improve model performance. In parallel to our work, S4L (Zhai et al.] 2019b) and CPC (Hénaff et al., 2019) also show significant improvements ImageNet.
8

---

Under review as a conference paper at ICLR 2020
Methods
SSL
10%
100%
ResNet-S0
$5.09 / 77.26
77.28 /9.73
x
w. RandAugment
58.84 / 80.56
78.43 94.37
UDA (RandAugment)
68.78 / 88.80
79.05 94.49
Table 5: Top-1 / top-5 accuracy on ImageNet with 10% and 100% of the labeled set. We use image size 224 and 331 for the 10% and 100% experiments respectively.
3.5 ABLATION STUDIES FOR TSA
Lastly, study the effect of TSA where we have only 2.5k labeled examples and 6m unlabeled examples. (b) CIFAR-I0 where we have 4k labeled examples and 50k unlabeled examples. For Yelp-5, we use randomly initialized transformer in this study to rule out factors of having a pre-trained representation.
tasks with different amounts of unlabeled data: (a) Yelp-5
As shown in Table 6 on Yelp-5, where there is lot more unlabeled data than labeled data, TSA reduces the eπror rate from 50.81 to 41.35 when compared to the baseline without TSA. More specifically, the best performance achieved when we choose to postpone releasing the supervised training signal to the end of the training, i.e, exp-schedule leads to the best performance. On the other hand, linear- schedule is the sweet spot on CIFAR- -10 in terms of the speed of releasing supervised training signals, where the amount of unlabeled data is comparable to that of supervised data.
TSA schedule
Yelp-5
CIFAR-10
50.81
5.67
log-schedule
49.06
5.67
linear-schedule
45.41
5.29
exp-schodule
41.35
7.81
Table 6: Ablation study for Training Signal Annealing (TSA) on Yelp-5 and CIFAR-10. The shown numbers are error rates.
RELATED WORK
Existing works in consistency training does make use of data augmentation (Laine & Aila 2016 Sajjadi et al. 2016); however, they only apply weak augmentation methods such as random transla- tions and cropping. In parallel to our work, ICT (Verma al.) 2019) and MixMatch (Berthelot al., 2019) also show improvements for semi-supervised learning. These methods employ mixup (Zhang al., 2017) on top of simple augmentations such as flipping and cropping: instead, UDA emphasizes on the use of state-of-the-art data augmentations, leading to significantly better results CIFAR-10 and SVHN. addition, UDA also applicable to language domain and can also scale well to more challenging vision datasets, such as ImageNet.
Other works in the consistency training family mostly differ how the noise is defined: Pseudo- ensemble Bachhaan et al., 2014) directly applies Gaussian noise and Dropout noise; VAT (Miyato ctt al. 2018|220110 2016) defines the noise by approximating the direction change in the input space that the model is most sensitive to; Cross-view training (Clark et al. 2018) masks part of the input data. Apart from enforcing consistency on the input examples and the hidden representations, an- other line of research enforces consistency on the model parameter space. Works in this category in- clude Mean Teacher (Tarvainen & Valpola (2017). fast-Stochastic Weight Averaging (Athiwaratkun al. 2018) and Smooth Neighbors Teacher Graphs (Luo al. 2018) For complete version of related work, see Appendix
5 CONCLUSION
In this paper, we show that data augmentation and semi-supervised learning are well connected: better data augmentation lead to significantly better semi-supervised learning. Our method,

---

Under review as a conference paper at ICLR 2020
UDA, employs ofate-of-the-ar data augmentation found in supervised learning to generate diverse and realistic noise and enforces the model to be consistent with respect to these noise. For text, UDA combines well with representation learning, c.g., BERT, and is very effective in low-data regime where state-of-the-ar performance is achieved on IMDb with only 20 examples. For vision, UDA outperforms prior works by clear margin and nearly matches the performance of the fully supervised models trained on the full labeled sets which are one order of magnitude larger. Lastly, UDA can effectively leverage out-of-domain unlabeled data and achieve improved performances on ImageNet where we have a large amount of supervised data. We hope that UDA will encourage future research transfer advanced supervised augmentation supessupee setting for different tasks.
REFERENCES
Ben Athiwaratkun, Mare Finzi, Pavel Izmailov, and Andrew Gordon Wilson. There many sistent explanations of unlabeled data: Why you should average. 2018.
Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. In Advances in Neural Information Processing Systems, pp. 3365-3373, 2014.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. arXiv preprint arXiv: 19X.::9905.0224, 2019.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C Duchi. Unlabeled data improves adversarial robustness. arXiv preprint arXiv:1905.137:0 2019.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning (chapelle, et al., eds.; 2006)]book reviews]. IEEE Transactions on Neural Networks, 20(3):542- 2009.
Kevin Clark, Minh-Thang Luong, Christopher D Manning, and Quoc V Le. Semi- supirsupee se- quence modeling with vross-view training. arXiv preprint arXiv: 180.08800 2018.
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pp. 160-167. ACM, 2008.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: 2018.
Learning augmentation policies from data. arXiv preprint
Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In Advances in neural infor- mation processing systems, pp. 3079-3087, 2015.
Zihang Dai, Zhilin Yang, Fan Yang, William W Cohen, and Ruslan R Salakhutdinov. Good semi- supervised learning that requires bad gan. In Advances in Neural Information Processing Sys- tems, pp. 6510-6520, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint / 2018.
Sergey Edunov, Myle Out, Michael Auli, and David Grangier. Understanding back-translation at scale. arXiv preprint arXiv: 093880 2018.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Ad- vances in neural information processing systems, pp. 529-536, 2005.
Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv: 142.v:1412.5567. 2014.
10

---

Under review as a conference paper at ICLR 2020
Ryuichiro Hataya and Hideki Nakayama. Unifying semi-supervised and robust learning by mixup. ICLR The 2nd Learning from Limited Labeled Data (LLD) Workshop, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Xuanli He, Gholamreza Haffari, and Mohammad Norouzi. Sequence to sequence mixture model for diverse machine translation. arXiv preprint arXiv:1810.0733 2018.
Olivier J Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint 1rXtv:1905.09272, 2019.
Alex Hernández-Garcia and Peter König. Data augmentation instead of regulaaiizau arXiv preprint arXiv: 03852, 2018.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), volume 1, pp. 328-339, 2018.
Weihua Hu, Takeru Miyato, Seiya Tokui, Eicchi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self elf-auugented training. In Proceedings of the 34th International Conference on Machine Learning- Voag-- 70, pp. 1558-1567. JMLR. org, 2017.
Jacob Jackson and John Schulman. Semi-supervised learning by label gradient alignment. arXiv preprint arXiv: 1902::900.02.... 2019.
Rie Johnson and Tong Zhang. Deep pyramid convolutional neural networks for text categorization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pp. 562-570, 2017
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pp- 3581-3589, 2014.
Thomas N Kipf and Max Welling. Sumi-super classification with graph convolutional net- works. arXiv preprint 1rXiv:1609.022 2016.
Wouter Kool, Herke van Hoof, and Max Welling. Stochastic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement. arXiv preprint 07X6v:1903.06059 2019.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint 2016.
Dong- Hyyu Lee. Pscudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning. ICML, volume 3, pp. 2, 2013.
Davis Liang. Zhiheng Huang, and Zachary C Lipton. Learning noise-invariant representations for robust speech recognition. In 2018 IEEE Spoken Language Technology Workshop (SI.T), pp- 56-63, IEEE, 2018.
Yucen Luo, Jun Zhu, Mengxi Li, Yong Ren, and Bo Zhang. Smooth neighbors teacher graphs for semi-supervised learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8896-8905, 2018
11

---

Under review as a conference paper at ICLR 2020
Lars Maalde, Casper Kaae Sønderby, Søren Kaae Sønderby, and Ole Winther, Auxiliary deep gen- erative models. arXiv preprint arXiv: 2016.
Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th amnual meeting of the association for computational linguistics: Human language technologies-volume I, pp. 142-150. Association for Computational Linguisties, 2011.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based rec- ommendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 43-52. ACM, 2015.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations words and phrases and their compositionality. In Advances in neural information pro- cessing systems, pp. 15111-1119 2013.
Takeru Miyato, Andrew M Dai, and Ian Goodfellow. Adversarial training methods for semi- supervised text classification. arXiv preprint arXiv:/605.07725, 2016.
Takeru Miyato, Shin-ichi Maeda, Shin Ishii, and Masanori Koyama. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realis- tic evaluation of deep semi-supervised learning algorithms. In Advances in Neural Information Processing Systems, pp.
2018.
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv: 1904.0870000077 2019.
Sungrae Park, JunKeon Park, Su-Jin Shin, and I1-Chul Moon. Adversarial dropout for supervised and semi-supervised learning. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint 1xXvv:8802.05365, 2018.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language un- derstanding by generative pre-training. URL. https://s3-us-westt amazonaws. com/openai- phersveerrrkooere//ggggguuurrrre
understanding paper. pdf. 2018.
Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi- supervised learning with ladder networks. In Advances in neural information processing systems, pp. 3546-3554, 2015.
Devendra Singh Sachan, Manzil Zaheer, and Ruslan Salakhutdinov. Revisiting Istm networks for semi-supervised text classification via mixed objective function. 2018
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep supervupee learning. In Advances in Neural Information Processing Systems, pp. 1163-1171, 2016.
Julian Salazar, Davis Liang, Zhiheng Huang, and Zachary C Lipton. Invariant representation leamn ing for robust deep networks. In Workshop on Integration of Deep Learning Theories, NeurIPS, 2018.
12

---

Under review as a conference paper at ICLR 2020
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2244,2242, 2016.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. arXiv preprint
2015.
Tianxiao Shen, Myle Ott, Michael Auli, and Marc' Aurelio Ranzato. Mixture models for diverse
machine translation: Tricks of the trade. arXiv preprint arXiv:1902.078/1 2019.
Patrice Y Simard, Yann A LeCun, John S Denker, and Bernard Victorri. Transformation invariance in pattern recognition traniittoonttnnnttttten distance and tangent propagation. In Neural networks: tricks of the trade, pp. 239-274. Springer, 1998.
Robert Stanforth, Albussein Fawzi, Pushmeet Kohli, et al. Are labels required for improving adver- sarial robustness? arXiv preprint orXiv:1905.13720 2019.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: tency targets improve semi-supervised deep learning results. In Advances in neural information processing systems, pp. 1195-129000 1204, 2017.
consis-
Trieu H Trinh, Minh-Thang Luong, and Quoc V Le. Selfie: Self-supervised pretraining for image embedding. arXiv preprint arXiv: 190.00240000 2019.
Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation con- sistency training for semi-supervised learning. arXiv preprint arXiv: 1903:03895 2019.
Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig. Switchout: an efficient data augmenta- tion algorithm for neural machine translation. arXiv preprint arXiv: 1808.00108. 2018.
Jason Weston, Frédéric Ratle, Hossein Mobahi, and Ronan Collobert. Deep learning via semi- supervised embedding. In Neural Networks: Tricks of the Trade, pp. 639-655. Springer, 2012.
Zhilin Yang. William W Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. arXiv preprini arXiv: (et::::0008000 2016.
Zhilin Yang, Junjie Hu, Ruslan Salakhutdinov, and William W Cohen. Semi-superviss qa with generative domain-adaptii nets. arXiv preprint (0t.:.22020200 2017.
Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6210-6219, 2019.
Adams Wei Yu, David Dohan, Minh-Thang
Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoe V Le. Qanet: Combining local convolution with global self-attention for reading compre- hension. arXiv preprint
2018
Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks.
arXiv preprint
arXiv:1605.07146, 2016.
Runtian Zhai, Tianle Cai, Di He, Chen Dan, Kun He, John Hopcroft, and Liwei Wang. Adversarially robust generalization just requires more unlabeled data, arXiv preprint arXiv: arXiv: 2019a.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S*1: Self-supervised semi- supervised learning, arXiv preprint arXiv:1905.03670 2019b.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization, arXiv preprint arXiv:1710.09411 2017.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. In Advances in neural information processing systems, pp. 649-657, 2015.
Xiaojin Zhu, Zoubin Ghahramani, and John D Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In Proceedings of the 20th International conference on Machine learning (ICML-03), pp. 912-919, 2003.
13

---

Under review as a conference paper at ICLR 2020
A MORE EXPERIMENTS
A.1 ABLATIONS STUDIES ON RANDAUGMENT
We hypothesize that the success RandAugment should be credited the diversity of the augmen- tation transformations. since Ragmentt works very well for multiple different datasets while does not require search algorithm find out the most effective policies. To verify this hypothesis, we test UDA's performance when we restrict the number of possible transformations used in Ran- dAugment. As shown in Figure 6 the performance gradually improves as we use more augmentation transformations.
6.2
6.0
Eror rate
5.8
5.6
5.4
2
4
6
8 10
12
14
16
Number possible tranformations
Figure 6: Error rate of UDA on CIFAR-10 with different numbers of possible transformations in RandAugment. UDA achieves lower error rate when we increase the number of possible transfor- mations, which demonstrates the importance of a rich set of augmentation transformations.
A.2 RESULTS ON CIFAR-10 10 AND SVHN WITH VARIED LABEL SET SIZES
CIFAR-10 In Table 7 show results for compared methods of Figure 4a and results of Pseudo-
Label LLee 2013), II-Model (Laine & Aila 2016). Mean Teacher (Tarvainen & Valpola (017). Fully supervised learning using 50,000 examples achieves an error rate of 5,36 and 4.23 with or without Rand Augmen. The performance of the baseline models are reported by MixMatch (Berth-
elot et al. 2019).
To make sure that the performance reported by MixMatch and our results are comparable, we reim- plement MixMatch in our codebase and find that the results in the original paper is comparable but slightly higher than our reimplementation, which results in more competitive comparison for UDA. For example, our reimplementation MixMatch achieves an error rate of 7.00 + 0.59 and 7.39 + 0.11 with 4,000 and 2,000 examples. MixMatch uses different model implementation and employs exponential moving average (EMA) on the model parameters, while we do not use EMA for our implementations.
14

---

Under review as a conference paper at ICLR 2020
Methods Sup
250
500
1,000
2,000
4,000
Pseudo-Label
49,98 1.17
40.55 1 1.70
30.91 1: 1.73
21.96 1 0.42
16.21 0.1.1
II-Model
53.02 2.05
41.82
31.53 = 0.98
23.07 0.66
17.41
Mcan Teacher
47.32 + 4.71
42.01 5.86
17.32 - 4.00
12.17
10.36
VAT
36.03 + 2.82
26.11 1.52
18.68 + 0,40
14.40 0.15
11.05
MixMatch
11.08 0.87
9.65 + 0.94
7.75 + 0.32
7.03 0.15
6.24 6.24
UDA (RandAugment)
8.76 0.90
6.68 + 0.24
5.87 + 0.13
5.51 0.21
5.29 + 0.25
Table 7: Error rate (%) for CIFAR-10.
SVHN In Table we similarly show results for compared methods Figure 4b and results of methods mentioned above. Fully supervised learning using 73,257 examples achieves an error rate of 2.84 and 2.28 with or without RandAugment. The performance of the baseline models are re- ported by MixMatch (Berthelot et al. 2019). Our reimplementation of Mix.aatch also resulted in comparable but higher error rates than the reported ones.
Methods / Sup
250
500
1,000
2,000
4,000
Pseudo-Label
21.16 1 0.88
14.35 1 0.37
10.19
0.41
7.54 1 0.27
5.71 1 0.07
II-Model
17.65 + 0.27
11.44 0.39
8.60
+.18
6.24
5.57
0.14
Mcan Teacher
6.45 + 2.43
3.82
0.10
3.51
+.39
VAT
8,41 1.01
7.44
5.98 0.21
4.85
0.23
4.20 +20000.11
MixMatch
3.78 + 0.26
3.64
0.46
3.27 + 0.31
3.04
+ 0.13
2.89 + 0.06
UDA (RandAuggeet)
2.76 + 0.17
2.70 + 0.09
2.55 + 0.09
2.57 + 0.09
2.47 + 0.15
Table 8: Error rate (%) for SVHN.
B ADDITIONAL TRAINING TECHNIQUES
While UDA generally works well, there several practical issues in consistency training that may dampen the performance gain if not carefully dealt with. This section presents additional techniques targeting at commonly encountered problems.
Sharpening Predictions.
Entropy minimization (Grandvalet Bengio 2005) has been shown to be effective in semi-supervised learning methods such as VAT (Miyato et al. 2018). To apply entropy minimization in our method, we simply add a loss term to the objective to regularize the predicted distributions on unlabeled examples have low entropy. Alternatively, when the number of la- beled examples are extremely small, we find it helpful to mask out examples that the current model not confident about and use a low Softmax temperature when computing the target distribution on unlabeled examples. Specifically, in each minibatch, the consistency loss term is computed only on examples whose highest probability among classification categories greater than a threshold.
Domain-relevance Data Filtering. Ideally, we would like to make use of out-of-domall unla- beled data since it is usually much easier to collect, but the class distributions of out-of-domain data are mismatched with those of in-domain data, which can result in performance loss if directly used (Oliver et al.] 2018). To obtain data relevant to the domain for the task hand, we adopt common technique for detecting out-of-domain data. We in-domain data infer the labels of data in a large out-of-domain dataset and pick out examples that the model is most confident about. Specifically, for each category, sort all examples based on the classified probabilities of being in that category and select the examples with the highest probabilities.
baseline model trained on the
C EXTENDED AUGMENTATION STRATEGIES FOR DIFFERENT TASKS
Discussion Trade-off Between Diversity and Validity for Data Augmentation. Despite that state-of-the-an data augmentation methods can generate diverse and valid augmented examples discussed in section 2.2. there is trade-off between diversity and validity since diversity is
15

---

Under review as a conference paper at ICLR 2020
achieved by changing a part of the original example, naturally leading to the risk of altering the ground-truti label. We find it beneficial to tune the trade-off between diversity and validity for data augmentation methods. For text classification, one hand, when we use temperature of 0, decoding by random sampling degenerates into greedy decoding and generates perfectly valid but identical paraphrases. On the other hand, when we use temperature of 1, random sampling generates very diverse but barely readable paraphrases. We find that setting the Softmax temperature 0.7, 0.8 or 0.9 leads to the best performances.
tune the temperature of random sampling. On the
RandAAggmet Details. In our implementation of RandAugment, each sub-policy is composed of two operations, where each operation is represented by the transformation name. probability, and magnitude that is specific to that operation. For example, a sub-policy can be [(Sharpness, 0.6, 2), (Posterize, 0.3, 9)].
For each operation, we randomly sample transformation from 15 possible transformations, mag- nitude in [1, 10) and fix the probability to 0.5. Specifically, we sample from the following 15 trans- formations: Invert, Cutout, Sharpness, AutoContrast, Posterize, ShearX, TranslateX, Translate ShearY, Rotate, Equalize, Contrast, Color, Solarize, Brightness. We find this setting to work well in our first try and did not tune the magnitude range and the probability. Tuning these hyperparameters might result in further gains in accuracy.
TF-IDF based word replacing Details. We describe the TF-IDF based word replacing data aug- mentation method in this section. Ideally, we would like the augmentation method to generate both diverse and valid examples. Hence, the augmentation is designed to retain keywords and replace uninformative words with other uninformative words. We use BERT's word tokenizer since BERT first tokenizes sentences into a sequence of words and then tokenize words into subwords although the model uses subwords as input.
Specifically, Suppose IDF(w) is the IDF score for word computed on the whole corpus, and TF(w) is the TF score for word w in sentence. We compute the TF-IDF score as TFIDF (w) = TF(w) (F(W)ID (w))DF(W. Suppose the maximum TF-IDF score in a sentence x C' max, TFIDF(z;). To make the probability of having word replaced to negatively correlate with its TF-IDF score, we set the probability to min(p(C TF:DF(x;))/2 1), where p a hyperparameter that controls the magnitude of the augmentation and Z - CC (FIIF((..)//) is the average score. p is set to 0.7 for experiments on DBPedia.
When a word is replaced, we sample another word from the whole vocabulary for the replace- ment. Intuitively, the sampled words should not be keywords to prevent changing the ground-truth labels of the sentence. To measure if word is keyword, the whole corpus. Specifically, we compute the score as S(w) = freq(w) (FW)) where freq(w) is the frequency of word the whole corpus. We set the probability of sampling word as (max S(w') - S(w))/Z² where Z' = E max S(w')- - S(w) a normalization term.
compute score of each word on
D EXTENDED RELATED WORK
Semi-supervised Learning. Due to the long history of semi-supervised learning (SSL), refer readers to (Chapelle et al., 2009) for a general review. More recently, many efforts have been made renovate classic ideas into deep neural instantiations. For example, graph-based label propaga- tion (Zhu et al., 2003) has been extended to neural methods via graph embeddings (Weston et al.) 2012 Yang al. 2016) and later graph convolutions (Kipt & Welling 2016). Similarly, with the variational auto- entooin framework and reinforce algorithm, classic graphical models based SSL methods with target variable being latent can also take advantage of deep architectures (Kingmal et al., 2014 Maaloe et al. 20166; Yang et al.) 2017). Besides the direct extensions, it was found that training neural classifiers to classify out-of examples into an additional class (Salimans)
al. 2016) works very well in practice. Later, Dai et al. (011)) shows that this can be seen as an instantiation of low-density separation.
Apart from enforcing consistency on the noised input examples and the hidden representations. an- other line of research enforces consistency under different model parameters, which is complemen- tary to our method. For example, Mean Teacher (Tarvainen & Valpola 2017) maintains teacher model with parameters being the ensemble of a student model's parameters and enforces the con- sistency between the predictions of the two models. Recently, Athiwaratkun al. (2088) propose
16

---

Under review as a conference paper at ICLR 2020
fast-SWA that improves Mean Teacher by encouraging the model to explore diverse set of plausi- ble parameters. In addition to parameter-level consistency, SNTG (Luo et al., 2018) also enforces input-level consistency by constructing similarity graph between unlabeled examples.
Data Augmentation. Also related to our work the field of data augmentation research. Besides the conventional approaches and two data augmentation methods mentioned in Section 2.1 a cent approach MixUp (Zhang et al.] 2017) goes beyond data augmentation from a single data point and performs interpolation of data pairs achieve augmentation. Recently, [Herández--Garíia & König 2018) have shown that data augmentation can be regarded as a kind of explicit regularization methods similar to Dropout.
Diverse Back Translation. Diverse paraphrases generated by tuck-translation has been a key com- ponent in the significant performance improvements in our text classification experiments. We use random sampling instead of beam search for decoding similar to the work by Edunov et al. (0188)
There are also recent works on generating diverse translations (He et al. 2018 [hhn et al. Kool et al. 2019) that might lead to further improvements when used as data augmentations.
2019
Unsupervised Representation Learning. Apart from semi-supervised learning, unsupervised rep- resentation learning offers another way to utilize unsupervised data. Collobert & Weston (2008) demonstrated that word embeddings learned by language modeling can improve the performance significantly on semantic role labeling. Later, the pre-training of word embeddings was simpli- fied and substantially scaled in Word2Vec (Mikolov et al. (2033) and Glove (Pennington et al.]
2014). More recently, Dai & Le (205)); (Peerrs et al. (2018); Radford et al. (Howard & Ruder (2018); Devlin et al. (2018) have shown that pre-training using language modeling and denoising auto-encoding leads to significant improvements on many tasks in the language domain. There
also a growing interest in self-supervised learning for vision (Zhai et al. 2019b Hénaff et al. 2019
Trinh al. 2019).
Consistency Training in Other Domains. Similar ideas of consistency training has also been ap- plied in other domains. For example, recently, enforcing adversarial consistency on unsupervised data has also been shown to be helpful in adversarial robustness (Stanforth et al., 2019 Zhai et al., 2019a Carmon et al., 2019). Enforcing consistency w.r.t data augmentation has also been shown to work well for representation learning (Hu et al. 2017; Ye et al.) (2019). Invariant representa- tion learning (Liang et al. 2018: [Salazar et al. 2018 applies the consistency loss not only to the predicted distributions but also to representations and has been shown significant improvements on speech recognition.
E EXPERIMENT DETAILS
In this section, we provide experiment details for the performed experiments.
E.1 TEXT CLASSIFICATIONS
Datasets. In our semi-supervised setting, we randomly sampled labeled examples from the full supervised set sef" and use the same number of examples for each category. For unlabeled data, we use the whole training set for DBPedia, the concatenation of the training set and the unlabeled set for IMDb and external data for Yelp-2, Yelp-5, Amazon-2 and Amazon-5 (McAuley et al. 2015 [ Note that for Yelp and Amazon based datasets, the label distribution of the unlabeled set might not match with that of labeled datasets since there are different number of examples in different categories. Nevertheless, we find it works well to use all the unlabeled data.
Preprocessing. We find the sequence length to be an important factor in achieving good perfor- mance. For all text classification datasets, we truncate the input to 512 subwords since BERT is pretrained with maximum sequence length of 512. Further, when the length of example is greater than 512, we keep the last 512 subwords instead of the first 512 subwords as keeping the latter part of the sentence lead to better performances IMDb.
http://bit. http://ttt..11y/kko.s// http://wwww..ppppaaaaaaaaaaarrad
ientt/ http:/ imcuu By ucsd.
https://www. www.w.aagll.... le, com / yel 1o--daattt///yg dataset
edu/data/amazonn
17

---

Under review as a conference paper at ICLR 2020
Fine-tuning BERT on in-domain unsupervised data. We fine-tune the BERT model on in-domain unsupervised data using the code released by BERT. We try learning rate of 2c-5, 5e-5 and le-4, batch size of 32, 64 and 128 and number of training steps of 30k, 100k and 300k. We pick the fine-tuned models by the BERT loss on a held-out set instead of the performance on a downstream
task.
Random initialized Transformer. For the experiments with randomly initialized Transformer, we adopt hyperparameters for BERT base except that we only use hidden layers and 8 attention heads. We also increase the dropout rate on the attention and the hidden states to 0.2, When we train UDA with randomly initialized architectures, we train UDA for 500k or 1M steps on Amazon-5 and Yelp- 5 where we have abundant unlabeled data.
BERT hy hyperparameters. Following the common BERT fine-tuning procedure, we keep a dropout rate of 0.1, and try learning rate of le-5, 2e-5 and 5e-5 and batch size of 32 and 128. We also tune the number of steps ranging from 30 100k for various data sizes.
UDA hyperparameters. We set the weight on the unsupervised objective À to 1 in all of our exper- iments. We use a batch size of 32 for the supervised objective since 32 the smallest batch size on v3-32 Cloud TPU Pod. We use batch size of 224 for the unsupervised objective when the Trans- former is initialized with BERT so that the model can be trained on more unlabeled data. We find that generating one augmented example for each unlabeled example is enough for BERTFINETUNE-
All experiments in this part are performed on a v3-32 Cloud TPU Pod.
E.2 SEMI-SUPERVISD LEARNING BENCIMARKS CIFAR-10 10 AND SVHN
Hyperparameters for Wide-ResNet-288. For hyperparameter tuning, for simplicity, we per- formed a random sampling search over hyperparameters and choose the best one based on validation sets (20% of the training sets with different sizes). We use the averaged results of multiple exper- iments to reduce the performance variance measured on the small validation sets. Specifically, tried the following ranges:
training steps: 50k, 100k;
learning rate: 0.03, 0.05, 0.1;
TSA schedules: log-schedule, linear-schedule, exp-schedule, not using TSA;
entropy minimization loss weight: O, 0.1, 0.3;
consistency loss weight: 1, 6;
unlabeled data batch size: 960, 1280;
weight decay rate: 5e-4, 7e-4, le-3;
softmax temperature: 1, 0.9;
confidence threshold: 0, 0.8.
where the values in bold black text are our default hyperparameters. We found that given a rea- sonably large labeled set, our method is robust pypar-pprameterr. Therefore, syper-paramettrs in these cases. Specifically, we use the above default hyperparameters for CIFAR- 10 with 4,000, 2,000, 1,000 and 500 examples. For SVHN with 4,000, 2.000, 1,000, 500, 250 examples, we additionally set the learning rate to 0.05 and unlabeled batch size to 1280. For the case of 250 examples on CIFAR-10, we use a different set of tlyper-parameten TSA schedule: log sccecuue, consistency loss coefficient: 6; weight decay: 7e-4; unlabeled data batch size: 1280; softmax temperature: 0.9; consistency threshold: 0.8.
the same
training steps: 50k;
Other hyperparameters not mentioned above are the same to the the original paper of Wide- ResNet (Zagoruyko & Komodakis 2016). In order to reduce training time, we generate augmented examples before training and dump them to disk. For CIFAR-10, we generate 100 augmented ex- amples for each unlabeled example. Note that generating augmented examples in an online fashion is always better as good as using dumped augmented examples since the model can different augmented examples in different epochs, leading to more diverse samples. We report the average performance and the standard deviation for 10 runs.
Hyperparameters for Shake-Shake and PyramidNet. For the experiments with Shake-Shake we train UDA for 300k steps and batch size of 128 for the supervised objective and use batch
18

---

Under review as a conference paper at ICLR 2020
size of 512 for the unsuperivsed objective. For the experiments with PyramidNet-ShakeDDng train UDA for 700k steps and use a batch size of 64 for the supervised objective and batch size of 128 for the unsupervised objective. For both models, we use learning rate of 0.03 and use cosine learning decay with one annealing cycle following Autonug.ent.
All experiments in this part performed on v3-32 Cloud TPU v3 Pod.
E.3
IMAGENET
10% Labeled Set Setting, Unless otherwise stated, we follow the standard hyperparameters used in an open-source implementation of ResNet" For the 10% labeled set setting, we use batch size of 512 for the supervised objective and batch size of 15,360 for the unsupervised objective. We use a base learning rate of 0.3 that decayed by 10 for four times and set the weight on the unsupervised objective À to 20. We mask out unlabeled examples whose highest probabilities across categories are less than 0.5 and set the Softmax temperature to 0.4. The model is trained for 40k steps. Experiments in this part are performed on a v3-64 Cloud TPU v3 Pod.
Full Labeled Set Setting, For experiments on the full ImageNet, we use a batch size of 8,192 for the supervised objective and batch size of 16,384 for the unsupervised objective. The weight the unsupervised objective À is set to 1. We use entropy minimization to sharpen the prediction. We use base learning rate of 1.6 and decay it by 10 for four times. Experiments in this part performed on v3-128 Cloud TPU v3 Pod.
19


Reviews:
[{"review_id": "rkxWsB9TKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors present a new perspective on how to effectively noise unlabeled examples and argue that the quality of noising plays a crucial role in semi-supervised learning. By substituting simple noising operations with advanced data augmentation methods, their method brings substantial improvements across six language and three vision tasks under the same consistency training framework. I think the topic itself is interesting and I have the following concerns.\n(1) The first is about the contribution of this paper. In this paper, all the results, including the augmented methods are all well established approaches. The authors have just employed them in solving a new problem, without support about why they work. Thus, the results are only strategies, without theoretical guarantee or insights. It is difficult to convince the reviewers.\n(2) Although the authors have achieved seemingly promising results, I think it can not convince me since the authors have not answered the questions about why and when. I think this paper likes a technical report, not a research paper.\n(3) I have also noticed the discussions among the authors and other readers. It seems that the large improvement depends on the parameters heavily. So, why not to share the parameters directly? ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1lbeiL5KH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper \"Unsupervised Data Augmentation for Consistency Training\" marries two recent ideas of \n1. \"Data Augmentation\" (DA) from supervised learning: The authors explore various methods for \"DA\" mostly inspired by much recent work such as Random image transformations, Backtranslation, and TF-IDF based word replacement.\n2. \"Consistency Training\" (CT) from semi-supervised learning: CT tries to minimize the divergence between the output distributions of the classifiers that are produced by adding noise to the input.\n\nThe key insight in this paper is that, data augmentation methods that work well during supervised training should also work equally well as the noise distribution for consistency training on unlabeled data. The authors support this claim empirically through the experiments in table 1 and 2. \n\nThe paper is well written and the authors present extensive comparative and ablation tests to demonstrate that their proposed method works well with both low and high amounts of labeled data.  This paper should be accepted into the conference.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Byxu8IcTKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to substitute simple noising operations with many data augmentation methods in consistency-based semi-supervised learning. The main idea is the same as previous work: constrain the model predictions of unlabeled examples to be invariant to different noise. The proposed UDA is evaluated on a wide range of language and vision tasks.\n\nOverall, the paper is well-written and clear. The most impressive point of this paper is its strong empirical results. However, it looks not surprising to me that more data augmentations found in supervised learning are also effective in semi-supervised learning. The paper fails to provide any theoretical insights but a thorough empirical evaluation.\n\nOne of my concerns is that the hyperparameters on vision tasks follow those of AutoAugment, which is carefully tuned on supervised tasks. Apparently, their hyperparameters are based on the whole labeled training dataset. In this case, the adopted hyperparameters include sort of information of the whole labeled dataset. Is it fair?\n\nAnother concern is how to control the strength of augmentations. For example, for digit images like SVHN, a \"6\" rotates by 180 degree is \"9\", whose prediction should change correspondingly. In this case, the assumption of invariance does not hold when the augmentation is too strong. \n\nI'm willing to increase my score if the authors address my concerns.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper shows that data augmentation methods work well for consistency training on unlabeled data in semi-supervised learning.

Reviewers and AC think that the reported experimental scores are interesting/strong, but scientific reasoning for convincing why the proposed method is valuable is limited. In particular, the authors are encouraged to justify novelty and hyper-parameters used in the paper. This is because I also think that it is not too surprising that more data augmentations in supervised learning are also effective in semi-supervised learning. It can be valuable if more scientific reasoning/justification is provided.

Hence, I recommend rejection.
Decision: Reject