SlowGuess/ACL_2020_OCR huggingface dataset:
-rel on Friday on rising fears of a winter fuel supply crunch and robust economic growth in China, the world's number two user", the augment texts through German and Russian are: "Oil prices surged to a record high above  $55 a barrel on Friday on growing fears of a winter slump and robust economic growth in world No.2 China" and "Oil prices soared to record highs above$ 55 per barrel on Friday amid growing fears over a winter reduction in U.S. oil inventories and robust economic growth in China, the world's second-biggest oil consumer".
To test the effectiveness of our method, we compared it with several recent models:
VAMPIRE (Gururangan et al., 2019): VAriational Methods for Pretraining In Resource-limited Environments(VAMPIRE) pretrained a unigram document model as a variational autoencoder on in-domain, unlabeled data and used its internal states as features in a downstream classifier.
- BERT (Devlin et al., 2019): We used the pretrained BERT-based-uncased model<sup>3</sup> and fine-tuned it for the classification. In details, we used average pooling over the output of BERT encoder and the same two-layer MLP as used in MixText to predict the labels.
- UDA (Xie et al., 2019): Since we do not have access to TPU and need to use smaller amount of unlabeled data, we implemented Unsupervised Data Augmentation(UDA) using pytorch by ourselves. Specifically, we used the same BERT-based-uncased model, unlabeled augment data and batch size as our MixText, used original unlabeled data to predict the labels with the same softmax sharpen temperature as our MixText and computed consistency loss between augmented unlabeled data.
5.3 Model Settings
We used BERT-based-uncased tokenizer to tokenize the text, bert-based-uncased model as our text encoder, and used average pooling over the output of the encoder, a two-layer MLP with a 128 hidden size and tanh as its activation function to predict the labels. The max sentence length is set as 256. We remained the first 256 tokens for sentences that exceed the limit. The learning rate is 1e-5 for BERT encoder, 1e-3 for MLP. For  $\alpha$  in the beta distribution, generally, when labeled data is fewer than 100 per class,  $\alpha$  is set as 2 or 16, as larger  $\alpha$  is more likely to generate  $\lambda$  around 0.5, thus creating "newer" data as data augmentations; when labeled data is more than 200 per class,  $\alpha$  is set to 0.2 or 0.4, as smaller  $\alpha$  is more likely to generate  $\lambda$  around 0.1, thus creating "similar" data as adding noise regularization.
For TMix, we only utilize the labeled dataset as the settings in Bert baseline, and set the batch size
3https://pypi.org/project/pytorch-transformers/
$^{2}$ https://github.com/pytorch/fairseq
Dataset	Label Type	Classes	Unlabeled	Dev	Test
AG News	News Topic	4	5000	2000	1900
DBpedia	Wikipedia Topic	14	5000	2000	5000
Yahoo! Answer	QA Topic	10	5000	5000	6000
IMDB	Review Sentiment	2	5000	2000	12500
Table 1: Dataset statistics and dataset split. The number of unlabeled data, dev data and test data in the table means the number of data per class.
Datset	Model	10	200	2500	Dataset	Model	10	200	2500
AG News	VAMPIRE	-	83.9	86.2	DBpedia	VAMPIRE	-	-	-
BERT	69.5	87.5	90.8	BERT	95.2	98.5	99.0
TMix*	74.1	88.1	91.0	TMix*	96.8	98.7	99.0
UDA	84.4	88.3	91.2	UDA	97.8	98.8	99.1
MixText*	88.4	89.2	91.5	MixText*	98.5	98.9	99.2
Yahoo!	VAMPIRE	-	59.9	70.2	IMDB	VAMPIRE	-	82.2	85.8
BERT	56.2	69.3	73.2	BERT	67.5	86.9	89.8
TMix*	58.6	69.8	73.5	TMix*	69.3	87.4	90.3
UDA	63.2	70.2	73.6	UDA	78.2	89.1	90.8
MixText*	67.6	71.3	74.1	MixText*	78.7	89.4	91.3
Table 2: Performance (test accuracy(%) ) comparison with baselines. The results are averaged after three runs to show the significance (Dror et al., 2018), each run takes around 5 hours. Models are trained with 10, 200, 2500 labeled data per class. VAMPIRE, Bert, and TMix do not use unlabeled data during training while UDA and MixText utilize unlabeled data. * means our models.
as 8. In MixText, we utilize both labeled data and unlabeled data for training using the same settings as in UDA. We set  $K = 2$ , i.e., for each unlabeled data we perform two augmentations, specifically German and Russian. The batch size is 4 for labeled data and 8 for unlabeled data. 0.5 is used as a starting point to tune temperature  $T$ . In our experiments, we set 0.3 for AG News, 0.5 for DBpedia and Yahoo! Answer, and 1 for IMDB.
We evaluated our baselines and proposed methods using accuracy with 5000 unlabeled data and with different amount of labeled data per class ranging from 10 to 10000 (5000 for IMDB).
5.4.1 Varying the Number of Labeled Data
The results on different text classification datasets are shown in Table 2 and Figure 3. All transformer based models (BERT, TMix, UDA and MixText) showed better performance compared to VAMPIRE since larger models were adopted. TMix outperformed BERT, especially when labeled data was limited like 10 per class. For instance, model accuracy improved from  $69.5\%$  to  $74.1\%$  on AG News with 10 labeled data, demonstrating the effective
ness of TMix. When unlabeled data was introduced in UDA, it outperformed TMix such as from  $58.6\%$  to  $63.2\%$  on Yahoo! with 10 labeled data, because more data was used and consistency regularization loss was added. Our proposed MixText consistently demonstrated the best performances when compared to different baseline models across four datasets, as MixText not only incorporated unlabeled data and utilized implicit relations between both labeled data and unlabeled data via TMix, but also had better label guessing on unlabeled data through weighted average among augmented and original sentences.
5.4.2 Varying the Number of Unlabeled Data
We also conducted experiments to test our model performances with 10 labeled data and different amount of unlabeled data (from 0 to 10000) on AG News and Yahoo! Answer, shown in Figure 4. With more unlabeled data, the accuracy became much higher on both AG News and Yahoo! Answer, which further validated the effectiveness of the usage of unlabeled data.
5.4.3 Loss on Development Set
To explore whether our methods can avoid overfitting when given limited labeled data, we plotted
AG News
Figure 3: Performance (test accuracy  $(\%)$ ) on AG News, DBpedia, Yahoo! Answer and IMDB with 5000 unlabeled data and varying number of labeled data per class for each model.
Yahoo! Answer
Figure 4: Performance (test accuracy  $(\%)$ ) on AG News ( $y$  axis on the right) and Yahoo! Answer ( $y$  axis on the left) with 10 labeled data and varying number of unlabeled data per class for MixText.
the losses on development set during the training on IMDB and Yahoo! Answer with 200 labeled data per class in Figure 5. We found that the loss on development sets tends to increase a lot in around 10 epochs for Bert, indicating that the model overfitted on training set. Although UDA can alleviate the overfitting problems with consistency regularization, TMix and MixText showed more stable trends and lower loss consistently. The loss curve for TMix also indicated that it can help solving overfitting problems even without extra data.
We performed ablation studies to show the effectiveness of each component in MixText.
5.5.1 Different Mix Layer Set in TMix
We explored different mixup layer set  $M$  for TMix and the results are shown in Table 3. Based on (Jawahar et al., 2019), the  $\{3,4,5,6,7,9,12\}$  are the most informative layers in BERT based model and each of them captures different types of informa
Figure 5: Loss on development set on IMDB and Yahoo! Answer in each epoch while training with 200 labeled data and 5000 unlabeled data per class.
tion (e.g., surface, syntactic, or semantic). We chose to mixup using different subsets of those layers to see which subsets gave the optimal performance. When no mixup is performed, our model accuracy was  $69.5\%$ . If we just mixup at the input and lower layers ( $\{0, 1, 2\}$ ), there seemed no performance increase. When doing mixup using different layer sets (e.g.,  $\{3,4\}$ , or  $\{6,7,9\}$ ), we found large differences in terms of model performances:  $\{3,4\}$  that mainly contains surface information like sentence length does not help text classification a lot, thus showing weaker performance. The 6th layer captures depth of the syntactic tree which also does not help much in classifications. Our model achieved the best performance at  $\{7, 9, 12\}$ ; this layer subset contains most of syntactic and semantic information such as the sequence of top level constituents in the syntax tree, the object number in main clause, sensitivity to word order, and the sensitivity to random replacement of a noun/verb.
Mixup Layers Set	Accuracy(%)
∅	69.5
{0,1,2}	69.3
{3,4}	70.4
{6,7,9}	71.9
{7,9,12}	74.1
{6,7,9,12}	72.2
{3,4,6,7,9,12}	71.6
Table 3: Performance (test accuracy  $(\%)$ ) on AG News with 10 labeled data per class with different mixup layers set for TMix.  $\varnothing$  means no mixup.
Model	Accuracy(%)
MixText	67.6
- weighted average	67.1
- TMix	63.5
- unlabeled data	58.6
- all	56.2
Table 4: Performance (test accuracy  $(\%)$ ) on Yahoo! Answer with 10 labeled data and 5000 unlabeled data per class after removing different parts of MixText.
5.5.2 Remove Different Parts from MixText
We also measured the performance of MixText by stripping each component each time and displayed the results in Table 4. We observed the performance drops after removing each part, suggesting that all components in MixText contribute to the final performance. The model performance decreased most significantly after removing unlabeled data which is as expected. Comparing to weighted average prediction for unlabeled data, the decrease from removing TMix was larger, indicating that TMix has the largest impact other than unlabeled data, which also proved the effectiveness of our proposed Text Mixup, an interpolation-based regularization and augmentation technique.
To alleviate the dependencies of supervised models on labeled data, this work presented a simple but effective semi-supervised learning method, MixText, for text classification, in which we also introduced TMix, an interpolation-based augmentation and regularization technique. Through experiments on four benchmark text classification datasets, we demonstrated the effectiveness of our proposed TMix technique and the Mixup model, which have better testing accuracy and more stable loss trend, compared with current pre-training and fine-tuning
models and other state-of-the-art semi-supervised learning methods. For future direction, we plan to explore the effectiveness of MixText in other NLP tasks such as sequential labeling tasks and other real-world scenarios with limited labeled data.
We would like to thank the anonymous reviewers for their helpful comments, and Chao Zhang for his early feedback. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan V GPU used for this research. DY is supported in part by a grant from Google.
Alan Akbik, Tanja Bergmann, and Roland Vollgraf. 2019. Pooled contextualized embeddings for named entity recognition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 724-728, Minneapolis, Minnesota. Association for Computational Linguistics.
David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. 2019. Mixmatch: A holistic approach to semi-supervised learning. CoRR, abs/1905.02249.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August 11-12, 2016, pages 10-21.
Ming-Wei Chang, Lev Ratinov, Dan Roth, and Vivek Srikumar. 2008. Importance of semantic representation: Dataless classification. In Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 2, AAAI'08, pages 830-835. AAAI Press.
Nitesh V. Chawla and Grigoris I. Karakoulas. 2011. Learning from labeled and unlabeled data: An empirical study across techniques and domains. CoRR, abs/1109.2047.
Jiaao Chen, Jianshu Chen, and Zhou Yu. 2019. Incorporating structured commonsense knowledge in story completion. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 6244-6251.
Jiaao Chen, Yuwei Wu, and Diyi Yang. 2020. Semi-supervised Models via Data Augmentation for Classifying Interactive Affective Responses. In Workshop On Affective Content Analysis, The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020.
Mingda Chen, Qingming Tang, Karen Livescu, and Kevin Gimpel. 2018. Variational sequential labelers for semi-supervised learning. In Proc. of EMNLP.
Kevin Clark, Minh-Thang Luong, Christopher D Manning, and Quoc V Le. 2018. Semi-supervised sequence modeling with cross-view training. arXiv preprint arXiv:1809.08370.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383-1392, Melbourne, Australia. Association for Computational Linguistics.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. CoRR, abs/1808.09381.
Yves Grandvalet and Yoshua Bengio. 2004. Semi-supervised learning by entropy minimization. In Proceedings of the 17th International Conference on Neural Information Processing Systems, NIPS'04, pages 529-536, Cambridge, MA, USA. MIT Press.
Suchin Gururangan, Tam Dang, Dallas Card, and Noah A. Smith. 2019. Variational pretraining for semi-supervised text classification. CoRR, abs/1906.02242.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339, Melbourne, Australia. Association for Computational Linguistics.
Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651-3657, Florence, Italy. Association for Computational Linguistics.
Ashutosh Kumar, Satwik Bhattamishra, Manik Bhandari, and Partha Talukdar. 2019. Submodular
optimization-based diverse paraphrasing and its effectiveness in data augmentation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3609-3619, Minneapolis, Minnesota. Association for Computational Linguistics.
Samuli Laine and Timo Aila. 2016. Temporal ensembling for semi-supervised learning. CoRR, abs/1610.02242.
Pooja Lakshmi Narayan, Ajay Nagesh, and Mihai Surdeanu. 2019. Exploration of noise strategies in semi-supervised named entity classification. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM* 2019), pages 186-191, Minneapolis, Minnesota. Association for Computational Linguistics.
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. CoRR, abs/1901.07291.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature Cell Biology, 521(7553):436-444.
Dong-Hyun Lee. 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. ICML 2013 Workshop: Challenges in Representation Learning (WREPL).
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT '11, pages 142-150, Stroudsburg, PA, USA. Association for Computational Linguistics.
Pablo N. Mendes, Max Jakob, and Christian Bizer. 2012. Dbpedia for nlp: A multilingual cross-domain knowledge base. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12), Istanbul, Turkey.
Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. 2018. Weakly-supervised neural text classification. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM '18, pages 983-992, New York, NY, USA. ACM.
Takeru Miyato, Andrew M Dai, and Ian Goodfellow. 2017. Adversarial training methods for semi-supervised text classification. In International Conference on Learning Representations.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. 2019. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell., 41(8):1979-1993.
Ajay Nagesh and Mihai Surdeanu. 2018. An exploration of three lightly-supervised representation learning approaches for named entity classification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2312-2324.
Luis Perez and Jason Wang. 2017. The effectiveness of data augmentation in image classification using deep learning. CoRR, abs/1712.04621.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018. Language models are unsupervised multitask learners.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Improving neural machine translation models with monolingual data. CoRR, abs/1511.06709.
Antti Tarvainen and Harri Valpola. 2017. Weight-averaged consistency targets improve semi-supervised deep learning results. CoRR, abs/1703.01780.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitlagkas, David Lopez-Paz, and Yoshua Bengio. 2019a. Manifold mixup: Better representations by interpolating hidden states. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6438-6447, Long Beach, California, USA. PMLR.
Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. 2019b. Interpolation consistency training for semi-supervised learning. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 3635-3641. International Joint Conferences on Artificial Intelligence Organization.
Vikas Verma, Meng Qu, Alex Lamb, Yoshua Bengio, Juho Kannala, and Jian Tang. 2019c. Graphmix: Regularized training of graph neural networks for semi-supervised learning. ArXiv, abs/1909.11715.
Jason W. Wei and Kai Zou. 2019. EDA: easy data augmentation techniques for boosting performance on text classification tasks. CoRR, abs/1901.11196.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. 2019. Unsupervised data augmentation for consistency training. arXiv preprint arXiv:1904.12848.
Diyi Yang, Jiaao Chen, Zichao Yang, Dan Jurafsky, and Eduard Hovy. 2019a. Let's make your request more persuasive: Modeling persuasive strategies via semi-supervised neural nets on crowdfunding platforms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3620-3630.
Diyi Yang, Miaomiao Wen, and Carolyn Rose. 2015. Weakly supervised role identification in teamwork interactions. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1671-1680.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019b. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR, abs/1906.08237.
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick. 2017. Improved variational autoencoders for text modeling using dilated convolutions. CoRR, abs/1702.08139.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 1480-1489.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. 2019. Cutmix: Regularization strategy to train strong classifiers with localizable features. CoRR, abs/1905.04899.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. 2017. mixup: Beyond empirical risk minimization. CoRR, abs/1710.09412.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. CoRR, abs/1509.01626.