SlowGuess/NAACL_2021_OCR huggingface dataset:
Sentence embedding space. Our data augmentation method is based on structuring a large external bank of text with a sentence embedding space. The sentence embedding method plays an essential role as shown in Table 6. We compare three embedding methods, the average of fastText (Mikolov et al., 2018) word embeddings (average-word2vec), the uSIF-ParaNMT embeddings (Ethayarajh, 2018b) and our own sentence encoder. We observe that uSIF-ParaNMT and paraembeddings - two sentence embedding methods that obtain state-of-the-art results on semantic textual similarity benchmarks - lead to stronger performance than the average-word2vec approach. Paraembeddings leads to the best performance and improves performance over uSIF by  $0.4\%$  on average.
Table 5: Impact of data augmentation technique.  $\mathcal{C}$  is the complexity,  $M$  the size of the bank of sentences,  $K$  the number of labels (or clusters),  $N$  the size of the downstream training set and  $d$  the embedding size.
Model	Embedding	dim	SST-5	CR	NER	Avg
RoBERTaLarge + ST	avg-w2v	300	59.4	95.2	92.9	82.5
RoBERTaLarge + ST	uSIF	300	59.9	95.0	93.1	82.7
RoBERTaLarge + ST	SASE	256	60.4	95.7	93.1	83.1
Scaling bank size. To demonstrate the importance of large-scale retrieval, we evaluate our method using an increasing amount of data for our bank, from fifty million sentences to five billion sentences (one hundred billion words). We observe a significant increase in performance from 50m to 1B in Table 7, but the improvement seems to saturate when going from 1B to 5B. However, the 5B external bank may however provide additional gains for tasks that are in rare domains and that can leverage the additional 4B sentences, which correspond to 342M additional CommonCrawl documents. Another effect of increasing the corpus size may be reducing diversity in the retrieved sentences. We leave experimenting with diversity-inducing enhancements to the retrieval for future work.
Table 6: Impact of sentence embedding method: average-word2vec, uSIF with ParaNMT and SASE.
Model	#lines	#words	SST-5	CR	NER	Avg
RoBERTaLarge + ST	50m	1B	59.5	95.4	92.8	82.6
RoBERTaLarge + ST	250m	5B	59.5	95.7	92.9	82.7
RoBERTaLarge + ST	1B	20B	60.4	95.7	93.1	83.1
RoBERTaLarge + ST	5B	100B	60.0	95.3	93.1	82.8
Continuous labels. In Table 8, we show that using class probabilities as synthetic labels leads to significantly better performance, outperforming discrete synthetic labels by  $0.9\%$  on average. We found very little gain when using self-training with discrete labels, contrary to previously published results in computer vision (Yalniz et al., 2019; Xie et al., 2020). A difference with previous work in computer vision is the number of classes of the supervised data. In that context, discrete labels provide even less information to the student model than continuous class probabilities.
Table 7: Impact of sentence bank size (number of lines and words) on self-training results.
Model	label type	SST-5	CR	NER	Avg
RoBERTaLarge + ST	discrete	59.1	94.7	92.8	82.2
RoBERTaLarge + ST	logits	60.4	95.7	93.1	83.1
Table 8: Impact of label type on self-training results.
Computational cost of self-training. SentAugment data prefiltering reduces the amount of data to be annotated by the teacher model and also filters based on the target domain. Filtering based solely on classifier confidence is significantly more expensive computationally, as annotating 10000 sentences with RoBERTa-Large takes approximately 3 seconds on a Volta-32GB GPU. This means that annotating 1B sentences takes 83 hours on a single GPU and much longer for models of larger size such as T5 (Raffel et al., 2019) or GPT-3 (Brown et al., 2020). On the other hand, using SentAugment based on a few task-specific query embedding (label-average) takes one minute for scoring 1B sentences. By only selecting the first few million top sentences, or less, to synthetically annotate, this greatly reduces computational cost and allows to scale to a larger bank of sentences, which in turn allows for more domains to be considered. Note that similarity search can be further sped up significantly by using fast nearest neighbor search such as product quantization with inverted files (Johnson et al., 2019).
BioNLP query: A single gene on chromosome 7 makes a protein called the cystic fibrosis transmembrane conductance regulator (CFTR).
Nearest neighbor: Cystic Fibrosis A mutation in the gene cystic fibrosis transmembrane conductance regulator (CFTR) in chromosome 7.
Financial Query: Google has entered into an agreement to buy Nest Labs for $3.2 billion.
Nearest neighbor: In January Google (NASDAQ:GOOG) reached an agreement to buy Nest Labs for $3.2 billion in cash.
Hate-speech Query: Average sentence embeddings of the "hateful" class of IMP
Nearest neighbor: fuzzy you are such a d* f* piece of s* just s* your g* d* mouth. – All you n* and s* are fucking ret*
Movie review Query: Average sentence embeddings of the "bad movie" class of SST-5
Nearest neighbor: This movie was terribly boring, but so forgettable as well that it didn't stand out for how awful it was..
Product review Query: Average sentence embeddings of the "positive" class of CR
Nearest neighbor: The phone is very good looking with superb camera setup and very lightweight.
Question type Query: Average sentence embeddings of the "location" class of TREC
Nearest neighbor: Lansing is the capital city of which state?
5 Analysis of similarity search
In this section, we present the results of our SentAugment sentence embedding (SASE) method on semantic textual similarity (STS) benchmarks and present examples of retrieved sentence based on large-scale similarity search.
5.1 Sentence embeddings (SASE)
In Table 10, we compare our sentence embedding method to previous approaches including BERT (Mean) (Devlin et al., 2018), InferSent (Conneau et al., 2017), GenSen (Subramanian et al., 2018), USE (Cer et al., 2018), Sentence-BERT (Reimers and Gurevych, 2019), uSIF (Ethayarajh, 2018a), Charagram (Wieting and Gimpel, 2017) and BGT (Wieting et al., 2019b). On average, our embeddings outperform previous approaches by  $0.2\%$  on STS 2012 to 2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), and by  $0.9\%$  on STS-Benchmark (Cer et al., 2017).
Table 9: Examples of nearest neighbors using a per-sentence or label-average query from different domains.
Model	Semantic	Textual	Similarity (STS)	Avg	STS-B
2012	2013	2014	2015	2016	2017
BERT (Mean)	48.8	46.5	54.0	59.2	63.4	54.4	-
InferSent	61.1	51.4	68.1	70.9	70.7	64.4	70.6
GenSen	60.7	50.8	64.1	73.3	66.0	63.0	-
USE	61.4	59.0	70.6	74.3	73.9	67.8	-
Sentence-BERT	66.9	63.2	74.2	77.3	72.8	70.9	-
uSIF-	68.3	66.1	78.4	79.0	-	-	79.5
Word, trigram	67.8	62.7	77.4	80.3	78.1	73.3	79.9
BGT	68.9	62.2	75.9	79.4	79.3	73.1	-
SASE (ours)	69.7	62.9	77.3	79.8	78.1	73.5	80.8
Table 10: Results of our sentence encoder (SASE) on STS benchmarks from 2012 to 2016 and on the test sets of the STS-Benchmark dataset, compared to previously published results. We report Pearson's  $r \times  {100}$  .
5.2 Examples of large-scale similarity search
SentAugment uses large-scale similarity search combined with an embedding space with billions of
sentences to find in-domain sentences. In Table 9, we show examples of nearest neighbors extracted from CommonCrawl based on sentence-level or label-level queries and for different domains such as biomedical, financial or hate-speech data. We see that retrieving nearest neighbors can lead to good paraphrases which either preserve the meaning or augment it with additional information. We also observe reformulation of the same input sentence. As for label-level queries, we observe that retrieved sentences match very well the domain of the downstream task. We also release as part of our work nearest-neighbor indexes for researchers to explore further large-scale similarity search of web data. These indexes provide more examples of how well the model performs when trying to find similar sentences in our corpus using our sentence embedding. We hope this will lead to an improved understanding of large-scale embedding spaces and also help the community analyze the content and biases of large-scale web corpora used to train language models.
Recent work in natural language understanding has focused on unsupervised pretraining. In this paper, we show that self-training is another effective method to leverage unlabeled data. We introduce SentAugment, a new data augmentation method for NLP that retrieves relevant sentences from a large web data corpus. Self-training is complementary to unsupervised pre-training for a range of natural language tasks and their combination leads to further improvements on top of a strong RoBERTa baseline. We also explore knowledge distillation and extend previous work on few-shot learning by showing that open domain data with SentAugment is sufficient for good accuracy.
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel M Cer, Mona T Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, et al. 2015. Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretability. In SemEval@ NAACL-HLT, pages 252-263.
Eneko Agirre, Carmen Baneab, Claire Cardiac, Daniel Cerd, Mona Diabe, Aitor Gonzalez-Agirre, Weiwei Guof, Rada Mihalceab, German Rigaua, and Janyce Wiebeg. 2014. Semeval-2014 task 10: Multilingual semantic textual similarity. SemEval 2014, page 81.
Eneko Agirre, Carmen Baneab, Daniel Cerd, Mona Diabe, Aitor Gonzalez-Agirre, Rada Mihalceab, German Rigaua, Janyce Wiebef, and Basque Country Donostia. 2016. Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. Proceedings of SemEval, pages 497-511.
Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalezagirre, and Weiwei Guo. 2013. sem 2013 shared task: Semantic textual similarity, including a pilot on typed-similarity. In In *SEM 2013: The Second Joint Conference on Lexical and Computational Semantics. Association for Computational Linguistics.
Eneko Agirre, Mona Diab, Daniel Cer, and Aitor Gonzalez-Agirre. 2012. Semeval-2012 task 6: A pilot on semantic textual similarity. In Proceedings of Semeval-2012, pages 385-393.
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In 5th International Conference on Learning Representations.
Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597-610.
Ondrej Bojar and Ales Tamchyna. 2011. Improving translation model by monolingual data. In Workshop on Statistical Machine Translation (WMT).
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder for english.
In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 169-174.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670-680, Copenhagen, Denmark. Association for Computational Linguistics.
Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. NeurIPS.
Mathias Johan Philip Creutz et al. 2018. Open subtitles paraphrase corpus for six languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA).
Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In Advances in neural information processing systems, pages 3079-3087.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL.
William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the International Workshop on Paraphrasing.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proc. of EMNLP.
Kawin Ethayarajh. 2018a. Unsupervised random walk sentence embeddings: A strong but simple baseline. ACL 2018, page 91.
Kawin Ethayarajh. 2018b. Unsupervised random walk sentence embeddings: A strong but simple baseline. In Proceedings of The Third Workshop on Representation Learning for NLP.
Junxian He, Jiatao Gu, Jiajun Shen, and Marc'Aurelio Ranzato. 2019. Revisiting self-training for neural sequence generation. arXiv preprint arXiv:1909.13788.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 328-339.
Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of SIGKDD, pages 168-177.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. IEEE Transactions on Big Data.
Jacob Kahn, Ann Lee, and Awni Hannun. 2020. Self-training for end-to-end speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7084-7088. IEEE.
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems, pages 3294-3302.
Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT Summit.
Pierre Lison, Jörg Tiedemann, Milen Kouylekov, et al. 2019. Open subtitles 2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora. In LREC 2018, Eleventh International Conference on Language Resources and Evaluation. European Language Resources Association (ELRA).
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Proc. of NAACL.
Tomáš Mikolov, Édouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.
Daniel S Park, Yu Zhang, Ye Jia, Wei Han, Chung-Cheng Chiu, Bo Li, Yonghui Wu, and Quoc V Le. 2020. Improved noisy student training for automatic speech recognition. arXiv preprint arXiv:2005.09629.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. arXiv.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.
Erik F Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. arXiv preprint cs/0306050.
H Scudder. 1965. Probability of error of some adaptive pattern-recognition machines. IEEE Transactions on Information Theory, 11(3):363-371.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment tree-bank. In EMNLP, pages 1631-1642.
Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. 2018. Learning general purpose distributed sentence representations via large scale multi-task learning. arXiv preprint arXiv:1804.00079.
Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Edouard Grave, Tatiana Likhomanenko, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert. 2019. End-to-end asr: from supervised to semi-supervised learning with modern architectures. arXiv preprint arXiv:1911.08460.
Ellen M Voorhees and Dawn M Tice. 2000. Building a question answering test collection. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pages 200-207. ACM.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzman, Armand Joulin, and Edouard Grave. 2019. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359.
John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Towards universal paraphrastic sentence embeddings. *ICLR*.
John Wieting and Kevin Gimpel. 2017. *Paranmt-50m: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations.* arXiv preprint arXiv:1711.05732.
John Wieting, Kevin Gimpel, Graham Neubig, and Taylor Berg-Kirkpatrick. 2019a. Simple and effective paraphrastic similarity from parallel translations. arXiv preprint arXiv:1909.13872.
John Wieting, Graham Neubig, and Taylor Berg-Kirkpatrick. 2019b. A bilingual generative transformer for semantic sentence embedding. arXiv preprint arXiv:1911.03895.
Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. Proceedings of the 2nd Workshop on Evaluating Vector-Space Representations for NLP.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. 2019. Unsupervised data augmentation for consistency training. arXiv preprint arXiv:1904.12848.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. 2020. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10687-10698.
I Zeki Yalniz, Herve Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. 2019. Billion-scale semi-supervised learning for image classification. arXiv preprint arXiv:1905.00546.
David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd annual meeting of the association for computational linguistics, pages 189-196.
Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin D Cubuk, and Quoc V Le. 2020. Rethinking pre-training and self-training. arXiv preprint arXiv:2006.06882.