SlowGuess/NAACL_2022_OCR huggingface dataset:
NLP datasets sometimes involve user generated text or sensitive information; therefore, data privacy can be a concern when training and deploying models with certain NLP datasets. In this section, we modify our experiment setting to a practical and privacy-preserving one. We assume the user has access to a public teacher model that is trained on private train dataset  $(D_{train})$ , but does not have access to  $D_{train}$  itself. This is realistic nowadays with the growth of public model hubs such as TensorFlow Hub<sup>5</sup> and Hugging Face Models<sup>6</sup>. After downloading the model, the user may wish to deploy a faster version of this model, or adapt the model to the user's own application domain.
Knowledge Distillation without  $D_{train}$ . To simulate the privacy-preserving setting, we remove  $D_{train}$  from the knowledge distillation stage in our experiment pipeline and only use the unlabeled corpus  $C$ . We use "DAN (KD) $^{\dagger}$ " to denote this model in Table 2. By comparing "DAN (KD)" and "DAN (KD) $^{\dagger}$ ", we found that the performance difference brought by task specific data  $D_{train}$  is small for all single-sentence tasks, with the largest gap being  $1.2\%$  on IMDB dataset. This suggests that the proposed pipeline is still useful in the absence of task-specific data.
<sup>5</sup>https://www.tensorflow.org/hub
Source	IMDB	SST-2
Target	SST-2	IMDB
DAN (from scratch, tar)	79.5	88.3
(1) RoBERTa-Large (src)	90.0	94.1
(2) DAN (KD)	81.9	92.0
(3) DAN (KD+FT)	88.4	93.0
(4) DAN (KD+FT w. re-init.)	86.7	92.8
RoBERTa-Large (tar)	96.2	96.3
Table 6: Domain Generalization and Adaptation Results. (1) We take the teacher model trained on the source dataset and evaluate it on the target dataset. (2) We obtain the student model "DAN (KD)" with unlabeled corpus  $C$  and knowledge distillation. (3) We further fine-tune the student model on the target dataset to obtain "DAN (KD+FT)". (4) The classification head is re-initialized before further fine-tuning.
Domain Generalization and Adaptation. We select the two sentiment analysis tasks: IMDB and SST-2, and further explore the domain generalization/adaptation setting. Specifically, during stage 1 of our three-stage pipeline (§2.4), we fine-tune the RoBERTa-Large model on a source dataset; during stage 2, we apply knowledge distillation with unlabeled corpus  $C$  only and get the student model; during stage 3, we further fine-tune the student model on the target dataset. The last step is optional and serves to simulate the situation where the user collects additional data for domain adaptation. We list the results in Table 6. With weakened assumptions about the teacher model and distillation supervision, we still have observations similar to those in our main experiments (§4.1): Performance of the final student model is significantly improved compared to DANs trained from scratch.
4.4 Limitations and Discussions
Extension to sentence-pair tasks. So far we have limited the scope to single-sentence classification tasks. We consider extending our sparse distillation framework to a sentence-pair task, Quora Question Pair (QQP)<sup>7</sup>, which aims to identify duplicated questions. We create pseudo sentence-pair data for knowledge distillation by randomly sampling 10 million question pairs from PAQ. To better model the relation between a pair of sentence, we modify DANs by introducing a concatenate-compare operator (Wang and Jiang, 2017), following the practice in (Tang et al., 2019). More
Figure 6: Analysis on N-gram Coverage. Left: Relation between n-gram coverage and cross-entropy loss w.r.t. ground truth labels. Each blue line represents the median loss in that n-gram coverage bucket. Right: Distribution of n-gram coverage.
specifically, the two input sentences,  $\mathbf{x}_1$  and  $\mathbf{x}_2$ , go through the embedding layer and average pooling independently, resulting in two sentence representations,  $\mathbf{h}_1$  and  $\mathbf{h}_2$ . We then apply the concatenate compare operator, i.e.,  $f(\mathbf{h}_1,\mathbf{h}_2) = [\mathbf{h}_1,\mathbf{h}_2,\mathbf{h}_1\odot \mathbf{h}_2,|\mathbf{h}_1 - \mathbf{h}_2|]$ , where  $\odot$  represents element-wise multiplication. Finally,  $f(\mathbf{h}_1,\mathbf{h}_2)$  goes through two fully connected layers for classification, the same as DANs for single-sentence tasks.
The results on QQP dataset is listed in the rightmost column in Table 2. Firstly, knowledge distillation still helps close the gap between RoBERTa-Large and DANs trained from scratch (2% improvement) and leads to a decent accuracy of  $84.2\%$ ; however the benefit brought by KD is not as strong as with single-sentence tasks. Secondly, the performance of  $\mathrm{DAN}(\mathrm{KD})^{\dagger}$  (i.e., without access to  $D_{train}$  during KD) is much worse than the performance of DAN(KD). We hypothesize that this is due to the quality and distribution of knowledge distillation corpus. We randomly sample questions pairs as the knowledge distillation examples, which may not carry sufficient supervision signals – more than  $99\%$  of them are negative ("not duplicated") examples. Creating more suitable distillation corpus for sentence-pair tasks is beyond the scope of our work, and we leave this as future work.
Impact of N-gram Coverage. One potential drawback of n-grams (based on white-space tokenization) is that they cannot directly handle out-of-vocabulary words, while WordPiece/BPE tokenization together with contextualization can better handle this issue. In Fig. 6, we quantify the influence of n-gram coverage on IMDB dev set. Here, n-gram coverage for an input sentence is defined as  $|G \cap V| / |V|$ , where  $G$  represents the set of n-grams
$^{7}$ https://quoradata.quora.com/First-Quora-DataSet-Release-Question-Pairs
Teacher	Student	Label	Sentence
Negative	Positive	Negative	I really wanted to love this film. ...
Negative	Positive	Negative	This movie is a great movie ONLY if you need something to sit and laugh at the stupidity of it. ...
Positive	Negative	Positive	...They are such bad actors and it made this movie so much funnier to watch. ...
Table 7: Case study on IMDB predictions. In these cases, the model can only make the correct predictions by understanding long contexts. Performance of DAN models are still limited as they only look at local n-grams.
in the sentence and  $V$  is the n-gram vocabulary (§2.5). We first group the instances into buckets of n-gram coverage (e.g., [40%, 50%), [50%, 60%)) and then compute the statistics of cross-entropy loss in each bucket. We observe that performance is worse on sentences with more out-of-vocabulary words. Future work may build upon this observation and improve DANs performance by addressing out-of-vocabulary words. For example, BPE-based n-grams may be used for creating the vocabulary.
Case study: What are DANs still not capable of? We take a closer look at the predictions made by our DAN model (student) and the RoBERTa-Large model (teacher) on the IMDB dataset. We list several representative cases in Table 7. These cases typically require understating of complex language phenomena, such as irony, conditional clauses, and slang. In addition, these phenomena typically occur in contexts longer than 4 words, which DANs are not capable of modeling by design. For example, "bad actors" can mean "good actors" based on the later context "much funnier to watch". We conclude that sparse distillation is not suitable to cases where modeling complex language phenomena has a higher priority than improving inference speed.
Understanding the performance gaps. Tay et al. (2021) advocate that architectural advances should not be conflated with pre-training. Our experiments further support this claim, if we consider knowledge distillation as a "substitute" for pre-training that provides the student model with stronger inductive biases, and interpret the remaining teacher-student performance gap as the difference brought by architectural advances. On the other hand, we believe the power of DANs are previously undermined due to the challenges in optimizing large sparse models with limited supervision. Our experiments show that knowledge distillation effectively densify the supervision and greatly improve the performance of DANs.
Additional Analysis and Specifications. Due to space limit, we leave some additional analysis and specifications in Appendix C. We discuss
tokenization speed (Table 9) and impact of  $n$  in n-grams (Table 10). We provide more detailed speed comparison in Table 12, model storage and memory usage information in Table 11. We provide fine-grained n-gram coverage information in Table 13.
Efficient Transformers. Recent work attempts to improve computation or memory efficiency of transformer models mainly from the following perspectives: (1) Proposing efficient architectures or self-attention variants, e.g., Linformer (Wang et al., 2020a), Longformer (Beltagy et al., 2020). Tay et al. (2020) provide a detailed survey along this line of work. (2) Model compression using knowledge distillation, e.g., DistillBERT (Sanh et al., 2019), MobileBERT (Sun et al., 2020), MiniLM (Wang et al., 2020b). These compressed models are typically task-agnostic and general-purpose, while in this work we focus on task-specific knowledge distillation. (3) Weight quantization and pruning, e.g., Gordon et al. (2020); Li et al. (2020); Kundu and Sundaresan (2021).
Task-specific Knowledge Distillation in NLP. Researchers explored distilling a fine-tuned transformer into the following lightweight architectures, including smaller transformers (Turc et al., 2019; Jiao et al., 2020), LSTMs (Tang et al., 2019; Adhikari et al., 2020) and CNNs (Chia et al., 2019). Wasserblat et al. (2020) distill BERT into an architecture similar to DAN, however they restrict the model to only take unigrams (thus having small student models), and adopt a non-standard low-resource setting. To summarize, existing work typically focuses on reducing both number of parameter and the amount of computation, while in the paper we study an under-explored area in the design space, where the amount of computation is reduced by training a larger student model.
Reducing Contextualized Representations to Static Embeddings. Related to our work, Ethayarajh (2019) and Bommasani et al. (2020) show
how static word embeddings can be computed from BERT-style transformer models. Ethayarajh (2019) suggest that less than  $5\%$  of the variance in a word's contextualized representation can be explained by a static embedding, justifying the necessity of contextualized representation. Bommasani et al. (2020) found that static embeddings obtained from BERT outperforms Word2Vec and GloVe in intrinsic evaluation. These two papers mainly focus on post-hoc interpretation of pre-trained transformer models using static embeddings. In our work we opt to use knowledge distillation to learn n-gram embeddings. Meanwhile we acknowledge that the technique in Ethayarajh (2019) and Bommasani et al. (2020) could be used as an alternative method to convert transformer models to fast text classifiers.
Sparse Architectures. In our work we aggressively cut off computation cost by compensating it with more parameters in the student model. Alternatively, one could fix the computational cost at the same level as a transformer while greatly expanding the parameter count, as explored in the Switch Transformer (Fedus et al., 2021). Both their work and ours agree in the conclusion that scaling up parameter count allows the model to memorize additional useful information.
6 Conclusions & Future Work
We investigated a new way of using knowledge distillation to produce a faster student model by reversing the standard practice of having the student be smaller than the teacher and instead allowed the student to have a large table of sparsely-activated embeddings. This enabled the student model to essentially memorize task-related information that if an alternate architecture were used would have had to be computed. We tested this method on six single-sentence classification tasks with models that were up to 1 billion parameters in size, approximately 3x as big as the RoBERTa-Large teacher model, and found that the student model was blazing fast and performed favorably.
We hope that our work can lead to further exploration of sparse architectures in knowledge distillation. There are multiple directions for future work, including extending the DAN architecture to better support tasks with long range dependencies like natural language inference or multiple inputs like text similarity. Additionally, more work is needed to test the idea on non-English languages where n-gram statistics can be different from English.
We would like to thank Robin Jia, Christina Sauper, and USC INK Lab members for the insightful discussions. We also thank anonymous reviewers for their valuable feedback. Qinyuan Ye and Xiang Ren are supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via Contract No. 2019-19051600007, the DARPA MCS program under Contract No. N660011924033, the Defense Advanced Research Projects Agency with award W911NF-19-20271, NSF IIS 2048211, NSF SMA 1829268.
Ashutosh Adhikari, Achyudh Ram, Raphael Tang, William L. Hamilton, and Jimmy Lin. 2020. Exploring the limits of simple learners in knowledge distillation for document classification with DocBERT. In 5th Workshop on Representation Learning for NLP, pages 72-77, Online. ACL.
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. ArXiv preprint, abs/2004.05150.
Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam Tauman Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 4349-4357.
Rishi Bommasani, Kelly Davis, and Claire Cardie. 2020. Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4758-4781, Online. Association for Computational Linguistics.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019. Nuanced metrics for measuring unintended bias with real data for text classification. In *Companion proceedings of the 2019 world wide web conference*, pages 491-500.
Thorsten Brants, Ashok C. Popat, Peng Xu, Franz J. Och, and Jeffrey Dean. 2007. Large language models in machine translation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 858-867, Prague, Czech Republic. Association for Computational Linguistics.
Christian Buck, Kenneth Heafield, and Bas van Ooyen. 2014. N-gram counts and language models from
the Common Crawl. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 3579-3584, Reykjavik, Iceland. European Language Resources Association (ELRA).
Jonathan P. Chang, Caleb Chiam, Liye Fu, Andrew Wang, Justine Zhang, and Cristian Danescu-Niculescu-Mizil. 2020. ConvoKit: A toolkit for the analysis of conversations. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 57-60, 1st virtual meeting. Association for Computational Linguistics.
Yew Ken Chia, Sam Witteveen, and Martin Andrews. 2019. Transformer to CNN: Label-scarce distillation for efficient text classification. ArXiv preprint, abs/1909.03508.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 55–65, Hong Kong, China. Association for Computational Linguistics.
William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. *ArXiv* preprint, abs/2101.03961.
Mitchell Gordon, Kevin Duh, and Nicholas Andrews. 2020. Compressing BERT: Studying the effects of weight pruning on transfer learning. In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 143-155, Online. Association for Computational Linguistics.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. *ArXiv* preprint, abs/1503.02531.
Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1681–1691, Beijing, China. Association for Computational Linguistics.
Xiaqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163-4174, Online. Association for Computational Linguistics.
Brendan Kennedy, Xisen Jin, Aida Mostafazadeh Davani, Morteza Dehghani, and Xiang Ren. 2020. Contextualizing hate speech classifiers with post-hoc explanation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5435-5442, Online. Association for Computational Linguistics.
Souvik Kundu and Sairam Sundaresan. 2021. Attentionlite: Towards efficient self-attention models for vision. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2225-2229.
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Kuttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. 2021. PAQ: 65 million probably-asked questions and what you can do with them. Transactions of the Association for Computational Linguistics, 9:1098-1115.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumont, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierrick Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Xin Li and Dan Roth. 2002. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics.
Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joseph E Gonzalez. 2020. Train large, then compress: Rethinking model size for efficient training and inference of transformers. ArXiv preprint, abs/2002.11794.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. ArXiv preprint, abs/1907.11692.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts.
2011. Learning word vectors for sentiment analysis. In ACL, pages 142-150, Portland, Oregon, USA.
Sebastian Nagel. 2016. Cc-news. URL: http://web.archive.org save/http://commoncrawl.org/2016/10/newsdatasetavailable.
Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 188-197, Hong Kong, China. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairseq: A fast, extensible toolkit for sequence modeling.* In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)*, pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. ArXiv preprint, abs/1910.01108.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053.
Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2888-2913, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2021. Fast WordPiece tokenization. In Proceedings of the 2021 Conference on