SlowGuess/NAACL_2022_OCR huggingface dataset:
Empirical Methods in Natural Language Processing, pages 2089-2103, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. MobileBERT: a compact task-agnostic BERT for resource-limited devices. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2158-2170, Online. Association for Computational Linguistics.
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019. Distilling task-specific knowledge from BERT into simple neural networks. *ArXiv* preprint, abs/1903.12136.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2020. Efficient transformers: A survey. ArXiv preprint, abs/2009.06732.
Yi Tay, Mostafa Dehghani, Jai Prakash Gupta, Vamsi Aribandi, Dara Bahri, Zhen Qin, and Donald Metzler. 2021. Are pretrained convolutions better than pretrained transformers? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4349-4359, Online. Association for Computational Linguistics.
Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models. ArXiv preprint, abs/1908.08962.
Shuohang Wang and Jing Jiang. 2017. A compare-aggregate model for matching text sequences. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. 2020a. Linformer: Self-attention with linear complexity. ArXiv preprint, abs/2006.04768.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020b. Minilm: Deep selfattention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Moshe Wasserblat, Oren Pereg, and Peter Izsak. 2020. Exploring the boundaries of low-resource BERT distillation. In Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, pages 35-40, Online. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen,
Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex machina: Personal attacks seen at scale. In WWW, WWW '17, page 1391-1399, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
Weiqiu You, Simeng Sun, and Mohit Iyyer. 2020. Hard-coded Gaussian attention for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7689-7700, Online. Association for Computational Linguistics.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 649-657.
Datasets and corpora used, and their specifications are previously listed in Table 1. Here we provide links to download these data.
- IMDB: https://ai.stanford.edu/~amaas/data/sentiment/
- SST-2: https://huggingface.co/datasets/glue
- AGNews: https://huggingface.co/datasets/ag_news
- TREC: https://huggingface.co/datasets/trec
- CivilComments: https://huggingface.co/datasets/civil/comments
- WikiToxic: https://www.tensorflow.org/datasets/catalog/wiki/toxicity_subtypes and https://meta.m.wikimedia.org/wiki/Research:Detox/Data_Release
- QQP: https://huggingface.co/datasets/glue
- Amazon Reviews: https://nijianmo.github.io/amazon/index.html
- PAQ: https://github.com/facebookresearch/PAQ
- Reddit News: https://zissou.infosci.cornell.edu/convokit/datasets/subreddit-corpus/corpus-zipped/newreddits_nsfw~news/news.corpus.zip
QQP dataset has 363,846 training instances and 40,430 development instances. The average input length is 13 tokens. We thank huggingface dataset team (Lhoest et al., 2021) for providing easy access to these datasets.
Licensing. For WikiToxic, the dataset is licensed under CC0, with the underlying comment text being governed by Wikipedia's CC-SA-3.0. The PAQ QA-pairs and metadata is licensed under CC-BY-SA. The licensing information of other datasets are unknown to us.
N-gram pre-processing are implemented with scikit-learn (Pedregosa et al., 2011). DistilBERT (Sanh et al., 2019) and MobileBERT baselines are implemented in huggingface transformers (Wolf et al., 2020). RoBERTa-Large, BiLSTM, CNN, and DAN experiments are implemented with fairseq (Ott et al., 2019).
For fine-tuning in stage 1, we select batch size from  $\{16,32\}$  and learning rate from  $\{1e - 5,2e - 5,5e - 5\}$  following the recommendations in (Liu et al., 2019). We train the model for 10 epochs on  $D_{train}$ . For knowledge distillation in stage 2, we set the batch size to be 2048, learning rate to be 5e-4, and total number of updates to be 1,000,000, as they work well in our preliminary experiments. The embedding table is randomly initialized and the embedding dimension  $d_{e}$  is set to 1,000, unless specified otherwise. For further fine-tuning in
stage 3, we set the batch size to be 32 and select the learning rate from  $\{3\mathrm{e} - 4,1\mathrm{e} - 4,3\mathrm{e} - 5\}$ . We train the model for 10 epochs on  $D_{train}$ . For all training procedures, we validate the model at the end of each epoch in the case of fine-tuning, or every 100,000 steps in the case of knowledge distillation. We save the best checkpoint based on dev accuracy. Due to the analysis nature of this work and the scale of experiments, performance are computed using dev set and based on one single run.
A.4 Hardware
Model Training. Except for the parallel training attempt in Table 5, all experiments are done on one single GPU. We train DAN models on either A100 40GB PCIe or Quadro RTX 8000 depending on availability. Knowledge distillation (Stage 2) with 1,000,000 updates typically finishes within 36 hours.
Inference Speed Tests. All inference speed tests are done with the batch size of 32. GPU inference is performed with one Quadro RTX 8000 GPU, and CPU inference is performed with 56 Intel Xeon CPU E5-2690 v4 CPUs.
B Additional Details
B.1 DAN Variations
Due to space limits we have omitted the details for the DAN variations we studied in  $\S 4.1$ . We introduce these variations in the following.
Attentive Pooling. We consider adding attentive pooling to the DAN model to capture more complicated relations in the input. Our attention layer is modified from the one in (Zhang et al., 2017). we use the representation  $\mathbf{h}$  after mean pooling as query, and each n-gram embedding  $\mathbf{e}_i = \mathrm{Emb}(g_i)$  as key. More specifically, for each n-gram  $g_{i}$  we calculate an attention weight  $a_{i}$  as:
$$
u _ {i} = \mathbf {v} ^ {\top} \tanh (\mathbf {W} _ {g} \mathbf {e} _ {i} + \mathbf {W} _ {h} \mathbf {h}) \qquad (1)
$$
$$
a _ {i} = \frac {\exp (u _ {i})}{\sum_ {j = 1} ^ {n} \exp (u _ {j})} \tag {2}
$$
Here  $\mathbf{W}_g, \mathbf{W}_h \in \mathbb{R}^{d_e \times d_a}$  and  $\mathbf{v} \in \mathbb{R}^{d_a}$  are learnable parameters.  $d_e$  is the dimension of the embedding table, and  $d_a$  is the size of the attention layer. To maintain an acceptable training speed, for attentive pooling, we use a batch size of 512 during knowledge distillation.
Parallel Training We try further scaling up the student model by splitting the gigantic embedding table to different GPUs and enable parallel training, as implemented in Megatron-LM (Shoeybi et al., 2019). We train a 2-billion parameter model in parallel on two GPUs. The embedding dimension is set to be 2,000 in total, and each GPU handles an embedding table of hidden dimension 1,000. The vocabulary size is 1 million.
B.2 Comments on SparseAdam
SparseAdam is a modified version of the regular Adam optimizer. For Adam, the first and second moment for each parameter is updated at every step. This can be costly, especially for DAN, as most parameters in the embedding layer are not used during the forward pass. SparseAdam computes gradients and updates the moments only for parameters used in the forward pass.
Speed Comparison. Table 12 is an extended version of Table 4 which contains inference speed comparison on IMDB and SST-2 dataset, in three different settings (GPU-FP32, GPU-FP16, CPU-FP32). Our major conclusion remains the same: DANs achieve excellent inference speed in various settings.
Vocabulary Size vs. Embedding Dimension Trade-off. Table 8 contains original results that were visualized in Fig. 4.
Param. 500m	Param. 1b	Param. 2b
|V|	de	Acc	de	Acc.	de	Acc.
1m	500	93.0	1000	93.2	-	-
2m	250	92.8	500	93.0	900	93.1
4m	125	92.7	250	92.9	500	93.1
5m	100	92.6	200	92.9	400	93.1
10m	50	92.3	100	92.5	200	92.9
20m	25	92.0	50	92.2	100	92.7
40m	-	-	25	92	50	92.4
Table 8: IMDB dev accuracy with different configurations of vocabulary size  $\left( {\left| V\right| }\right)$  and embedding table dimension  $\left( {d}_{e}\right)$  . Performance grows with larger embedding tables, and the best performing model has  $\left| V\right|  = {1m}$  and  ${d}_{e} = 1,{000}$  .
N-gram Coverage Statistics. In our work, we opt to determine the n-gram vocabulary with the training set  $D_{train}$  and the corpus  $C$ , by selecting the top 1 million n-grams according to frequency.
N-gram range is set to be within 1 to 4. For reference, we list statistics about the n-gram vocabulary in Table 13. It is possible that adjustments to this pre-processing step (e.g., up-weighting n-grams in  $D_{train}$  and down-weighting n-grams in  $C$ ) will further improve performance, however we stop further investigation.
Tokenization Speed. The speed comparison in our work does not take pre-processing process into account. When the inference speed is at millisecond level (e.g., with our DAN model), preprocessing time can become non-negligible. For reference, in Table 9 we report the tokenization time on the 25,000 training instances in the IMDB dataset with (1) n-gram tokenization (used by DAN, implemented with scikit-learn); (2) BPE tokenization (used by RoBERTa/DistilRoBERTa, implemented with fairseq); (3) WordPiece tokenization (used by DistilBERT, implemented with huggingface transformers).
Tokenization Method	Time	Complexity
BPE	26.46	O(nlg n) or O(|V|n) (Song et al., 2021)
WordPiece	20.60	O(n2) or O(mn) (Song et al., 2021)
N-gram	16.45	O(n)
Table 9: Comparison of tokenization speed and complexity. Time is computed for tokenization the train set of IMDB dataset (25,000 instances) with one single worker. Time is averaged across 5 runs.  $n$  represents input length.
First of all, by setting the number of workers to be equal to the batch size (32) we use in the speed test, the tokenization speed will be 48632 instances/sec ( $= 25000 / 16.45^{*}32$ ), which is roughly  $3x$  faster than the inference speed. Tokenization speed is non-negligible in this case. Still, the main conclusion from the speed comparison remains the same: DANs are typically  $10x - 100x$  faster than the compared models.
Secondly, DAN models still have better tokenization speed than transformer models that use BPE/WordPiece tokenization. This is because our DAN model computes n-grams based on whitespace tokenization, which can be done in linear time when the n-gram to id mapping is implemented with a hashmap, i.e.,  $O(n)$  where  $n$  is the input length. BPE/WordPiece tokenization has higher complexity according to Song et al. (2021).
We would also like to emphasize that this part is also highly dependent on the design choice and implementation. For example, the user could im
plement a DAN model with BPE tokenization. The choice and optimization of tokenization is beyond the scope of this work.
Impact of  $n$  in  $\mathbf{n}$ -grams. Similar to the post-hoc pruning experiments in §4.2, we gradually disable the usage of four-grams, trigrams and bigrams at inference time, and report the performance in Table 10.
IMDB	AGNews
|V|	Acc.	|V|	Acc.
n = 1	54,089	74.86	81,796	91.32
n ≤ 2	446,793	92.09	541,431	92.93
n ≤ 3	835,403	93.33	882,489	93.03
n ≤ 4 (all)	1,000,000	93.47	1,000,000	92.99
Table 10: Impact of  $n$  in  $\mathbf{n}$  -grams. We disable usage of longer  $\mathrm{n}$  -grams in the DAN(KD+FT) model.  $\left| V\right|$  is the size of the vocabulary after disabling.
Model Storage. In Table 11 we provide more details about the disk space and memory required for using DAN models and the baseline models. Note that the GPU memory listed below is the memory used to load the static model. During training, more memory will be dynamically allocated during forward and backward passes. DAN uses smaller memory during training because only a small portion of the parameters are activated and trained (see the last row in Table 13). In this way we are able to use batch sizes as large as 2048 to train DANs on one single GPU, which is not possible for transformer based models.
#Param	GPU Memory	Disk Space	Source
RoBERTa-Large	355M	2199MB	711MB	fairseq (fp16)
RoBERTa-Large	355M	2199MB	1.33GB	HF transformers (fp32)
DistilBERT	66M	1123MB	256MB	HF transformers
MobileBERT	25M	973MB	140MB	HF transformers
DistilRoBERTa	85M	1181MB	316MB	HF transformers
LSTM (21-128d)	53M	1051MB	212MB	fairseq (fp32)
CNN (11-256d)	53M	1119MB	213MB	fairseq (fp32)
DAN	1001M	4655MB	3.99GB	fairseq (fp32)
Table 11: Disk space and GPU memory required for each model.
It is risky to deploy DAN models to high-stakes applications (e.g., medical decisions) as the model lacks the ability of understanding long context (see case study in §4.4). DANs may raise fairness concerns: it lacks ability to understand the meaning of words in context, so it may learn spurious correlations such as overemphasis on group identifiers.
Parameter Count	IMDB	SST-2
Total	Sparse	Dense	Acc.	GPU-fp32	GPU-fp16	CPU-fp32	Acc.	GPU-fp32	GPU-fp16	CPU-fp32
RoBERTa-Large	355M	51M	304M	96.3	28.9 (1x)	92.3 (1x)	1.4 (1x)	96.2	267.3 (1x)	610.2 (1x)	22.2 (1x)
DistillBERT	66M	23M	43M	92.2	175.8 (6x)	334.7 (4x)	10.7 (8x)	90.8	828.5 (3x)	1117.3 (2x)	60.6 (3x)
MobileBERT	25M	4M	21M	93.6	157.7 (5x)	200.3 (2x)	7.7 (6x)	90.9	574.5 (2x)	545.8 (1x)	89.4 (4x)
*DistillRoBERTa	83M	39M	44M	95.9	176.4 (6x)	569.8 (6x)	7.8 (6x)	94.2	636.5 (2x)	771.7 (1x)	185.9 (8x)
*LSTM (2l-512d)	62M	51M	11M	95.9	361.5 (12x)	594.5 (6x)	30.6 (22x)	93.9	4222.1 (14x)	6281.4 (8x)	394.3 (18x)
*LSTM (2l-256d)	56M	51M	5M	95.8	665.2 (23x)	788.0 (9x)	51.9 (37x)	93.3	6361.5 (21x)	7080.6 (9x)	678.5 (31x)
*LSTM (2l-64d)	53M	51M	2M	94.0	818.5 (28x)	808.5 (9x)	101.4 (73x)	92.8	7075.8 (24x)	7384.1 (9x)	1378.5 (62x)
*LSTM (2l-4d)	52M	51M	<1M	93.1	812.9 (28x)	817.0 (9x)	146.4 (105x)	88.3	7026.3 (24x)	7521 (9x)	2014.6 (91x)
*CNN (1l-256d)	53M	51M	2M	89.2	3410.7 (109x)	8427.1 (91x)	251.2 (181x)	82.8	1323.5 (5x)	1563.9 (3x)	3820.4 (172x)
*DAN (ours)	1001M	1000M	1M	93.5	17557.9 (607x)	20888.1 (226x)	922.6 (663x)	88.5	1745.5 (7x)	1865.9 (3x)	16478.6 (741x)
Table 12: Model Size and Inference Speed Comparison. We report accuracy, inference speed (unit: samples per second) and relative speed compared to the teacher model (RoBERTa-Large). Our DAN model achieves competitive accuracy while achieving significant inference speed-up in various settings.  $\star$  indicates the model is trained with task-specific distillation; no  $\star$  indicates the model is trained with direct fine-tuning.
Notation	Description	IMDB	SST-2	TREC	AGNews	CCom	WToxic
V0	Top 1 million n-grams in C and Dtrain	1,000,000
V1	All n-grams in Dtrain	10,109,522	262,417	89,358	7,156,063	116,143,462	15,805,923
V2	All n-grams in Ddev	9,843,369	39,666	5,995	662,665	8,987,055	6,958,457
V3	V0∩V1	805,360	76,370	31,770	486,438	983,843	828,302
|V0∩V1|/|V0| (%)	80.54%	7.64%	3.18%	48.44%	98.38%	82.83%
|V0∩V1|/|V1| (%)	7.97%	29.10%	35.56%	6.80%	0.85%	5.24%
V4	V0∩V2	792,251	15,395	3,461	123,247	740,286	671,985
|V0∩V2|/|V0| (%)	79.22%	1.54%	0.35%	12.32%	74.03%	67.20%
|V0∩V2|/|V2| (%)	8.05%	38.81%	57.73%	18.60%	8.23%	9.18%
V5	V0∩V1∩V2	690,790	8,804	1,840	113,311	739,920	638,833
|V0∩V1∩V2|/|V2| (%)	7.01%	22.20%	30.69%	17.10%	8.23%	9.18%
-	Average # activated n-grams per instance	496	16	17	68	103	144
Table 13: Size of different sets of n-gram and their statistics of n-gram coverage.
We believe a thorough analysis is needed and bias mitigation methods such as (Bolukbasi et al., 2016; Kennedy et al., 2020) are necessary for combating these issues.