SlowGuess/ACL_2020_OCR huggingface dataset:
(b) F1 drop versus speedup on SQuAD for DeFormer-BERT-large without auxiliary supervision.
Figure 4: F1 drop versus speedup of DeFormer-BERT model (without auxiliary supervision) when separating at different layers.
BERT-base model, and compute their distance of the vector representations at each layer.
Figure 5 shows the averaged distances of both the question and passage at different layers. The lower layer representations of the passage and questions for both models remain similar but the upper layer representations differ significantly, supporting the idea that lack of cross-attention has less impact in the lower layers than in the higher ones. Also, using the auxiliary supervision of upper layers has the desired effect of forcing DeFormer to produce representations that are closer to the original model. This effect is less pronounced for the question representations.
4.2 Inference Cost
DeFormer enables caching of text representations that can be computed offline. While a full-scale analysis of the detailed trade-offs in storage versus latency is beyond the scope of this paper, we present a set of basic calculations to illustrate that the storage cost of caching can be substantially smaller compared to the inference cost. Assuming a use case of evaluating one million question-passage pairs daily, we first compute the storage requirements of the representations of these passages. With the BERT-base representations we estimate this to be 226KB per passage and 226GB in total for 1 million passages. The cost of storing this data and the added compute costs and reading these passages at the current vendor rates amounts to a total of $61.7 dollars per month. To estimate inference cost, we use the compute times we obtain from our calculations and use current vendor rates for GPU workloads which amounts to $148.5 dollars to support the 1 million question-passage pair workload. The substantial reduction in cost
is because the storage cost is many orders of magnitude cheaper than using GPUs. Details of these calculations are listed in the Appendix.
Speeding up inference in a model requires reducing the amount of compute involved. There are two broad related directions of prior work:
(i) Compression techniques can be used to reduce model size through low rank approximation (Zhang et al., 2015; Kim et al., 2015; Tai et al., 2015; Chen et al., 2018), and model weights pruning (Guo et al., 2016; Han et al., 2015), which have been shown to help speedup inference in CNN and RNN based models. For Transformers, Michel et al. (2019) explore pruning the attention heads to gain inference speedup. This is an orthogonal approach that can be combined with our decomposition idea. However, for the paired-input tasks we consider, pruning heads only provides limited speedup. In more recent work Ma et al. (2019) propose approximating the quadratic attention computation with a tensor decomposition based multilinear attention model. However, it is not clear how this multi-linear approximation can be applied to pre-trained Transformers like BERT.
(ii) Distillation techniques can be used to train smaller student networks to speedup inference. Tang et al. (2019) show that BERT can be used to guide designing smaller models (such as single-layer BiLSTM) for multiple tasks. But for the tasks we study, such very small models suffer a significant performance drop. For instance there is a  $13\%$  accuracy degradation on MNLI task. Another closely related recent work is DistillBERT (Sanh et al., 2019), which trains a smaller BERT model (half the size of BERT-base) that runs 1.5 times faster
(a) Passage distance comparison
(b) Question distance comparison
Figure 5: Representation distance of BERT vs DeFormer-BERT and distance of BERT vs DeFormer-BERT w/o auxiliary loss/supervision
than the original BERT-base. However, the distilled model incurs a significant drop in accuracy. While more recent distillation works such as (Jiao et al., 2019) and (Sun et al., 2020) further improve the speedups, our decomposition also achieves similar accuracy performance. More importantly, this distillation model usually undergo expensive pretraining on the language modeling tasks before they can be fine-tuned for the downstream tasks.
Previous QA neural models like BIDAF(Seo et al., 2016), QANet(Yu et al., 2018) and many others contain decomposition as part of their neural architecture design. In contrast, the focus of our work is to show that large pre-trained Transformer models can be decomposed at the fine-tuning stage to bring effectiveness of SOTA pre-trained transformers at much lower inference latency.
In this work, we ask if can we speedup the inference of Transformer models without compressing or removing model parameters. Part of the massive success of pre-trained Transformer models for many NLP task is due to a large amount of parameters capacity to enable complex language representations. The decomposition we propose makes minimal changes retaining the overall capacity and structure of the original model but allows for faster inference by enabling parallel processing and caching of segments.
DeFormer applies to settings where the underlying model relies on input-wide self-attention layers. Even with models that propose alternate ways to improve efficiency, as long as the models use input-wide self-attention, DeFormer can be applied as a complementary mechanism to further improve inference efficiency. We leave an evaluation of applying DeFormer on top of other recent efficiency optimized models for future work.
Transformers have improved the effectiveness of NLP tools by their ability to incorporate large contexts effectively in multiple layers. This however imposes a significant complexity cost. In this work, we showed that modeling such large contexts may not always be necessary. We build a decomposition of the transformer model that provides substantial improvements in inference speed, memory reduction, while retaining most of the original model's accuracy. A key benefit of the model is that its architecture remains largely the same as the original model which allows us to avoid repeating pretraining and use the original model weights for fine-tuning. The distillation techniques further reduce the performance gap with respect to the original model. This decomposition model provides a simple yet strong starting point for efficient QA models as NLP moves towards increasingly larger models handling wider contexts.
We thank Google for supporting this research through the Google Cloud Platform credits.
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiao-
qiang Zheng. 2015. TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow.org.
Qingqing Cao, Noah Weber, Niranjan Balasubramanian, and Aruna Balasubramanian. 2019. DeQA: On-Device Question Answering. In Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services - MobiSys '19, pages 27-40, Seoul, Republic of Korea. ACM Press.
Patrick Chen, Si Si, Yang Li, Ciprian Chelba, and Cho-Jui Hsieh. 2018. Groupreduce: Block-wise low-rank approximation for neural language model shrinking. In Advances in Neural Information Processing Systems, pages 10988-10998.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019a. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019b. What does bert look at? an analysis of berts attention. Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Google. 2019. Understanding searches better than ever before. https://blog.google/products/search/search-language-understanding-bert/.
Yiwen Guo, Anbang Yao, and Yurong Chen. 2016. Dynamic network surgery for efficient dnns. In Advances In Neural Information Processing Systems, pages 1379-1387.
Song Han, Huizi Mao, and William J Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149.
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. 2019. Visualizing and understanding the effectiveness of bert.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network.
Shankar Iyer, Nikhil Dandekar, and Kornl Csernai. 2019. Quora question pairs. https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pa
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. Tinybert: Distilling bert for natural language understanding.
Yong-Deok Kim, Eunhyeok Park, Sungwoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. 2015. Compression of deep convolutional neural networks for fast and low power mobile applications. arXiv preprint arXiv:1511.06530.
NVIDIA Performance Lab. 2019. Considerations for scalinggpu-ready data centers. https://www.nvidia.com/content/g/pdfs/GPU-Ready-Data-Center-Tech-Overview.pdf.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Xindian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Dawei Song, and Ming Zhou. 2019. A tensorized transformer for language modeling. arXiv preprint arXiv:1906.09777.
Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? arXiv preprint arXiv:1905.10650.
Jonas Mockus. 1975. On bayesian methods for seeking the extremum. In *Optimization Techniques IFIP Technical Conference*, pages 400-404. Springer.
Nogueira, Fernando. 2019. Bayesianoptimization. https://github.com/fmfn/BayesianOptimization. [Online; accessed 22-September-2019].
Google Cloud Platform. 2019. Cloud pricing. https://cloud.google.com/compute/all-pricing#gpus,https://cloud.google.com/storage/pricing.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad:  $100,000+$  questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.
Victor Sanh, Lysandre Debut, and Thomas Wolf. 2019. Introducing distilbert, a distilled version of bert. https://medium.com/huggingface/distilbert-8cf3380435b5.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint. s. arXiv:1611.01603.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. Mobilebert: a compact task-agnostic bert for resource-limited devices.
Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, et al. 2015. Convolutional neural networks with low-rank regularization. arXiv preprint arXiv:1511.06067.
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019. Distilling task-specific knowledge from bert into simple neural networks. ArXiv, abs/1903.12136.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. Bert rediscovers the classical nlp pipeline. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237.
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. 2018. Qanet: Combining local convolution with global self-attention for reading comprehension. arXiv preprint arXiv:1804.09541.
Xiangyu Zhang, Jianhua Zou, Xiang Ming, Kaiming He, and Jian Sun. 2015. Efficient and accurate approximations of nonlinear convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and pattern Recognition, pages 1984-1992.
Data centers often use GPUs for inference workloads (Lab, 2019), we use the GPUs by default for both models. We use  $g_{u}$  to denote the cost of using one GPU per hour,  $n_{seq}$  to stand for the number of input sequences to process,  $b$  for the GPU batch size, and  $t_{b}$  is the time (in seconds) take to process  $b$  sequences,  $s$  denotes the storage size of the cached representations,  $s_{u}$  denotes the cost of storage per month,  $r_{u}$  is the cost of performing 10,000 reading operations (such as loading cached representations from the disk).
The total cost of the original model  $Cost_{original}$  is the cost of using GPUs and is given by the formula as below:
$$
C o s t _ {o r i g i n a l} = t _ {b} \cdot \frac {n _ {s e q}}{b} \cdot \frac {g _ {u}}{3 6 0 0}
$$
And the total cost of the decomposed model  $Cost_{decomp}$  includes three parts: using GPUs, storing representations on disk and loading them into memory. It is formulated as:
$$
\begin{array}{l} C o s t _ {d e c o m p} = t _ {b} \cdot \frac {n _ {s e q}}{b} \cdot \frac {g _ {u}}{3 6 0 0} + \frac {n _ {s e q}}{b} \cdot \frac {r _ {u}}{1 0 , 0 0 0} \\ + \frac {s \cdot s _ {u}}{3 0 * 2 4 * 3 6 0 0} \\ \end{array}
$$
We assume a passage has 150 tokens on average (The number is calculated based on the SQuAD dataset).
We take one cloud service provider (Platform, 2019) to instantiate  $g_{u}$ ,  $s_{u}$ , and  $r_{u}$ : one Tesla V100 GPU (16GB memory) costs $2.48 USD per hour ( $g_{u} = 2.48$ ), 1GB storage takes $0.02 per month ( $s_{u} = 0.02$ ) and additional $0.004 per 10,000 read operations ( $r_{u} = 0.004$ ).
It takes 226KB to store the vectors for 150 tokens  $^{8}$ , and the total storage for 1 million sequences is 226GB. The Tesla V100 GPU allows a maximum batch size of  $640^{9}$ . We measure the  $t_b = 4.6$  for the original BERT-base model and  $t_b = 1.4$  for the decomposed BERT-base model. Then  $Cost_{original} = 30 * 4.6 * 1,000,000 / 640 * 2.48 / 3600 = \\(148.5$ , and  $Cost_{decomp} = 30 * 1.4 * 1,000,000 / 640 * 2.48 / 3600 + 30 * 1,000,000 / 10,000 * 0.004 + 226 * 0.02 = \$ 61.7.\)
Hyper-parameters We set the final  $\alpha = 1.1$ ,  $\beta = 0.5$  and  $\gamma = 0.7$  for supervising BERT-base model on the SQuAD dataset,  $\alpha = 0.4$ ,  $\beta = 0.4$  and  $\gamma = 0.7$  and on the RACE dataset. For XLNet, we find that simple default parameters ( $\alpha = 1.1$ ,  $\beta = 0.5$  and  $\gamma = 0.7$ ) work well for both SQuAD and BoolQ datasets.
Class B operations on GCP
8 vector dimension=768, bfloat16 format
$9 > 640$  batch size will cause V100 GPU out of memory