SlowGuess/ACL_2020_OCR huggingface dataset:
In this section, we present our experimental results. We apply Algorithm 1 on BERT base. For pre-training BERT base we use MXNET based gluon-nlp repository that uses the hyper-parameters suggested in the original BERT paper. Besides pretraining, our algorithm has three hyper-parameters: regularization coefficient  $\gamma$ , learning rate for prunable parameters, and the number of steps for regularizing prune parameters- Equation (2). We run hyper-parameter optimization on these parameters to get the best results. For regularization loss (2), we use the same training data that we use for pre-training, BooksCorpus (800M words) and English Wikipedia (2,500M words). However, we run the regularization step for  $1/1000$ th steps as used for pre-training. We finet-une the pruned BERT
Algorithm 1 Pruning Transformers
Input: A Transformer model, minimization objective (FLOPs/Params/Latency), target fraction  $\eta$ , number of iterations  $T$ .
Output: A optimally pruned Transformer model. pre-training: Train the network using loss Equation (1).
Repeat  $T$  times:
Initialize prunable parameters  $\alpha_{h},\alpha_{a_{i}},\alpha_{k_{i}},\alpha_{v_{i}},\alpha_{f_{i}}\to 1$
- Multiply prunable parameters with network parameter
$$
\widetilde {K _ {i} , Q _ {i} , V _ {i} , P _ {i} , D _ {i} , G _ {i}} \rightarrow \widetilde {K _ {i} , \widetilde {Q _ {i} , \widetilde {V _ {i} , \widetilde {P _ {i} , \widetilde {D _ {i} , \widetilde {G _ {i}}}}}}}
$$
- Train the network using loss Equation (2)
- Set the  $\zeta$  smallest prunable parameters to zero to achieve  $\eta / T$  reduction in the target objective value
- Offset zero and non-zero prunable parameters into model parameters
$$
\widetilde {K _ {i}}, \widetilde {Q _ {i}}, \widetilde {V _ {i}}, \widetilde {P _ {i}}, \widetilde {D _ {i}}, \widetilde {G _ {i}} \rightarrow K _ {i}, Q _ {i}, V _ {i}, P _ {i}, D _ {i}, G _ {i}
$$
$$
K _ {i}, Q _ {i}, \bar {V} _ {i}, P _ {i}, D _ {i}, G _ {i} \rightarrow \widehat {K} _ {i}, \widehat {Q} _ {i}, \widehat {V} _ {i}, \widehat {P} _ {i}, \widehat {D} _ {i}, \widehat {G} _ {i}
$$
- Create smaller model parameter tensors by removing all-zero rows/columns
- Finetune the model using loss Equation (1)
by training for  $1 / 20$ th of the steps used for pretraining.
We provide accuracy results for schuBERT on the following downstream tasks- question answering datasets- SQuAD v1.1, SQuAD v2.0; and GLUE datasets - MNLI, MRPC, SST-2 and RTE. For these downstream tasks, we use the fine-tuning hyper-parameters as suggested in the BERT paper.
We create six schuBERTs by pruning one or all of the design dimensions. Accuracy of the downstream tasks on these schuBERTs are given in Tables 6-13. The BERT base has 108 million parameters. The schuBERT sizes 88, 66, 43 million are chosen to match the number of parameters in BERT with  $\ell \in \{9,6,3\}$  layers.
We use schuBERT- $x$  notation for  $x \in \{h, f, a\}$  to denote a schuBERT obtained by only pruning  $h$ -hidden size,  $f$ -filter size of feed-forward,  $a$ -number of attention heads respectively. We use schuBERT-all to denote the case when all the design dimensions-  $h, f, a, k, v$ , except  $\ell$  are pruned.
We compare our results with original BERT base, and by varying its number of encoder layers  $\ell \in \{12,9,6,3\}$ . We denote these results by BERT- $\ell$ . Since ALBERT reduces parameters by factorizing the embedding matrix, we denote its results by
model	SQuAD v1.1	SQuAD v2.0	MNLI	MRPC	SST-2	RTE	Avg
BERT-base (108M)	90.2/83.3	80.4/77.6	84.1	87.8	92.1	71.4	84.3
# parameters = 99M
schuBERT-all	89.8/83.0	80.1/77.6	83.9	87.5	92.4	71.1	84.1
schuBERT-f	89.8/82.9	79.6/77.3	83.5	87.4	91.6	70.7	83.8
schuBERT-h	89.6/82.6	79.9/77.5	83.7	87.3	91.5	70.4	83.7
BERT-all uniform	89.7/82.7	79.8/77.3	83.7	87.2	92.0	69.8	83.7
schuBERT-a	89.3/82.3	79.1/77.4	83.3	86.8	91.1	69.1	83.1
Table 6: Accuracy results on SQuAD and GLUE datasets obtained by fine-tuning BERT and schuBERTs with total of 99 million parameters.
l	1	2	3	4	5	6	7	8	9	10	11	12
f =	2022	2222	2344	2478	2576	2530	2638	2660	2748	2792	2852	2974
a =	12	12	12	12	11	12	12	12	12	12	12	12
k =	64	64	64	64	64	64	64	64	64	64	64	64
v =	54	54	46	58	52	60	64	64	64	64	64	62
number of encoder layers  $\ell = 12$  , number of hidden units  $h = 768$
Table 7: Design dimensions of schuBERT-all for 99 million parameters.
ALBERT- $e$ . ALBERT provided results only for 88 million parameter model, not for any smaller models. Further, we also compare with the baseline case when all the design dimensions are pruned uniformly. We denote these results by BERT-all uniform.
For 99M model, Table 6, schuBERT-all beats the baseline BERT-all uniform by  $0.4\%$  higher average accuracy and performs better than schuBERT- $f / h / a$ . Moreover, the loss in performance in comparison to BERT base with 108 million parameters is only  $0.2\%$ . Table 7 gives exact design dimensions for schuBERT-all with 99 million parameters. We see that number of hidden units remain same as in BERT base,  $h = 768$ . Parameter reduction primarily comes from feed-forward layers. Moreover, filter size of feed-forward layer  $-f$  has a clear increasing pattern across the layers.
For 88M model, Table 8, again schuBERT-all beats all the other models. It gives  $1.1\%$  higher average accuracy than BERT-  $\ell$  with 9 layers. ALBERT-  $\mathcal{E}$  performs better on SQuAD datasets, but performs significantly worse on MNLI and SST2 datasets. Note ALBERT's approach is complementary to our approach and it can be incorporated into our schubERTs. schubERT-  $a$  performs significantly worse than schubERT-all which implies that pruning only number of attention heads is highly sub-optimal, as is recently done in Michel et al. (2019). Table 9 provides the exact design dimensions for schubERT-all with 88 million parameters.
Similar to 99M model, filter size of feed-forward layer  $f$  has a clear increasing pattern across the layers.
For heavily pruned models - 77M, 66M, 55M and 43M models - accuracy results are shown in Table 10, Table 11, Table 12 and Table 13 respectively. In all these models schuBERT-  $h$  beats all the other models. For 66M model, schuBERT-  $h$  gives  $1.9\%$  higher average accuracy than BERT-  $\ell$  with 6 layers. For 43M model, schuBERT-  $h$  gives  $6.6\%$  higher average accuracy than BERT-  $\ell$  with 3 layers. That is reducing the hidden units is way better than to reduce the number of layers to create a light BERT model. Ideally, we would expect schuBERT-all to perform better than schuBERT-  $h$ , but marginally worse performance of schuBERT-all can be attributed to the high complexity of pruning all the design dimensions together.
Table 14 provides best schuBERT architectures when the number of model parameters are restricted to different values. For smaller models, schuBERT- $h$  outperforms all other schuBERTs including schuBERT-all. Note that our schuBERT architectures are smaller in size as well as they yield lower latency.
7 schuBERT
Based on the above described experimental results, we provide following insights on the design dimensions of schuBERT architecture.
Slanted Feed-forward Layer. The fully-
model	SQuAD v1.1	SQuAD v2.0	MNLI	MRPC	SST-2	RTE	Avg
BERT-base (108M)	90.2/83.3	80.4/77.6	84.1	87.8	92.1	71.4	84.3
# parameters = 88M
BERT-ℓ	88.4/80.9	78.8/77.2	83.8	85.6	91.3	68.2	82.7
schuBERT-all	89.4/82.5	79.8/77.1	84.1	87.6	92.3	69.7	83.8
schuBERT-f	89.2/82.2	79.5/77.5	83.7	87.4	92.2	69.3	83.6
BERT-all uniform	89.1/82.0	79.6/77.6	83.7	87.5	91.7	68.9	83.4
schuBERT-h	89.1/82.0	79.4/77.3	83.6	87.2	91.5	69.2	83.3
schuBERT-a	85.1/77.1	74.1/72.4	82.2	85.2	90.9	67.0	80.8
ALBERT-e	89.9/82.9	80.1/77.8	82.9	-	91.5	-	-
Table 8: Accuracy results on SQuAD and GLUE datasets obtained by fine-tuning BERT, ALBERT, and schuBERTs with total of 88 million parameters.
l	1	2	3	4	5	6	7	8	9	10	11	12
f =	1382	1550	1672	1956	2052	2030	2210	2314	2474	2556	2668	2938
a =	12	12	11	12	11	12	12	12	12	12	12	12
k =	64	64	64	64	64	64	64	64	64	64	64	64
v =	46	48	42	52	46	54	64	62	64	64	64	40
number of encoder layers  $\ell = 12$  , number of hidden units  $h = 756$
Table 9: Design dimensions of schuBERT-all for 88 million parameters.
connected component applied to each token separately plays a much more significant role in the top layers as compared to the bottom layers. Figure 3 shows pattern of filter size of feed-forward layer across the encoder cells for various schuBERT-all models. In each of them, filter size follows an increasing pattern with min-max ratio ranging from 1.5 to 4, as opposed to same value across all the layers.
Tall and Narrow BERT. When we aim to obtain a computationally lighter model, using a 'tall and narrow' architecture provides better performance than a 'wide and shallow' architecture. Our results in Tables 8, 11, 13 demonstrate that schubert with  $\ell = 12$  encoder layers significantly outperforms BERT with  $\ell \in \{9,6,3\}$  layers for the same number of parameters.
Expansive Multi-head Attention. The ratio of the design dimensions within a BERT encoder layer can be modified to obtain a better performing layer architecture. Transformer design dimensions suggested in (Vaswani et al., 2017) are sub-optimal.
Following the original Transformer architecture described in (Vaswani et al., 2017), BERT and other Transformer based models set key-query  $k$  and value  $v$  dimension for multi-head attention to  $k = v = h / a$ , where  $h$  is the size of the hidden representation, and  $a$  is the number of attention heads. Also, following the same architecture (Vaswani
et al., 2017), BERT sets feed-forward filter size  $f = 4h$ . Although there is no restriction in using different output dimensions  $k, v$  and filter size  $f$ , without changing the behaviour of the attention mechanism, we are not aware of any study questioning this 'default value' of  $k = v = h / a$  and  $f = 4h$ .
Our schuBERT architecture for various model sizes given in Table 14, show that for smaller models  $k, v$  should be much larger than  $h / a$ . For 43M schuBERT model  $h / a = 25.3$  whereas  $k = v = 64$ . Also,  $f$  should be much larger than  $4h$ . For the same 43M schuBERT model  $4h = 936$  whereas  $f = 3072$ . Table 13 shows that 43M schuBERT ( $\ell = 12, h = 304, a = 12, k = v = 64, f = 3072$ ) significantly outperforms BERT- $\ell$  ( $\ell = 3, h = 768, a = 12, k = v = h / a, f = 4h$ ).
Figure 3: Feed-forward size across the encoder layers in schuBERT-all for various model sizes.
parameters  $= 77\mathrm{M}$
model	SQuAD v1.1	SQuAD v2.0	MNLI	MRPC	SST-2	RTE	Avg
schuBERT-h	88.8/81.6	78.6/76.3	84.0	87.2	91.5	68.9	83.2
BERT-all uniform	88.8/81.6	78.4/76.0	83.7	86.6	91.9	68.9	83.1
schuBERT-f	88.8/81.4	78.8/76.1	83.2	86.5	92.2	67.7	82.9
schuBERT-all	88.8/81.6	78.6/76.2	83.8	86.6	92.2	66.4	82.7
schuBERT-a	82.6/74.2	73.1/68.9	82.0	84.9	89.6	66.4	79.8
Table 10: Accuracy results on SQuAD and GLUE datasets obtained by fine-tuning BERT and schuBERTs with total of 77 million parameters.
parameters  $= 66\mathrm{M}$
model	SQuAD v1.1	SQuAD v2.0	MNLI	MRPC	SST-2	RTE	Avg
BERT-ℓ	85.3/77.1	75.3/72.5	82.3	84.4	91.1	67.6	81.0
schuBERT-h	88.1/80.7	78.4/74.7	83.8	86.7	91.7	68.5	82.9
schuBERT-all	88.0/80.7	78.2/74.5	83.2	87.2	91.3	67.8	82.6
BERT-all uniform	87.7/80.3	77.8/74.0	83.6	86.2	91.3	68.1	82.4
schuBERT-f	87.6/80.0	77.6/74.1	83.0	86.8	90.6	68.1	82.3
Table 11: Accuracy results on SQuAD and GLUE datasets obtained by fine-tuning BERT and schuBERTs with total of 66 million parameters.
parameters  $= 55\mathrm{M}$
model	SQuAD v1.1	SQuAD v2.0	MNLI	MRPC	SST-2	RTE	Avg
schuBERT-h	87.6/80.3	77.4/74.6	83.5	86.3	90.9	66.7	82.1
schuBERT-all	86.8/79.3	76.6/73.5	83.4	86.3	90.9	66.8	81.8
BERT-all uniform	86.2/78.5	76.9/72.2	83.2	84.0	90.5	67.1	81.3
schuBERT-f	85.8/77.5	75.8/71.8	81.8	84.4	90.2	67.3	80.9
Table 12: Accuracy results on SQuAD and GLUE datasets obtained by fine-tuning BERT and schuBERTs with total of 55 million parameters.
parameters  $= 43\mathrm{M}$
model	SQuAD v1.1	SQuAD v2.0	MNLI	MRPC	SST-2	RTE	Avg
BERT-ℓ	75.6/65.8	65.9/57.8	78.5	79.5	87.3	63.8	75.1
schuBERT-h	86.7/79.0	76.9/73.8	83.4	84.8	90.9	67.3	81.7
schuBERT-all	86.0/77.9	76.7/72.8	82.6	84.2	90.5	66.2	81.0
BERT-all uniform	85.0/77.2	75.3/72.4	82.2	83.4	90.6	67.2	80.6
schuBERT-f	84.2/75.5	74.7/69.8	80.3	77.1	89.7	58.7	77.5
Table 13: Accuracy results on SQuAD and GLUE datasets obtained by fine-tuning BERT and schuBERTs with total of 43 million parameters.
# parameters	BERT	99M	88M	77M	66M	55M	43M	33M
l =	12	12	12	12	12	12	12	12
h =	768	768	756	544	466	390	304	234
f(min - max) =	3072	2022 - 2974	1382 - 2938	3072	3072	3072	3072	3072
a(min - max) =	12	11 - 12	11 - 12	12	12	12	12	12
k(min - max) =	64	64	64	64	64	64	64	64
v(min - max) =	64	46 - 64	40 - 64	64	64	64	64	64
Table 14: Best schuBERT architectures for different number of model parameters. BERT base has 108M parameters.
Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. 2017. Structured pruning of deep convolutional neural networks. ACM Journal on Emerging Technologies in Computing Systems (JETC), 13(3):32.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326.
Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In Advances in neural information processing systems, pages 3079-3087.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Song Han, Huizi Mao, and William J Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149.
Babak Hassibi and David G Stork. 1993. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in neural information processing systems, pages 164-171.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2017. Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, 18(1):6869-6898.
Yoon Kim and Alexander M Rush. 2016. Sequence-level knowledge distillation. arXiv preprint arXiv:1606.07947.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942.
Yann LeCun, John S Denker, and Sara A Solla. 1990. Optimal brain damage. In Advances in neural information processing systems, pages 598-605.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. 2016. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2018. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? arXiv preprint arXiv:1905.10650.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. 2016. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440.
Kenton Murray and David Chiang. 2015. Auto-sizing neural networks: With applications to n-gram language models. arXiv preprint arXiv:1508.05051.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. arXiv preprint arXiv:1806.00187.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365.
Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean. 2018. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv:1802.03268.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, Technical report, OpenAI.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. 2016. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pages 525-542. Springer.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
Abigail See, Minh-Thang Luong, and Christopher D. Manning. 2016. Compression of neural machine translation models via pruning. arXiv preprint arXiv:1606.09274.
Shashank Singh, Ashish Khetan, and Zohar Karnin. 2019. Darc: Differentiable architecture compression. arXiv preprint arXiv:1905.08170.
Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-informed self-attention for semantic role labeling. arXiv preprint arXiv:1804.08199.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237.
Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. 2016. Trained ternary quantization. arXiv preprint arXiv:1612.01064.
Figure 4, Figure 5 and Figure 6 show the pattern of number of heads, key-query dimension, and value dimension across the encoder layers for various schuBERT-all architectures respectively. There is not much significant pattern in these design dimensions across the layers. The number of attention heads drastically reduce to 1 in the top layer for very small models. Same is true for key-query and value dimensions. Key-query remains almost same as their original value 64 even when the models are pruned heavily, except in the top layer. Whereas value dimension does decrease significantly from their original value when the models are pruned heavily.
Figure 4: Number of multi-attention heads across the encoder layers in schuBERT-all for various model sizes.
Figure 5: Dimension of key-query vectors across the encoder layers in schuBERT-all for various model sizes.
Figure 6: Dimension of value vectors across the encoder layers in schuBERT-all for various model sizes.