SlowGuess/ACL_2020_OCR huggingface dataset:
For instance, let  $\alpha = 0.3$ , batch size  $b = 128$ ; further, recall that, in our experiments, a CT consists of 5 cascade rerankers. Then, after layer 4, the size of the batch gets reduced to  $90\left(\lfloor 0.3 \cdot 128\rfloor = 38 \text{ candidates are discarded by the first classifier}\right)$ . After the second classifier (layer 6),  $\lfloor 0.3 \cdot 90\rfloor = 27$  examples are further removed, for an effective batch size of 63. By layer 12, only 31 samples are left, i.e., the instance number scored by the final classifier is reduced by more than 4 times.
Our approach has the effect of improving the throughput of a transformer model by reducing the average batch size during inference: the throughput of any neural model is capped by the maximum number of examples it can process in parallel (i.e., the size of each batch), and said number is usuallyCeiled by the amount of memory available to the model (e.g., RAM on GPU). The monolithic models have a constant batch size at inference; however, because the batch size for a cascade model varies while processing a batch, we can size our network with respect to its average batch size, thus increasing the number of samples we initially have in a batch. In the example above, suppose that the hardware requirement dictates a maximum batch size of 84 for the monolithic model. As the average batch size for the cascading model is  $(4\cdot 128 + 2\cdot 90 + 2\cdot 63 + 2\cdot 44 + 2\cdot 28) / 12 =$ $80.2 < 84$  , we can process a batch of 128 instances without violating memory constrains, increasing throughput by  $52\%$
We remark that using a fixed  $\alpha$  is crucial to obtain the performance gains we described: if we were to employ a score-based thresholding ap
ASNQ	GPD	TRECQA	WikiQA
TRAIN	Questions	57,242	1,000	1,227	873
Avg cand.	413.3	99.8	39.2	9.9
Avg corr.	1.2	4.4	4.8	1.2
DEV	Questions	1,336	340	65	126
Avg cand.	403.6	99.7	15.9	9.0
Avg corr.	3.2	2.85	2.9	1.1
TEST	Questions	1,336	440	68	243
Avg cand.	400.5	101.1	20.0	9.7
Avg corr.	3.2	8.13	3.4	1.2
Table 1: Datasets statistics: ASNQ and GPD have more sentence candidates than TRECQA and WikiQA.
proach (that is, discard all candidates with score below a given threshold), we could not determine the size of batches throughout the cascade, thus making it impossible to efficiently scale our system. On the other hand, we note that nothing in our implementations prevents potentially correct candidates from being dropped when using CT. However, as we will show in Section 5, an opportune choice of a threshold and good accuracy of early classifiers ensure high probability of having at least one positive example in the candidate set for the last classifier of the cascade.
We present three sets of experiments designed to evaluate CT. In the first (Section 5.3), we show that our proposed approach without any selection produces comparable or superior results with respect to the state of the art of AS2, thanks to its stability properties; in the second (Section 5.4), we compare our Cascade Transformer with a vanilla transformer, as well as a sequence of transformer models trained independently; finally, in the third (Section 5.5), we explore the tuning of the drop ratio,  $\alpha$ .
TRECQA & WikiQA Traditional benchmarks used for AS2, such as TRECQA (Wang et al., 2007) and WikiQA (Yang et al., 2015), typically contain a limited number of candidates for each question. Therefore, while they are very useful to compare accuracy of AS2 systems with the state of the art, they do not enable testing large scale passage reranking, i.e., inference on hundreds or thousand of answer candidates. Therefore, we evaluated our approach (Sec. 4.3) on two datasets: ASNQ, which is publicly available, and our GPD dataset. We still leverage TRECQA and WikiQA to show that our
5702
cascade system has comparable performance to state-of-the-art transformer models when no filtering is applied.
ASNQ The Answer Sentence Natural Questions dataset (Garg et al., 2020) is a large collection (23M samples) of question-answer pairs, which is two orders of magnitude larger than most public AS2 datasets. It was obtained by extracting sentence candidates from the Google Natural Question (NQ) benchmark (Kwiatkowski et al., 2019). Samples in NQ consists of tuples <question, answerlong, answershort, label>, where answerlong contains multiple sentences, answershort is fragment of a sentence, and label is a binary value indicating whether answerlong is correct. The positive samples were obtained by extracting sentences from answerlong that contain answershort; all other sentences are labeled as negative. The original release of ANSQ<sup>7</sup> only contains train and development splits; we further split the dev. set to both have dev. and test sets.
GPD The General Purpose Dataset is part of our efforts to study large scale web QA and evaluate performance of AS2 systems. We built GPD using a search engine to retrieve up to 100 candidate documents for a set of given questions. Then, we extracted all candidate sentences from such documents, and rank them using a vanilla transformer model, such as the one described in Sec. 4.1. Finally, the top 100 ranked sentences were manually annotated as correct or incorrect answers.
We measure the accuracy of our approach on ASNQ and GPD using four metrics: Mean Average Precision (MAP), Mean Reciprocal Rank (MRR), Precision at 1 of ranked candidates  $(\mathrm{P}@\mathrm{1})$  and Normalized Discounted Cumulative Gain at 10 of retrieved candidates (nDCG@10). While the first two metrics capture the overall system performance, the latter two are better suited to evaluate systems with many candidates, as they focus more on Precision. For WikiQA and TRECQA, we use MAP and MRR.
5.2 Models and Training
Our models are fine-tuned starting from a pretrained RoBERTa encoder (Liu et al., 2019c). We chose this transformer model over others due to its strong performance on answer selection tasks (Garg et al., 2020). Specifically, we use the BASE
Model	WikiQA	TRECQA
MAP	MRR	MAP	MRR
CA1 (Wang and Jiang, 2016)	74.3	75.4	-	-
CA2 (Yoon et al., 2019)	83.4	84.8	87.5	94.0
\( TANDA_{BASE} \) (Garg et al., 2020)	88.9	90.1	91.4	95.2
4 layers TANDA	80.5	80.9	77.2	83.1
6 layers TANDA	82.1	82.9	78.5	88.4
8 layers TANDA	85.7	86.7	88.2	94.7
10 layers TANDA	89.0	90.0	90.5	95.9
Our \( TANDA_{BASE} \)	89.1	90.1	91.6	96.0
CT (4 layers, α = 0.0)	60.1	60.2	67.9	74.7
CT (6 layers, α = 0.0)	79.8	80.3	89.7	95.0
CT (8 layers, α = 0.0)	84.8	85.4	92.3	95.3
CT (10 layers, α = 0.0)	89.7	89.8	92.3	95.6
CT (12 layers, α = 0.0)	89.9	91.0	92.4	96.7
Table 2: Comparison on two AS2 academic datasets. With the exception of a 4-layer transformer, both the partial and final classifiers from CT achieve comparable or better performance than state of the art models.
variant (768-dimensional embeddings, 12 layers, 12 heads, and 3072 hidden units), as it is more appropriate for efficient classification. When applicable $^{8}$ , we fine-tune our models using the two-step "transfer and adapt" (TANDA) technique introduced by Garg et al. (2020).
As mentioned in Section 4.3, we optimize our model in a multi-task setting; that is, for each minibatch, we randomly sample one of the output layers of the CT classifiers to backpropagate its loss to all layers below.
While we evaluated different sampling techniques, we found that a simple uniform distribution is sufficient and allows the model to converge quickly.
Our models are optimized using Adam (Kingma and Ba, 2014) using triangular learning rate (Smith, 2017) with a 4,000 updates ramp-up<sup>9</sup>, and a peak learning rate  $l_r = 1e^{-6}$ . Batch size was set to up to 2,000 tokens per mini-batch for CT models. For the partial and final classifiers, we use 3-layers feedforward modules with 768 hidden units and tanh activation function. Like the original BERT implementation, we use dropout value of 0.1 on all dense and attention layers. We implemented our system using MxNet 1.5 (Chen et al., 2015) and GluonNLP 0.8.1 (Guo et al., 2019a) on a machine with 8 NVIDIA Tesla V100 GPUs, each with 16GB of memory.
When fine-tuning on GPD, TRECQA, and WikiQA, we perform a "transfer" step on ASNQ before "adapting" to our target dataset; for ASNQ, we directly fine-tune on it.
${}^{9}$  On ASNQ,it is roughly equivalent to  ${950k}$  samples or about 4% of the training set.
<sup>7</sup>https://github.com/alexa/wqa_tanda
5703
Method	Model	α	ASNQ	GPD	Cost reduction per batch
MAP	nDCG@10	P@1	MRR	MAP	nDCG@10	P@1	MRR
Monolithic transformer (MT)	4 layers TANDA	-	31.5	30.8	25.9	30.8	38.9	50.1	40.8	54.0	-67%
6 layers TANDA	-	60.2	58.7	47.2	59.2	51.4	64.1	56.1	67.6	-50%
8 layers TANDA	-	63.9	62.2	49.2	62.4	56.3	68.7	61.2	70.4	-33%
10 layers TANDA	-	65.3	64.5	52.0	64.1	57.2	71.3	64.9	72.7	-20%
TANDABASE	-	65.5	65.1	52.1	64.7	58.0	72.2	67.5	76.8	baseline
Sequential Ranker (SR)	MT models, 4 to 12 layers, in sequence	0.3	65.4	65.1	52.1	64.8	55.8	70.2	66.2	74.3	+53%
0.4	64.9	64.2	51.6	64.2	53.8	69.6	65.6	73.0	+18%
0.5	64.6	63.4	50.8	63.5	52.2	68.4	63.0	72.3	-10%
Cascade transformer (CT)	4 layers CT	0.0	22.0	19.3	10.2	18.3	32.7	38.9	35.2	42.6	-67%
6 layers CT	0.0	49.1	47.2	32.7	47.7	44.8	56.0	47.3	58.5	-50%
8 layers CT	0.0	62.8	61.5	48.7	61.9	53.8	71.7	61.2	69.1	-33%
10 layers CT	0.0	65.6	65.1	53.0	65.2	55.8	72.0	63.1	72.1	-20%
0.0	66.3	66.1	53.2	65.4	57.8	71.9	67.5	76.9	-0%
Full CT (12 layers)	0.3	65.3	65.3	52.9	65.3	55.7	69.8	66.2	75.1	-37%
0.4	64.8	65.0	52.5	64.8	52.8	68.6	65.6	74.3	-45%
0.5	64.1	65.0	52.4	64.5	50.2	66.1	62.4	72.9	-51%
Table 3: Comparison of Cascade Transformers with other models on the ASNQ and GPD datasets. "Monolithic transformer" refers to a single transformer model trained independently; "sequential ranker" (ST) is a sequence of monolithic transformer models of size 4, 6, ..., 12 trained independently; and "Cascade Transformer" (CT) is the approach we propose. This can train models that equal or outperform the state of the art when no drop is applied (i.e.,  $\alpha = 0.0$ ); with drop, they obtain the same performance with  $37\%$  to  $51\%$  fewer operations.
5.3 Stability Results of Cascade Training
In order to better assess how our training strategy for CT models compare with a monolithic transformer, we evaluated the performance of our system on two well known AS2 datasets, WikiQA and TRECQA. The results of these experiments are presented in Table 2. Note how, in this case, we are not applying any drop to our cascade classifier, as it is not necessary on this dataset: all sentences fit comfortably in one mini batch (see dataset statistics in Table 1), so we would not observe any advantage in pruning candidates. Instead, we focus on evaluating how our training strategy affects performance of partial and final classifiers of a CT model.
Our experiment shows that classifiers in a CT model achieve competitive performance with respect to the state of the art: our 12-layer transformer model trained in cascade outperforms  $\mathrm{TANDA}_{\mathrm{BASE}}$  by 0.8 and 0.9 absolute points in MAP (0.9 and 0.7 in MRR). 10, 8, and 6 layer models are equally comparable, differing at most by 2.3 absolute MAP points on WikiQA, and outscoring TANDA by up to 11.2 absolute MAP points on TRECQA. However, we observed meaningful differences between the performance of the 4-layers cascade model and its monolithic counterparts. We hypothesize that this is due to the fact that lower layers are not typically well suited for classification when used as part of a larger model (Peters et al., 2019); this observation is reinforced by the fact that the 4 layers TANDA model shown in Table 2
takes four times the number of the iterations of any other model to converge to a local optimum.
Overall, these experiments show that our training strategy is not only effective for CT models, but can also produce smaller transformer models with good accuracy without separate fine-tuning.
5.4 Results on Effectiveness of Cascading
The main results for our CT approach are presented in Table 3: we compared it with (i) a state-of-the-art monolithic transformer  $(\mathrm{TANDA}_{\mathrm{BASE}})$ , (ii) smaller, monolithic transformer models with 4-10 layers, and (iii) a sequential ranker (SR) consisting of 5 monolithic transformer models with 4, 6, 8, 10 and 12 layers trained independently. For CT, we report performance of each classifier individually (layers 4 up to 12, which is equivalent to a full transformer model). We test SR and CT with drop ratio  $30\%$ ,  $40\%$ ,  $50\%$ . Finally, for each model, we report the relative cost per batch compared to a base transformer model with 12 layers.
Overall, we observed that our cascade models are competitive with monolithic transformers on both ASNQ and GPD datasets. In particular, when no selection is applied ( $\alpha = 0.0$ ), a 12 layer cascade model performs equal or better to  $\mathrm{TANDA}_{\mathrm{BASE}}$ : on ASNQ, we improve P@1 by  $2.1\%$  (53.2 vs 52.1), and MAP by  $1.2\%$  (66.3 vs 65.5); on GDP, we achieve the same P@1 (67.5), and a slightly lower MAP (57.8 vs 58.0). This indicates that, despite the multitasking setup, out method is competitive with the state of the art.
A drop rate  $\alpha > 0.0$  produces a small degradation in accuracy, at most, while significantly reducing the number of operations per batch  $(-37\%)$ . In particular, when  $\alpha = 0.3$ , we achieve less than  $2\%$  drop in P@1 on GPD, when compared to  $\mathrm{TANDA}_{\mathrm{BASE}}$ ; on ANSQ, we slightly improve over it (52.9 vs 52.1). We observe a more pronounced drop in performance for MAP, this is to be expected, as intermediate classification layers are designed to drop a significant number of candidates.
For larger values of  $\alpha$ , such as 0.5, we note that we achieve significantly better performance than monolithic transformer of similar computational cost. For example, CT achieves an  $11.2\%$  improvement in  $\mathbf{P}@\mathbf{1}$  over a 6-layers TANDA model (62.4 vs 56.1) on GPD; a similar improvement is obtained on ANSQ  $(+11.0\%, 52.4$  vs 47.2).
Finally, our model is also competitive with respect to a sequential transformer with equivalent drop rates, while being between 1.9 to 2.4 times more efficient. This is because an SR model made of independent TANDA models cannot re-use encodings generated by smaller models as CT does.
5.5 Results on Tuning of Drop Ratio  $\alpha$
Finally, we examined how different values for drop ratio  $\alpha$  affect the performance of CT models. In particular, we performed an exhaustive grid-search on a CT model trained on the GPD dataset for drop ratio values  $\{\alpha_{p_1},\alpha_{p_2},\alpha_{p_3},\alpha_{p_4}\}$ , with  $\alpha_{p_k} \in \{0.1, 0.2,\dots,0.6\}$ . The performance is reported in Figure 2 with respect to the relative computational cost per batch of a configuration when compared with a TANDA<sub>BASE</sub> model.
Overall, we found that CT models are robust with respect to the choice of  $\{\alpha_{p_k}\}_{k = 1}^4$ . We observe moderate degradation for higher drop ratio values (e.g., P@1 varies from 85.6 to 80.0). Further, as expected, performance increases for models with higher computational cost per batch, although they taper off for CT models with relative cost  $\geq 70\%$ . On the other hand, the grid search results do not seem to suggest an effective strategy to pick optimal values for  $\{\alpha_{p_k}\}_{k = 1}^4$ , and, in our experiments, we ended up choosing the same values for all drop rates. In the future, we would be like to learn such values while training the cascade model itself.
This work introduces CT, a variant of the traditional transformer model designed to improve inference
Figure 2: Grid search plot on the GPD validation set. Each point corresponds to a configuration of drop ratios  $\{\alpha_{p_1},\ldots ,\alpha_{p_4}\}$  with  $\alpha_{p_k}\in \{0.1,0.2,\dots ,0.6\}$ ; values on the  $x$  -axis represent the relative computational cost per batch of a configuration compared to TANDA<sub>BASE</sub>. The three runs reported in Table 3 correspond to  $\triangle (\alpha = 0.3)$ ,  $\spadesuit$  ( $\alpha = 0.4$ ), and  $\bullet$  ( $\alpha = 0.5$ ).
throughput. Compared to a traditional monolithic stacked transformer model, our approach leverages classifiers placed at different encoding stages to prune candidates in a batch and improve model throughput. Our experiments show that a CT model not only achieves comparable performance to a traditional transformer model while reducing computational cost per batch by over  $37\%$ , but also that our training strategy is stable and jointly produces smaller transformer models that are suitable for classification when higher throughput and lower latency goals must be met. In future work, we plan to explore techniques to automatically learn where to place intermediate classifiers, and what drop ratio to use for each one of them.
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In ICLR.
Arvind Agarwal, Hema Raghavan, Karthik Subbian, Prem Melville, Richard D. Lawrence, David C. Gondek, and James Fan. 2012. Learning to rank for robust question answering. In Proceedings of the 21st ACM International Conference on Information and Knowledge Management, CIKM '12, pages 833-842, New York, NY, USA. ACM.
Nima Asadi and Jimmy Lin. 2013. Effectiveness/efficiency tradeoffs for candidate generation in multi-stage retrieval architectures. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '13, pages 997-1000, New York, NY, USA. ACM.
Yonatan Belinkov, Lluis Marquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2017. Evaluating layers of representation in neural machine translation on part-of-speech and semantic tagging tasks. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1-10, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Weijie Bian, Si Li, Zhao Yang, Guang Chen, and Zhiqing Lin. 2017. A compare-aggregate model with dynamic-clip attention for answer selection. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM '17, pages 1987-1990, New York, NY, USA. ACM.
Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. 2015. Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv:1512.01274.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy. Association for Computational Linguistics.
Kevin Clark, Minh-Thang Luong, Christopher D Manning, and Quoc Le. 2018. Semi-supervised sequence modeling with cross-view training. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1914-1925.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. CoRR, abs/1901.02860.
Van Dang, Michael Bendersky, and W. Bruce Croft. 2013. Two-stage learning to rank for information retrieval. In Proceedings of the 35th European Conference on Advances in Information Retrieval, ECIR'13, pages 423-434, Berlin, Heidelberg. Springer-Verlag.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Luke Gallagher, Ruey-Cheng Chen, Roi Blanco, and J. Shane Culpepper. 2019. Joint optimization of cascade ranking models. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, WSDM '19, pages 15-23, New York, NY, USA. ACM.
Siddhant Garg, Thuy Vu, and Alessandro Moschitti. 2020. Tanda: Transfer and adapt pre-trained transformer models for answer sentence selection. In AAAI.
Jian Guo, He He, Tong He, Leonard Lausen, Mu Li, Haibin Lin, Xingjian Shi, Chenguang Wang, Jun-yuan Xie, Sheng Zha, et al. 2019a. GluonCV and GluonNLP: Deep learning in computer vision and natural language processing. arXiv preprint arXiv:1907.04433.
Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. 2019b. Starttransformer. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1315-1325.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907-926.
Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651-3657, Florence, Italy. Association for Computational Linguistics.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
5706