SlowGuess/NAACL_2022_OCR huggingface dataset:
linearly with input length, reducing overheads of self-attention. Refer to Appendix C.3 for details.
Qualitative Examples: We present some qualitative examples from the three AS2 datasets highlighting cases where the pairwise RoBERTa-Base model is unable to rank the correct answer on the top position, but our pre-trained joint model (Joint MSPP  $\mathrm{IE}_k\rightarrow \mathrm{FT}$ $\mathrm{IE}_k$ ) can do this using supporting information from other candidates in Table 4.
In this paper we have presented a multi-sentence cross-encoder for performing inference jointly on multiple sentences for tasks like answer sentence selection and fact verification. We have proposed a novel pre-training task to capture paragraph-level semantics. Our experiments on three answer selection and one fact verification datasets show that our pre-trained joint models can outperform pairwise cross-encoders and pre-trained LMs when directly used as joint models.
Qingyao Ai, Keping Bi, Jiafeng Guo, and W. Bruce Croft. 2018. Learning a deep listwise context model for ranking refinement. The 41st International ACM SIGIR Conference on Research and Development in Information Retrieval.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150.
Weijie Bian, Si Li, Zhao Yang, Guang Chen, and Zhiqing Lin. 2017. A compare-aggregate model with dynamic-clip attention for answer selection. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM '17, page 1987–1990, New York, NY, USA. Association for Computing Machinery.
Daniele Bonadiman and Alessandro Moschitti. 2020. A study on efficiency, accuracy and document structure for answer sentence selection. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5211-5222, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew Peters, Arie Cattan, and Ido Dagan. 2021. CDLM: Cross-document language modeling. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 2648-2662, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pretraining text encoders as discriminators rather than generators. In International Conference on Learning Representations.
Nicki Skafte Detlefsen, Jiri Borovec, Justus Schock, Ananya Harsh Jha, Teddy Koker, Luca Di Liello, Daniel Stancl, Changsheng Quan, Maxim Grechkin, and William Falcon. 2022. Torchmetrics - measuring reproducibility in pytorch. Journal of Open Source Software, 7(70):4101.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Siddhant Garg and Alessandro Moschitti. 2021. Will this question be answered? question filtering via answer model distillation for efficient question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7329-7346, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Siddhant Garg, Thuy Vu, and Alessandro Moschitti. 2020. Tanda: Transfer and adapt pre-trained transformer models for answer sentence selection. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7780-7788.
Dvir Ginzburg, Itzik Malkiel, Oren Barkan, Avi Caciularu, and Noam Koenigstein. 2021. Self-supervised document similarity ranking via contextualized language models and hierarchical inference. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP* 2021, pages 3088-3098, Online. Association for Computational Linguistics.
John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. 2021. DeCLUTR: Deep contrastive learning for unsupervised textual representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 879-895, Online. Association for Computational Linguistics.
Aaron Gokaslan and Vanya Cohen. 2019. OpenwebText corpus. http://Skylion007.github.io/OpenWebTextCorpus.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations.
Luca Di Liello, Matteo Gabburo, and Alessandro Moschitti. 2021. Efficient pre-training objectives for transformers.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2020. Fine-grained fact verification with kernel graph attention network. In Proceedings of ACL.
Gehui Shen, Yunlun Yang, and Zhi-Hong Deng. 2017. Inter-weighted alignment network for sentence pair modeling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1179-1189, Copenhagen, Denmark. Association for Computational Linguistics.
Luca Soldaini and Alessandro Moschitti. 2020. The cascade transformer: an application for efficient answer sentence selection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5697-5708, Online. Association for Computational Linguistics.
Dominik Stammbach and Elliott Ash. 2020. e-fever: Explanations and summaries for automated fact checking. In TTO.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERIFICATION. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
Kateryna Tymoshenko and Alessandro Moschitti. 2021. Strong and light baseline models for fact-checking joint inference. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP* 2021, pages 4824-4830, Online. Association for Computational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Computational Linguistics.
Mengqiu Wang, Noah A. Smith, and Teruko Mitamura. 2007. What is the Jeopardy model? a quasi-synchronous grammar for QA. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 22-32, Prague, Czech Republic. Association for Computational Linguistics.
Wei Wang, Bin Bi, Ming Yan, Chen Wu, Jiangnan Xia, Zuyi Bao, Liwei Peng, and Luo Si. 2020. Structbert: Incorporating language structures into pre-training for deep language understanding. In International Conference on Learning Representations.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. 2021. PRIMER: pyramid-based masked sentence pre-training for multi-document summarization. CoRR, abs/2110.08499.
Yi Yang, Scott Wen-tau Yih, and Chris Meek. 2015. Wikiqa: A challenge dataset for open-domain question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. ACL - Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontonon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33.
Zeyu Zhang, Thuy Vu, and Alessandro Moschitti. 2021. Joint models for answer verification in question answering systems. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3252-3262, Online. Association for Computational Linguistics.
Chen Zhao, Chenyan Xiong, Corby Rosset, Xia Song, Paul Bennett, and Saurabh Tiwary. 2020. Transformer-xh: Multi-evidence reasoning with extra hop attention. In The Eighth International Conference on Learning Representations (ICLR 2020).
Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin.
2020. Reasoning over semantic-level graph for fact checking. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6170-6180, Online. Association for Computational Linguistics.
Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2019. GEAR: Graph-based evidence aggregating and reasoning for fact verification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 892-901, Florence, Italy. Association for Computational Linguistics.
We present the complete details for all the datasets used in this paper along with links to download them for reproducibility of results.
We use the Wikipedia $^2$ , BookCorpus $^3$ , OpenWebText (Gokaslan and Cohen, 2019) and CC-News $^4$  datasets for performing pre-training of our joint transformer models. We do not use the STORIES dataset as it is no longer available for research use $^5$ . After decompression and cleaning we obtained 6GB, 11GB, 38GB and 394GB of raw text respectively from the BookCorpus, Wikipedia, OpenWebText and CC-News.
A.2 Finetuning Datasets
We evaluate our joint transformers on three AS2 and one Fact Verification datasets. The latter differs from the former in not selecting the best candidate, but rather explicitly using all candidates to predict the target label. Here are the details of the finetuning datasets that we use for our experiments along with data statistics for each dataset:
Dataset	Split	# Questions	# Candidates	Avg. # C/Q
ASNQ	Train	57,242	20,377,568	356.0
Dev	1,336	463,914	347.2
Test	1,336	466,148	348.9
WikiQA	Train	2,118	20,360	9.6
Dev	122	1,126	9.2
Test	237	2,341	9.9
TREC-QA	Train	1,226	53,417	43.6
Dev	69	1,343	19.5
Test	68	1,442	21.2
- ASNQ: A large-scale AS2 dataset (Garg et al., 2020)<sup>6</sup> where the candidate answers are from Wikipedia pages and the questions are from search queries of the Google search engine. ASNQ is a modified version of the Natural Questions
(NQ) (Kwiatkowski et al., 2019) dataset by converting it from a machine reading to an AS2 dataset. This is done by labelling sentences from the long answers which contain the short answer string as positive correct answer candidates and all other answer candidates as negatives. We use the dev. and test splits released by Soldaini and Moschitti<sup>7</sup>.
- WikiQA: An AS2 dataset released by Yang et al.8 where the questions are derived from query logs of the Bing search engine, and the answer candidate are extracted from Wikipedia. This dataset has a subset of questions having no correct answers (all-) or having only correct answers  $(\mathrm{all}+)$ . We remove both the all- and all+ questions for our experiments ("clean" setting).
- TREC-QA: A popular AS2 dataset released by Wang et al.. For our experiments, we trained on the train-all split, which contains more noise but also more question-answer pairs. Regarding the dev. and test sets we removed the questions without answers, or those having only correct or only incorrect answer sentence candidates. This setting refers to the "clean" setting (Shen et al., 2017), which is a TREC-QA standard.
- FEVER: A popular benchmark for fact extraction and verification released by Thorne et al. The aim is to retrieve evidences given a claim, and then identify whether the retrieved evidences support or refute the claim or if there is not enough information to make a choice. For supporting/refuting a claim, at least one of the retrieved evidences must support/retrieve the claim. Note that the performance on FEVER depends crucially on the retrieval system and the candidates retrieved. For our experiments, we are interested only in the fact verification sub-task and thus we exploit the evidences retrieved by Liu et al. using a BERT-based DocIR<sup>9</sup>.
Table 5: Statistics for ASNQ, WikiQA and TREC-QA datasets.
Split	# Claims	# Evidences	Avg. # E/C
Train	145,406	722,473	4.97
Dev	19,998	98,915	4.95
Test	19,998	98,839	4.94
Table 6: Statistics for the FEVER dataset where evidences has been retrieved using (Liu et al., 2020).
$^{2}$ https://dumps.wikimedia.org/enwiki/20211101/  
 $^{3}$ https://huggingface.co/datasets/bookcorpusopen  
 $^{4}$ https://commoncrawl.org/2016/10/news-dataset-available/  
 $^{5}$ https://github.com/tensorflow/models/tree/archive/research/lm_commonsense#1-download-data-files  
 $^{6}$ https://github.com/alexa/wqa_tanda
$^{7}$ https://github.com/alexa/  
wqa-cascade-transformers  
 $^{8}$ http://aka.ms/WikiQA  
 $^{9}$ https://github.com/thunlp/KernelGAT/tree/master/data
B.1 Complete Experimental Details
Following standard practice, the token ids, positional ids and token type ids are embedded using separate embedding layers, and their sum is fed as the input to the transformer layers. We use  $k = 5$  for our experiments (following Zhang et al.; Tymoshenko and Moschitti), and perform continuous pre-training starting from the RoBERTa-Base checkpoint using a combination of MLM and our MSPP pre-training objective for 100,000 steps with a batch size of 4096. We use a triangular learning rate with 10,000 warmup steps and a peak value of  $5 * 10^{-5}$ . We use Adam optimizer with  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$  and  $\epsilon = 10^{-8}$ . We apply a weight decay of 0.01 and gradient clipping when values are higher than 1.0. We set the dropout ratio to 0.1 and we use two different prediction heads for pre-training:  $\mathrm{IE}_k$  and  $\mathrm{AE}_k$ . We follow the strategy of (Devlin et al., 2019; Lan et al., 2020), and equally weight the the two pre-training loss objectives: MLM and MSPP.
For evaluation, we fine-tune all models on the downstream AS2 and FEVER datasets: using the same  $\mathrm{IE}_k$  and  $\mathrm{AE}_k$  prediction heads exploited in pre-training for AS2 and using either  $\mathrm{IE}_1$  or  $\mathrm{AE}_1$  prediction heads for FEVER. We finetune every model with the same maximum sequence length equal to  $64 * (k + 1) = 384$  tokens. For ASNQ we train for up to 6 epochs with a batch size of 512 and a learning rate of  $10^{-5}$  with the same Adam optimizer described above but warming up for only 5000 steps. We do early stopping on the MAP of the development set. For WikiQA and TREC-QA, we created batches of 32 examples and we used a learning equal to  $2 * 10^{-6}$  and 1000 warm up steps. We train for up to 40 epochs again with early stopping on the MAP of the development set. On FEVER, we use a batch size of 64, a learning rate of  $10^{-5}$ , 1000 warm up steps and we do early stopping checking the Accuracy over the development set. We implemented our code based on HuggingFace's Transformers library (Wolf et al., 2020).
For AS2, we consider two baselines: (i) pairwise RoBERTa-Base model when used as a cross-encoder for AS2, and (ii) RoBERTa-Base LM when used as a joint model with  $\mathrm{IE}_k$  and  $\mathrm{AE}_k$  prediction heads independently for AS2 tasks.
For FEVER, we use several recent baselines
from Tymoshenko and Moschitti: (i) GEAR (Zhou et al., 2019), (ii) KGAT (Liu et al., 2020), (iii) Transformer-XH (Zhao et al., 2020), (iv) joint RoBERTa-Base with  $\mathrm{IE}_1$  prediction head (Tymoshenko and Moschitti, 2021), (v) pairwise RoBERTa-Base when used as a cross-encoder with max-pooling head (Tymoshenko and Moschitti, 2021), (vi) pairwise RoBERTa-Base when used as a cross-encoder with weighted-sum head (Tymoshenko and Moschitti, 2021).
We used metrics from Torchmetrics (Detlefsen et al., 2022) to compute MAP, MRR, Precision@1 and Accuracy.
B.3 Metrics
The performance of AS2 systems in practical applications is typically (Garg and Moschitti, 2021) measured using the Accuracy in providing correct answers for the questions (the percentage of correct responses provided by the system), also called the Precision-at-1  $(\mathrm{P}@\mathrm{1})$ . In addition to  $\mathrm{P}@\mathrm{1}$ , we use Mean Average Precision (MAP) and Mean Reciprocal Recall (MRR) to evaluate the ranking produced of the set of candidates by the model.
For FEVER, we measure the performance using Label Accuracy (LA), a standard metric for this dataset, that measures the accuracy of predicting support/refute/neither for a claim using a set of evidences.
C Complete Results and Discussion
C.1 Results on AS2 with cascaded pairwise and Joint re-ranker
Below we present results of evaluating our joint models to re-rank the top- $k$  candidates ranked by the pairwise RoBERTa-Base cross-encoder. Our joint models can significantly improve the  $\mathrm{P}@\mathrm{1}$ , MAP and MRR over the baseline for all datasets. The performance gap stems from questions for which the pairwise RoBERTa model was unable to rank the correct answer at the top position, but support from other candidates in the top- $\mathrm{k}$  helped the joint model rank it in the top position.
C.2 Results on FEVER
Here we present complete results on the FEVER dataset in Table 8, by also presenting some additional baselines such as: (i) pairwise BERT-Base cross-encoder (Tymoshenko and Moschitti, 2021), (ii) joint BERT-Base cross-encoder with  $\mathrm{IE}_1$  prediction head, (iii) DOMLIN++ (Stammbach and
Model	ASNQ	WikiQA	TREC-QA
P@1	MAP	MRR	P@1	MAP	MRR	P@1	MAP	MRR
Pairwise RoBERTa-Base	61.8	66.9	73.1	77.1	85.3	86.5	87.9	89.3	93.1
Joint RoBERTa-Base → FT IEk	25.2	44.0	45.6	24.6	49.3	49.7	57.6	73.7	74.6
Joint RoBERTa-Base → FT AEk	25.4	44.8	46.2	26.4	50.6	51.1	60.9	74.6	76.7
(Ours) Joint MSPP IEk → FT IEk	63.9	71.3	73.1	82.7	88.5	89.0	92.2	93.5	95.4
(Ours) Joint MSPP AEk → FT AEk	64.3	71.5	73.4	82.1	87.9	88.7	91.2	93.5	94.9
Table 7: Complete results of our joint models for AS2 datasets when re-ranking the answer candidates ranked in top-k by Pairwise RoBERTa-Base. MSPP, FT refer to our pre-training task and finetuning respectively. We indicate the prediction head  $(\mathrm{IE}_k / \mathrm{AE}_k)$  used for both pre-training and finetuning.
Model	Dev	Test
GEAR	70.69	71.60
KGAT with RoBERTa-Base	78.29	74.07
Transformer-XH	78.05	72.39
Pairwise BERT-Base	73.30	69.75
Pairwise RoBERTa-Base + MaxPool	79.82	-
Pairwise RoBERTa-Base + WgtSum	80.01	-
Joint BERT-Base	73.67	71.01
Joint RoBERTa-Base + FT IE1	79.25	73.56
(Ours) Joint Pre IEk + FT IE1	81.21 (0.24)	74.39
(Ours) Joint Pre IEk + FT AE1	81.10 (0.15)	74.25
(Ours) Joint Pre AEk + FT IE1	81.18 (0.14)	73.77
(Ours) Joint Pre AEk + FT AE1	81.21 (0.16)	74.13
Methods with larger models and/or sophisticated retrieval
DOMLIN++	77.48	76.60
DREAM	79.16	76.85
Table 8: Complete Results on FEVER dev and test sets. For our method, prediction heads  $\mathrm{(IE_1 / AE_1)}$  are only used for finetuning (FT), while for pre-training (Pre) we use the  $\mathrm{(IE_k / AE_k)}$  heads. ’-’ denotes models that are not publicly released and have no reported results on the test split in their published paper. Statistically significant results (T-Test  $95\%$  ) are underlined.
Ash, 2020) which uses additional DocIR components and data (MNLI (Williams et al., 2018)) for fine-tuning, (iv) DREAM (Zhong et al., 2020) that uses the XL-Net model. Note that comparing our joint models with (iii) and (iv) is unfair since they use additional retrieval components, datasets and larger models. We just include these results here for the sake for completeness. Interestingly, our joint models outperform DREAM and DOMLIN++ on the dev set without using additional retrieval and larger models.
C.3 Compute Overhead of Joint Models
Change in Number of Model Parameters: The transformer block of our joint inference model is identical to pre-trained models such as RoBERTa, and contains the exact same number of parameters. Classification heads  $IE_{1}, IE_{k}$  and  $AE_{1}$  all operate on the embedding of a single token, and are identical to the classification head of RoBERTa ( $AE_{k}$  operates on the concatenation of two token embeddings, and contains double the number of
parameters as the RoBERTa). The maximum sequence length allowed for both the models is the same (512). The exact number of parameters of our joint model with  $AE_{k}$  and the RoBERTa model are 124, 062, 720 and 124, 055, 040 respectively.
Change in Inference Latency: While our joint model provides a longer input sequence to the transformer, it also reduces the number of forward passes that need to be done by a pairwise cross-encoder. A simplified latency analysis for AS2 (assuming each sentence has a length  $L$ ): pairwise cross-encoder will need to make  $k$  forward passes of the transformer with a sequence of length  $2L$  ( $q$  with each candidate  $c_i$ ), while our joint model will only need to make 1 forward pass of the transformer with input length  $(k + 1) \times L$  ( $q$  with  $k$  candidates). Transformer self-attention is quadratic in input sequence length, so this should lead to the inference time of out joint model being  $\frac{(k + 1)^2}{4k}$  times the inference time of the cross-encoder. However, the input embedding layer and the feedforward layers are linear in input sequence length, so this should lead to a reduction in the inference time of our joint model by  $\frac{(k + 1)}{2k}$  times the inference time of the cross-encoder. Empirically, when we fine-tune for WikiQA on one A100-GPU, we only observe latency increasing from  $71s \rightarrow 81s$  (increase of only  $14.1\%$ ).