SlowGuess/NAACL_2022_OCR huggingface dataset:
Luca Di Liello $^{1*}$ , Siddhant Garg $^{2}$ , Luca Soldaini $^{3\dagger}$ , Alessandro Moschitti $^{2}$
<sup>1</sup>University of Trento, <sup>2</sup>Amazon Alexa AI, <sup>3</sup>Allen Institute for AI
luca.diliello@unitn.it
{sidgarg,amosch}@amazon.com
lucas@allenai.org
Inference tasks such as answer sentence selection (AS2) or fact verification are typically solved by fine-tuning transformer-based models as individual sentence-pair classifiers. Recent studies show that these tasks benefit from modeling dependencies across multiple candidate sentences jointly. In this paper, we first show that popular pre-trained transformers perform poorly when used for fine-tuning on multi-candidate inference tasks. We then propose a new pre-training objective that models the paragraph-level semantics across multiple input sentences. Our evaluation on three AS2 and one fact verification datasets demonstrates the superiority of our pre-training technique over the traditional ones for transformers used as joint models for multi-candidate inference tasks, as well as when used as cross-encoders for sentence-pair formulations of these tasks.
Pre-trained transformers (Devlin et al., 2019; Liu et al., 2019; Clark et al., 2020) have become the de facto standard for several NLP applications, by means of fine-tuning on downstream data. The most popular architecture uses self-attention mechanisms for modeling long range dependencies between compounds in the text, to produce deep contextualized representations of the input. There are several downstream NLP applications that require reasoning across multiple inputs candidates jointly towards prediction. Some popular examples include (i) Answer Sentence Selection (AS2) (Garg et al., 2020), which is a Question Answering (QA) task that requires selecting the best answer from a set of candidates for a question; and (ii) Fact Verification (Thorne et al., 2018), which reasons whether a claim is supported/refuted by multiple evidences. Inherently, these tasks can utilize information from multiple candidates (answers/evidences) to support the prediction of a particular candidate.
Pre-trained transformers such as BERT are used for these tasks as cross-encoders by setting them as sentence-pair classification problems, i.e., aggregating inferences independently over each candidate. Recent studies (Zhang et al., 2021; Tymoshenko and Moschitti, 2021) have shown that these tasks benefit from encoding multiple candidates together, e.g., encoding five answer candidates per question in the transformer, so that the cross-attention can model dependencies between them. However, Zhang et al. only improved over the pairwise cross-encoder by aggregating multiple pairwise cross-encoders together (one for each candidate), and not by jointly encoding all candidates together in a single model.
In this paper, we first show that popular pretrained transformers such as RoBERTa perform poorly when used for jointly modeling inference tasks (e.g., AS2) using multi-candidates. We show that this is due to a shortcoming of their pre-training objectives, being unable to capture meaningful dependencies among multiple candidates for the fine-tuning task. To improve this aspect, we propose a new pre-training objective for 'joint' transformer models, which captures paragraph-level semantics across multiple input sentences. Specifically, given a target sentence  $s$  and multiple sentences (from the same/different paragraph/document), the model needs to recognize which sentences belong to the same paragraph as  $s$  in the document used.
Joint inference over multiple-candidates entails modeling interrelated information between multiple short sentences, possibly from different paragraphs or documents. This differs from related works (Beltagy et al., 2020; Zaheer et al., 2020; Xiao et al., 2021) that reduce the asymptotic complexity of transformer attention to model long contiguous inputs (documents) to get longer context for tasks such as machine reading and summarization.
We evaluate our pre-trained multiple-candidate based joint models by (i) performing AS2 on
*Work done as an intern at Amazon Alexa AI
†Work completed at Amazon Alexa AI
Human Language Technologies, pages 2521 - 2531
ASNQ (Garg et al., 2020), WikiQA (Yang et al., 2015), TREC-QA (Wang et al., 2007) datasets; and (ii) Fact Verification on the FEVER (Thorne et al., 2018) dataset. We show that our pre-trained joint models substantially improve over the performance of transformers such as RoBERTa being used as joint models for multi-candidate inference tasks, as well as when being used as cross-encoders for sentence-pair formulations of these tasks.
Multi-Sentence Inference: Inference over a set of multiple candidates has been studied in the past (Bian et al., 2017; Ai et al., 2018). The most relevant for AS2 are the works of Bonadiman and Moschitti (2020) and Zhang et al. (2021), the former improving over older neural networks but failing to beat the performance of transformers; the latter using task-specific models (answer support classifiers) on top of the transformer for performance improvements. For fact verification, Tymoshenko and Moschitti (2021) propose jointly embedding multiple evidence with the claim towards improving the performance of baseline pairwise cross-encoder transformers.
Transformer pre-training Objectives: Masked Language Modeling (MLM) is a popular transformer pre-training objective (Devlin et al., 2019; Liu et al., 2019). Other models are trained using token-level (Clark et al., 2020; Joshi et al., 2020; Yang et al., 2019; Liello et al., 2021) and/or sentence-level (Devlin et al., 2019; Lan et al., 2020; Wang et al., 2020) objectives. REALM (Guu et al., 2020) uses a differentiable neural retriever over Wikipedia to improve MLM pre-training. This differs from our pre-training setting as it uses additional knowledge to improve the pre-trained LM. DeCLUTR (Giorgi et al., 2021) uses a contrastive learning objective for cross-encoding two sentences coming from the same/different documents in a transformer. DeCLUTR is evaluated for sentence-pair classification tasks and embeds the two inputs independently without any cross-attention, which differs from our setting of embedding multiple candidates jointly for inference.
Modeling Longer Sequences: Beltagy et al. (2020); Zaheer et al. (2020) reduce the asymptotic complexity of transformer attention to model very long inputs for longer context. For tasks with short sequence lengths, LongFormer works on par or slightly worse than RoBERTa (attributed to re
Figure 1: Multi-sentence 'Joint' transformer model.  $E_{i}$  refers to embedding for the question/each candidate.
duced attention computation). These works encode a single contiguous long piece of text, which differs from our setting of having multiple short candidates, for a topic/query, possibly from different paragraphs and documents. DCS (Ginzburg et al., 2021) proposes a cross-encoder for the task of document-pair matching. DCS is related to our work as it uses a contrastive pre-training objective over two sentences extracted from the same paragraph, however different from our joint encoding of multiple sentences, DCS individually encodes the two sentences and then uses the InfoNCE loss over the embeddings. CDLM (Caciularu et al., 2021) specializes the Longformer for document-pair matching and cross-document coreference resolution. While the pre-training objective in CDLM exploits information from multiple documents, it differs from our setting of joint inference over multiple short sentences.
3 Multi-Sentence Transformers Models
3.1 Multi-sentence Inference Tasks
AS2: We denote the question by  $q$ , and the set of answer candidates by  $C = \{c_{1},\dots c_{n}\}$ . The objective is to re-rank  $C$  and find the best answer  $A$  for q. AS2 is typically treated as a binary classification task: first, a model  $f$  is trained to predict the correctness/incorrectness of each  $c_{i}$ ; then, the candidate with the highest likelihood of being correct is selected as an answer, i.e.,  $A = \operatorname{argmax}_{i=1}^{n} f(c_{i})$ . Intuitively, modeling interrelated information between multiple  $c_{i}$ 's can help in selecting the best answer candidate (Zhang et al., 2021).
Fact Verification: We denote the claim by  $F$ , and the set of evidences by  $C = \{c_1 \ldots c_n\}$  that are retrieved using DocIR. The objective is to predict whether  $F$  is supported/refuted/neither using  $C$  (at least one evidence  $c_i$  is required for supporting/refuting  $F$ ). Tymoshenko and Moschitti (2021) jointly model evidences for supporting/refuting a claim as they can complement each other.
3.2 Joint Encoder Architecture
For jointly modeling multi-sentence inference tasks, we use a monolithic transformer cross-encoder to encode multiple sentences using self-attention as shown in Fig 1. To perform joint inference over  $k$  sentences for question  $q$  or claim  $F$ , the model receives concatenated sentences  $[s_0 \ldots s_k]$  as input, where the first sentence is either the question or the claim  $(s_0 = q$  or  $s_0 = F)$ , and the remainder are  $k$  candidates  $s_i = c_i$ ,  $i = \{1 \ldots k\}$ . We pad (or truncate) each sentence  $s_i$  to the same fixed length  $L$  (total input length  $L \times (k + 1)$ ), and use the embedding for the [CLS] / [SEP] token in front of each sentence  $s_i$  as its embedding (denoted by  $E_i$ ). Similar to Devlin et al., we create positional embeddings of tokens using integers 0 to  $L(k + 1) - 1$ , and extend the token type ids from  $\{0, 1\}$  to  $\{0 \ldots k\}$  corresponding to  $(k + 1)$  input sentences.
3.3 Inference using Joint Transformer Model
We use the output embeddings  $[E_0\ldots E_k]$  of sentences for performing prediction as following:
Predicting a single label: We use two separate classification heads to predict a single label for the input to the joint model  $[s_0\ldots s_k]$ : (i)  $\mathbf{IE}_1$ : a linear layer on the output embedding  $E_0$  of  $s_0$  (similar to BERT) referred to as the Individual Evidence  $(\mathrm{IE}_1)$  inference head, and (ii)  $\mathbf{AE}_1$ : a linear layer on the average of the output embeddings  $[E_0,E_1,\dots ,E_k]$  to explicitly factor in information from all candidates, referred to as the Aggregated Evidence  $(\mathrm{AE}_1)$  inference head. For Fact Verification, we use prediction heads  $\mathrm{IE}_1$  and  $\mathrm{AE}_1$ .
Predicting Multiple Labels: We use two separate classification heads to predict  $k$  labels, one label each for every input  $[s_1 \ldots s_k]$  specific to  $s_0$ : (i)  $\mathbf{IE}_k$ : a shared linear layer applied to the output embedding  $E_i$  of each candidate  $s_i$ ,  $i \in \{1 \ldots k\}$  referred to as  $k$ -candidate Individual Evidence  $(\mathrm{IE}_k)$  inference head, and (ii)  $\mathbf{AE}_k$ : a shared linear layer applied to the concatenation of output embedding  $E_0$  of input  $s_0$  and the output embedding  $E_i$  of each candidate  $s_i$ ,  $i \in \{1 \ldots k\}$  referred to as  $k$ -candidate Aggregated Evidence  $(\mathrm{AE}_k)$  inference head. For AS2, we use prediction heads  $\mathrm{IE}_k$  and  $\mathrm{AE}_k$ . Prediction heads are illustrated in Figure 2.
3.4 Pre-training with Paragraph-level Signals
Long documents are typically organized into paragraphs to address the document's general topic from different viewpoints. The majority of trans
Figure 2: Inference heads for joint transformer model.  $E_{i}$  refers to embedding for the question/each candidate.
former pre-training strategies have not exploited this rich source of information, which can possibly provide some weak supervision to the otherwise unsupervised pre-training phase. To enable joint transformer models to effectively capture dependencies across multiple sentences, we design a new pre-training task where the model is (i) provided with  $(k + 1)$  sentences  $\{s_0\ldots s_k\}$ , and (ii) tasked to predict which sentences from  $\{s_1\ldots s_k\}$  belong to the same paragraph  $P$  as  $s_0$  in the document  $D$ . We call this pre-training task Multi-Sentences in Paragraph Prediction (MSPP). We use the  $\mathrm{IE}_k$  and  $\mathrm{AE}_k$  prediction heads, defined above, on top of the joint model to make  $k$  predictions  $p_i$  corresponding to whether each sentence  $s_i$ ,  $i\in \{1\ldots k\}$  lies in the same paragraph  $P\in D$  as  $s_0$ . More formally:
$$
p _ {i} = \left\{ \begin{array}{l l} 1 \text {i f} s _ {0}, s _ {i} \in P \text {i n} D \\ 0 \text {o t h e r w i s e} \end{array} \right. \quad \forall i = \{1, \ldots , k \}
$$
We randomly sample a sentence from a paragraph  $P$  in a document  $D$  to be used as  $s_0$ , and then (i) randomly sample  $k_{1}$  sentences (other than  $s_0$ ) from  $P$  as positives, (ii) randomly sample  $k_{2}$  sentences from paragraphs other than  $P$  in the same document  $D$  as hard negatives, and (iii) randomly sample  $k_{3}$  sentences from documents other than  $D$  as easy negatives (note that  $k_{1} + k_{2} + k_{3} = k$ ).
We evaluate our joint transformers on three AS2 and one Fact Verification datasets<sup>1</sup>. Common LM benchmarks, such as GLUE (Wang et al., 2018), are not suitable for our study as they only involve sentence pair classification.
Pre-training: To eliminate any improvements stemming from usage of more data, we perform pre-training on the same corpora as RoBERTa: English Wikipedia, the BookCorpus, OpenWebText and CC-News. For our proposed pre-training, we randomly sample sentences from paragraphs as  $s_0$ ,
<sup>1</sup>We will release the code and all pre-trained model checkpoints at https://github.com/alexa/wqa-multi-sentence-inference
Model	ASNQ	WikiQA	TREC-QA
P@1	MAP	MRR	P@1	MAP	MRR	P@1	MAP	MRR
Pairwise RoBERTa-Base	61.8 (0.2)	66.9 (0.1)	73.1 (0.1)	77.1 (2.1)	85.3 (0.9)	86.5 (1.0)	87.9 (2.2)	89.3 (0.9)	93.1 (1.0)
Joint RoBERTa-Base → FT IEk	3.4 (2.3)	8.0 (1.9)	10.0 (2.4)	19.7 (1.9)	39.4 (1.6)	40.3 (1.8)	30.9 (5.4)	41.9 (2.4)	50.8 (3.9)
Joint RoBERTa-Base → FT AEk	3.6 (2.7)	8.0 (2.2)	10.2 (2.8)	18.7 (3.9)	39.0 (2.8)	39.7 (2.9)	29.7 (6.9)	42.3 (3.2)	49.2 (5.0)
(Ours) Joint MSPP IEk → FT IEk	63.0 (0.3)	67.2 (0.2)	73.7 (0.2)	82.7 (2.2)	88.5 (1.5)	89.0 (1.5)	91.7 (2.2)	91.1 (0.5)	95.2 (1.3)
(Ours) Joint MSPP AEk → FT AEk	63.0 (0.3)	67.3 (0.2)	73.7 (0.2)	81.9 (2.6)	87.9 (1.4)	89.0 (1.5)	88.7 (0.8)	90.1 (1.0)	93.6 (0.6)
and choose  $k_{1} = 1, k_{2} = 2, k_{3} = 2$  as the specific values for creating positive and negative candidates for  $s_0$ . For complete details refer to Appendix A.
Fine-tuning: For AS2, we compare performance with MAP, MRR and Precision of top ranked answer  $(\mathrm{P}@\mathrm{l})$  .For fact verification, we measure Label Accuracy (LA). Brief description of datasets is presented below (details in Appendix A):
- ASNQ: A large AS2 dataset (Garg et al., 2020) derived from NQ (Kwiatkowski et al., 2019), where the candidate answers are from Wikipedia pages and the questions are from search queries of the Google search engine. We use the dev. and test splits released by Soldaini and Moschitti.
- WikiQA: An AS2 dataset (Yang et al., 2015) where the questions are derived from query logs of the Bing search engine, and the answer candidate are extracted from Wikipedia. We use the most popular clean setting (questions having at least one positive and one negative answer).
- TREC-QA: A popular AS2 dataset (Wang et al., 2007) containing factoid questions. We only retain questions with at least one positive and one negative answer in the development and test sets.
- FEVER: A dataset for fact extraction and verification (Thorne et al., 2018) to retrieve evidences given a claim and identify if the evidences support/refute the claim. As we are interested in the fact verification sub-task, we use evidences retrieved by Liu et al. using a BERT-based DocIR.
4.2 Experimental Details and Baselines
We use  $k = 5$  for our experiments (following (Zhang et al., 2021) and (Tymoshenko and Moschitti, 2021)), and perform continued pre-training starting from RoBERTa-Base using a combination of MLM and our MSPP pre-training for 100k steps with a batch size of 4,096. We use two different prediction heads,  $\mathrm{IE}_k$  and  $\mathrm{AE}_k$ , for pre-training. For evaluation, we fine-tune all models on the downstream AS2 and FEVER datasets using the corre
Table 1: Results (std. dev. in parenthesis) on AS2 datasets. MSPP, FT refer to our pre-training task and fine-tuning respectively. We indicate the prediction head  $(\mathrm{IE}_k / \mathrm{AE}_k)$  used for both pre-training and fine-tuning. We underline statistically significant gains over the baseline (Student  $t$  -test with  $95\%$  confidence level).
Model	ASNQ	WikiQA	TREC-QA
Pairwise RoBERTa-Base	61.8 (0.2)	77.1 (2.1)	87.9 (2.2)
Joint RoBERTa-Base → FT IEk	25.2 (3.1)	24.6 (3.1)	57.6 (4.8)
Joint RoBERTa-Base → FT AEk	25.4 (3.3)	26.4 (2.2)	60.9 (4.9)
(Ours) Joint MSPP IEk → FT IEk	63.9 (0.8)	82.7 (3.0)	92.2 (0.8)
(Ours) Joint MSPP AEk → FT AEk	64.3 (1.1)	82.1 (1.1)	91.2 (2.9)
Table 2: P@1 of joint models for AS2 when re-ranking answers ranked in top-5 by pairwise RoBERTa-Base. Statistically significant results (Student t-test  $95\%$ ) are underlined. Complete results in Appendix C.
sponding  $\mathrm{IE}_k$  and  $\mathrm{AE}_k$  prediction heads. We consider the pairwise RoBERTa-Base cross-encoder and RoBERTa-Base LM used as a joint model with  $\mathrm{IE}_k$  and  $\mathrm{AE}_k$  prediction heads as the baseline for AS2 tasks. For FEVER, we use several baselines: GEAR (Zhou et al., 2019), KGAT (Liu et al., 2020), Transformer-XH (Zhao et al., 2020), and three models from (Tymoshenko and Moschitti, 2021): (i) Joint RoBERTa-Base with  $\mathrm{IE}_1$  prediction head, (ii) Pairwise RoBERTa-Base with max-pooling, and (iii) weighted-sum heads. For complete experimental details, refer to Appendix B.
Answer Sentence Selection: The results for AS2 tasks are presented in Table 1, averaged across five independent runs. From the table, we can see that the RoBERTa-Base when used as a joint model for multi-candidate inference using either the  $\mathrm{IE}_k$  or  $\mathrm{AE}_k$  prediction heads performs inferior to RoBERTa-Base used as a pairwise cross-encoder. Across five experimental runs, we observe that finetuning RoBERTa-Base as a joint model faces convergence issues (across various hyper-parameters) indicating that the MLM pre-training task is not sufficient to learn text semantics which can be exploited for multi-sentence inference.
Our MSPP pre-trained joint models (with both  $\mathrm{IE}_k$ ,  $\mathrm{AE}_k$  heads) get significant improvements over the pairwise cross-encoder baseline and very large improvements over the RoBERTa-Base joint model. The former highlights modeling improvements stemming from joint inference over multiple
Model	Dev	Test
GEAR	70.69	71.60
KGAT with RoBERTa-Base	78.29	74.07
Transformer-XH	78.05	72.39
Pairwise RoBERTa-Base + MaxPool	79.82	-
Pairwise RoBERTa-Base + WgtSum	80.01	-
Joint RoBERTa-Base + FT IE1	79.25	73.56
(Ours) Joint Pre IEk + FT IE1	81.21 (0.24)	74.39
(Ours) Joint Pre IEk + FT AE1	81.10 (0.15)	74.25
(Ours) Joint Pre AEk + FT IE1	81.18 (0.14)	73.77
(Ours) Joint Pre AEk + FT AE1	81.21 (0.16)	74.13
Table 3: Results on FEVER dev and test sets. For our method, prediction heads  $(\mathrm{IE}_1 / \mathrm{AE}_1)$  are only used for fine-tuning (FT), while for pre-training (Pre) we use  $(\mathrm{IE}_k / \mathrm{AE}_k)$  heads. \* denotes models not released publicly, and results not reported in the paper. Statistically significant results (Student  $t$  -test  $95\%$  ) are underlined.
candidates, while the latter highlights improvements stemming from our MSPP pre-training strategy. Across all three AS2 datasets, our joint models are able to get the highest P@1 scores while also improving the MAP and MRR metrics.
To demonstrate that our joint models can effectively use information from multiple candidates towards prediction, we perform a study in Table 2 where the joint models are used to re-rank the top- $k$  candidates ranked by the pairwise RoBERTa-Base cross-encoder. Our joint models can significantly improve the  $\mathrm{P@1}$  over the baseline for all datasets. The performance gap stems from questions for which the pairwise RoBERTa model was unable to rank the correct answer at the top position, but support from other candidates in the top- $k$  helped the joint model rank it in the top position.
Fact Verification: The results for the FEVER task are presented in Table 3 and show that our joint models (pre-trained with both the  $\mathrm{IE}_k$  and  $\mathrm{AE}_k$  heads and fine-tuned with the  $\mathrm{IE}_1$  and  $\mathrm{AE}_1$  heads) outperform all previous baselines considered, including the RoBERTa-Base joint model directly applied for multi-sentence inference.
Compute Overhead: We present a simplified latency analysis for AS2 (assuming sentence length  $L$ ) as follows: a pairwise cross-encoder uses  $k$  transformer steps with input length  $2L$ , while our model uses 1 step with input length  $(k + 1)\times L$ . Since transformer attention scales quadratic on input length, our model should take  $\frac{(k + 1)^2}{4k}$  times the inference time of the cross-encoder, which is 1.8 when  $k = 5$ . However, when we fine-tune for WikiiQA on one A100-GPU, we only observe latency increasing from  $71s\rightarrow 81s$  (only  $14.1\%$  increase). The input embeddings and feedforward layers vary
ASNQ
Q: Who invented the submarine during the civil war?
A1: H.L. Hunley, often referred to as Hunley, was a submarine of the Confeder
A2: Hunley, McClintock, and Baxter Watson first built Pioneer, which was tested in February 1862 in the Mississippi River and was later towed to Lake Pontchartrain for additional trials.
A3: She was named for her inventor, Horace Lawson Hunley, shortly after she was taken into government service under the control of the Confederate States Army at Charleston, South Carolina.
A4: 1864 painting of H.L. Hunley by Conrad Wise Chapman History Confederate States Name : H.L. Hunley Namesake : Horace Lawson Hunley Builder : James McClintock Laid down : Early 1863 Launched : July 1863 Acquired : August 1863 In service : February 17, 1864 Out of service : February 17, 1864 Status : Awaiting conservation General characteristics Displacement : 7.5 short tons ( 6.8 metric tons ) Length : 39.5 ft  
A5: Johan F. Carlsen was born in Årskøbing April 9, 1841.
WikiQA
Q: What is the erb/heart?
A1: Heart valves are labeled with "B", "T", "A", and "P". First heart sound: caused by atrioventricular valves - Bicuspid/Mitral (B) and Tricuspid (T).
A2: Second heart sound caused by semilunar valves - Aortic (A) and Pulmonary/ Pulmonic (P).
A3: Front of thorax, showing surface relations of bones, lungs (purple), pleura (blue), and heart (red outline).
A4: In cardiology, Erb's point refers to the third intercostal space on the left sternal border where sS2 is best auscultated.
A5: It is essentially the same location as what is referred to with left lower sternal border (LLSB).
TREC-QA
Q: When was the Khmer Rouge removed from power?
A1: Sihanouk was named head of state after the Khmer Rouge seized power in 1975, but was locked in his palace by the communists as they embarked on their brutal attempt to create an agrarian utopia.
A2: When a Vietnamese invasion drove the Khmer Rouge from power in 1979, Duch fled with other Khmer Rouge leaders into the jungles.
A3: Religious practices were revived after the Khmer Rouge were driven from power by a Vietnamese invasion in 1979
A4: Moreover, 20 years after the Khmers Rougees were ousted from power, Cambodia still struggles on the brink of chaos , ruled by the gun , not by law .
A5: Sihanouk resigned in 1976, but the Khmer Rouge kept him under house arrest until they were driven from power by an invading Vietnamese army in 1979.
Table 4: Examples from AS2 datasets where the pairwise RoBERTa-Base model is unable to rank a correct answer for the question at the top position, but our joint model (Joint MSPP  $\mathrm{IE}_k\rightarrow \mathrm{FT}\mathrm{IE}_k$ ) can. We present answers  $\{A1,\ldots ,A5\}$  in their ranked order by the pairwise RoBERTa-Base model. For all these examples we highlight the top ranked answer by the pairwise RoBERTa-Base model in red since it is incorrect.