SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3 shows that PMR significantly surpasses RoBERTa on all six OOD datasets (+4.5 F1 on average), although they have similar in-domain performance on SQuAD. This finding verifies that our PMR with MRC-style pre-training can help capture QA patterns that are more generalizable to unseen domains. In addition, PMR with less than half the parameters of T5-v1.1, achieves a better generalization capability.
Full-resource Results. Although using the full-resource training data can alleviate the pretraining-finetuning discrepancy, MRC-style continual pre-training still delivers reasonable performance gains. As shown in Table 4, PMR achieves 0.9 and 1.5 F1 improvements over RoBERTa on EQA and NER, respectively. Further analysis shows that PMR can do better at comprehending the input text (Appendix 5.3). We also explore the upper limits of PMR by employing a larger and stronger MLM, i.e.  $\mathrm{ALBERT}_{\mathrm{xxlarge}}$  [27], as the backbone. The results show additional improvements of our PMR over  $\mathrm{ALBERT}_{\mathrm{xxlarge}}$  on EQA (Appendix A.6).
Table 3: Performance on OOD EQA. We used the SQuAD training data to train the models and evaluate them on MRQA OOD dev sets.
Model	Size	Unified	EQA	NER
RBT-Post	355M	×	81.9	79.8
SpanBERT	336M	×	81.7	77.3
T5-v1.1	800M	✓	82.0	76.0
UIE	800M	✓	-	79.6
RoBERTa	355M	×	84.0	80.8
PMR	355M	✓	84.9	82.3
Table 4: Full-resource results on EQA and NER. For EQA, we reported the average F1 score on six MRQA in-domain dev sets. For NER, we used four datasets.
Size	RACE	DREAM	MCTest	SST-2	MNLI	Avg.
T5-v1.1	800M	81.7	75.7	86.7	96.7	90.4	86.2
RBT-Post	355M	80.6	81.3	86.9	96.1	89.7	86.9
RoBERTa	355M	83.2	84.2	89.3	96.4	90.1	88.6
PMR	355M	82.8	83.8	92.3	96.5	89.9	89.1
Table 5: Full-resource performance (Acc.) on sequence classification tasks.
Input and extracted rationales	Label
It's all pretty cynical and condescending, too.	Negative
Perhaps the heaviest, most joyless movie ever made about giant dragons taking over the world.	Negative
This is the best American movie about troubled teens since 1998's whatever.	Positive
An experience so engrossing it is like being buried in a new environment.	Positive
Table 6: Case study on SST-2. PMR can additionally extract token-level rationales (in red and bold) to support the sequence classification. A total of 224 rationales out of 300 (74%) are reasonable.
5.1 Explainable Sequence Classification
Settings. The relevance classification between the query and the context in our WAE objective can be inherited to tackle downstream sequence classification problems, another important topic of NLU. To demonstrate, we considered two major sequence classification tasks in our MRC formulation. The first is sequence classification with pre-defined task labels, such as sentiment analysis. Each task label is used as a query for the input text (i.e. the context in  $\mathrm{PMR})^4$ ; The second is sequence classification with natural questions on multiple choices, such as multi-choice QA (MCQA). We treated the concatenation of the question and one choice as the query for the given passage (i.e., the context). In the output space, these problems are tackled by conducting relevance classification on  $S_{1,1}$  (extracting [CLS] if relevant). Examples are included in Appendix A.2. We tested PMR on (1) three MCQA tasks: DREAM [53], RACE [26], and MCTest [47]; (2) a sentence-pair classification task: MNLI [61]; and (3) a sentence classification task: SST-2 [52].
Results. Table 5 presents the results of PMR and previous strong baselines on the above-mentioned tasks. Compared with RoBERTa, PMR exhibits comparable results on all tasks and achieved slightly better average performance. This indicates that solving sequence classification in an MRC manner is feasible and effective.
Explainability. In sequence classification, we restricted the extraction space to the [CLS] token and used the extraction probability of this token to determine which class label (i.e. the query) the input text (i.e. the context) corresponds to. Note that while predicting the extraction probability of [CLS], PMR also calculates the extraction scores of context spans, just as done for the span extraction tasks. Therefore, we leveraged the fine-tuned PMR to additionally extract a span with the highest  $S_{i,j}$  score from the input of SST-2, as exemplified in Table 6. The extracted spans are clear rationales that support the sequence-level prediction (i.e., the overall sentiment of the input sentence). To verify this, we conducted a quantitative analysis by randomly checking 300 test instances. The results show that approximately  $74\%$  of the extracted spans are reasonable rationales for sentiment prediction. These findings suggest that the fine-tuned PMR on SST-2 effectively preserves the conditional span extraction capability inherited from the MRC pre-training. Such a capability shows that the model captures the semantic interactions between relevant context spans and the label-encoded query (e.g., "Negative. Feeling bad"). In addition to the performance improvement, PMR provides high explainability for the classification results.
${}^{4}$  Sentence-pair classification is classified into this type,where the concatenation of two sentences denotes the context.
5.2 Unifying Extraction and Classification with PMR
In the previous sections, we demonstrate that various extraction and classification tasks can be separately tackled in the same MRC formulation. We further explore the potential that fine-tuning a unified model for solving multiple tasks of different types.
Settings. We use two datasets, one from CoNLL NER (of extraction type) and the other from DREAM (of classification type), to train a multi-task model. For evaluation, we conduct three groups of experiments, where the models are evaluated on (1) Held-in: testing sets from training tasks, (2) Held-out Datasets: testing sets from other tasks of the same type with training tasks, and (3) Held-out Tasks: testing sets from unseen tasks
Results. As shown in Figure 4, the held-in results show that the multi-task fine-tuned RoBERTa suffers from a significant performance drop on DREAM compared to the single-task fine-tuned RoBERTa. This indicates that multi-task learning is difficult for a discriminative model if the task head is not well-trained. In contrast, the multi-task PMR is on par with PMR on CoNLL and slightly underperforms PMR on DREAM. Such a finding suggests that the MRC-style pre-training enhances PMR's capability to learn extraction and classification patterns from downstream NLU tasks, enabling PMR to serve as a unified model for solving various NLU tasks. Though generative models like T5-v1.1 (800M) could
also unify NLU tasks through a conditional generation manner [44, 38], the overall performance, especially the held-out performance, is lower than the smaller-sized PMR (355M). This suggests that the discriminative PMR may be better than generative models at unifying NLU tasks.
Figure 4: The results of Held-in, Held-out Datasets, and Held-out Tasks evaluation. The six data points in the Held-in group denote the single-task fine-tuned performance of three models on two tasks respectively.
5.3 Better Comprehending capability
To verify that PMR can better comprehend the input text, we feed the models with five different query variants during CoNLL evaluation. The five variants are:
- Defaulted query:
"[Label]". [Label description]
- Query template (v1):
What is the "[Label]" entity, where [Label description]?
- Query template (v2):
Identify the spans (if any) related to "[Label]" entity. Details: [Label description]
- Paraphrasing label description with ChatGPT (v1):
"[Label]". [Paraphrased Label description v1]
- Paraphrasing label description with ChatGPT (v1):
"[Label]". [Paraphrased Label description v2]
In Figure 5, we show the statistic results of the three models on CoNLL when five different query templates are used respectively during evaluation. Among the models, PMR demonstrated significantly higher and more stable performance than RoBERTa and T5-v1.1. Such a finding verifies our assumption that PMR can effectively comprehend the latent semantics of the input text despite being rephrased with varying lexical usage from the default query used for fine-tuning models.
Gaps between Pre-training and Fine-tuning. The prevailing approaches tackle span extraction tasks in a discriminative manner with tailored task-specific classifiers (e.g. tagging or MRC head)
Figure 5: CoNLL performance when the models are fed with five different templates respectively during evaluation.
[19, 9, 12]. Recent research reported that generative fine-tuning methods can effectively bridge the gaps between pre-training and fine-tuning and achieve remarkable few-shot NLU performance by representing downstream NLU tasks as the same language model pre-training problem [49, 50, 34, 16, 35]. In the scope of span extraction, these works can be classified into three categories: generating label tokens based on the prompt span [10, 40], generating span tokens based on the prompt label [6, 38], and directly generating label-span pairs with soft prompt [8]. Another way to mitigate the gap is to adapt MLMs into an appropriate paradigm for solving span extraction. Typically, this can be achieved through the use of task-specific data from similar tasks, referred to as "pre-finetuning" [24, 1, 68]. However, these methods may not be effective in domain-specific tasks or for non-English languages due to the lack of labeled data. Several studies leveraged abundant raw text to construct MRC examples for retrofitting MLMs [30, 18, 46]. By contrast, PMR employs ubiquitous hyperlink information to construct MRC data, which guarantees highly precise MRC triplets. In addition, PMR also provides a unified model for both sequence classification and span extraction, thereby enabling strong explainability through the extraction of high-quality rationale phrases.
Hyperlinks for NLP. Hyperlinks are utilized in two ways. First, hyperlinks can be regarded as a type of relevance indicator in model pre-training [67], passage retrieval [7, 51], and multi-hop reasoning [2, 66]. Second, the anchors labeled by hyperlinks can serve as entity annotations for representation learning [64, 5]. PMR is the first one to combine the advantages of both scenarios. In this work, we paired MRC query and context based on the relevance of hyperlinks and automatically labeled the anchors as MRC answers.
This work presents a novel MRC-style pre-training model called PMR. PMR can fully resolve the learning objective and model architecture gaps that frequently appear in fine-tuning existing MLMs. Experimental results from multiple dimensions, including effectiveness in solving few-shot tasks and OOD generalization, show the benefits of bridging the gap between pre-training and fine-tuning for span extraction tasks. PMR also shows promising potential in explaining the sequence classification process and unifying NLU tasks.
[1] Armen Aghajanyan, Anchit Gupta, Akshit Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. Muppet: Massive multi-task representations with pre-finetuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5799-5811, November 2021.
[2] Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. Learning to retrieve reasoning paths over wikipedia graph for question answering. In International Conference on Learning Representations, 2020.
A.1 Limitations
Multilinguality Although constructing large-scale MRC-style training data is feasible for resource-rich languages, such as English, extending this idea to resource-poor languages might be difficult due to the relatively small amount of anchors in their corresponding Wikipedia articles. Exploring other data resources to automatically construct large-scale pre-training data can remedy this issue. For example, given a word in the monolingual dictionaries, we can regard the word itself, the definition of this word, and the example sentence of this word as the MRC answer, query, and context respectively. We believe our MRC-style pre-training is still applicable for low-resource languages with such dictionaries.
Comparison with Large Language Models In this paper, we did not compare PMR with large language models (LLM) for the following two reasons. First, existing MLMs are small in scale. Therefore, we are unable to find a suitable MLM to make a fair comparison with LLMs. Second, studies have shown that LLMs yield inferior results compared to smaller MLMs on span extraction tasks, particularly those involving structured prediction [41, 43, 60, 31]. Based on this fact, we mainly compare with existing strong generative methods of comparable model size.
Few-shot NER results of SpanBERT We ran SpanBERT [20] in our NER few-shot settings. However, its performance was below our expectations. In all our few-shot settings, SpanBERT achieved an F1 score of 0 on CoNLL and WNUT datasets. Additionally, its performance on ACE04 and ACE05 datasets was significantly lower than RoBERTa [36]. Based on these outcomes, we only compare PMR with SpanBERT in the NER full-resource setting.
A.2 Fine-tuning Tasks
For EQA, we use the MRQA benchmark [15], including SQuAD [45], TriviaQA [21], NaturalQuestion [25], NewQA [55], SearchQA [14], HotpotQA [66], BioASQ [56], DROP [13], DuoRC [48], RACE [26], RelationExtraction [28], TextbookQA [22]. EQA has always been treated as an MRC problem, where the question serves as the MRC query, and the passage containing the answers serves as the MRC context. For NER, We follow MRC-NER [32] to formulate NER into the MRC paradigm, where the entity label together with its description serves as the MRC query, and the input text serves as the MRC context. The goal is to extract the corresponding entities as answers. We use the Eq. 4 as the learning objective, where  $Y_{i,j}^{ext}$  indicates that the input span  $X_{i:j}$  is an answer/entity.
For sequence classification tasks, we construct the MRC query and context as followed. MCQA: The query is the concatenation of the question and one choice, and the context is the supporting document. MNLI: The query is the entailment label concatenated with the label description, and the context is the concatenation of the premise and hypothesis. SST-2: The query is the sentiment label concatenated with the label description, and the context is the input sentence. We use Eq. 3 to fine-tune the classification tasks. Note that only the correct query-context pair would get  $Y^{cls} = 1$ . Otherwise, the supervision is  $Y^{cls} = 0$ . During inference, we select the query-context pair with the highest  $S_{1,1}$  among all MRC examples constructed for the sequence classification instance as the final prediction. We show concrete examples for each task in Table 7 and Table 8.
A.3 Implementations
We download the 2022-01-01 dump<sup>5</sup> of English Wikipedia. For each article, we extract the plain text with anchors via WikiExtractor [3] and then preprocess it with NLTK [4] for sentence segmentation and tokenization. We consider the definition articles of entities that appear as anchors in at least 10 other articles to construct the query. Then, for each anchor entity, we pair its query from the definition article with 10 relevant contexts from other mention articles that explicitly mention the corresponding anchors and construct answerable MRC examples as described in Sec. 2. Unanswerable examples are formed by pairing the query with 10 irrelevant contexts.
<sup>5</sup>https://dumps.wikimedia.org/enwiki/latest
Task	Example Input	Example Output
EQA(SQuAD)	Ori.	Question: Which NFL team represented the NFC at Super Bowl 50? Context: Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers to earn their third Super Bowl title.	Answer: "Carolina Panthers"
PMR	[CLS] Which NFL team represented the NFC at Super Bowl 50? [SEP] [SEP] Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season . The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers to earn their third Super Bowl title . [SEP]	(53,54) - "Carolina Panthers"
NER(CoNLL)	Ori.	Two goals in the last six minutes gave holders Japan an uninspiring 2-1 Asian Cup victory over Syria on Friday.	("Japan", LOC); ("Syria", LOC); ("Asian Cup", MISC)
PMR	[CLS] "ORG". Organization entities are limited to named corporate, governmental, or other organizational entities. [SEP] [SEP] Two goals in the last six minutes gave holders Japan an uninspiring 2-1 Asian Cup victory over Syria on Friday . [SEP]	∅
[CLS] "PER". Person entities are named persons or family. [SEP] [SEP] Two goals in the last six minutes gave holders Japan an uninspiring 2-1 Asian Cup victory over Syria on Friday . [SEP]	∅
[CLS] "LOC". Location entities are the name of politically or geographically defined locations such as cities , countries . [SEP] [SEP] Two goals in the last six minutes gave holders Japan an uninspiring 2-1 Asian Cup victory over Syria on Friday . [SEP]	(32,32) - "Japan"; (40,40) - "Syria"
[CLS] "MISC". Examples of miscellaneous entities include events , nationalities , products and works of art . [SEP] [SEP] Two goals in the last six minutes gave holders Japan an uninspiring 2-1 Asian Cup victory over Syria on Friday . [SEP]	(34,35) - "Asian Cup"
Table 7: MRC examples of span extraction. Ori. indicates the original data format of these NLU tasks.
We use Huggingface's implementations of RoBERTa [62] as the MLM backbone. During the pretraining stage, the window size  $W$  for choosing context sentences is set to 2 on both sides. We use the first  $T = 1$  sentence as the MRC query. Sometimes, the sentence segmentation would wrongly segment a few words to form a sentence, which is not meaningful enough to serve as an MRC query. Therefore, we continue to include subsequent sentences to form the query as long the query length is short than 30 words. The learning rate is set to 1e-5, and the training batch size is set to 40 and 24 for  $\mathrm{PMR}_{\text{base}}$  and  $\mathrm{PMR}_{\text{large}}$  respectively in order to maximize the usage of the GPU memory. We follow the default learning rate schedule and dropout settings used in RoBERTa. We use AdamW [37] as our optimizer. We train both  $\mathrm{PMR}_{\text{base}}$  and  $\mathrm{PMR}_{\text{large}}$  for 3 epochs on 4 A100 GPU. Since the WAE is a discriminative objective, the pre-training is extremely efficient, which tasks 36 and 89 hours to finish all training processes for two model sizes respectively. We also reserve 1,000 definition articles to build a dev set (20,000 examples) for selecting the best checkpoint. Since the queries constructed by these definition articles have never been used in training, they can be used to estimate the general language understanding ability of the model instead of hand match. The hyper-parameters of  $\mathrm{PMR}_{\text{large}}$  on downstream NLU tasks can be found in Table 9 and Table 11 for full-supervision and few-shot settings respectively.
Task	Example Input	Example Output
MCQA(OBQA)	Ori.	Question: A positive effect of burning biofuel is:(A) shortage of crops for the food supply.(B) an increase in air pollution(C) powering the lights in a home.(D) deforestation in the amazon to make room for crops(Context: Biofuel is used to produce electricity by burning.	Answer Choice: C
PMR	[CLS] A positive effect of burning biofuel is shortage of crops for the food supply . [SEP] [SEP] Biofuel is used to produce electricity by burning . [SEP]	∅
[CLS] A positive effect of burning biofuel is an increase in air pollution . [SEP] [SEP] Biofuel is used to produce electricity by burning . [SEP]	∅
[CLS] A positive effect of burning biofuel is powering the lights in a home . [SEP] [SEP] Biofuel is used to produce electricity by burning . [SEP]	(0,0) - "[CLS]"
[CLS] A positive effect of burning biofuel is deforestation in the amazon to make room for crops . [SEP] [SEP] Biofuel is used to produce electricity by burning . [SEP]	∅
Sentence Classification(SST-2)	Ori.	This is one of Polanski's best films.	Positive
PMR	[CLS] Negative , feeling not good . [SEP] [SEP] This is one of Polanski 's best films . [SEP]	∅
[CLS] Positive , having a good feeling . [SEP] [SEP] This is one of Polanski 's best films . [SEP]	(0,0) - "[CLS]"
Sen. Pair Classification(MNLI)	Ori.	Hypothesis: You and your friends are not welcome here, said Premise: Severn said the people were not welcome there.	Severn. Entailment
PMR	[CLS] Neutral. The hypothesis is a sentence with mostly the same lexical items as the premise but a different meaning . [SEP] [SEP] Hypothesis : You and your friends are not welcome here, said Severn . Premise : Severn said the people were not welcome there . [SEP]	∅
[CLS] Entailment . The hypothesis is a sentence with a similar meaning as the premise . [SEP] [SEP] Hypothesis : You and your friends are not welcome here, said Severn . Premise : Severn said the people were not welcome there . [SEP]	(0,0) - "[CLS]"
[CLS] Contradiction . The hypothesis is a sentence with a contradictory meaning to the premise . [SEP] [SEP] Hypothesis : You and your friends are not welcome here, said Severn . Premise : Severn said the people were not welcome there . [SEP]	∅
Table 8: MRC examples of sequence classification.
A.4 Analysis of Data Construction
In addition to the defaulted way of constructing MRC examples (the first sentence in the definition article is the query, and randomly find 10 contexts for pairing 10 MRC examples), we compare with some advanced strategies to pair the query and the context, including:
- Q-C Relevance: We still use the first sentence from the definition article as the query, but we only select the top  $\mathrm{P\%}$  or top  $\mathrm{P}$  most similar contexts to the query, where the similarity score is computed as the combination of BM25 and SimCSE [17].
Dataset	CoNLL03	WNUT	ACE04	ACE05	MRQA	RACE	DREAM	MCTest	MNLI	SST-2
Query Length	32	32	64	64	64	128	128	128	64	64
Input Length	192	160	192	192	384	512	512	512	192	192
Batch Size	32	16	64	32	16	8	2	2	16	16
Learning Rate	2e-5	1e-5	2e-5	2e-5	2e-5	2e-5	2e-5	1e-5	1e-5	2e-5
Epoch	10	5	10	5	4	4	3	8	3	2
Table 9: Hyper-parameters settings in fine-tuning downstream tasks in full-supervision settings.
ID	Strategy	Query	Context	CoNLL	SQuAD	DREAM	SST-2
0	RoBERTa base	N.A.	N.A.	92.3	91.2	66.4	95.0
1	Random	First 1	Random 10	93.2	92.2	66.7	94.8
2	Q-C Relevance (top P%)	First 1	top 30%	93.0	91.9	65.5	95.3
3	Q-C Relevance (top P)	First 1	top 10	93.2	92.1	65.8	94.8
4	Random (Defaulted)	First 1	Random 10 + Unanswerable	93.1	92.1	70.7	94.6
5	Q-C Relevance (top P)	First 1	top 10 + Unanswerable	93.1	92.2	69.7	94.7
6	Q Diversity	Random 5	Random 10 + Unanswerable	93.2	92.2	70.6	94.8
7	C Diversity	First 1	Cluster 10 + Unanswerable	92.8	92.2	70.5	95.1
- Q Diversity: In searching for an anchor, we hope the query should be diverse enough such that the model would not make a hard match between the fixed query and the anchor. Therefore, we randomly select one sentence from the first P sentences in the definition article to serve as the query for the anchor, while we keep the same context selection strategy.
- C Diversity: We hope the contexts should also be diverse enough such that they provide more possible usages of an anchor. Therefore, We use K-means to cluster all contexts containing the anchor into P clusters and randomly select 1 context in each cluster. Similar scores in K-means are also obtained via SimCSE.