Title: From Cloze to Comprehension: Retrofitting Pre-trained Masked Language Models to Pre-trained Machine Reader
Venue: NeurIPS 2023
Accepted: yes
Keywords: Machine Reading Comprehension

Pre-training

Natural Language Understanding
Abstract: We present Pre-trained Machine Reader (PMR), a novel method for retrofitting pre-trained masked language models (MLMs) to pre-trained machine reading comprehension (MRC) models without acquiring labeled data.
PMR can resolve the discrepancy between model pre-training and downstream fine-tuning of existing MLMs.
To build the proposed PMR, we constructed a large volume of general-purpose and high-quality MRC-style training data by using Wikipedia hyperlinks and designed a Wiki Anchor Extraction task to guide the MRC-style pre-training.
Apart from its simplicity, PMR effectively solves extraction tasks, such as Extractive Question Answering and Named Entity Recognition. PMR shows tremendous improvements over existing approaches, especially in low-resource scenarios.
When applied to the sequence classification task in the MRC formulation, PMR enables the extraction of high-quality rationales to explain the classification process, thereby providing greater prediction explainability. PMR also has the potential to serve as a unified model for tackling various extraction and classification tasks in the MRC formulation.

From Cloze to Comprehension: Retrofitting Pre-trained Masked Language Models to Pre-trained Machine Reader ∗

Abstract

We present Pre-trained Machine Reader (PMR), a novel method for retrofitting pre-trained masked language models (MLMs) to pre-trained machine reading comprehension (MRC) models without acquiring labeled data. PMR can resolve the discrepancy between model pre-training and downstream fine-tuning of existing MLMs. To build the proposed PMR, we constructed a large volume of general-purpose and high-quality MRC-style training data by using Wikipedia hyperlinks and designed a Wiki Anchor Extraction task to guide the MRC-style pretraining. Apart from its simplicity, PMR effectively solves extraction tasks, such as Extractive Question Answering and Named Entity Recognition. PMR shows tremendous improvements over existing approaches, especially in low-resource scenarios. When applied to the sequence classification task in the MRC formulation, PMR enables the extraction of high-quality rationales to explain the classification process, thereby providing greater prediction explainability. PMR also has the potential to serve as a unified model for tackling various extraction and classification tasks in the MRC formulation. 2

1 Introduction

Span extraction, such as Extractive Question Answering (EQA) and Named Entity Recognition (NER), is a sub-topic of natural language understanding (NLU) with the goal of detecting token spans from the input text according to specific requirements like task labels or questions [45, 54]. Discriminative methods were used to execute such tasks and achieved state-of-the-art performance. As shown in the left part of Figure 1, these works tailored a task-specific fine-tuning head on top of pre-trained language models (PLMs) to perform sequence tagging or machine reading comprehension (MRC) [12, 36, 27]. The base PLMs are usually selected from pre-trained masked language models (MLM), such as RoBERTa [36] or BART [29] due to their comprehensive bi-directional modeling for the input text in the encoder. However, given the disparate nature of the learning objectives and different model architectures of MLM pre-training and task-specific fine-tuning, the discriminative

Pre-training

Fine-tuning

Fine-tuning

Fine-tuning

with PMR

Generative

Fine-tuning

It is a chemical element with the

symbol Si and a semiconductor

.

Tom develops silicon technology .

15

,

15

(silicon)

MLM

MRC Head

PMR

MLM

Tom [Mask] silicon

technology .

MLM

Tom [Mask] silicon

technology .

develops

develops

MLM

MLM

MRC Head

Tom will fly to London .

Who is the father of Tom ?

Tom's father is Bob .

B-PER O O O B-LOC O

10

,

10

Tagging Head

MLM

Tom will fly to London .

"Tom" is a [Mask] entity.

person

-> PER

location

-> LOC

none

-> O

MLM

Tom's father is Bob.

Who is the father of Tom?

The answer is [Mask]

("will", "Tom will", ...)

Person ?

Tom will fly

to London .

2

,

2

Who is the father of Tom ?

Tom's father is Bob .

10

,

10

MLM

MRC Head

MLM

MRC Head

PMR

PMR

methods are less effective for adapting MLMs to downstream tasks when there is limited fine-tuning data available, leading to poor low-resource performance [6].

As shown in the middle part of Figure 1, generative fine-tuning is a popular solution to mitigate the gap between pre-training and fine-tuning [49, 50, 34]. This solution achieves remarkable few-shot performance in various span extraction tasks [10, 6, 38]. Specifically, generative methods formulate the downstream tasks as a language modeling problem in which PLMs generate response words for a given prompt (i.e., a task-specific template) as task prediction. Despite its success, tackling extraction tasks in a generative manner leads to several disadvantages. First, if it is used to generate the label token (e.g., 'person' for PER entities) for a candidate span, the generative method needs to enumerate all possible span candidates to query PLMs [10]. This requirement can be computationally expensive for tasks with a long input text, such as EQA. Second, if the desired predictions are target spans (e.g., the 'answer' in the EQA task), generative methods usually need to explore a large search space to generate span tokens. Moreover, it is also challenging to accurately generate structured outputs, e.g., the span-label pairs in the NER task, with PLMs originally trained on unstructured natural language texts. These limitations impede PLMs from effectively learning extraction patterns from increased volumes of training data. As a result, even instruction-tuned large language models like ChatGPT 3 are less effective than discriminative methods with smaller MLMs on extraction tasks [41, 43, 60, 31].

To bridge the gap between pre-training and fine-tuning without suffering from the aforementioned disadvantages, we propose a novel Pre-trained Machine Reader (PMR) as a retrofit of pre-trained MLM for more effective span extraction. As shown in the right part of Figure 1, PMR resembles common MRC models and introduces an MRC head on top of MLMs. But PMR is further enhanced by a comprehensive continual pre-training stage with large-scale MRC-style data. By maintaining the same MRC-style learning objective and model architecture as the continual pre-training during finetuning, PMR facilitates effective knowledge transfer in a discriminative manner and thus demonstrates great potential in both low-resource and rich-resource scenarios. Given that MRC has been proven as a universal paradigm [32, 33, 63, 23], our PMR can be directly applied to a broad range of span extraction tasks without additional task design.

To establish PMR, we constructed a large volume of general-purpose and high-quality MRC-style training data based on Wikipedia anchors (i.e., hyperlinked texts). As shown in Figure 2, for each Wikipedia anchor, we composed a pair of correlated articles. One side of the pair is the Wikipedia article that contains detailed descriptions of the hyperlinked entity, which we defined as the definition

Silicon

(

Q670

)

Silicon is a chemical element with

the symbol Si and atomic number

14.  It  is  a  hard,  brittle  crystalline

solid

with

a

blue-grey

metallic

luster,

and

is

a

tetravalent

metalloid  and  semicondu-ctor.

It

is  a  member  of  group  14  in  the

periodic table: carbon is above it;

and  germanium,  tin,

lead,  and

flerovium

are

below

it.

It

is

unreactive.  Because  of  its  high

chemical affinity ...

Semiconductor

(

Q11456

)

…

the

most

commercially

important  of  these  elements  are

silicon

Q670

and

germanium.

.

Silicon

Q670

and  germanium  are

used here effectively ...

Definition Article

Mention Article

Integrated circuit

(

Q80831

)

…  (also  referred  to  as  an  IC,  a

chip,  or  a  microchip)  is  a  set  of

electronic  circuits  on  one  small

flat

piece

of

semiconductor

material, usually

silicon

Q670

…

🔗

🔗

PMR

[CLS]  Silicon  It  is  a  chemical  element  with  the

symbol Si and atomic number 14 ... [SEP] … the

most commercially important of these elements

are

silicon

and

germanium

.

Silicon

and

germanium are used here effectively ... [SEP]

[CLS]  Silicon  It  is  a  chemical  element  with  the

symbol Si and atomic number 14 ... [SEP] … on

one small flat piece (or "chip") of semiconductor

material , usually silicon … [SEP]

[CLS]  Silicon  It  is  a  chemical  element  with  the

symbol Si and atomic number 14 ... [SEP] The

pizza is delicious ... [SEP]

Answerable / multi-span extraction

Answerable / single-span extraction

Unanswerable

0,0 ([CLS])

29,29 (silicon)

PMR

0,0 ([CLS])

26,26 (silicon)

30,30 (Silicon)

PMR

article . The other side of the pair is the article that mentions the specific anchor text, which we defined as the mention article . We composed an MRC-style training instance in which the anchor is the answer, the surrounding passage of the anchor in the mention article is the context, and the definition of the anchor entity in the definition article is the query. Based on the above data, we then introduced a novel Wiki Anchor Extraction (WAE) problem as the pre-training task of PMR. In this task, PMR determines whether the context and the query are relevant. If so, PMR extracts the answer from the context that satisfies the query description.

We evaluated PMR on two representative span extraction tasks: NER and EQA. The results show that PMR consistently obtains better extraction performance compared with the vanilla MLM and surpasses the best baselines by large margins under almost all few-shot settings (up to 6.3 F1 on EQA and 16.3 F1 on NER). Additionally, we observe that sequence classification can be viewed as a special case of extraction tasks in our MRC formulation. In this scenario, it is surprising that PMR can identify high-quality rationale phrases from input text as the justifications for classification decisions. Furthermore, PMR has the potential to serve as a unified model for addressing various extraction and classification tasks in the MRC formulation.

In summary, our contributions are as follows. Firstly, we constructed a large volume of generalpurpose and high-quality MRC-style training data to retrofit MLMs to PMRs. Secondly, by unifying pre-training and fine-tuning as the same discriminative MRC process, the proposed PMR obtains state-of-the-art results under all few-shot NER settings and three out of four few-shot EQA settings. Thirdly, with a unified MRC architecture for solving extraction and classification tasks, PMR also shows promising potential in explaining the sequence classification predictions and unifying NLU tasks.

2 PMR

This section describes PMR from the perspectives of model pre-training and downstream fine-tuning. For pre-training, we first introduce the proposed model with the training objective of WAE and then describe the curation procedure of WAE pre-training data from Wikipedia. For fine-tuning, we present how PMR can seamlessly be applied to various extraction tasks and solve them in a unified MRC paradigm.

2.1 Pre-training of PMR

PMR receives MRC-style data in the format of ( Q , C , { A k } K k =1 ), where Q is a natural language query and C is the input context that contains the answers { A k } K k =1 to the query. Each answer is a consecutive token span in the context, and zero ( K = 0 ) or multiple ( K > 1 ) answers may exist.

Model Architecture. PMR has two components: an MLM encoder and an extractor (Figure 3). The encoder receives the concatenation of query Q and context C as input X and represents each input token as hidden states H .

PMR

Extractor

[CLS]

Tok 1

Tok |Q|

[SEP]

Tok 1

[SEP]

Query

Context

Answers

S

1,1

S

1,

N

+1

S

1,

N

+2

S

1,

M

SN

+1,

N

+1

SN

+1,

N

+2

SN

+1,

M

SN

+1,

N

+2

SN

+2,

M

SM

,

M

1

-

-

-

1

-

0

-

0

-

Y

S

MLM

where [ CLS ] and [ SEP ] are special tokens inserted into the sequence, M is the sequence length, and d is the dimension of the hidden states. The encoder MLM ( · ) denotes any pre-trained text encoder for retrofitting, e.g. RoBERTa.

The extractor receives the hidden states of any two tokens and predicts the probability score that tells if the span between the two tokens should be output as an answer. We applied the general way to compute the score matrix S [39]:

where FFN is the feed-forward network [57], and S i,j is the probability to extract the span X i : j as output. The general way avoids creating a large R M × M × 2 d -shape tensor of the concatenation way [32], achieving higher training efficiency with fewer resources.

Training Objective. PMR is pre-trained with the WAE task, which checks the existence of answers in the context and extracts the answers if they exist. For the first goal, PMR determines whether the context contains spans that can answer the query:

where CE is the cross-entropy loss and S 1 , 1 at the [CLS] token denotes the query-context relevance score. If Y cls = 1 , the query and the context are relevant (i.e. answers exist). This task mimics the downstream situation in which there may be no span to be extracted in the context (e.g. NER) and encourages the model to learn through the semantic relevance of two pieces of text to recognize the unextractable examples.

Secondly, the model is expected to extract all correct spans from the context as answers, which can be implemented by predicting the answer positions:

where Y ext i,j = 1 indicates that X i : j is an answer to Q , and N is the positional offset of the context in X . Note that only X i : j with N < i ≤ j < M are legal answer span candidates (i.e., spans from the context). MRC-NER [32] predicted the start and end probabilities as two additional objectives. However, we find that these objectives are redundant for our matrix-based objective and incompatible with multi-span extraction.

The overall pre-training objective L wae is:

Data Preparation. MLMtraining can be easily scaled to millions of raw texts with a self-supervised learning objective [12]. In contrast, training PMR in the MRC paradigm requires labeled triplets (query, context, and answers) as supervision signals, which is expensive to prepare for large-scale pre-training. To address this limitation, we automated the construction of general-purpose and high-quality MRC-style training data by using Wikipedia anchors.

As illustrated in Figure 2, a Wikipedia anchor hyperlinks two Wikipedia articles: the definition article that provides detailed descriptions of the anchor entity 'Silicon', and the mention article where the anchor is mentioned. We leveraged the large scale of such hyperlink relations in Wikipedia as the distant supervision to automatically construct the MRC triplets. Specifically, we regarded an anchor as the MRC answer for the following context and query pair. The sentences surrounding the anchor in the mention article serve as the MRC context. The sentences from the first section of the definition article, which usually composes the most representative summary for the anchor entity [7], comprise the query. The query provides a precise definition of the anchor entity, and thus serves as a good guide for PMR to extract answers (i.e., anchor text) from the context.

Concretely, we considered sentences within a window size W of the anchor as the MRC context and used the first T sentences from the definition article as the query. Note that the context may cover multiple mentions of the same anchor entity. In this case, we treated all mentions as valid answers (i.e., K > 1 ) to avoid confusing the model training. More importantly, the preceding scenario naturally resembles multi-span extraction tasks like NER. To prevent information leakage, we anonymized the anchor entity in the query by using 'it' to substitute text spans that overlapped more than 50% with the anchor entity name. we did not use the '[MASK]' token because it does not exist in the data of downstream tasks.

In addition to the above answerable query and context pairs prepared through hyperlink relation, we introduced unanswerable examples by pairing a context with an irrelevant query (i.e., query and context pairs without the hyperlink association). The unanswerable examples are designed to help the model learn the ability to identify passage-level relevance and avoid extracting any answer (i.e., K = 0 ) for such examples.

2.2 Fine-tuning PMR for Extraction Tasks

We unified downstream span extraction tasks in our MRC formulation, which typically falls into two categories: (1) span extraction with pre-defined labels (e.g., NER) in which each task label is treated as a query to search the corresponding answers in the input text (context) and (2) span extraction with natural questions (e.g., EQA) in which the question is treated as the query for answer extraction from the given passage (context). Then, in the output space, we tackled span extraction problems by predicting the probability S i,j of context span X i : j being the answer. The detailed formulation and examples are provided in Appendix A.2.

3 Experimental Setup

Implementation. We used the definition articles of the entities that appear as anchors in at least 10 other articles to construct the query. As mentioned in Sec. 2, we prepared 10 answerable query and context pairs for each anchor entity. Then, we paired the query with 10 irrelevant contexts to formulate unanswerable MRC examples. The resulting pre-training corpus consists of 18 million MRC examples (6.4 billion words). We also tried various advanced data construction strategies, such as relevance-driven and diversity-driven ones, to construct query and context pairs. However, no significant performance gain is observed. A detailed comparison is provided in Appendix A.4.

The encoder of PMR is initialized with RoBERTa, a popular MLM with competitive downstream performance. The extractor is randomly initialized, introducing additional 1.6M parameters. In terms of the pre-training efficiency, with four A100 GPUs, it only takes 36 and 89 hours to complete 3-epoch training of PMR for base-sized and large-sized models, respectively. Additional data preprocessing details and hyper-parameter settings can be found in Appendix A.3.

Downstream Extraction Tasks. We evaluated two extraction tasks: EQA and NER.

EQA: We evaluated PMR on MRQA benchmark [15]. For the few-shot setting, we used the few-shot MRQAdatasets sampled by Splinter [46]. Although BioASQ and TbQA are originally used for OOD

evaluation, Splinter [46] constructed few-shot training sets for both datasets by sampling examples from their original dev sets. We followed FewshotQA [6] to build the dev set that has the same size as the training set for model selection. For the full-resource experiments, we followed MRQA [15] in both in-domain and OOD evaluations.

NER: We evaluated PMR on two flat NER datasets, namely, CoNLL and WNUT, and two nested NER datasets, namely, ACE04 and ACE05 [54, 11, 42, 58]. For the few-shot setting, we constructed five splits of K-shot training data, where K ∈ {4,8,32,64} sentences are sampled for each tag [38]. We also constructed the dev set that has the same size as the training set.

Baselines. We compared PMR with (1) vanilla MLMs: RoBERTa [36] and SpanBERT [20] with a randomly-initialized extractor, (2) RBT-Post, a continually pre-trained RoBERTa using our Wikipedia data but with the MLM objective; and (3) models bridging the gaps between pre-training and finetuning, namely, Splinter [46], FewshotBART [6], EntLM [40], T5-v1.1 [44], and UIE [38], where the latter four are generative methods.

4 Main Results

Few-shot Results. The few-shot results of EQA and NER are presented in Tables 1 and 2. The poor few-shot capability of RoBERTa suggests that fine-tuning an MRC extractor from scratch with limited data is extremely challenging. Our PMR achieves notably better results on all few-shot levels of the two tasks than RoBERTa, obtaining an average improvement of 34.8 F1 and 18.6 F1 on 32-shot EQA and 4-shot NER, respectively. Other models that bridge the gaps between pre-training and fine-tuning also perform much better than RoBERTa and RBT-Post, which is consistent with our findings. For EQA, although FewshotBART benefits from a larger output space of the generative

model to achieve better transferability in an extremely low-resource EQA setting (i.e., 16 shot), processing such large output is far more complicated than the discriminative MRC and thus is prone to overfitting. MRC-based PMR demonstrates higher effectiveness in learning extraction capability from more training examples than FewshotBART, consequently yielding better performance on EQA when at least 32 examples are provided. Note that the compared baselines for EQA and NER are slightly different due to their different applicability. For example, UIE mainly emphasizes a structured prediction and is not applicable to complicated extraction tasks like EQA. EntLM, which aims to generate label tokens, is also not applicable to EQA. The findings further reveal that PMR can work reasonably well as a zero-shot learner (Appendix A.5).

OODGeneralization. Domain generalization is another common low-resource scenario in which the knowledge can be transferred from a resource-rich domain to a resource-poor domain. We evaluated the domain generalization capability of the proposed PMR on the MRQA benchmark. The MRQAbenchmark provides meaningful OOD datasets that are mostly converted from other tasks (e.g., multi-choice QA) and substantially differ from SQuAD in terms of text domain.

Table 3 shows that PMR significantly surpasses RoBERTa on all six OOD datasets (+4.5 F1 on average), although they have similar in-domain performance on SQuAD. This finding verifies that our PMR with MRC-style pre-training can help capture QA patterns that are more generalizable to unseen domains. In addition, PMR with less than half the parameters of T5-v1.1, achieves a better generalization capability.

Full-resource Results. Although using the full-resource training data can alleviate the pretraining-finetuning discrepancy, MRC-style continual pre-training still delivers reasonable performance gains. As shown in Table 4, PMR achieves 0.9 and 1.5 F1 improvements over RoBERTa on EQA and NER, respectively. Further analysis shows that PMR can do better at comprehending the input text (Appendix 5.3). We also explore the upper limits of PMR by employing a larger and stronger MLM, i.e. ALBERT xxlarge [27], as the backbone. The results show additional improvements of our PMR over ALBERT xxlarge on EQA (Appendix A.6).

Table 6: Case study on SST-2. PMR can additionally extract token-level rationales (in red and bold) to support the sequence classification. A total of 224 rationales out of 300 (74%) are reasonable.

5 Discussions

5.1 Explainable Sequence Classification

Settings. The relevance classification between the query and the context in our WAE objective can be inherited to tackle downstream sequence classification problems, another important topic of NLU. To demonstrate, we considered two major sequence classification tasks in our MRC formulation. The first is sequence classification with pre-defined task labels, such as sentiment analysis. Each task label is used as a query for the input text (i.e. the context in PMR) 4 ; The second is sequence classification with natural questions on multiple choices, such as multi-choice QA (MCQA). We treated the concatenation of the question and one choice as the query for the given passage (i.e., the context). In the output space, these problems are tackled by conducting relevance classification on S 1 , 1 (extracting [CLS] if relevant). Examples are included in Appendix A.2. We tested PMR on (1) three MCQA tasks: DREAM [53], RACE [26], and MCTest [47]; (2) a sentence-pair classification task: MNLI [61]; and (3) a sentence classification task: SST-2 [52].

Results. Table 5 presents the results of PMR and previous strong baselines on the above-mentioned tasks. Compared with RoBERTa, PMR exhibits comparable results on all tasks and achieved slightly better average performance. This indicates that solving sequence classification in an MRC manner is feasible and effective.

Explainability. In sequence classification, we restricted the extraction space to the [CLS] token and used the extraction probability of this token to determine which class label (i.e. the query) the input text (i.e. the context) corresponds to. Note that while predicting the extraction probability of [CLS] , PMR also calculates the extraction scores of context spans, just as done for the span extraction tasks. Therefore, we leveraged the fine-tuned PMR to additionally extract a span with the highest S i,j score from the input of SST-2, as exemplified in Table 6. The extracted spans are clear rationales that support the sequence-level prediction (i.e., the overall sentiment of the input sentence). To verify this, we conducted a quantitative analysis by randomly checking 300 test instances. The results show that approximately 74% of the extracted spans are reasonable rationales for sentiment prediction. These findings suggest that the fine-tuned PMR on SST-2 effectively preserves the conditional span extraction capability inherited from the MRC pre-training. Such a capability shows that the model captures the semantic interactions between relevant context spans and the label-encoded query (e.g., 'Negative. Feeling bad.'). In addition to the performance improvement, PMR provides high explainability for the classification results.

5.2 Unifying Extraction and Classification with PMR

In the previous sections, we demonstrate that various extraction and classification tasks can be separately tackled in the same MRC formulation. We further explore the potential that fine-tuning a unified model for solving multiple tasks of different types.

Settings. We use two datasets, one from CoNLL NER (of extraction type) and the other from DREAM (of classification type), to train a multi-task model. For evaluation, we conduct three groups of experiments, where the models are evaluated on (1) Held-in: testing sets from training tasks, (2) Held-out Datasets: testing sets from other tasks of the same type with training tasks, and (3) Held-out Tasks: testing sets from unseen tasks

Results. As shown in Figure 4, the heldin results show that the multi-task fine-tuned RoBERTa suffers from a significant performance drop on DREAM compared to the single-task fine-tuned RoBERTa. This indicates that multi-task learning is difficult for a discriminative model if the task head is not well-trained. In contrast, the multi-task PMR is on par with PMR on CoNLL and slightly underperforms PMR on DREAM. Such a finding suggests that the MRC-style pre-training enhances PMR's capability to learn extraction and classification patterns from downstream NLUtasks, enabling PMR to serve as a unified model for solving various NLU tasks. Though generative models like T5-v1.1 (800M) could

CoNLL

WNUT

RACE

SQuAD

SST-2

Datasets

0

20

40

60

80

100

Scores

Held-in

Held-out Datasets

Held-out Tasks

T5-v1.1_large

RoBERTa_large

PMR_large

also unify NLU tasks through a conditional generation manner [44, 38], the overall performance, especially the held-out performance, is lower than the smaller-sized PMR (355M). This suggests that the discriminative PMR may be better than generative models at unifying NLU tasks.

5.3 Better Comprehending capability

To verify that PMR can better comprehend the input text, we feed the models with five different query variants during CoNLL evaluation. The five variants are:

•

•

•

[Label description]

Defaulted query: "[Label]". [Label description]

Query template (v1):

What is the "[Label]" entity, where [Label description]?

Query template (v2): Identify the spans (if any) related to "[Label]" entity. Details:

Paraphrasing label description with ChatGPT (v1):

"[Label]". [Paraphrased Label description v1]

Paraphrasing label description with ChatGPT (v1):

"[Label]". [Paraphrased Label description v2]

In Figure 5, we show the statistic results of the three models on CoNLL when five different query templates are used respectively during evaluation. Among the models, PMR demonstrated significantly higher and more stable performance than RoBERTa and T5-v1.1. Such a finding verifies our assumption that PMR can effectively comprehend the latent semantics of the input text despite being rephrased with varying lexical usage from the default query used for fine-tuning models.

6 Related Work

Gaps between Pre-training and Fine-tuning. The prevailing approaches tackle span extraction tasks in a discriminative manner with tailored task-specific classifiers (e.g. tagging or MRC head)

PMR

RoBERTa

Model

82

84

86

88

90

92

94

F1

[19, 9, 12]. Recent research reported that generative fine-tuning methods can effectively bridge the gaps between pre-training and fine-tuning and achieve remarkable few-shot NLU performance by representing downstream NLU tasks as the same language model pre-training problem [49, 50, 34, 16, 35]. In the scope of span extraction, these works can be classified into three categories: generating label tokens based on the prompt span [10, 40], generating span tokens based on the prompt label [6, 38], and directly generating label-span pairs with soft prompt [8]. Another way to mitigate the gap is to adapt MLMs into an appropriate paradigm for solving span extraction. Typically, this can be achieved through the use of task-specific data from similar tasks, referred to as 'pre-finetuning' [24, 1, 68]. However, these methods may not be effective in domain-specific tasks or for non-English languages due to the lack of labeled data. Several studies leveraged abundant raw text to construct MRC examples for retrofitting MLMs [30, 18, 46]. By contrast, PMR employs ubiquitous hyperlink information to construct MRC data, which guarantees highly precise MRC triplets. In addition, PMR also provides a unified model for both sequence classification and span extraction, thereby enabling strong explainability through the extraction of high-quality rationale phrases.

Hyperlinks for NLP. Hyperlinks are utilized in two ways. First, hyperlinks can be regarded as a type of relevance indicator in model pre-training [67], passage retrieval [7, 51], and multi-hop reasoning [2, 66]. Second, the anchors labeled by hyperlinks can serve as entity annotations for representation learning [64, 5]. PMR is the first one to combine the advantages of both scenarios. In this work, we paired MRC query and context based on the relevance of hyperlinks and automatically labeled the anchors as MRC answers.

7 Conclusions

This work presents a novel MRC-style pre-training model called PMR. PMR can fully resolve the learning objective and model architecture gaps that frequently appear in fine-tuning existing MLMs. Experimental results from multiple dimensions, including effectiveness in solving few-shot tasks and OOD generalization, show the benefits of bridging the gap between pre-training and fine-tuning for span extraction tasks. PMR also shows promising potential in explaining the sequence classification process and unifying NLU tasks.

References

Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. Muppet: Massive multi-task representations with pre-finetuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 5799-5811, November 2021.

Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. Learning to retrieve reasoning paths over wikipedia graph for question answering. In International Conference on Learning Representations , 2020.

Giusepppe Attardi. Wikiextractor. https://github.com/attardi/wikiextractor , 2015.

Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit . " O'Reilly Media, Inc.", 2009.

Iacer Calixto, Alessandro Raganato, and Tommaso Pasini. Wikipedia entities as rendezvous across languages: Grounding multilingual language models by predicting Wikipedia hyperlinks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , June 2021.

Rakesh Chada and Pradeep Natarajan. FewshotQA: A simple framework for few-shot learning of question answering tasks using pre-trained text-to-text models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 6081-6090, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.

Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. Pre-training tasks for embedding-based large-scale retrieval. In International Conference on Learning Representations , 2020.

Xiang Chen, Lei Li, Shumin Deng, Chuanqi Tan, Changliang Xu, Fei Huang, Luo Si, Huajun Chen, and Ningyu Zhang. LightNER: A lightweight tuning paradigm for low-resource NER via pluggable prompting. In Proceedings of the 29th International Conference on Computational Linguistics , pages 2374-2387, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics.

Jason P.C. Chiu and Eric Nichols. Named entity recognition with bidirectional LSTM-CNNs. Transactions of the Association for Computational Linguistics , 4:357-370, 2016.

Leyang Cui, Yu Wu, Jian Liu, Sen Yang, and Yue Zhang. Template-based named entity recognition using BART. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , August 2021.

Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. Results of the WNUT2017 shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User-generated Text , September 2017.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , June 2019.

Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 2368-2378, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.

Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Güney, Volkan Cirik, and Kyunghyun Cho. Searchqa: A new q&a dataset augmented with context from a search engine. ArXiv , abs/1704.05179, 2017.

Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering , November 2019.

Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 3816-3830, Online, August 2021. Association for Computational Linguistics.

Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , November 2021.

Michael Glass, Alfio Gliozzo, Rishav Chakravarti, Anthony Ferritto, Lin Pan, G P Shrivatsa Bhargav, Dinesh Garg, and Avi Sil. Span selection pre-training for question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 2773-2782, Online, July 2020. Association for Computational Linguistics.

Karl Moritz Hermann, Tomáš Koˇ cisk` y, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume 1 , pages 1693-1701, 2015.

Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics , 8, 2020.

Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , July 2017.

Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern recognition , pages 4999-5007, 2017.

Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. Unifying question answering, text classification, and regression via span extraction, 2020.

Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020 , November 2020.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics , 7, 2019.

Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , September 2017.

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. ArXiv , abs/1909.11942, 2020.

Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017) , pages 333-342, Vancouver, Canada, August 2017. Association for Computational Linguistics.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , July 2020.

Patrick Lewis, Ludovic Denoyer, and Sebastian Riedel. Unsupervised question answering by cloze translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4896-4910, Florence, Italy, July 2019. Association for Computational Linguistics.

Bo Li, Gexiang Fang, Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, and Shikun Zhang. Evaluating chatgpt's information extraction capabilities: An assessment of performance, explainability, calibration, and faithfulness, 2023.

Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. A unified MRC framework for named entity recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , July 2020.

Jian Liu, Yubo Chen, Kang Liu, Wei Bi, and Xiaojiang Liu. Event extraction as machine reading comprehension. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , November 2020.

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586 , 2021.

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv:2103.10385 , 2021.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019.

Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. Unified structure generation for universal information extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , May 2022.

Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , September 2015.

Ruotian Ma, Xin Zhou, Tao Gui, Yiding Tan, Linyang Li, Qi Zhang, and Xuanjing Huang. Template-free prompt tuning for few-shot NER. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , Seattle, United States, July 2022. Association for Computational Linguistics.

Yubo Ma, Yixin Cao, YongChing Hong, and Aixin Sun. Large language model is not a good few-shot information extractor, but a good reranker for hard samples!, 2023.

Alexis Mitchell, Stephanie Strassel, Shudong Huang, and Ramez Zakhary. Ace 2004 multilingual training corpus, 2005.

Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt a general-purpose natural language processing task solver?, 2023.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21, 2020.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , November 2016.

Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Globerson, and Omer Levy. Few-shot question answering by pretraining span selection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 3066-3079, Online, August 2021. Association for Computational Linguistics.

Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , October 2013.

Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, and Karthik Sankaranarayanan. DuoRC: Towards complex language understanding with paraphrased reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 16831693, Melbourne, Australia, July 2018. Association for Computational Linguistics.

Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume , April 2021.

Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , June 2021.

Yeon Seonwoo, Sang-Woo Lee, Ji-Hoon Kim, Jung-Woo Ha, and Alice Oh. Weakly supervised pre-training for multi-hop retriever. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , August 2021.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , October 2013.

Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. DREAM: A challenge data set and models for dialogue-based reading comprehension. Transactions of the Association for Computational Linguistics , 7, 2019.

Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Languageindependent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003 , 2003.

Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP , August 2017.

George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, et al. An overview of the bioasq large-scale biomedical semantic indexing and question answering competition. BMC bioinformatics , 16(1):1-28, 2015.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V on Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30, 2017.

Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. Ace 2005 multilingual training corpus, 2006.

Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. Improving named entity recognition by external context retrieving and cooperative learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , August 2021.

Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, Yong Jiang, and Wenjuan Han. Zero-shot information extraction via chatting with chatgpt, 2023.

Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , June 2018.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , October 2020.

Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. CorefQA: Coreference resolution as querybased span prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , July 2020.

Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. LUKE: Deep contextualized entity representations with entity-aware self-attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , November 2020.

Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, and Xipeng Qiu. A unified generative framework for various NER subtasks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , August 2021.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , OctoberNovember 2018.

Michihiro Yasunaga, Jure Leskovec, and Percy Liang. LinkBERT: Pretraining language models with document links. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , May 2022.

Wanjun Zhong, Yifan Gao, Ning Ding, Yujia Qin, Zhiyuan Liu, Ming Zhou, Jiahai Wang, Jian Yin, and Nan Duan. ProQA: Structural prompt-based pre-training for unified question answering. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 4230-4243, Seattle, United States, July 2022. Association for Computational Linguistics.

A Appendix

A.1 Limitations

Multilinguality Although constructing large-scale MRC-style training data is feasible for resourcerich languages, such as English, extending this idea to resource-poor languages might be difficult due to the relatively small amount of anchors in their corresponding Wikipedia articles. Exploring other data resources to automatically construct large-scale pre-training data can remedy this issue. For example, given a word in the monolingual dictionaries, we can regard the word itself, the definition of this word, and the example sentence of this word as the MRC answer, query, and context respectively. We believe our MRC-style pre-training is still applicable for low-resource languages with such dictionaries.

Comparison with Large Language Models In this paper, we did not compare PMR with large language models (LLM) for the following two reasons. First, existing MLMs are small in scale. Therefore, we are unable to find a suitable MLM to make a fair comparison with LLMs. Second, studies have shown that LLMs yield inferior results compared to smaller MLMs on span extraction tasks, particularly those involving structured prediction [41, 43, 60, 31]. Based on this fact, we mainly compare with existing strong generative methods of comparable model size.

Few-shot NER results of SpanBERT We ran SpanBERT [20] in our NER few-shot settings. However, its performance was below our expectations. In all our few-shot settings, SpanBERT achieved an F1 score of 0 on CoNLL and WNUT datasets. Additionally, its performance on ACE04 and ACE05 datasets was significantly lower than RoBERTa [36]. Based on these outcomes, we only compare PMR with SpanBERT in the NER full-resource setting.

A.2 Fine-tuning Tasks

For EQA, we use the MRQA benchmark [15], including SQuAD [45], TriviaQA [21], NaturalQuestion [25], NewQA [55], SearchQA [14], HotpotQA [66], BioASQ [56], DROP [13], DuoRC [48], RACE [26], RelationExtraction [28], TextbookQA [22]. EQA has always been treated as an MRC problem, where the question serves as the MRC query, and the passage containing the answers serves as the MRC context. For NER, We follow MRC-NER [32] to formulate NER into the MRC paradigm, where the entity label together with its description serves as the MRC query, and the input text serves as the MRC context. The goal is to extract the corresponding entities as answers. We use the Eq. 4 as the learning objective, where Y ext i,j indicates that the input span X i : j is an answer/entity.

For sequence classification tasks, we construct the MRC query and context as followed. MCQA: The query is the concatenation of the question and one choice, and the context is the supporting document. MNLI: The query is the entailment label concatenated with the label description, and the context is the concatenation of the premise and hypothesis. SST-2: The query is the sentiment label concatenated with the label description, and the context is the input sentence. We use Eq. 3 to fine-tune the classification tasks. Note that only the correct query-context pair would get Y cls = 1 . Otherwise, the supervision is Y cls = 0 . During inference, we select the query-context pair with the highest S 1 , 1 among all MRC examples constructed for the sequence classification instance as the final prediction. We show concrete examples for each task in Table 7 and Table 8.

A.3 Implementations

We download the 2022-01-01 dump 5 of English Wikipedia. For each article, we extract the plain text with anchors via WikiExtractor [3] and then preprocess it with NLTK [4] for sentence segmentation and tokenization. We consider the definition articles of entities that appear as anchors in at least 10 other articles to construct the query. Then, for each anchor entity, we pair its query from the definition article with 10 relevant contexts from other mention articles that explicitly mention the corresponding anchors and construct answerable MRC examples as described in Sec. 2. Unanswerable examples are formed by pairing the query with 10 irrelevant contexts.

Table 7: MRC examples of span extraction. Ori. indicates the original data format of these NLU tasks.

We use Huggingface's implementations of RoBERTa [62] as the MLM backbone. During the pretraining stage, the window size W for choosing context sentences is set to 2 on both sides. We use the first T = 1 sentence as the MRC query. Sometimes, the sentence segmentation would wrongly segment a few words to form a sentence, which is not meaningful enough to serve as an MRC query. Therefore, we continue to include subsequent sentences to form the query as long the query length is short than 30 words. The learning rate is set to 1e-5, and the training batch size is set to 40 and 24 for PMR base and PMR large respectively in order to maximize the usage of the GPU memory. We follow the default learning rate schedule and dropout settings used in RoBERTa. We use AdamW [37] as our optimizer. We train both PMR base and PMR large for 3 epochs on 4 A100 GPU. Since the WAE is a discriminative objective, the pre-training is extremely efficient, which tasks 36 and 89 hours to finish all training processes for two model sizes respectively. We also reserve 1,000 definition articles to build a dev set (20,000 examples) for selecting the best checkpoint. Since the queries constructed by these definition articles have never been used in training, they can be used to estimate the general language understanding ability of the model instead of hand match. The hyper-parameters of PMR large on downstream NLU tasks can be found in Table 9 and Table 11 for full-supervision and few-shot settings respectively.

A.4 Analysis of Data Construction

In addition to the defaulted way of constructing MRC examples (the first sentence in the definition article is the query, and randomly find 10 contexts for pairing 10 MRC examples), we compare with some advanced strategies to pair the query and the context, including:

Q-C Relevance: We still use the first sentence from the definition article as the query, but we only select the top P% or top P most similar contexts to the query, where the similarity score is computed as the combination of BM25 and SimCSE [17].

Table 10: We try various advanced strategies to pair the query and the context to form an MRC example. the Query and Context columns indicate how to select possible query and context for pairing. + Unanswerable indicates that PMR also uses Unanswerable examples and is also trained with L cls . Models are base-sized.

Q Diversity: In searching for an anchor, we hope the query should be diverse enough such that the model would not make a hard match between the fixed query and the anchor. Therefore, we randomly select one sentence from the first P sentences in the definition article to serve as the query for the anchor, while we keep the same context selection strategy.

C Diversity: We hope the contexts should also be diverse enough such that they provide more possible usages of an anchor. Therefore, We use K-means 6 to cluster all contexts containing the anchor into P clusters and randomly select 1 context in each cluster. Similar scores in K-means are also obtained via SimCSE.

We compare those advanced strategies with our defaulted one in Table 10, where two span extraction and sequence classification tasks are selected for evaluating the effectiveness of these strategies. First, we make a fast evaluation with only L ext without unanswerable examples (i.e. Strategy 1,2,3). Comparing Q-C Relevance (top P%) against Q-C Relevance (top P), we can observe that it is better to sample contexts based on absolute values. In Wikipedia, the reference frequency of anchor entities is extremely unbalanced, where some frequent anchor entities such as "the United States" are referenced more than 200,000 times, while other rare anchor entities are only mentioned once or twice in other articles. Therefore, Q-C Relevance (top P%) would waste too much focus on the well-learned frequent anchor entities and affect the learning of other less frequent anchor entities.

Then, when trained on both answerable and unanswerable examples as well well guided with both L cls and L ext , we only sample an absolute number of contexts. However, comparing among Strategy 4,5,6,7, no significant difference between these strategies and our random sampling is observed. We

How many solo tackles did Von Miller make at Super Bowl 50?

Gold

: five solo tackles

RoBERTa

: forced him into three turnovers, including

: context: context: context: context: context: context: context:

FewshotBART

:

PMR : five solo tackles (✓)

( × )

( × )

∅

The Broncos took an early lead in Super Bowl 50 and never trailed.  Newton  was  limited  by  Denver's  defense,  which sacked him seven times and forced him into three turnovers, including a  fumble which  they  recovered  for  a  touchdown. Denver  linebacker  Von  Miller  was  named  Super  Bowl  MVP, recording five solo tackles , 2 ½ sacks, and two forced fumbles.

F1

EM

RoBERTa

7.3

0.1

12.6

0.0

FewshotBART

0.8

0.3

PMR

17.2

10.4

Which Newton turnover resulted in seven points for Denver?

Gold

: a fumble

RoBERTa :  trailed.  Newton  was  limited  by  Denver's  defense,  which  sacked him  seven  times  and  forced  him  into  three  turnovers,  including  a  fumble which they recovered ( × )

T5-v1.1 : . context: Newton's first Super Bowl touchdown came in Super Bowl 50. context: ( × )

FewshotBART : Denver linebacker Von PMR : two forced fumbles ( × ) ( × )

suggest that the benefits from these heuristic strategies are marginal in the presence of large-scale training data. Therefore, in consideration of the implementation simplicity, we just use the Random strategy as our final PMR implementation.

A.5 Zero-shot Learning

To reveal PMR's inherent capability from its MRC-style pretraining, we show its zero-shot performance in Figure 6, where the F1 and Exact Match (EM) scores on the entire SQuAD dev set and a case study in answering several questions are presented. Without any fine-tuning, our PMR achieves 10.4 EM, whereas T5 and RoBERTa can barely provide a meaningful answer, as shown by their near-zero EM scores. In the case study, our PMR correctly answers the first question. For the second question, although PMR gives an incorrect answer, the prediction is still a grammatical phrase. In contrast, RoBERTa and T5-v1.1 always perform random extractions and generations. Such a phenomenon verifies that PMR obtains a higher-level language digest capability from the MRC-style pretraining and can directly tackle downstream tasks to some extent.

A.6 Fully-Resource Results

Table 12 compares PMR with strong approaches in full-resource settings. On EQA and NER, PMR can significantly and consistently outperform previous approaches, where PMR large achieves up to 3.7 and 2.6 F1 improvements over RoBERTa large on WNUT and SearchQA, respectively. For the base-sized models, the advantage of PMR is more obvious, i.e. 1.4 F1 over RoBERTa base . Apart from those, we also observe that: (1) PMR can also exceed strong generative approaches (i.e. UIE, T5-v1.1) on most tasks, demonstrating that the MRC paradigm is more suitable to tackle NLU tasks. (2) RoBERTa-Post, which leverages our Wikipedia corpus (a subset of its original pre-training data) for MLM-style continued-pretraining, performs poorly on most tasks, especially those with natural-question queries (i.e. EQA and MCQA). (3) PMR can be applied on even larger MLM such as ALBERT xxlarge [27] to gain stronger representation capability and further improve the performance of downstream tasks. Such findings suggest that with our MRC data format and WAE objective, PMR can leverage the same data to learn a high level of language understanding ability, beyond language representation.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- Summary:\n\nThis paper presents a novel pretraining method for span-extraction tasks, such as machine reading comprehension and NER.\nBy appending labels to the inputs, classification tasks can also be solved in span-extraction way.\n\n- Method:\n\nThis paper proposes to use the hyper links in wikipedia to construct a large scale dataset for span-extraction pretraining. By continue training a pretrained MLM model with the constructed data, the model can learn to extract entity spans given the entity description and the mention article. \n\n- Experiments:\n\nBy pretraining RoBERTa with their constructed dataset, various machine reading comprehension tasks and NER tasks are improved.\nAlso, by applying their model to test classification tasks, their model can extract the supporting span for classification.\n\nSTRENGTHS:\n- The data for pretraining is collected unsupervisely, which make the data collection process more attractive to inspire other research work. \n- The proposed method is straightforward and effective. The evaluation and results are comprehensive.\n\nWEAKNESSES:\n- The constructed dataset can also be used to finetune seq2seq model like T5 for span-extraction. But only MLM models are covered in this work.\n- This paper claims their model support both span extraction and classification tasks. But there are less classification experiments in this submission. For example, full-resource results on classification.\n\nQUESTIONS:\n1. For an article with multiple different hyperlink anchors, which one is used when building the pretraining dataset?\n2. Many downstream tasks are evaluated in this paper. Can you discuss the SOTA method/model for each of them? such as the advantage and disadvantage of the proposed method compared to these SOTA method/models?\n3. Can you discuss if there is a domain issue of the proposed method? as the pretraining data is only from wikipedia.\n\nLIMITATIONS:\nlimitations discussed in the appendix", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to retrofit pre-trained language models to pre-trained machine reading comprehension models by constructing training data from Wikipedia. This technique can alleviate the discrepancy between pre-trained models and downstream tasks, especially extraction-based tasks. Specifically, the anchors of Wikipedia articles are leveraged. The definition of the anchor entity is regarded as query while the surrounding text of the anchor is regarded as context. The PMR obtains state-of-the-art performance on several datasets of NER and EQA under few-shot setting.\n\nSTRENGTHS:\n1. The proposed pre-trained machine reader achieved state-of-the-art performance on few-shot NER task and few-shot Extractive Question Answering task. The retrofitted pre-trained machine reader has competitive OOD generalization ability as T5-v1.1. \n2. Extensive experiments were conducted to prove the effectiveness of the retrofitting method. Also, the experiments on explainability were also interesting.\n3. It is interesting to integrate the unanswerable examples into the pre-training.\n\nWEAKNESSES:\nThere are no major weaknesses. However, there are several comments here.\n1. I think that the comparison with fewshotBART is not that direct. Even though fewshotBART and PMR were both designed for bridging the gaps between pre-training and fine-tuning. However, the PMR leveraged extra training data (unlabeled) while fewshotBART improved the input-output format without further pre-training. So it seems that for extremely low-resource setting, the fewshowBART has more advantages (no need for extra training).\n2. The idea is quite similar to Splinter. In the related work, it would improve the paper if the authors can better explain the differences. E.g. These two works both use the entity recurring for constructing the dataset. Difference: For Splinter, it leverages the recurring of the original paragraph, while for PMR, a synthetic paragraph (abstract of entity + original paragraph) is constructed. \n3. It would be great if the model size of the existing work can be shown in the table 1, for a more fair comparison. E.g. in the fewshotBART paper, they used the base model (6 layers of encoder and 6 layers of decoder -- 12 layers in total) -- which should be compared to PMR_base model.\n\nQUESTIONS:\nIs simple string matching used for finding other mentions of anchor entity in text?\n\nLIMITATIONS:\nNo limitation is discussed in this work. No potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper introduces PMR, a pretrained machine reader. PMR combines a MLM and extractor modules in an MRC-style using a synthetic dataset from Wikipedia in order to bring the gap between pretrain and finetuning for span extraction tasks. Experimental results in few shot settings across various benchmarks (EQA, NER, Sequence classification) highlight PMR's effectiveness compared to similarly sized MLM models. Furthermore, PMR's performance in out-of-distribution (OOD) scenarios is demonstrated through additional experiments.

STRENGTHS:
- Motivations are clear and narrow their contributions to a specific field (Span extraction)
- The construction of a synthetic dataset from Wikipedia for MRC-style pretraining looks interesting.
- The proposed method is tested on several benchmarks.
- The paper is easy to read, and generally well-written.

WEAKNESSES:
- Authors decided not to include LLMs as baselines for generative finetuning approaches, however, there are smaller versions e.g. opt-350m, T5Flan-small, etc. 
- The paper presents experiments on EQA for a medium-big number of shots. I would suggest adding results for 4 and 8 samples to verify if the number of few shot examples affects the performance of PMR in comparison to the other baselines.
- (Minor) It will be better if the authors mention limitations in the main section of the paper as is important for readers to quickly notice the limitations of the work.

QUESTIONS:
As mentioned in the weakness:
- I would suggest adding baselines for small versions of LLMs to make the paper more solid.
- Also include experiments in smaller few-shot settings for EQA benchmarks.
- (Minor) Include limitations in the main section of the paper.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-17

comments: SUMMARY:
The paper proposes Pre-trained Machine Reader (PMR) which allows for retrofitting MLMs for better transferability to downstream tasks, especially under low-resource scenarios. PMR introduces an MRC-style head on top of MLMs and is pre-trained with a large volume of MRC-style data constructed from Wikipedia anchors and hyperlinks. The pertaining task (Wiki Anchor Extraction (WAE)) is to determine if the context and the query are relevant, and if so answer extraction from the context. PMR is evaluated on span extraction tasks like named entity recognition (NER) and extractive question answering (EQA). It achieves significantly better performance than vanilla MLMs under low-resource settings.

STRENGTHS:
The idea of using Wikipedia anchors to construct MRC-style data is interesting, and the paper is well-written. The results show that PMR performs significantly better than vanilla MLMs under low-resource settings.

WEAKNESSES:
It is unclear to me if the gains come from the WAE pretraining task itself or just having more MRC-style training data. The performance gains could stem from PMR having access to orders of magnitude more MRC examples during training with WAE rather than WAE specifically being a better task. Comparing WAE pretraining to an equivalent amount of pretraining data from existing MRC datasets would better isolate the impact of the pretraining task design and Wikipedia data.

QUESTIONS:
does WAE training return better performance on downstream tasks than training on other MRC tasks on the same amount of data ? I want to know this to see WAE pretraining task is a better task, or if gains could be matched by an equivalent amount of pretraining data using other tasks such as NER.

LIMITATIONS:
Please add a limitations section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-27

comments: SUMMARY:
To improve the performance of tasks that can be formulated as span extraction (e.g., extractive MRC and NER), especially in low-resource settings, this paper proposes to automatically construct data MRC-style data based on Wikipedia for model pre-training. Another advantage of this proposed method is that when applied to classification tasks, spans that can be used to explain the answer will be extracted for better explainability.

STRENGTHS:
1) Several backbone models are used for comparison.

2) The authors show MRC-style formulation can address several types of NLP tasks.

WEAKNESSES:
1) The proposed work focuses on tasks that can be formatted as MRC tasks. The transformation may be not efficient (e.g., several follow-up questions may be asked and answered to arrive at the final solution) or effective for tasks that involve complex structures or a large number of types such as fine-grained NER, event extraction, and text generation tasks.

2) It may be useful to explore other types of models such as decoder-only and encoder-decoder models. 

3) Evaluation: it might be better if the authors can compare the proposed method PMR with previous ones on these datasets. Are these baselines reported in Table 1 and Table 2 implemented by the authors? If not, references should be clearly stated. If the CoNLL refers to CoNLL-03, UIE only reports one/five/ten-shot results (46.43/67.09/73.9) on this dataset. This paper starts with four-shot experiments (65.7), making results not easily compared.

QUESTIONS:
Please check the above comments.

LIMITATIONS:
The proposed work focuses on tasks that can be easily formatted as MRC tasks.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-28

comments: SUMMARY:
The paper proposed a Pre-trained Machine Reader to close the gap the gap between pretraining and finetuning.

STRENGTHS:
This paper presents a comprehensive and novel method to tackle the extractive tasks by pretraining a reader from curated large corpus from wikipedia. The methodology is sound to the specific task and the results are solid and strong.

WEAKNESSES:
This paper and the methodology mostly focus on the extractive tasks instead of generative tasks, limiting its scope and applicability.

QUESTIONS:
Why does the model generalize to unseen domains? Any insights or explaination?

LIMITATIONS:
n/a
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-08-01