SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We compare those advanced strategies with our defaulted one in Table 10, where two span extraction and sequence classification tasks are selected for evaluating the effectiveness of these strategies. First, we make a fast evaluation with only  $L_{ext}$  without unanswerable examples (i.e. Strategy 1,2,3). Comparing Q-C Relevance (top P%) against Q-C Relevance (top P), we can observe that it is better to sample contexts based on absolute values. In Wikipedia, the reference frequency of anchor entities is extremely unbalanced, where some frequent anchor entities such as "the United States" are referenced more than 200,000 times, while other rare anchor entities are only mentioned once or twice in other articles. Therefore, Q-C Relevance (top P%) would waste too much focus on the well-learned frequent anchor entities and affect the learning of other less frequent anchor entities.
Then, when trained on both answerable and unanswerable examples as well well guided with both  $L_{cls}$  and  $L_{ext}$ , we only sample an absolute number of contexts. However, comparing among Strategy 4,5,6,7, no significant difference between these strategies and our random sampling is observed. We
Table 10: We try various advanced strategies to pair the query and the context to form an MRC example. the Query and Context columns indicate how to select possible query and context for pairing. + Unanswerable indicates that PMR also uses Unanswerable examples and is also trained with  $L_{cls}$ . Models are base-sized.
Dataset	EQA	NER
Query Length	64	32
Input Length	384	192
Batch Size	12	12
Learning Rate	{5e-5,1e-4}	{5e-5,1e-4}
Max Epochs/Steps	12/200	20/200
Table 11: Hyper-parameters settings in fine-tuning downstream tasks in few-shot settings.
$^{6}$ https://github.com/subhadarship/kmeans_pytorch
The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five solo tackles, 2½ sacks, and two forced fumbles.	F1	EM
RoBERTa	7.3
T5-v1.1	12.6
FewshotBART	0.8
PMR	17.2
1. How many solo tackles did Von Miller make at Super Bowl 50?
Gold: five solo tackles
RoBERTa: forced him into three turnovers, including (X)
T5-v1.1: context: context: context: context: context: context: (X)
FewshotBART: ∅
PMR: five solo tackles (✓)
2. Which Newton turnover resulted in seven points for Denver?
Gold: a fumble
RoBERTa: trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him into three turnovers, including a fumble which they recovered (X)
T5-v1.1: . context: Newton's first Super Bowl touchdown came in Super Bowl 50. context: (X)
FewshotBART: Denver linebacker Von (X)
PMR: two forced fumbles (X)
Figure 6: Zero-shot performance on SQuAD and a case study. The F1/EM scores are shown in the left-top corner.
suggest that the benefits from these heuristic strategies are marginal in the presence of large-scale training data. Therefore, in consideration of the implementation simplicity, we just use the Random strategy as our final PMR implementation.
A.5 Zero-shot Learning
To reveal PMR's inherent capability from its MRC-style pretraining, we show its zero-shot performance in Figure 6, where the F1 and Exact Match (EM) scores on the entire SQuAD dev set and a case study in answering several questions are presented. Without any fine-tuning, our PMR achieves 10.4 EM, whereas T5 and RoBERTa can barely provide a meaningful answer, as shown by their near-zero EM scores. In the case study, our PMR correctly answers the first question. For the second question, although PMR gives an incorrect answer, the prediction is still a grammatical phrase. In contrast, RoBERTa and T5-v1.1 always perform random extractions and generations. Such a phenomenon verifies that PMR obtains a higher-level language digest capability from the MRC-style pretraining and can directly tackle downstream tasks to some extent.
A.6 Fully-Resource Results
Table 12 compares PMR with strong approaches in full-resource settings. On EQA and NER, PMR can significantly and consistently outperform previous approaches, where  $\mathrm{PMR}_{\text{large}}$  achieves up to 3.7 and 2.6 F1 improvements over RoBERTa $_{\text{large}}$  on WNUT and SearchQA, respectively. For the base-sized models, the advantage of PMR is more obvious, i.e. 1.4 F1 over RoBERTa $_{\text{base}}$ . Apart from those, we also observe that: (1) PMR can also exceed strong generative approaches (i.e. UIE, T5-v1.1) on most tasks, demonstrating that the MRC paradigm is more suitable to tackle NLU tasks. (2) RoBERTa-Post, which leverages our Wikipedia corpus (a subset of its original pre-training data) for MLM-style continued-pretraining, performs poorly on most tasks, especially those with natural-question queries (i.e. EQA and MCQA). (3) PMR can be applied on even larger MLM such as  $\mathrm{ALBERT}_{\text{xxlarge}}$  [27] to gain stronger representation capability and further improve the performance of downstream tasks. Such findings suggest that with our MRC data format and WAE objective, PMR can leverage the same data to learn a high level of language understanding ability, beyond language representation.
EQA	Size	Unified	SQuAD	NewsQA	TriviaQA	SearchQA	HotpotQA	NQ	Avg.
RBT-Postlarge	355M	✘	93.0	70.9	80.9	86.8	79.8	79.9	81.9
SpanBERTlarge [20]	336M	✘	93.1	72.3	78.1	83.2	80.9	82.3	81.7
LUKElarge [64]	483M	✘	94.5	72.1	NA	NA	81.9	83.3	-
T5-v1.1large [44]	800M	△	93.9	69.8	77.8	87.1	81.9	81.6	82.0
RoBERTabase	125M	✘	91.2	69.0	79.3	85.0	77.9	79.7	80.4
PMRbase (OURS)	125M	✓	92.1	71.9	81.5	86.4	80.6	81.0	82.3
RoBERTalarge	355M	✘	94.2	73.8	85.1	85.7	81.6	83.3	84.0
PMRlarge (OURS)	355M	✓	94.5	74.0	85.1	88.3	83.6	83.8	84.9
ALBERTxxlarge	223M	✘	94.7	75.3	86.0	89.4	83.8	83.8	85.5
PMRxlarge (OURS)	223M	✓	95.0	75.4	86.7	89.6	84.5	84.8	86.0
NER	Size	Unified	CoNLL	WNUT	ACE04	ACE05	Avg.
Robertalarge+Tagging [36]	355M	✘	92.4	55.4	-	-	-
RBT-Postlarge	355M	✘	92.7	53.8	86.6	86.2	79.8
SpanBERTlarge	336M	✘	90.3	47.2	86.4	85.4	77.3
LUKElarge [64]	483M	✘	92.4†	55.2†	-	-	-
CL-KLlarge [59]	550M	✘	93.2†	59.3†	-	-	-
BARTNERlarge [65]	406M	✘	93.2‡	-	86.8‡	84.7‡	-
T5-v1.1large [44]	800M	✓	90.5	46.7	83.9	82.8	76.0
UIELarge [38]	800M	✓	93.2▲	52.5	86.9▲	85.8▲	79.6
RoBERTabase	125M	✘	92.3	53.9	85.8	85.2	79.3
PMRbase (OURS)	125M	✓	93.1	57.6	86.1	86.1	80.7
RoBERTalarge	355M	✘	92.6	57.1	86.3	87.0	80.8
PMRlarge (OURS)	355M	✓	93.6	60.8	87.5	87.4	82.3
ALBERTxxlarge	223M	✘	92.8	54.0	86.8	87.7	80.3
PMRxlarge (OURS)	223M	✓	93.2	58.3	88.4	87.9	82.0
Table 12: Performance on EQA (F1), and NER (F1). The best models are bolded. For EQA, as done in MRQA [15], we report the F1 on dev set and produce the results of SpanBERT and LUKE following the same protocol. Although we try hard to produce the results of LUKE for TriviaQA and SearchQA, its performance is unreasonably low. For CoNLL, we assume there is no additional context available and therefore we retrieve the results of CL-KL w/o context from [59]. Results labeled by  $\dagger$ ,  $\ddagger$ , and  $\spadesuit$  are cited from [59, 65, 38], respectively.