SlowGuess/NAACL_2022_OCR huggingface dataset:
Sanjay Subramanian and Dan Roth. 2019. Improving generalization in coreference resolution via adversarial training. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM* 2019), pages 192-197, Minneapolis, Minnesota. Association for Computational Linguistics.
Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018. What makes reading comprehension questions easier? In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4208-4219, Brussels, Belgium. Association for Computational Linguistics.
Saku Sugawara, Pontus Stenetorp, Kentaro Inui, and Akiko Aizawa. 2020. Assessing the benchmarking capacity of machine reading comprehension datasets. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8918-8927.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5754-5764.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Computational Linguistics.
Qinyuan Ye, Belinda Z Li, Sinong Wang, Benjamin Bolte, Hao Ma, Wen-tau Yih, Xiang Ren, and Madian Khabsa. 2021. On the influence of masking policies in intermediate pre-training. ArXiv preprint, abs/2104.08840.
Xiaorui Zhou, Senlin Luo, and Yunfang Wu. 2020. Coattention hierarchical network: Generating coherent long distractors for reading comprehension. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9725-9732.
A Details for MRC Model Training
We train all MRC models using mixed precision, with batch size of 16 sequences for 4 epochs. The maximum sequence length is set to 256 tokens. We use the AdamW optimizer (Loshchilov and Hutter, 2019) with an initial learning rate of 2e-5 that is linearly decayed to 0 during finetuning.
B Perturbable Span Identification
For PER, we only consider names with one or two words. A one-word name is considered as a first name, while a two-word name is considered as a full name, with the first word being the first name and the second word being the last name. We infer the gender of the detected name to be male, female, or neutral with gender-guesser<sup>5</sup>.
For GPE, we detect its contained country names, state names, and city names by string matching with the Countries States Cities Database<sup>6</sup>.
For ORG, besides mentions of GPE names, we include two additional types of perturbable words identified using Penn Treebank (PTB) (Marcus et al., 1993). Words that are annotated as NNP(S) for more than  $90\%$  of the time in PTB are considered as proper nouns (dented as  $\langle \mathrm{NNP}\rangle$ ), which are usually specialized for naming an entity. Words outside PTB are considered as rare words (denoted as  $\langle \mathrm{Rare}\rangle$ ), which are likely to be invented by people to name an entity. These two kinds of words are weakly related to the characteristics of the entity and thus can be flexible.
C Evaluation of Perturbation Quality
The accuracy of step 2 is evaluated based on whether the perturbable spans and their corresponding span types are all correct for an instance, which also implies the quality of step 1 ("Answer Entity Recognition") as different entity types have different applicable span types.
The accuracy of step 4 is evaluated based on whether the string mapping function successfully locates all mentions of the perturbable spans in the passage to perform string mapping.
The quality of step 3 can be inferred from the accuracy of step 2 for InDistName perturbation. For DBName, we assume the database is of acceptable quality in the sense that all names it provides belongs to the correct span type, which is guaranteed by the source of the data—PTB is annotated
by human experts, U.S. names come from official statistics, and GPE names are actively maintained by its creator and the community for more than 3 years. RandStr is proposed to simulate the extreme case, and we therefore do not evaluate its quality.
D Details for Continual Pretraining
MLM (vanilla) refers to the masking strategy used by BERT (Devlin et al., 2019), where the masked tokens are randomly sampled. MLM (whole word) always masks all tokens corresponding to a word at once. MLM (span) uses the masking strategy proposed by Joshi et al. (2020), which masks random spans rather than individual whole words or tokens. MLM (entity) masks a random entity for  $50\%$  of the time, and uses MLM (span) for the other  $50\%$  of the time. The idea is inspired by salient span masking proposed in Guu et al. (2020). All strategies mask  $15\%$  of the training tokens in total.
To eliminate domain shift during continual pretraining as a possible explanation for any improvements, we keep the corpus for continual pretraining consistent with the pretraining corpus used by SpanBERT, which is the concatenation of BookCorpus and English Wikipedia. We train using mixed precision, with effective batch size of 2,048 sequences for 8,000 steps, with 256 tokens per sequence. We use the AdamW optimizer with a constant learning rate of 1e-4.
5https://pypi.org/project/gender-guesser/
$^{6}$ https://countrystatecity.in/