SlowGuess/NAACL_2021_OCR huggingface dataset:
HotpotQA (Yang et al., 2018) is an extractive question answering dataset, which requires multi-hop reasoning over multiple supporting documents for answering 113k questions. We adopt the full-wiki setting for HotpotQA, which requires models to find answers from a large-scale corpus. We employ the retriever and re-ranker in Transformer-XH (Zhao et al., 2020) to obtain question-related paragraphs and merge them into one document. In HotpotQA, models are required to reason over bridge entities or check multiple properties in different positions, which brings challenges to the token selection of TR-BERT in considering the global information. We concatenate two positive paragraphs and several negative paragraphs to make the reading material for training, which contains up to 1,024 tokens. And we concatenate the re-rank passages in order for evaluating. For the vanilla BERT, we apply a shared-normalization training objective (Clark and Gardner, 2018) to produce a global answer candidate comparison across two parts of the document.
TriviaQA (Joshi et al., 2017) has more than  $95\mathrm{k}$  question-answer pairs authored by Trivia enthusiasts. We use the Wikipedia setting of TriviaQA, which provides question-retrieved paragraphs from Wikipedia. We use the linear passage re-ranker in DocQA (Clark and Gardner, 2018) to re-rank these retrieved paragraphs and finally concatenate the first 1,024 tokens as a new reading document. We also employ the shared-normalization training objective (Clark and Gardner, 2018) for the vanilla BERT.
WikiHop (Welbl et al., 2018) consists of 51k questions, candidate answers, and supporting paragraphs triples. It requires models to find multi-hop reasoning chains for choosing the correct answer. Due to the long length of the concatenation of supporting paragraphs, we follow the processing tactic in Longformer (Beltagy et al., 2020), which splits the document into several parts and then averages their candidate scores.
YELP.F (Zhang et al., 2015) contains 1,569k samples with review texts, which are obtained from the YELP Dataset Challenge in 2015. Yelp Review Full (YELP.F) contains five star classes.
20NewsGroups (Lang, 1995) comprises around 18k newsgroups posts on 20 topics.
IMDB (Maas et al., 2011) consists of  $50\mathrm{k}$  informal movie reviews from the Internet Movie Database. Each review is annotated as positive or negative.
Hyperpartisan (Kiesel et al., 2019) aims to decide whether a news article text follows a hyperpartisan argumentation. Hyperparisan only contains 645 documents, which makes it a good testbed in a low-resource scenario. For the vanilla BERT, we adopt the max-pooling results of BERT sliding window. We split the data into training / validation / test set with a ratio of 9:1:1, run each model five times, and report the median performance.
B Training Configuration
We follow the configuration from previous work (Devlin et al., 2019; Beltagy et al., 2020; Liu et al., 2020) for fine-tuning BERT, DistilBERT and  $\mathrm{BERT_L}$ . Hyperparameters are shown in Table 5.
To train TR-BERT, we first initialize TR-BERT with corresponding fine-tuned models, which are trained with a task-specific objective for  $N$  epochs. After that, we maintain the same learning rate, warmup proportion and batch size for the latter two-step training: (1) Freeze all the parameters
Dataset	Epoch	LR	WP	BSZ	Optimizer
SQuAD	2	3e-5	10%	32	Adam
NewsQA	5	3e-5	10%	32	Adam
NaturalQA	5	3e-5	10%	32	Adam
RACE	5	5e-5	10%	32	Adam
HotpotQA	6	5e-5	7.3%	32	Adam
TriviaQA	5	3e-5	10.6%	32	Adam
WikiHop	15	3e-5	1%	32	Adam
YELPF	3	3e-5	10%	32	Adam
20NewsGroups	5	3e-5	10%	32	Adam
IMDB	5	3e-5	10%	32	Adam
Hyperparisan	15	3e-5	10%	32	Adam
Table 5: Hyperparameters of all the models in different datasets. LR: Learning rate; BSZ: Batch size; WP: Warmup proportion.
except that of the policy network and conduct reinforcement learning to update the policy network for  $\lceil (N + 1) / 2\rceil$  epochs; (2) Unfreeze all parameters and train the entire network with the task-specific knowledge distillation objective and the reinforcement learning objective simultaneously for  $N$  epochs.
C Actual Wall Time