SlowGuess/NAACL_2021_OCR huggingface dataset:
Shuohang Wang, Mo Yu, Jing Jiang, Wei Zhang, Xiaoxiao Guo, Shiyu Chang, Zhiguo Wang, Tim Klinger, Gerald Tesauro, and Murray Campbell. 2018b. Evidence aggregation for answer re-ranking in open-domain question answering. In International Conference on Learning Representations (ICLR).
Yizhong Wang, Kai Liu, Jing Liu, Wei He, Yajuan Lyu, Hua Wu, Sujian Li, and Haifeng Wang. 2018c. Multi-passage machine reading comprehension with cross-passage answer verification. In Association for Computational Linguistics (ACL).
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020. Scalable zero-shot entity linking with dense entity retrieval. In Empirical Methods in Natural Language Processing (EMNLP).
A Computing Infrastructure Used
All experiments were run on a machine with 2 chips of Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz with 20 cores (40 threads) each, equipped with 8 NVIDIA TESLA V100 GPUs, each with 32 GB of memory.
B Average Run-time and #Parameters
We report average run-times for training and inference on NQ (TRIVIAQA is similar), as well as number of model parameters, in Table 4. WEBQ and TREC are much smaller datasets and have lower runtimes.
Model	Params	Train	Inf.
DPR-BERTlarge	335M	37 h	2.8 h
RECONSIDERbase	109M	13 h	2 m
RECONSIDERlarge	335M	28 h	2 m
Table 4: Runtime for training and inference, and number of parameters of the models that we executed, on NQ. Runtimes for RECONSIDER do not include the time required to train and obtain predictions from DPR.
Table 8 presents validation set performance for the experiments that we ran for this paper.
For training RECONSIDER, we use top-100 predictions of the baseline MRC model. This was chosen based on validation set accuracy, and other values that were experimented with were 50 and 75. For training RECONSIDER we use 1 positive and  $M - 1$  negatives during each iteration. We tried values of  $M$  between 5 and 40 in increments of 5 and chose  $M = 30$  based on validation set accuracy (see Table 6). Similarly, we re-rank  $K = 5$  candidates
K	Accuracy (%)
3	41.54
5	42.50
10	42.26
15	42.03
20	41.91
Table 5: Hyper-parameter tuning for  $K$  at inference time on the validation set of NQ.
Train-M	Accuracy (%)
25	42.34
30	42.48
40	41.25
Table 6: Hyper-parameter tuning for the number of negative passages i.e. train-  $M$  ,on the validation set of NQ.
during inference, and this value was chosen by experimenting with values 2, 3, 4 and values between 5 and 20, in increments of 5 (see Table 5).
Dataset	Train	Dev	Test
NQ	67,098	8,757	3,610
TRIVIAQA	67,975	8,837	11,313
WEBQ	2,898	361	2,032
TREC	1,240	133	694
Table 7: Training, validation and testing set sizes for the four open-domain QA tasks evaluated in our paper.
Table 7 presents the number of examples in the training, validation and testing splits of the four open-domain QA datasets that we use, based on the dataset prepared by Karpukhin et al. (2020).
Model	NQ	TRIVIAQA	WEBQ	TREC
RECONSIDERbase (Ours)	42.5	60	46.3	49.6
DPR-BERTlarge (Karpukhin et al., 2020)	42.2	60.1	43.8	54.9
RECONSIDERlarge (Ours)	44.2	61.7	46	54.9
Table 8: Validation set performance for our experiments.