SlowGuess/NAACL_2021_OCR huggingface dataset:
Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020. ERNIE 2.0: A continual pre-training framework for language understanding. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8968-8975. AAAI Press.
Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, and Bing Xiang. 2019. Multi-passage BERT: A globally normalized BERT model for open-domain question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 5877-5881.
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020. Scalable zero-shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6397-6407.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020. Approximate nearest neighbor negative contrastive learning for dense text retrieval. CoRR, abs/2007.00808.
Ming Yan, Chenliang Li, Chen Wu, Bin Bi, Wei Wang, Jiangnan Xia, and Luo Si. 2019. IDST at TREC 2019 deep learning track: Deep cascade ranking with generation-based document expansion and pre-trained language modeling. In Proceedings of the Twenty-Eighth Text REtrieval Conference, TREC 2019, Gaithersburg, Maryland, USA, November 13-15, 2019, volume 1250 of NIST Special Publication.
Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the use of lucene for information retrieval research. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017, pages 1253-1256.
A.1 The Effects of Pre-trained LMs
We notice that previous work use different pre-trained LMs. As shown in Table 6, DPR (Karpukhin et al., 2020) uses  $\mathrm{BERT}_{\mathrm{base}}$ . ANCE (Xiong et al., 2020) uses RoBERTa $_{\mathrm{base}}$ , and ME-BERT (Luan et al., 2020) uses  $\mathrm{BERT}_{\mathrm{large}}$ . We mainly use ERNIE $_{\mathrm{base}}$  in our experiments. In this section, we try to examine the effects of pre-trained LMs for RocketQA. Specifically, we use  $\mathrm{BERT}_{\mathrm{base}}$  to replace ERNIE $_{\mathrm{base}}$ , and apply it to the first step of RocketQA. From Table 6 (see the forth row and the fifth row), we can observe that the performance slightly decreases when using  $\mathrm{BERT}_{\mathrm{base}}$ . In other words, comparing to  $\mathrm{BERT}_{\mathrm{base}}$ , ERNIE $_{\mathrm{base}}$  brings gains about 0.6 in terms of MRR@10 on MSMARCO, and 1.6 in terms of R@100 on NQ, respectively. However, RocketQA trained only with cross-batch negatives is already comparable to previous work, including DPR, ANCE and ME-BERT (although they employ better pre-trained LMs). We conclude that our approach is still effective when using different pre-trained LMs.
Methods	PLMs	MSMARCO
MRR@10	NQ
R@100
DPR (single)	BERTbase	-	85.4
ANCE (single)	RoBERTa base	33.0	87.5
ME-BERT	BERTlarge	33.8	-
RocketQASTEPI	BERTbase	32.7	86.0
RocketQASTEPI	ERNIEbase	33.3	87.6
RocketQA	ERNIEbase	37.0	88.5
A.2 The Effectiveness of The Three Training Strategies on NQ
In this section, we examine the effectiveness of the three proposed training strategies on NQ dataset. From Table 7, we can observe that all the three strategies are effective. The findings are similar to the results on MSMARCO.
Table 6: The effects of pre-trained LMs. Note that we directly copy the reported numbers from the original papers and leave the blanks if they were not reported.
Strategy	R@5
In-batch negatives	68.5
Cross-batch negatives (i.e. STEP 1)	68.9
Hard negatives w/o denoising	68.0
Hard negatives w/ denoising (i.e. STEP 3)	73.2
Data augmentation (i.e. STEP 4)	74.0
Table 7: The experiments to examine the effectiveness of the three proposed training strategies in RocketQA on NQ.