SlowGuess/NAACL_2022_OCR huggingface dataset:
Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. 2021a. Augmented SBERT: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 296–310, Online. Association for Computational Linguistics.
Nandan Thakur, Nils Reimers, Andreas Rückle, Abhishek Srivastava, and Iryna Gurevych. 2021b. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, et al. 2015. An overview of the bioasq large-scale biomedical semantic indexing and question answering competition. BMC bioinformatics, 16(1):138.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. ArXiv preprint, abs/1807.03748.
Ellen Voorhees. 2005. Overview of the trec 2004 robust retrieval track. Special Publication (NIST SP), National Institute of Standards and Technology, Gaithersburg, MD.
Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R. Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2021. TREC-COVID: Constructing a pandemic information retrieval test collection. SIGIR Forum, 54(1).
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534-7550, Online. Association for Computational Linguistics.
Kexin Wang, Nils Reimers, and Iryna Gurevych. 2021. TSDAE: Using transformer-based sequential denoising auto-encoder for unsupervised sentence embedding learning. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 671-688, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Doug Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Michael Kinney, Yunyao Li, Ziyang Liu, William Merrill, Paul Mooney, Dewey A. Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex D. Wade, Kuansan Wang, Nancy Xin Ru Wang, Christopher Wilhelm, Boya Xie, Douglas M. Raymond, Daniel S. Weld, Oren Etzioni, and Sebastian
Kohlmeier. 2020. CORD-19: The COVID-19 open research dataset. In Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020, Online. Association for Computational Linguistics.
Ji Xin, Chenyan Xiong, Ashwin Srinivasan, Ankita Sharma, Damien Jose, and Paul N. Bennett. 2021. Zero-shot dense retrieval with momentum adversarial domain invariant representations. ArXiv preprint, abs/2110.07581.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
A Performance of Using Different Retrievers for Negative Mining in GPL
The performance of using different retrievers (BM25, dense, BM25 + dense and single dense retrievers) for mining hard negatives in GPL is shown in Table 6. The results show GPL performs best when using hard negatives mined by both the two dense retrievers.
Dataset
Method	FiQA	SciFact	BioASQ	TRECC.	CQADup.	Robust04	Avg.
GPL (w/ BM25 + dense)	32.9	64.4	61.1	68.6	33.8	41.3	50.4
GPL (w/ BM25)	31.1	60.9	57.8	67.5	33.5	35.9	47.8
GPL (w/ dense)	32.8	66.4	61.0	72.6	34.5	41.4	51.5
GPL (w/ msmarco-distilbert-base-v3)	32.1	64.7	60.9	70.8	34.3	41.5	50.7
GPL (w/ msmarco-MiniLM-L-6-v3)	32.7	64.6	61.7	69.7	35	40.4	50.7
MS MARCO	26.7	57.1	52.9	66.1	29.6	39.0	45.2
Table 6: Performance (nDCG@10) of using different retrievers for hard-negative mining in GPL. The scores of the baseline MS MARCO and the scores of GPL with dense retrievers are copied from Table 1. "Dense" represents using both of the two dense retrievers msmarco-distilbert-base-v3 and msmarco-MiniLM-L-6-v3.
B Performance of the Zero-Shot Retrievers in Hard-Negative Mining
The performance of directly using the zero-shot retrievers for hard-negative mining in GPL is shown in Table 7. Compared with the strong baseline (MS MARCO in Table 7) trained with MarginMSE, msmarco-distilbert-base-v3 and msmarco-MiniLM-L-6-v3 are much worse in terms of zero-shot generalization on each dataset. This comparison supports GPL can indeed train powerful domain-adapted dense retrievers with minimum reliance on choices of the retrievers for hard-negative mining.
Dataset
Method	FiQA	SciFact	BioASQ	TRECC.	CQADup.	Robust04	Avg.
msmarco-distilbert-base-v3	24.0	52.3	45.6	61.1	24.3	30.6	39.7
msmarco-MiniLM-L-6-v3	23.3	48.8	41.9	57.9	24.3	28.5	37.5
MS MARCO	26.7	57.1	52.9	66.1	29.6	39.0	45.2
Table 7: Performance (nDCG@10) of different zero-shot retrievers. msmarco-distilbert-base-v3 and msmarco-MiniLM-L-6-v3 are used in GPL for hard-negative mining. The scores of the baseline MS MARCO are copied from Table 1.
C Target Datasets
FiQA is for the task of opinion question answering over financial data. It contains 648 queries and 5.8K passages from StackExchange posts under the Investment topic in the period between 2009 and 2017. The labels are binary (relevant or irrelevant) and there are 2.6 passages in average labeled as relevant for each query.
SciFact is for the task of verifying scientific claims using evidence from the abstracts of the scientific papers. It contains 300 queries and 5.2K passages built from S2ORC (Lo et al., 2020), a publicly-available corpus of millions of scientific articles. The labels are binary and there are 1.1 passages in average labeled as relevant for each query.
BioASQ is for the task of biomedical question answering. It originally contains 500 queries and 15M articles from PubMed<sup>15</sup>. The labels are binary and it has 4.7 passages in average labeled as relevant for each query. For efficient training and evaluation, we randomly remove irrelevant passages to make the final corpus size to 1M.
TREC-COVID is an ad-hoc search challenge for scientific articles related to COVID-19 based on the CORD-19 dataset (Wang et al., 2020). It originally contains 50 queries and 171K documents. The original corpus has many documents with only a title and an empty body. We remove such documents and the
15https://pubmed.ncbi.nlm.nih.gov/
final corpus size is 129.2K. The labels in TREC-COVID are 3-level (i.e. 0, 1 and 2) and there are 430.8 passages in average labeled as 1 or 2 in the clean-up version.
CQADupStack is a dataset for community question-answering, built from 12 StackExchange subforums: Android, English, Gaming, Gis, Mathematica, Physics, Programmers, Stats, Tex, Unix, Webmasters and WordPress. The task is to retrieve duplicate question posts with both a title and a body text given a post title. It has 13.1K queries and 457.2k passages. The labels are binary and there are 1.4 passages in average labeled as relevant for each query. As in Thakur et al. (2021b), the average score of the 12 sub-tasks is reported.
Robust04 is a dataset for news retrieval focusing on poorly performing topics. It has 249 queries and 528.2K passages. The labels are 3-level and there are in average 69.9 passages labeled as relevant for each query.
The detailed statistics of these target datasets are shown in Table 8.
Statistics Dataset	Domain	Title	Relevancy	#Queries	#Passages	PPQ	Query Len.	Passage Len.
FiQA	Financial	X	Binary	648	57.6K	2.6	10.8	132.2
SciFact	Scientific	✓	Binary	300	5.2K	1.1	12.4	213.6
BioASQ	Bio-Medical	✓	Binary	500	1.0M	4.7	8.1	204.1
BioASQ*	Bio-Medical	✓	Binary	500	14.9M	4.7	8.1	202.6
TREC-COVID	Bio-Medical	✓	3-Level	50	129.2K	430.8	10.6	210.3
TREC-COVID*	Bio-Medical	✓	3-Level	50	171.3K	493.5	10.6	160.8
CQADupStack	Forum	✓	Binary	13,145	457.2K	1.4	8.6	129.1
Robust04	News	X	3-Level	249	528.2K	69.9	15.3	466.4
Table 8: Statistics of the target datasets used in the experiments. Column Title indicates whether there is (✓) a title for each passage or not (X). Column PPQ represents number of Passages Per Query. Query/passage lengths are counted in words. Symbol * marks the original version from the BeIR benchmark (Thakur et al., 2021b)
We also evaluate the models trained in this work on the original version of BioASQ and TREC-COVID datasets from BeIR (Thakur et al., 2021b). The results are shown in Table 9.
D Results on full BeIR
We also evaluate the models on all the 18 BeIR datasets. We include DocT5Query (Nogueira et al., 2019a), the strong baseline based on document expansion with the T5 query generator (also used in GPL for query generation) + BM25 (Anserini). We also include the powerful zero-shot model TAS-B (Hofstätter et al., 2021), which is trained on MS MARCO with advanced knowledge-distillation techniques into comparison. Viewing TAS-B as the base model and also the negative miner, we apply QGen and GPL on top of them, resulting in TAS-B + QGen and TAS-B + GPL, resp.
The results are shown in Table 9. We find both DocT5Query and BM25 (Anserini) outperform MS MARCO, TSDAE and QGen, in terms of both average performance and average (performance) rank. QGen struggles to beat MS MARCO, the zero-shot baseline and it even significantly harms the performance on many datasets, e.g. TREC-COVID, FEVER, HotpotQA, NQ. Thakur et al. (2021b) also observes the same issue, claiming that the bad generation quality on these corpora is the key to the failure of QGen. On the other hand, GPL significantly outperforms these baselines above, achieving average performance rank 5.2 and can consistently improve the performance over the zero-shot model on all the datasets. For TSDAE, TSDAE + QGen and TSDAE + GPL, the conclusion remains the same as in the main paper.
For the powerful zero-shot model TAS-B, it outperforms QGen and performs on par with TSDAE + QGen. When building on top of TAS-B, GPL can also yield significant performance gain by up-to 21.5 nDCG@10 points (on TREC-COVID) and 4.6 nDCG@10 points on average. This TAS-B + GPL model performs the best over all these retriever models, achieving the averaged performance rank equal to 3.2. However, when applying QGen on top of TAS-B, it cannot improve the overall performance but also harms the individual performance on many datasets, instead.
Method Dataset	BM25 (Anserini)	DocT5-Query	MS MARCO	TSDAE	QGen	TSDAE + QGen (Ours)	GPL	TSDAE + GPL	TAS-B	TAS-B + QGen	TAS-B + GPL	BM25 + CE (Upperbound)
FiQA	23.6	29.1	26.7	29.3	28.7	31.4	32.8	34.4	29.8	30.1	34.4	34.7
SciFact	66.5	67.5	57.1	62.8	63.8	66.7	66.4	68.9	63.5	65.3	67.4	68.8
BioASQ*	46.5	43.1	33.6	37.3	36.9	38.5	41.2	40.9	36.2	38.5	44.2	52.3
TRECC.*	65.6	71.3	66.1	70.8	56.0	58.4	71.8	74.9	48.5	56.6	70.0	75.7
CQADup.	29.9	32.5	29.6	31.8	33.0	35.3	34.5	35.1	31.5	33.7	35.7	37.0
Robust04	40.8	43.7	39.0	39.4	38.1	37.4	41.4	43.0	42.4	39.4	43.7	47.5
ArguAna	41.4†	46.9†	33.9	37.5	52.4	54.7	48.3	51.2	43.4	51.8	55.7	41.7†
Climate-F.	21.3	20.1	20.0	16.8	22.5	22.6	22.7	22.2	22.1	24.4	23.5	25.3
DBPedia	31.3	33.1	34.2	35.4	33.1	33.2	36.1	36.1	38.4	32.7	38.4	40.9
FEVER	75.3	71.4	76.5	64.0	63.8	64.2	77.9	78.6	69.5	63.9	75.9	81.9
HotpotQA	60.3	58.0	55.4	63.8	51.4	52.2	56.5	57.2	58.4	52.0	58.2	70.7
NFCorpus	32.5	32.8	27.7	31.2	31.4	33.7	34.2	33.9	31.9	33.4	34.5	35.0
NQ	32.9	39.9	45.6	47.1	35.4	34.6	46.7	47.1	46.3	36.3	48.3	53.3
Quora	78.9	80.2	81.2	83.3	85.0	85.7	83.2	83.1	83.5	85.3	83.6	82.5
SciDocs	15.8	16.2	13.6	15.4	15.5	17.1	16.1	16.8	14.9	16.4	16.9	16.6
Signal-1M	33.0	30.7	24.4	25.9	26.8	26.8	26.5	27.6	28.9	26.6	27.6	33.8
TRECN.	39.8	42.0	36.0	35.0	36.0	38.3	40.7	41.5	37.7	38.0	42.1	43.1
Touché20	36.7	34.7	19.6	21.8	17.1	17.2	23.1	23.5	16.2	17.5	25.5	27.1
Avg.	42.9	44.1	40.0	41.6	40.4	41.6	44.5	45.3	41.3	41.2	45.9	48.2
Avg. Rank	7.6	6.2	9.8	8.2	8.9	6.5	5.2	4.2	7.8	7.3	3.2	2.4
Table 9: Performance (nDCG@10) on all the original 18 BeIR datasets. The results of MS MARCO, TSDAE, QGen, TSDAE + QGen, GPL and TSDAE + GPL on FiQA, SciFact, CQADupStack and Robust04 are copied from Table 1. The results of BM25, DocT5Query and BM25 + CE come from Thakur et al. (2021b). † marks correction over the original scores, where identical IDs between queries and passages are removed. TRECN. is short for TREC-NEWS. Avg. Rank is the average over the rank of the performance on each dataset over the different models (the lower, the better).
E Performance of Unsupervised Pre-Training
The performance of the unsupervised pre-training methods without access to the MS MARCO data is shown in Table 10. We find ICT is the best method, achieving highest scores on all the datasets. However, all the unsupervised pre-training methods cannot directly yield improvement in performance compared with the zero-shot baseline.
Dataset Method	FiQA	SciFact	BioASQ	TRECC.	CQADup.	Robust04	Avg.
CD	6.6	0.6	0.3	9.8	8.1	3.8	4.9
CT	0.2	0.7	0.0	2.5	0.9	0.0	0.7
MLM	5.4	27.8	4.7	16.0	8.5	6.1	11.4
TSDAE	7.8	37.2	6.9	9.4	14.3	10.1	14.3
SimCSE	5.5	25.0	13.1	26.0	14.6	9.8	15.7
ICT	10.2	42.6	39.0	47.5	23.0	16.5	29.8
MS MARCO	26.7	57.1	52.9	66.1	29.6	39.0	45.2
Table 10: Performance (nDCG@10) of unsupervised pre-training methods with only access to the target corpus as the training data. The scores of the zero-shot baseline MS MARCO are copied from Table 1.
F Examples of Generated Queries under Different Temperatures
The generation temperature controls the sharpness of the next-token distribution. The examples for one passage from FiQA are shown in Table 11 Higher temperature results in longer and less duplicate queries under more risk of generating non-sense texts.
Item	Text	Pseudo Label
Input Passage	You can never use a health FSA for individual health insurance premiums. Moreover, FSA plan sponsors can limit what they are will to reimburse. While you can’t use a health FSA for premiums, you could previously use a 125 cafeteria plan to pay premiums, but it had to be a separate election from the health FSA. However, under N. 2013-54, even using a cafeteria plan to pay for individual premiums is effectively prohibited.	-
Temperature 0.1	can you use a cafeteria plan for premiums	9.1
can you use a cafeteria plan for premiums	9.1
can you use a cafeteria plan for premiums	9.1
Temperature 1.0	can i use my fsa to pay for a health plan	9.7
can i use my health fsa for an individual health plan?	9.9
can fsa pay premiums	9.2
Temperature 3.0	cafe a number cafe plan is used by	-10.5
what type of benefits do the health savings accounts cover when applying for medical terms health insurance	-7.2
why can’t an individual file medical premium on their insurance account with an fsa plan instead of healthcare policy.	6.0
Temperature 5.0	which one does not apply after an emergency medical	-11.1
is medicare cafe used exclusively as plan funds (health savings account	-7.2
how soon to transfer coffee bean fses to healthcare	-11.0
Temperature 10.0	will employer limit premiums reimbursement on healthcare expenses with caeatla cafeturil and capetarians account on my employer ca. plans and deductible accounts a.f, haaq and asfrhnta,	-2.5
kfi what is allowed as personal health account or ca	-10.2
do people put funds back to buy plan plans before claiming an deductible without the provider or insurance cover f/f associator funds of the person you elect? healthfin depto of benefit benefits deduct all oe premiumto payer for individual care	-4.5
Table 11: Examples of generated queries under different temperature value for a passage from FiQA.