SlowGuess/NAACL_2022_OCR huggingface dataset:
Wizard of Wikipedia [Dinan et al., 2018] is the dialog dataset. The input is a short dialog history ending with the information seeker's turn. The expected output is a fact presented conversationally or just an utterance or question mentioning content from a relevant Wikipedia page. It is the smallest dataset with approximately 3k instances in development and test and 64k in train.
For all tasks, systems are expected to produce the target output as well as justify it with provenance information from the KILT knowledge source. The metrics of R-Precision and Recall@5 measure the correctness of the provenance. R-Precision measures what fraction of the  $R$  documents in the ground truth provenance  $(|\mathbf{Prov}| = R)$  are present in the top- $R$  documents returned by the system. Accuracy and (token-level) F1 measure the cor
rectness of the generated output. For Wizard of Wikipedia, Rouge-L [Lin, 2004] is used instead of accuracy, since systems are very unlikely to generate the exact target output. The metrics of KILT-Accuracy, KILT-F1 and, for Wizard of Wikipedia, KILT-Rouge-L are the underlying metric (e.g. Accuracy) for instances where R-Precision is one, otherwise zero. These metrics indicate output correctness when provenance is also correctly supplied.
Table 1 shows the performance of  $\mathrm{Re}^2\mathrm{G}$  on the KILT leaderboard. We achieved  $9\%$ ,  $31\%$ ,  $34\%$ ,  $22\%$  and  $10\%$  relative gains over the previous state-of-the-art on the headline KILT metrics for T-REx, Natural Questions, TriviaQA, FEVER, and Wizard of Wikipedia, respectively. Furthermore,  $\mathrm{Re}^2\mathrm{G}$  has held the lead in the headline KILT metrics in all datasets except for Wizard of Wikipedia where it is now second best.
Since our submission to the KILT leaderboard for the Wizard of Wikipedia, a new system called Hindsight [Paranjape et al., 2021] achieved even better results on the generation metrics on that particular task. The new system of SEAL has also achieved top results for some metrics on the Natural Questions and TriviaQA benchmarks.
Table 2 examines how the retrieval improves through each step of training. In the first half of the table we consider the initial retrieval alone. DPR Stage 1 is the DPR training described earlier - training only from the provenance ground truth with batch negatives and hard negatives from BM25.  $\mathrm{KGI}_0$  further trains the query encoder of DPR Stage 1 through its impact in generating the target output. Finally  $\mathrm{Re}^2\mathrm{G}$  extends the training of DPR with online knowledge distillation from the reranker. This step is beneficial in two of the three datasets, while the previous steps improve performance across all datasets.
In the second half of the table we examine the improvement in reranking. The baseline of  $\mathrm{KGI_0}$  DPR+BM25 merges the results of  $\mathrm{KGI_0}$ 's DPR and BM25 by scoring each passage by the sum of the inverse rank from each method. For both T-REx and FEVER, even this simple approach to assembling DPR and BM25 improves Recall@5, although not R-Precision. Following reranker training using the provenance ground truth (Reranker Stage 1), we find improvement over DPR across all five datasets on both retrieval metrics. The reranker's improve
T-REx	NQ	TriviaQA	FEVER	WoW
R-Prec	R@5	R-Prec	R@5	R-Prec	R@5	R-Prec	R@5	R-Prec	R@5
BM25	46.88	69.59	24.99	42.57	26.48	45.57	42.73	70.48	27.44	45.74
DPR Stage 1	49.02	63.34	56.64	64.38	60.12	64.04	75.49	84.66	34.74	60.22
KGI0 DPR	65.02	75.52	64.65	69.60	60.55	63.65	80.34	86.53	48.04	71.02
Re2G DPR	67.16	76.42	65.88	70.90	62.33	65.72	84.13	87.90	47.09	69.88
KGI0 DPR+BM25	60.48	80.06	36.91	66.94	40.81	64.79	65.95	90.34	35.63	68.47
Reranker Stage 1	81.22	87.00	70.78	73.05	71.80	71.98	87.71	92.43	55.50	74.98
Re2G Reranker	81.24	88.58	70.92	74.79	60.37	70.61	90.06	92.91	57.89	74.62
Table 2: Development Set Results for Retrieval
ment following end-to-end training is mixed. In FEVER and Wizard of Wikipedia there is substantial gain in R-Precision, approximately  $2\%$ . T-REx and Natural Questions are flat. However, there is a sharp decline in the performance of TriviaQA, in retrieval metrics. This is true despite the fact that retrieving these passages greatly improves answer accuracy and F1. This suggests some incompleteness in the provenance ground truth for TriviaQA.
Table 3 explores ablations of the  $\mathrm{Re}^2\mathrm{G}$  system. The point estimates and  $95\%$  confidence intervals are reported.  $\mathrm{Re}^2\mathrm{G}$ -KD excludes the online knowledge distillation, instead freezing the query encoder when training the reranker and generator during end-to-end training.  $\mathrm{Re}^2\mathrm{G}$ -BM25 excludes BM25 results, fetching 24 passages from DPR rather than 12 from DPR and 12 from BM25. The passages are still reranked.  $\mathrm{KGI}_0$  is the baseline system, without a reranker and therefore also without BM25 results or online knowledge distillation during training.
Both online knowledge distillation and ensembling with BM25 improve performance in four out of five datasets. Online knowledge distillation failed to improve for Wizard of Wikipedia and ensembling with BM25 failed to improve for Natural Questions.
Since the  $\mathrm{Re}^2\mathrm{G}$  model differs from the KGI model only in the retrieval phase, we hypothesized that its gains in output quality are driven by its better retrieval quality. To test this hypothesis we considered all cases where the  $\mathrm{Re}^2\mathrm{G}$  model produces better output than the  $\mathrm{KGI}_0$  model and calculated the fraction of such cases where  $\mathrm{Re}^2\mathrm{G}$ 's rank for the first correct passage is lower than  $\mathrm{KGI}_0$ 's.
We find that for T-REx, NQ, and FEVER the fractions of output gains that could be attributed to
improved retrieval and ranking are  $67.73\%$ ,  $61.08\%$  and  $66.86\%$  respectively. While for TriviaQA and Wizard of Wikipedia only  $36.86\%$  and  $27.74\%$  of output improvements were accompanied by improved ranking for the correct passage. It is important to note that in Wizard of Wikipedia, many of these improved outputs have only a small gain in token-level F1.
While much of the gain in output quality is attributable to improved recall, at least a third is not. This reinforces an observation of Glass et al. [2021], that models trained with better retrieval can produce better output even when the retrieved passages are equivalent at test time.
5.1 Slot filling error analysis
To understand the types of errors  $\mathrm{Re}^2\mathrm{G}$  makes we sampled 50 instances of the development set of the T-REx dataset where the Accuracy and token-level F1 score was zero.
Interestingly, the most common class of error (33/50) was due to the incompleteness of the ground truth. Often the head entity is ambiguous (19/50), or the relation has multiple fillers (16/50). As an example, consider the following where there are two Joe O'Donnell notable for sports in the passages retrieved, and each played for at least two different teams.
Joe O'Donnell [SEP] member of sports team Target: Buffalo Bills
$\mathbf{Re}^2\mathbf{G}$ : Dumbarton F.C.
- Joe O'Donnell (footballer) / Joe O'Donnell (footballer) Joseph 'Joe' O'Donnell (born 3 March 1961) was a Scottish footballer who played for Dumbarton and Stranraer.
Joe O'Donnell (American football) / ... fullback, guard and tackle for the University of Michigan from 1960 to 1963. He also played professional football as a guard and tackle for eight seasons for the Buffalo Bills...
When  $\mathrm{Re}^2\mathrm{G}$  produces genuine errors it is usually
T-REx	(Slot Filling)
R-Prec	Recall@5	Accuracy	F1	KILT-AC	KILT-F1
Re2G	81.24±1.08	88.58±0.84	86.60±0.94	89.20±0.81	75.66±1.19	77.08±1.15
Re2G-KD	81.08±1.09	88.84±0.83	87.00±0.93	89.46±0.80	75.72±1.19	77.00±1.15
Re2G-BM25	71.92±1.25	78.67±1.10	79.48±1.12	82.52±1.00	66.58±1.31	67.93±1.28
KGI0	65.02±1.32	75.52±1.16	77.52±1.16	80.91±1.03	60.18±1.36	61.38±1.34
Natural Questions (Question Answering)
R-Prec	Recall@5	Accuracy	F1	KILT-AC	KILT-F1
Re2G	70.92±1.67	74.79±1.27	46.70±1.84	62.44±1.65	39.23±1.80	50.90±1.76
Re2G-KD	69.72±1.69	73.73±1.30	46.56±1.84	61.68±1.67	38.24±1.79	49.93±1.76
Re2G-BM25	70.88±1.67	74.39±1.28	46.70±1.84	61.98±1.66	39.41±1.80	50.91±1.76
KGI0	64.65±1.76	69.60±1.39	40.50±1.81	55.07±1.71	32.96±1.73	42.87±1.75
TriviaQA (Question Answering)
R-Prec	Recall@5	Accuracy	F1	KILT-AC	KILT-F1
Re2G	72.01±1.20	73.16±0.98	74.01±1.17	80.86±0.99	56.04±1.33	60.91±1.27
Re2G-KD	72.01±1.20	73.16±0.98	73.80±1.18	80.62±1.00	56.04±1.33	60.84±1.28
Re2G-BM25	71.10±1.21	68.60±1.03	68.59±1.24	76.68±1.08	52.85±1.34	58.37±1.29
KGI0	61.13±1.31	63.12±1.08	60.68±1.31	66.61±1.20	44.00±1.33	47.35±1.31
FEVER (Fact Checking)
R-Prec	Recall@5	Accuracy	F1	KILT-AC
Re2G	90.06±0.53	92.91±0.47	91.05±0.55	80.56±0.76
Re2G-KD	89.85±0.54	92.48±0.48	90.78±0.55	80.14±0.77
Re2G-BM25	88.36±0.57	88.46±0.59	90.63±0.56	78.74±0.78
KGI0	80.34±0.73	86.53±0.63	87.84±0.63	70.06±0.88
Wizard of Wikipedia (Dialog)
R-Prec	Recall@5	Rouge-L	F1	KILT-RL	KILT-F1
Re2G	56.48±1.76	74.00±1.56	17.29±0.52	19.35±0.57	11.37±0.58	12.75±0.63
Re2G-KD	57.89±1.75	74.62±1.54	17.26±0.52	19.39±0.57	11.61±0.58	13.14±0.64
Re2G-BM25	55.83±1.76	72.72±1.58	17.15±0.51	19.17±0.56	11.13±0.57	12.52±0.63
KGI0	48.04±1.77	71.02±1.61	16.75±0.48	19.04±0.53	9.48±0.53	10.74±0.59
Table 3: Development Set Results for  ${\mathrm{{Re}}}^{2}\mathrm{G}$  Variations
because it has selected some entity as a filler related in a different way (6/17) or it has failed to retrieve the necessary passage (9/17).
$\mathrm{Re}^{2}\mathrm{G}$  considerably advanced the state-of-the-art across five KILT datasets, and still holds the top position in four of the five. Relative to previous work, such as RAG or KGI,  $\mathrm{Re}^{2}\mathrm{G}$  substantially improves both in retrieval and end-to-end performance on slot filling, question answering, fact checking, and dialog. The reranker alone improves performance and enables the inclusion of multiple sources of initial retrieval. This architecture permits us to integrate results from BM25, further improving accuracy. Our online knowledge distillation is able to improve the performance of DPR in four of the
five datasets, despite the loss in end-to-end training not depending on the DPR scores. Similarly, the ensembling of DPR and BM25, which is enabled by our incorporation of a reranker, benefits four of the five datasets tested. We have directed our efforts towards improving the retrieval of relevant knowledge. This also enables improvement in end-to-end performance by supplying better passages to the generation component. Further experiments on domain adaptation of  $\mathrm{Re}^2\mathrm{G}$  on tasks like question answering or dialog might provide useful insight on the application of this technology to real world use cases. We are releasing our source code as open source (Apache 2.0 license) to enable further research.
Michele Bevilacqua, Giuseppe Ottaviano, Patrick Lewis, Wen tau Yih, Sebastian Riedel, and Fabio Petroni. Autoregressive search engines: Generating substrings as document identifiers. ArXiv, abs/2204.10628, 2022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, 2020.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. Autoregressive entity retrieval. In International Conference on Learning Representations. OpenReview.net, 2021. URL https://openreview.net/forum?id=5k8F6UU39V.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge-powered conversational agents. In International Conference on Learning Representations, 2018.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge-powered conversational agents. In ICLR (Poster). OpenReview.net, 2019.
Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. T-REx: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Association (ELRA). URL https://aclanthology.org/L18-1544.
P. Ferragina and G. Manzini. Opportunistic data structures with applications. In Proceedings 41st Annual Symposium on Foundations of Computer Science, pages 390-398, 2000. doi: 10.1109/SFCS.2000.892127.
Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, and Alfio Gliozzo. Robust retrieval augmented generation for zero-shot slot filling. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1939-1949, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.148. URL https://aclanthology.org/2021.emnlp-main.148.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrievalaugmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015. URL http://arxiv.org/abs/1503.02531.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 874-880, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.eacl-main.74. URL https://aclanthology.org/2021.eacl-main.74.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. arXiv preprint arXiv:1702.08734, 2017.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1147. URL https://aclanthology.org/P17-1147.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77, 2020. doi: 10.1162/tacl_a_00300. URL https://aclanthology.org/2020.tacl-1.5.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.550. URL https://aclanthology.org/2020.emnlp-main.550.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, March 2019. doi: 10.1162/tacl_a_00276. URL https://aclanthology.org/Q19-1026.
Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, and Danqi Chen. Learning dense representations of phrases at scale. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6634-6647, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.518. URL https://aclanthology.org/2021.acl-long.518.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 333-342, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/v1/K17-1034. URL https://aclanthology.org/K17-1034.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online, July 2020a. Association for Computational Linguistics. doi: 10.18653/v1/2020. acl-main.703. URL https://aclanthology.org/2020.acl-main.703.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Roektaschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 9459-9474. Curran Associates, Inc., 2020b.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
Tie-Yan Liu. Learning to rank for information retrieval. Information Retrieval, 3(3):225-331, 2009.
Jean Maillard, Vladimir Karpukhin, Fabio Petroni, Wentauu Yih, Barlas Oguz, Veselin Stoyanov, and Gargi Ghosh. Multi-task retrieval for knowledge-intensive
tasks. In ACL/IJCNLP (1), pages 1098-1111. Association for Computational Linguistics, 2021.
Yu A Malkov and Dmitry A Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence, 42(4):824-836, 2018.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human generated machine reading comprehension dataset. In CoCo@ NIPS, 2016.
Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with bert. arXiv preprint arXiv:1901.04085, 2019.
Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. Document ranking with a pretrained sequence-to-sequence model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 708-718, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.63. URL https://aclanthology.org/2020-findings-emnlp.63.
Ashwin Paranjape, Omar Khattab, Christopher Potts, Matei Zaharia, and Christopher D Manning. Hind-sight: Posterior-guided training of retrievers for improved open-ended generation. arXiv preprint arXiv:2110.07752, 2021.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Roktaschel, and Sebastian Riedel. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2523-2544, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.nacl-main.200. URL https://aclanthology.org/2021.nacl-main.200.
Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Dmytro Okhonko, Samuel Broscheit, Gautier Izacard, Patrick Lewis, Barlas Oguz, Edouard Grave, Wentau Yih, et al. The web is your oyster-knowledge-intensive nlp against a very large web corpus. arXiv preprint arXiv:2112.09924, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2020.
Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3(4):333-389, April 2009. ISSN 1554-0669. doi: 10.1561/1500000019. URL http://dx.doi.org/10.1561/1500000019.
Cole Thienes and Jack Pertschuk. Nboost: Neural boosting search results. https://github.com/koursaros-ai/nboost, 2019.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: a large-scale dataset for fact extraction and verification. In NAACL-HLT, pages 809-819. Association for Computational Linguistics, 2018a.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. Fever: a large-scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809-819, 2018b.
James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. The fact extraction and verification (FEVER) shared task. CoRR, abs/1811.10971, 2018c.
James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. The fever2.0 shared task. In Proceedings of the Second Workshop on Fact Extraction and VERIFICATION (FEVER), pages 1-6, 2019.
Lidan Wang, Jimmy Lin, and Donald Metzler. A cascade ranking model for efficient ranked retrieval. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pages 105-114, 2011.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4003-4012, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-34-4. URL https://aclanthology.org/2020.lrec-1.494.
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. Scalable zero-shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6397-6407, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.519. URL https://aclanthology.org/2020.emnlp-main.519.
We have not done hyperparameter tuning for DPR Stage 1, Generation, or Reranking training. Instead we used hyperparameters similar to the original
works on training DPR, BERT reranking and RAG. Table 4 shows the hyperparameters used in our experiments.
For knowledge distillation we used the same hyperparameter settings as Generation. For the additional hyperparameters in online knowledge distillation: temperature and KD learn rate scaling, we experimented with temperatures of 10 and 40 and KD learn rate scaling of 1.0 and 0.1. For our reported results we used a temperature of 10.0 and a learn rate scaling of 1.0.
When training using online knowledge distillation, there is a separate optimizer for the query encoder while training generation. This optimizer uses the same hyperparameter settings.
Table 6 shows the settings for retrieval and generation used for all datasets.
All results are from a single run. The random seed for python, numpy and pytorch was 42.
B Software Details
We used the following software versions: