SlowGuess/NeurIPS_2023_OCR huggingface dataset:
MS MARCO. MS MARCO is a collection of queries and web pages from Bing search. To create the document collections, akin to NQ320k and following [46], we sample a subset of original documents by retaining the top-1 document for each query. We evaluate the models on the queries of the MS MARCO dev set and retrieval on the sampled document subset. We did not split the dev set into seen and unseen because  $84\%$  of the queries in the MS MARCO dev set are unseen.
BEIR. BEIR is a collection of datasets for heterogeneous retrieval tasks. In this paper, we evaluate the models on 6 BEIR datasets, which include distinct retrieval tasks and document collections from NQ and MS MARCO: (i) BEIR-Arg retrieves a counterargument to an argument; (ii) BEIR-Covid retrieves scientific articles about the COVID-19 pandemic; (iii) BEIR-NFC retrieves medical documents from PubMed; (iv) BEIR-SciFact retrieves scientific papers for fact-checking; (v) BEIR-SciDocs retrieves citations for scientific papers; (vi) BEIR-FiQA retrieves financial documents. All the queries in BEIR test set are unseen [38].
We summarize the statistics of above datasets in Table 4.
<sup>5</sup>A document in our context typically corresponds to a webpage, while a passage is a piece of text within a document. Our study focuses on building docid for documents. Using document retrieval tasks allows us to make fair comparisons with existing methods that utilize document URLs, titles, etc. And it also aligns with the settings of NQ320K and BEIR. Passage retrieval, as a different task, could lead to different evaluation results.
B Baselines
The sparse retrieval baselines are as follows:
- BM25, uses the tfidf feature to measure term weights; we use the implementation from http://pyserini.io/.
- DocT5Query, expands a document with possible queries predicted by a finetuned T5 with this document as the input.
The dense retrieval baselines are as follows:
- DPR [13], a dual-encoder model using the representation of the [CLS] token of BERT.
- ANCE [42], an asynchronously updated ANN indexer is utilized to mine hard negatives for training a RoBERTa-based dual-encoder model.
- Sentence-T5 [22], a dual-encoder model that uses T5 to produce continuous sentence embeddings. We reproduce Sentence-T5 (ST5 for short) on our datasets, the model is based on T5-Base EncDec model and is trained with in-batch negatives.
- GTR [23], a state-of-the-art dense retrieval model that pre-trains sentence-T5 on billions of paired data using contrastive learning.
- Contriever [11], a dual-encoder model pre-trained using unsupervised contrastive learning with independent cropping and inverse cloze task.
And the generative retrieval baselines are as follows:
- GENRE [5], an autoregressive retrieval model that generates the document's title. The original GENRE is trained on the KILT dataset [25] using BART, and we reproduce GENRE on our datasets using T5 for a fair comparison. For datasets without title, we use the first 32 tokens of the document as pseudo-title.
- DSI [37], which represents documents using hierarchical K-means clustering results, and indexes documents using the first 32 tokens as pseudo-queries. As the original code is not open source, we reproduce DSI using T5-base and the docids of NCI [41].
- SEAL [1] uses arbitrary n-grams in documents as docids, and retrieves documents under the constraint of a pre-built FM-indexer. We refer to the results reported by Wang et al. [41].
- CGR-Contra [17], a title generation model with a contextualized vocabulary embedding and a contrastive learning loss.
- DSI-QG [47], uses a query generation model to augment the document collection. We reproduce the DSI-OG results using T5 and our dataset.
- NCI [41], uses a prefix-aware weight-adaptive decoder and various query generation strategies, including DocAsQuery and DocT5Query. In particular, NCI augments training data by generating 15 queries for each document.
- Ultron [46], uses a three-stage training pipeline and represents the document as three types of identifiers, including URL, PQ, and Atomic.
C Performance on retrieving new documents
In this experiment, we investigate the impact of various document tokenization techniques on the ability of generative retrieval models to retrieve new documents. The generative models with different tokenization methods are trained on NQ320K data, excluding unseen documents, and are evaluated on NQ320K Unseen test set and BEIR-{Arg, NFC, SciDocs} datasets. For the baseline methods, which use rule-based document tokenization methods, the docids are generated for the target document collection using their respective tokenization techniques. In contrast, our proposed method uses a tokenization model to tokenize the documents in the target collection, producing the docids. However, our method may result in duplicate docids. In such cases, all corresponding documents are retrieved and shuffled in an arbitrary order. The results of this evaluation are summarized in Table 5.
Table 5: Zero-shot evaluation on retrieving new documents with different document tokenization methods. The second column indicates the type of docid, where BERT-HC denotes BERT-Hierarchical-Clustering [37], Prefix-HC denotes Prefix-aware BERT-Hierarchical-Clustering [41], and dAE denotes discrete auto-encoding.
Method	Docid	NQ (R@1) Unseen	BEIR (nDCG@10)
Arg	NFC	SciDocs
DSI-Naive† [37]	Naive String	0.0	0.1	1.0	0.1
DSI-Atomic† [37]	Atomic	0.0	0.2	0.8	0.1
GENRE† [5]	Title	6.0	0.0	2.4	0.6
DSI† [37]	BERT-HC	1.3	1.8	11.1	5.9
NCI [41]	Prefix-HC	15.5	0.9	4.3	1.2
Ours	dAE	34.2	12.1	12.1	12.3
Document tokenization methods that do not consider the semantic information of the documents, such as Naive String and Atomic, are ineffective in retrieving new documents without model updating. Methods that consider the semantic information of the documents, such as those based on title or BERT clustering, show some improvement. Our proposed document tokenization method significantly improves over these existing rule-based document tokenization methods. For instance, when the model trained on NQ - a factoid QA data based on Wikipedia documents - is applied to a distinct retrieval task on a different document collection, BEIR-SciDocs, a citation retrieval task on a collection of scientific articles, our proposed document tokenization model still showed promising results with an nDCG@10 of 12.3, which is comparable to those models trained on the target document collection. This suggests that our proposed method effectively encodes the semantic information of documents in the docid and leads to a better fit between the docid and the generative retrieval model.
Table 6: Efficiency analysis.
Method	Memory	Time (Offline)	Top-K	Time (Online)
ANCE	1160MB	145min	100	0.69s
GTR-Base	1430MB	140min	100	1.97s
GENRE	851MB	0min	100	1.41s
10	0.69s
DSI	851MB	310min	100	0.32s
10	0.21s
Ours	860MB	220min	100	0.16s
10	0.10s
D Efficiency analysis
In Table 6, we compare GENRET with baseline models on MS MARCO (323,569 documents) in terms of memory footprint, offline indexing time (not including the time for neural network training), and online retrieval latency for different Top-K values. We have four observations: (i) The memory footprint of generative retrieval models (GENRE, DSI-QG, and the proposed model) is smaller than of dense and sparse retrieval methods. The memory footprint of generative retrieval models is only dependent on the model parameters, whereas dense and sparse retrieval methods require additional storage space for document embeddings, which increases linearly with the size of the document collection. (ii) DSI and GENRET take a longer time for offline indexing, as DSI involves encoding and clustering documents using BERT, while GENRET requires tokenizing documents using a tokenization model. Dense retrieval's offline time consumption comes from document encoding; GENRE uses titles hence no offline computation. (iii) The online retrieval latency of the generative retrieval model is associated with the beam size (i.e., Top-K) and the length of the docid. GENRET utilizes diverse clustering to generate a shorter docid, resulting in improved online retrieval speed compared to DSI and GENRE.
E Embedding visualization
Embedding at position 1
Embedding at position 2
Document embedding
Codebook embedding
Figure 5: t-SNE visualization of the codebook embedding and document embedding on the NQ320K dataset. The codebook embedding is uniformly scattered in the document representation space.
Embedding at position 3