Autoregressive Search Engines: Generating Substrings as Document Identifiers
Authors: Michele Bevilacqua
Giuseppe Ottaviano
Patrick Lewis
Scott Yih
Sebastian Riedel
Fabio Petroni
Venue: NeurIPS 2022 Accept (2022)
Keywords: Natural Language Processing
Information Retrieval
Question Answering

Abstract: Knowledge-intensive language tasks require NLP systems to both provide the correct answer and retrieve supporting evidence for it in a given corpus. Autoregressive language models are emerging as the de-facto standard for generating answers, with newer and more powerful systems emerging at an astonishing pace. In this paper we argue that all this (and future) progress can be directly applied to the retrieval problem with minimal intervention to the models' architecture. Previous work has explored ways to partition the search space into hierarchical structures and retrieve documents by autoregressively generating their unique identifier. In this work we propose an alternative that doesn't force any structure in the search space: using all ngrams in a passage as its possible identifiers. This setup allows us to use an autoregressive model to generate and score distinctive ngrams, that are then mapped to full passages through an efficient data structure. Empirically, we show this not only outperforms prior autoregressive approaches but also leads to an average improvement of at least 10 points over more established retrieval solutions for passage-level retrieval on the KILT benchmark, establishing new state-of-the-art downstream performance on some datasets, while using a considerably lighter memory footprint than competing systems. Code available in the supplementary materials. Pre-trained models will be made available.

Autoregressive Search Engines: Generating Substrings as Document Identifiers
Michele Bevilaequa 1.2 Giuseppe Ottaviano Patrick Lewis²
Wen-tau Yih Sebastian Riedel2.3 Fabio Petroni Sapienza University of Rome Meta AI UUniversity College London
Abstract
Knowledge-intensiin intege-veteeeive language tasks require NLP systems both provide the cor- rect answer and retrieve supporting evidence for it in given corpus. Autoregressive language models are emerging as the de-facto standard for generating answers, with newer and more powerful systems emerging at astonishing pace. In this paper we argue that all this (and future) progress can be directly applied to the retrieval problem with minimal intervention to the models' architecture. Previous work has explored ways to partition the search space into hierarchical structures and retrieve documents by autoregressively generating their unique identifier. In this work we propose an alternative that doesn't It force any structure the search space: using all ngrams in passage its possible identifiers. This setup allows us to use an autoregressive model to generate and score distinctive ngrams, that are then mapped to full passages through efficient data structure. Empirically, we show this not only outperforms prior autoregressive approaches but also leads an average improvement of at least 10 points over more established retrieval solutions for passage-level retrieval on the KILT benchmark, establishing new state- of of-the-art downstream performance on some datasets, while using a considerably lighter memory footprint than competing systems. Code and pre-trained models are available at hhttss: /tttts:////ii htw.ch/...c.eaceeoooeeeeeeaaeeee CEEALL
Introduction
Surfacing knowledge from large corpora is crucial step when dealing with knowledge intensive
language tasks [Levy et al. 2017 Dinan et al. 2019 Elsahar et al. 2018 Petroni et al. 2021 such as open-domain question answering Voorhhes et 1999 Joshi al. 2017 Yang et al. 2018 Kwiatkowski et al. 2019] and fact checking [Thoree et al. 2018). A popular paradigm to approach such tasks is to combine search engine with a machine reader component. The former retrieves relevant context, usually in the form of short passages, which the latter then examines to produce
answers Chen al. 201 Lewis et 2020 Izacard and Grave 2021
In recent years we have witnessed a surge of research and development in autoregressive language models (Radford
2019 BBrwnn al. 2020 Rae al. 2021 Artetxe et al. 202 Smith et al. 2022). with ever increasing size and natural language understanding (NLU) capabilities Such models currently the de-facto implementation of the machine reader component in retrieval-reader architectures, and have contributed to rapid progress on a wide range of benchmarks [Joshi al.) [2017] Kwiatkowski et al. 2019 Petroni et al. 2021] However, these tremendous advances in aggressive modelling has yet to bring similar transformational changes in how retrieval is approached.
2019 Lewis et al. 2019 Raffel et
Transferring the NLU capabilities of modern autoregressive models to retrieval is non-trivial. Some works have demonstrated that knowledge stored in the parameters of these models can be retrieved
to some extend by directly generating evidence given a query (Petroni et al. 2019 2020 Roberts
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

FM-index
Carbon footprint
nateling
ngaan
Cartion diocids is reiraced naturilly
decompor
relesse
parton
Query:
Autoregressive
carbin
sloockk
burning foodll fuels, deforestation, and cement cementpreductbron production... Mathane (CH4) large'y relezsed coul, oll and gas industi Although
of os2
language model
Carbon
atom
docie
mass-pre prass-producedii
decdd
carbon diocida, very prevalent
Jords
which
Conetroined gereration
bon diocdds omissions by burning fossil fuels
SEAL
Figure 1: High-level SEAL architecture, composed of an autoregressive LM paired with an FM-Index. for which we show the first (F) and last (L. columns of the underlying matrix (more details in Sec 3.1) The FM-index constrains the autoregressive generation (e.g., after carbon the model contrained to generate either lax, dioxide atom in the example) and provides the documents matching (i.e., containing) the generated ngram (at each decoding step).
et al., 2020]. However, such approaches have been shown to be unreliable because of their tendency hallucinate non-factual content [Massarelli et al. 2019 Metzler al., [021] Ji al., 2022]. To alleviate this issue, previous work proposes to only use generation for query expansion in traditional search engines [Mao al. 202T], but these solutions don't exploit the full potential of autoregressive architecture, such as word order sensitivity and conditional probability modeling, and still lag behind vector-based approaches (Karpukhin al. 2020).
Recently, another line of work has investigated using autoregressive language models to generate identifier strings for documents as an intermediate target for retrieval, such as Wikipedia page titles [De Cao et al.) 2021b), or root-to-leaf paths in hierarchical cluster tree (Tay et al. 2022]. Employing identifiers, rather than generating evidence directly, induces structure in the search space, (i.e., index documents by their title or their cluster tree) which can be easier to memorize, learn, and retrieve from, than full unstructured passages. Moreover, it is relatively easy to constrain decoding with prefix tree "index" so that only valid identifiers are generated. As downside, if appropriate metadata (e.g., titles) are not available, one needs to create the identifiers, hence the structure (e.g., with hierarchical clustering), which has not been thoroughly evaluated on large-scale benchmark.
In this work, we propose solution that does not force any structure in the search space, but rather uses all the ngrams occurring in a document as its identifiers. Concretely, we introduce Search Engines with Autoregressive LMs (SEAL), retrieval solution that combines an autoregressive model, i.e. BART [Lewis et al. 2019 with compressed full-text substring index, i.., the FM-Index (Ferragina) and Manzini] 2000 see Figure for an high-level overview. This configuration comes with a twofold benefit: i) we can constram BART's generations with the FM-Index, hence preventing generation of invalid identifiers (i.e., ngrams not occurring any document); ii) the FM-Index provides information on all documents in the corpus containing specific ngram (for every decoding step), thus allowing to retrieve them. This setup allows SEAL to generate any span from any position in the corpus, without needing to explicitly encode all substrings in document. Morcover, a novel scoring function to intersect the results of multiple ngrams combining LM probabilities with FM-index frequencies (i.e., number of occurrences of the ngram in the whole corpus).
design
Our experimental evaluation shows that SEAL matches or outperforms recent retrieval solutions (including autoregressive ones) on Natural Questions (Kwiatkowski et al., 2019], while requiring substantially less memory (~22 7 times smaller in footprint). Moreover, SEAL's intersection passage-level retrieval by more than 10 points on the
formulation improves the state-of-the-art KILT benchmark (Petroni al., [02T], contributing in establishing new state-of-thh-an downstream results on multiple datasets when paired with existing reader technologies.
2 Related Work
One way to approach retrieval with autoregressive models makes use of unique identifiers, Lee, string pointers to documents that are in some way easier to generate than the full document itself (De Cao

---

et al,, 2021b Tay et al., 2022]. In our work the identifiers are generated ngrams, which do not necessarly occur in just one document, Note that the idea of using phrases retrieve passages has shown promise already in the context of dense retrieval [Lee et al. 202
Our approach is quite conceptually distinct from document NNgueira et al. 2019 Nogueira and Lin 2021] and query expansion approaches [Mao et al., 2021]. that use autoregressive models to boost search, but still rely on black-box systems like BM25 [Robertson and Zaragoza 2009) for the retrieval: in our work the boundary between generation and retrieval is blurred, since we generate grounded passage spans. The same reasoning applies models that generate queries for web search
engines (Komelli et al. 2021 Shuster et al. 2022 Nakano et al. 2021 Lazaridou et al., 2022).
Virtually all modern approaches
matihhintttiiig---i baaschhing--aeed retrieval use bag-of-words representations. Many recent works propose learned contextualized weighting for both queries and documents
terms Dai and Callan 2019 Gao et al. 2021 Lin and Ma [021 |Mallia et al. 2021 Dai and Callan 2020 Bai et al. 2020 Zhao et al. 2021 Formal et al. 2 Many of these methods can also weigh terms that are not present in the query, addressing so-called vocabulary mismatch. In contrast, SEAL generates (and scores) ngrams of arbitrary size. These approaches are partly orthogonal to SEAL, as many of the proposed techniques could be used rescore higher-order ngrams,
Finally, a connected strand of research is that of query likelihood models, which use autoregressive models to (re)rank passages according to the probability P(glp) of query q given the passage [Nogueira dos Santos et al. 2020)] Zhuang and Zuccon [2021] Lesota et al.] [202]] [Sachan et al.)
2022]. In our case, the autoregressive architecture models the likelihood of ngram given the query, Le., P(nlg).
3 Background
In retrieval, the automatic system is required to return an ordered list of documents d1, d2, dn from a retrieval corpus R, given a query q. Both queries and documents are texts, i.e., lists of tokens (t1, .., .ty), where each token t drawn from a vocabulary V. A span of tokens called an ngram; ngrams of size are known as unigrams. We denote with F(n, R) the frequency of an ngram n in R, i.e, the total number of times appears in the whole retrieval corpus.
text is
3.1 The FM-Index
Our method requires a data structure that can support the efficient identification of occurring substrings to guarantee that all decoded sequences are located somewhere in the retrieval corpus. Moreover, to perform retrieval, we require the ability to identify which documents the generated ngrams appear in. Neither inverted indices (which have efficient way search for phrases of arbitrary length), nor prefix trees (which would force us to explicitly encode all & suffixes in a document), are viable options. The core data structure that satisfies our requirements the FM-index [Ferragina and Manzini 22000. i.e., compressed suffix array that, as self-index, requires no additional storage for the original text. FM-index space requirements are linear in the size of the corpus, and, with small vocabularies such as those used by modern subword-based based language models, is thus usually significuntly smaller than the uncompressed corpus. The FM-index can be used to count the frequency of any sequence tokens n in O(n//log/V/ i.e,, independently from the size of the corpus itself. For constrained decoding, the VV). Internally, the FM-index relies on
list of possible token successors can be obtained in oooV the Burrows- Wheeler Transform [Burrows and Wheeler 1994]. BWT, invertible transformation that permutes a string to make easier to compress, defined as follows: all the rotations of the string are sorted lexicographically and laid out in matrix; the last column of the matrix is the strings's s BWT For example, given the string CABAC, the corresponding matrix would be:
F
L
86
C
A
C
4
B
A
C
$
C
4
C
C
B3
B3
A
C
$
C
A²
C
8
C
A
B
A
Ca
A
B
A
C
86
Since corpus contains multiple documents, concatenate them with separator token.
3

---

where $ is a special end-of-string token. The first (F) and last (L) columns are the only ones that will be explicitly stored in the FM-index; F is just an array of runs (i.e., sequences of repeated tokens), due to the rotations being sorted, so it can be represented with one count for each alphabet symbol: L, the string's 's BWT, will stored in data structure known as the Wavelet Tree [Girossi et al., 2003). which allows efficient the transformation. FM-indices have the useful property that for each symbol, the relative rank stays the same: that is, the ith occurrence of symbol in F points to the same location in the corpus of the ith occurrence a in L. Thanks to this property, we can locate any string (0l, ........ . in the index by starting from on and going backwards. First, we select the contiguous range of rows corresponding to the symbol on in F, then we check the ranks of the first and last occurrences of n- in the same range of rows in L. We use the ranks select new, smaller equal range of rows looking up the symbol nn-1 in F. The procedure can be applied iteratively to find ngrams of any size.
queries, while exploiting the compressibility induced by
4 Method
In our retrieval methodology. SEAL, we generate multiple ngrams, conditioning on a query. The ngrams then used to find the documents they appear in within the corpus, which are then returned to the user. In Figure we we show this process at a high-level. We use our indexing structure, Le., the FM-index do constrained decoding so that each ngram occurs at least once in the retrieval corpus. Jointly, we use the FM-index to efficiently find matching documents. Documents are ranked using the scores of the generated ngrams.
Autoregressive Retrieval We generate ngrams identifiers with constrained beam search, using the FM-index to identify the set of possible next tokens in at most O V log V(): tokens corresponding to unattested continuations are blocked by masking the logit to co. As a result, after a single decoding pass, get set of ngrams (K), along with their autoregressivelyycompuie to the model. It is also trivial to find the positions in the corpus where the decoded ngrams appear, as constrained decoding already requires selecting the relevant range of rows in the FM-index. Note that autoregressive scoring entails monotonically decreasing scores-aue -ny string will be assigned a lower probability than any of its prefixes. To address this issue, use fixed-length ngrams. Each document is assigned the score (P) of its most probable decoded occurring ngram, i.c. the probability of the ngram as autoregressively computed by the decoder, conditioned on the encoder input We refer to this as the LM scoring.
probabilities according
Factoring in FM-index frequencies To counterbalance the monotonic probability decrease, we integrate in scoring unconditional ngram probabilities, computed as normalized index frequencies:
P(n) F(n, R) Eder [d]
This also enables us to promote distinctive ngrams, i.e., those that have high probability according to the model and low probability according the FM-index. We take inspiration from the theory behind TF-IDF and BM25 [Robertson and Zaragoza 2009 and use the following scoring function:
P(nlg) ((()))
w(n, q) max 10 (
(2)
This formulation addresses the problem of length, as the unconditional probability an ngram will also be equal or lower than that of any of its prefixes. To make better use of the computational resources, we slightly modify the beam search implementation to keep track of all the partially decoded sequences that have been considered. Thanks to this, score larger number of ngrams than the size of the beam. We refer to this formulation as the LM+FM scoring.
An Intersective Scoring for Multiple Ngrams One problem with the previous scoring formulations that it is impossible to break ties among documents whose highest scoring ngram is the same, as they receive exactly the same score. Moreover, it might be difficult to capture all relevant information within a document by considering only single ngram, for instance when salient ngrams are non- contiguous (e.g., separated by unrelated text). To address these issues propose novel scoring
4

---

formulation that aggregates the contribution of multiple ngrams contained in the same document. To avoid repeated scoring of overlapping ngrams, for each document d R we only consider a subset of the generated ngrams K(d) C. K. An ngram belongs ((d) if there is at least one occurrence of n in d that does not overlap in the corpus with an occurrence of another ngram n' such that a) n' E K(d) b) w(n', q) > w(n, q). The document-level score, then, is the weighted sum of all ngrams in k(dd);
W W(d,9)
w(n, 9)a cover(n, k(()) NEK(O
E
where or is a hyperparameter and the coverage weight cover(n, K) (controlled by the second hyperpa- rameter 3) a function of how many ngram tokens are not included in the coverage set C(n, K) C V. Le., the union of all tokens in ngrams with higher score. We define this coverage weight follows:
cover(n, K) = --+ (set(n) \ C(n, K)]
[set(n)]
where set(n) is the set of all tokens in n. The purpose of the coverage weight is to avoid the overscoring of very repetitive documents, where many similar ngrams are matched. Note that by saving the probability distribution at the first decoding step we can compute scores for all unigrams with no additional forward pass. We refer this last approach, which can be thought of higher-order generalization of the bag-of-words assumption, as the LM+FM intersective scoring.
5 Experimental Setting
Our experimental setting evaluates SEAL on English knowledge-intensive NLP tasks. Each con- sidered dataset is collection of queries, each of which be answered by looking for piece(s) of evidence in the corpus. We consider both an in vivo evaluation, in which we assess the model by looking how well the document ranking matches with the ground truth, and, in addition, we perform a downstream evaluation, in which we feed the retrieved documents to a trained reader, that uses the documents generate the answer.
5.1 Data
Natural Questions Natural Questions (NQ) is dataset containing query-document pairs, where the query is question (e.g., "who wrote photograph by ringo starr"), and the document is Wikipedia page, in which span is marked as an answer [Kwiatkowski et al. 2019). We experiment on both the customary retrieval setup used by, among others, Karpukhin et al. 2020) and Mao al. [202T). and the substantially different setup used by [Ta et al. [2022]. We refer to these two settings as, respectively, NQ and NQ320k. NQ, retrieval is performed on an entire Wikipedia dump, chunked in around 21M passages of 100 tokens. Performance is measured as accuracy @ k; i.e., the fraction of instances for which least one of the top-k retrieved passages contains the answer. NQ320k: is much more restricted setting, in which the retrieval set is limited to the union of all ground truth document in the training, dev or test set. Different revisions of the same Wikipedia page count as different documents. Note that the exact splits used by Tay et al. 2022), the retrieval corpus and the preprocessing code have not been yet released at the time of writing. Therefore, we have tried to not precisely comparable with
replicate the setting as closely possible, but the exact numbers those reported in the original paper. In NQ320k, performance is measured as hits @ k. i.e, the fraction of instances for which at least one of the top-k retrieved passages is in the ground truth.
KILT is a comprehensive benchmark collecting different datasets including question answering, fact checking, dialogue, slot filling, and entity linking (Petroni al. [021]. All these tasks are solvable by retrieving information from a unified corpus- a Wikipedia dump. In KILT, the evidence usually the paragraph that contains the answer. Following Maillard et al.] [202T), we have re- chunked KILT's retrieval corpus, which is originally paragraph- -aseea,p--aadd,,... in around 36M passages of 100 tokens. We do not use the entity linking and ELI5 KILT tasks, where a ground truth passage is not provided in the training set. KILT's retrieval performance is measured with R R-precision, precision-orientee measure that considers only gold documents as correct answers, not just any document containing the answer. R-precision can be computed at either passage level or page level.
5

---

Table 1: Language model and index size on Natural Questions (around 21M passages). SEAL's
index is -1.5 times smaller than uncompressed plain text.
Model Params
Index Params
System
Size
GPU?
plain
13,4GB
DPR
220M
64.6 GB
16.1B
BM25
18.8 GB
X
GAR
406M
18.8 GBB
X
DSI BART
406M
SEAL
406M
8.8GB
x
5.2 SEAL configuration
Training We finetune BART large [Lewis et 2019 generate ngrams of length k 10 from the ground truth document. Since there are c k ngrams in document d, we sample (with replacement) 10 ngrams from it, biasing the distribution favor of ngrams with a high character overlap with the query. We also add the title of the document to the set of training ngrams. To expose the model more possible pieces of evidence, we also add different "unsupervised" examples for each document in the retrieval corpus to the training set. In each of these examples the model takes as input a uniformly sampled span from the document, and predicts either another sampled span, or the title of the page. We append special tokens to the input to signal to the model a) whether the pair comes from the supervised or unsupervised training pairs (in the same spirit as the co-training task prompts used by Tay et al. [2022] )) b) whether title or span is expected as output. On KILT we train SEAL on datasets at once. We report training and inference hyperameters in Appendix A$
How to choose ngrams In preliminary experiments, we have found that training SEAL with ngrams of fixed length sampled uniformly from the training chunk did not lead to good performances. Instead, we have biased the ngram distribution towards ngrams that were relevant to the query. As simple proxy for relevance, we have use the Levenshtein (character-based distance between the query and the ngram overlap. The final distribution to sample from is defined by the following formula:
e Llgddatt)/
(5)
d k+1 (Lndgss4)/s
j=1
where L is the Levenshtein distance and 7 (= 1.5 in our experiments) is temperature parameter, controlling the peakiness of the distribution.
Index We the C++ FM-index implementation in sdsl-lite, While the FM-index construction (which requires a sort of all rotations) takes around 6 hours in our single-threaded implementation, parallel algorithms are available [Labeit al. 2017]. Each document encoded as the subword tokenization of the concatenation of the title and the passage, separated by a special token. We report in Table ! the index statistics for Natural Questions. As can seen, SEAL's 's FM-index more than 7 times lighter compared to DPR's s full document embeddings for exact inner product search, and needs neither GPU for search on top of that, nor separate storage for the text itself. While vector compression methods can reduce dense retrievers' index size, this still comes at the expense of performance (Yamada et al. [202T] Lewis et al., [221a]. In addition, our the size of our index is less than 50% of that of the well-optimized Lucene BM25 index used by pyserini. but also roughly 65% the uncompressed plain text itself.
5.3 Retriever Baselines
We compare SEAL against well-established systems in the literature on each benchmark. On NQ and NQ320k we also compare against our BART-based replication of DSI [Tay et al. 2022 DSI-BARTY On NQ320K, page-level benchmark, we include our own replication of GENRE De Cao et al. 2021b). Unless otherwise specified, we use pyserini to compute the BM25 baseline. For other systems, we either take figures from the literature, or use publicly released model predictions.
We report reproduction details in Appendix B

---

Table 2: Results on NQ320k. Reporting hits@ and hits 10. Best bold.
hits@K
System
10
BM25 (gensim)
15.3
44.5
BM25
22.7
59.0
DSI-BART
25.0
63.6
GENRE
26.3
71.2
SEAL (LM, [n] 3)
21.3
66.5
SEAL (LM, y 4)
22.2
68.2
SEAL (LM, ( = 5)
22.6
68.7
SEAL (LM+FM)
25.3
72.0
SEAL (LM+FM, intersect.)
26.3
74.5
Table 3: Retrieval results on the NQ test set. Column blocks (left to right): retrieval results (aceu- racy @ 5/20/100); retrieval results the test splits of Lewis al. [2021b], partitioned according to whether the query/answer is paraphrase of one in the training set; downstream performances (exact match). Except for Izacard and Grave [021], J, all downstream results computed with the same FiD reader trained on DPR predictions. Best in bold.
accuracy @k
Overlap? (A @ 100)
System
EM
5
20
100
ans. d
X
ques. /
BM25
43.6
62.9
78.1
82.9
70.1
80.9
76.6
40.4
DPR Karpukhin et al.
2020
68.3
80.1
86.1
91.4
76.8
93.2
83.2
47.2
GAR Mao al. 2021
59.3
73.9
85.0
91.6
74.4
94.1
80.4
46.2
DSI-BART
28.3
47.3
65.5
77.8
44.2
84.9
57.7
31.4
Izacard and Grave 2021
48.2
SEAL (LM, |n|
40.5
60.2
73.1
82.2
57.1
85.2
64.9
36.0
SEAL (LM+FM)
43.9
65.8
81.1
86.9
70.9
89.5
78.1
42,9
SEAL (LM+FM, intersective)
61.3
76.2
86.3
91.2
77.7
93.2
84.1
48.0
5.4 Reader
For downstream results, use the -Deeooee abstractive reader [Izacard and Grave 2021]
which takes the query along with 100 contexts and produces a task-specific answer. We train FiD on training set predictions.
Results
SEAL outperforms BM25 and DSI-BART in hits@) 10 in all its formulations. When taking into account ngram frequencies (i.e., LM+FM). SEAL achieves even higher results than GENRE, despite the fact that this benchmark only requires page-level retrieval capabilities (that is the focus of GENRE). Finally, our intersective formulation achieves the highest results, both hits@1 and @ 10, indicating that multiple ngrams identifiers might capture complementary information, which can be aggregated for stronger performances.
NQ320k We report results on NQ320k in Table 2
Natural Questions We report in Table 3 the results of our evaluation on Natural Questions, a
passage-level retrieval benchmark with larger collection of documents (i.e., -21M w.r.t. 200k in NQ320k). In this setting, the gap in performance between DSI-BART and SEAL is larger, possibly because memorizing documents identifiers in the parameters of the model becomes more challenging with larger corpora. Remarkably, the intersective formulation of SEAL achieves results comparable or superior to more established retrieval paradigms (e.g., BM25, DPR and GAR), at high-recall (accuracy @ 100). To better understand the generalization capabilities of our retrieval solution we use the question/answer overlap split of Lewis et al. [2021b). This study reveals that SEAL achieves highest performance for question/answer pairs never seen during training (i.e., no overlap), suggesting a better ability to generalize to completely novel questions with novel answers (e.g., 3.5 points better than GAR on average).

---

Table 4: Retrieval results individual KILT dev set(s), with the average in the rightmost column. Reporting passage-level R-precision (higher better). We mark model that are also trained on additional synthetic data [Lewis et al. 2021e with f- All SEAL models multitask. Best among models trained only on KILT queries in bold.
Model
FEV
T-REx
zsRE
NQ
HoPo
TQA
WoW
AVG
BM25
40.1
51.6
53.0
14.2
38.4
16.2
18.4
33.1
DPR Maillard et al.
2021
43.9
58.5
78.8
28.
43.5
23.8
20.7
42.5
MT-DPR Mailard et al.
2021
52.1
53,5
41.7
28.8
38.4
34,2
24.1
39.0
MT-DPR Oguz
52.1
61.4
54.1
40.1
41.0
34.2
24.6
43.9
MT-DPR Oğuz et al.
2021
61.4
68.4
73.3
44.1
44.6
38.9
26.5
51.0
MT-DPR (large) Oguz
2021
62.8
66.6
66.9
42.6
42.1
37.9
23.4
48.9
SEAL (LM+FM)
31.5
42.0
34.0
21.7
24.7
21.4
17.6
27.6
SEAL (LM+FM, intersective)
67.8
58.9
78.8
43.6
54.3
41.8
36.0
54.5
Table Downstream results on the KILT test set(s). Downstream metrics are accuracy (FEVER, T-REx, zero-shot RE), exact match (Natural Questions, HotpotQA, TríviaQA), or FI (Wizard of Wikipedia). Best in bold. t: result taken from the eval KILT leaderboard.
FEV
T-REx
zsRE
NQ
HoPo
TQA
WoW
System
ACC
ACC
ACC
EM
EM
F1
KGI Glass et al. 2021
85.6
84.4
72.6
45.2
61.0
18.6
Hindsight Paranjape al.
2021
19.2
DPR+BART Petroni
202T
86.7
59.2
30.4
41.3
25.2
58.6
15.2
RAG Petroni et al. 2021
86.3
44.7
44.4
27.0
13.1
MT-DPR+BARI Maillard al.
2021
86.3
58.0
39.8
31.8
59.6
15.3
MT-DPR+FiD [Piktus el al. 2021
89.0
82.5
71.7
49.9
36.9
71.0
15.7
MT-DPR-WEEB-TE WEB+FID [Pikuus et
2021
89.0
81.7
74.2
51.6
38.3
72.7
15.5
SEAL+FiD (LM+FM)
87.9
83.7
74.2
47.3
37.6
65.8
17.5
SEAL+FDD (LM+FM, intersective)
89.5
83.6
74.7
53.7
40.5
70.9
18.3
Table 6: Retrieval results on the NQ test set with different model sizes. DPR (large) performance from [Oguz al. 2021].
System
# Par.
@ 20
A 100
DPR (basc)
-220M
80.1
86.1
DPR (large)
-350M
80.2
86.7
SEAL (large)
-400M
76.2
86.3
We report retrieval results passage level on the KILT benchmark in Table 41 SEAL outperforms DPR by more than 10 points on average in passage-level R-precision, indicating that our method is more precise surfacing ground truth evidence as the first result. Moreover, SEAL also performs better than MT-DPR (multi-task DPR) even when the latter is pretrained on tens of millions of questions from PAQ Lewis et al.) 2021c). J. technique that can drastically improve results and that could potentially bring benefits to method well (a task leave for future work). When it comes to downstream performances (Table 5). FiD with passages retrieved by intersective SEAL establishes state-of- the-ar art on 4 datasets out of (FEVER, zsRE, NQ, HoPo), and achieves very competitive results on the remaining 3.
KILT
Speed and constrained decoding The inference speed of SEAL is directly proportional to the beam size, with limited overhead added by constrained decoding. On the Natural Questions test set, for instance, retrieval with the intersective scoring requires 16 minutes and ~35 minutes with, respectively. a beam size of 5 or 15. Mao et al. [202T] report lower runtime for GAR (-5 minutes), and a comparable one for DPR (-30 minutes). Note that more efficient approaches to constrained decoding have been proposed (e.g., De Cao et al. 202
GPU evaluation setup
We report page-level and KILT-score results in the Appendix

---

Table 7: Ablation on Natural Questions. SEAL when using (V) or not using (X) FM-index constrained decoding, for beam size values in f3, 5, 10, 15). Reporting accuracy @ k.
System
Constr.
Beam
A A@
100
SEAL (LM+FM)
15
65.8
81.1
15
65.3
80.1
63.3
64.7
79.9
V
10
65.4
80.8
SEAL
15
76.2
(LM+FM,
X
15
76.2
86.2
intersective)
75.2
84.9
5
75.9
85.8
76.4
86.4
Table 8: Performance Natural Questions with different ngrams sizes, using SEAL (LM+FM, intersective).
Length
A@20 20
@
100
64.7
74.8
73.6
83.7
10
76.2
86.3
and we leave their application to SEAL as future work. Moreover, generation is becoming the de facto standard approach NLP, just as the method for materializing final outputs, but also for modeling the computational process needed before computing the answer (Wei et al. 2022]. As such, we expect generation latency will improve significantly and increasingly over time as result of this growing interest. Any improvement will be directly applicable to SEAL.
Model size In Table we show that parameter size is not a crucial factor behind the good per- formance of the method that we propose: SEAL (-400M) outperforms all model from Oguz et al. 2002I). including the MT-DPR large model (-350M) trained on NQ and PAQ [Lewis et al., 202 In fact, the results of MT-DPR are lower with bigger saabooee-wiich could point an increased difficulty training larger models for dense retrieval. In the same direction, on vanilla NQ (Table 6). where DPR large only slightly outperforms DPR base, the performance of SEAL is in the same ballpark.
Ablation studies In Table we report results NQ for various configurations of SEAL. While, in general, performances increase with a larger beam, diminishing returns (or even a performance decrease) are found between value of 10 and 15. Disabling constrained decoding and discarding posteriori all generated ngrams that don't appear in the corpus results slightly lower performances. We have found the decoding maximum length have erucial impact the retrieval performance of SEAL, since shorter ngrams tend to be less informative than longer ones. We report in Table S performances the NQ test when using, respectivaly, 3, and 10 ngram maximum length. Every other SEAL figure reported in this paper uses 10.
Qualitative Analysis In Table 9 we show examples of ngrams predicted by SEAL (trained on KILT) given the query "can you predict earthquakes". SEAL is able to rephrase the query in ways that preserve its lexical material producing ngrams such as earthquakes can be predicted, used to predict earthquakes etc. Morevoer, the model is also able to explore more diverse regions of the output space, overcoming the vocabulary mismatch problem: ngrams contain related tokens like subword seism- and the word forecast. SEAL's 's LM+FM scoring is also able to assign score below (and, thus, exclude from the search), unrelated ngrams that are considered by the beam because of their promising start, such as "Seismic risk in Malta @ @

---

Table 9: Best (top) and worst (bottom) generated keys for the query "can you predict earthquakes" (left), and retrieved documents (right). Matched ngrams in bold. @ @ separates title and body.
score
identifier
doc #1
doc #2
273.2
earthquakes can be predicted
Seismology @ for precise earth- quake predictions, including the VAN method. Most seismologists do not believe that system to pro- vide timely warnings for individual earthquakes has been developed, and many believe that such sys- tem would be unlikely to give useful warning of impending seismic events. However, more general forecasts rou- tinely predict seismic hazard. Such forecasts estimate the probability of an earthquake of particular [...]
Earthquake prediction @ reliably identified across significant spatial and temporal scales. While part of the scientific community hold that, taking into account non-seismic precursors and given enough resourees study them extensively, prediction might be possible, most scientists are pes- simistic and some maintain that earth- quake prediction is inherently impos- sible. Predictions
272.7
Earthquake prediction @ @
269.9
3
predicted carthquakes
229.7
Earthquake forecusting @ @ prediction Earthquake used to predict earthquakes
217.2
2
211.5
1
205.3
7
carthquakes. Farthquake
-77.0
9
Seismic metamaterial @@@ Seismic risk Malta @
-97.4
-113.4
3
Quaternary (FP) @ used to predict the location...]
deemed signif- icant if they shown to be suc- cessful beyond random chance.[...
-150.3
1
-301.5
Precipice (Battlestar Gala[...]
7 Discussion
With SEAL we present solution that could potentially find applications outside information retrieval (e.g., enforce generated substrings come from a white list of trusted sources). While we conduct our experiments with model of -400M parameters (i.e., BART) for fast iterations, believe the use of larger models could considerably improve performance. Changing the model would not affect the size of the index nor the cost of using it ( [log|V|) for finding an ngram Moreover, we believe that indexing very large corpora (e.g.. the web) could be done more efficiently than existing attempts (e.g.,(iiktus et al. (202]) given the light memory footprint. Finally, dynamic variants [Gerlach] [2007] Salson et al. 20099 could allow the update of the FM-index on the fly without the need of re- re-indexing. While out of the scope of the current paper, plan to tackle some of these scaling challenges in future work.
8 Conclusion
In this paper we present SEAL,, a novel retrieval system that combines an autoregressive language model with compressed full-text substring index. Such combination allows to constrain the genera- tion to existing ngrams in a corpus and to jointly retrieve documents containing them. Empirically, we show an improvement by more than 10 points in average passage-level R- R-precision on KILT, and establish new state-of-the- model. While our results show that SEAL could already compete with more established retrieval systems, we believe there is potential in exploring existing (or yet to come) larger autoregressive models.
downstream performance on 4 out 7 datasets when paired with a reader
Broader Impact
While our decoding methodology provides a way enforce vorpus-based generation constraints, this does not fully prevent the generation and retrieval of non-factual or abusive text, since the corpus itself may well contain non-factual or abusive text. Moreover, even if given corpus misinformative or abusive, the language model could still produce undesirable text by "misquoting" material from it, e.g., selectively copying an utterance that is being criticized in the original context of appearance. Applications should allow user to check the original supporting text. Furthermore, while we have only used a relatively small language model, scaling up the autoregressive model SEAL LL would come with the same environmental risks that any other large system would pose, due to the more demanding energy requirements.
Acknowledgments and Disclosure of Funding
We thank Aleksandra Piktus, Edoardo Barba, Niccold Campolungo, and Pere-Lluis Huguet Cabot for their helpful comments and suggestions.
10

---

References
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, al. Efficient large scale language modeling with mixtures of experts. CoRR, abs/21 12.16644, 2021.
Yang Bai, Xiaoguang Li, Gang Wang, Chaoliang Zhang, Lifeng Shang, Jun Xu, Zhaowei Wang, Fangshan Wang, and Qun Liu. Sparterm: Learning term-based sparse representation for fast text retrieval. CoRR,
2020. URL lattts://aaxiiii wttp:/iww..rg/bb/////00/
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
M. Burrows and J. Wheeler. A block-sorting lossless data compression algorithm. Technical report, 1994.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading wikipedia answer open- domain questions. CoRR,
2017. httttp:///arxi..orr/bbb..
00051
Zhuyun Dai and Jamie Callan. Context-aware sentence/passage term importance extiatatttteteetttttiioiiiit estimation for first stage retrieval. CoRR, abs/1910.10687, 2019.
Zhuyun Dai and Jamie Callan. Context-aware document term weighting for ad-hoc search. In Yennun Huang, Irwin King, Tie- RenYan Liu, and Maarten van Steen, editors, WWW '20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020, pages 1897-1907. ACM / IW3C2, 2020. doi: (0.1145/33664233388822 URL /tt.///oi.ooo/ http//ii.orgggg//// wtt. 1145...1111/1331201 3380258
Nicola De Cao. Wilker Aziz, and Ivan Titov. Highly parallel autoregressive entity linking with discriminative correction. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 7662-7669, Online and Punta Cana, Dominican Republic, November 2021a. Association for Computational Linguistics. doi: 10. 121186611/22211111 http:////222..m//mmmmm.6664 URL
https://aclanthology) org org/2021. emnlp-main. .604
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. Autoregressive entity re- trieval. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021b. /RL/httts://opperevraie wwwp:/neeeeeeeeeee net/forum?id= 5K8F6UU39V
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wiz- ard of wikipedia: Knowledge -ooweeeee- conversational agents. Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Elena Simperl, and Frederique Laforest. T-rex: A large scale alignment of natural language with knowledge base triples. LREC, 2018
P. Ferragina and G. Manzini. Opportunistic data structures with applications. In Proceedings 41st Annual Symposium on Foundations of Computer Science, pages 390-398, 2000. doi: 10.1 109/ SFCSS2000.892127.
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. SPLADE v2: Sparse lexical and expansion model for information retrieval. CoRR, abs/2109.10086, 2021a. URL
https://arxiv.com org/ hbtpg//bss/./..... 10086
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. SPLADE: sparse lexical and expansion model for first stage ranking. In Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai, editors, SIGIR '21: The 44th International ACM SIGIR Conference on Reseurch and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, 1, pages 2288 2292. ACM, 2021b. doi: 10. 14553444444555 3463099 //doi //doi.org// 10. 1145/ 11453/44444 3404835 3463098
URL https

---

Luyu Gao, Zhuyun Dai, and Jamie Callan. COIL: Revisit exact lexical match in information retrieval with contextualized inverted list. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3030-3042, Online, June 2021. Association for Computational Linguistics. doi: 10.18v33/v1/ 10221naacl-mann mand-mm1i.2......
URL https: http:////aaaaatoooooyooo org/ 2021. naacl-main 241
Wolfgang Gerlach. Dynamic fm-index for collection of texts with application space-efficient construction of the compressed suffix array diplomarbeit im fach. Master's thesis, University of Bielefeld, 2007.
Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, and Alfio Gliozzo. Robust retrieval augmented generation for zero-shot slot filling. In Proceedings of the 2021 Conference Empirical Methods in Natural Language P Processing, pages 1939-1949, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/ org/ 2021. emnlp-main. 148
.00222eemllpmaaiin 148.. URL
Roberto Grossi, Ankur Gupta, and Jeffrey Scott Vitter. High-order
text in- dexes. In Proceedings of the Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA '03, page 841-850, USA, 2003. Society for Industrial and Applied Mathematics. ISBN 0898715385.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 874-880, Online, April 2021, Association for Computational Linguistics. doi:
74. URL
https://aclanthollee hww.tchhooogyoooooooooooooo www.....almmai.......
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. CoRR, abs/2202.03629, 2022.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601 1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/ v1/P17-1147. URL httpt::t////// wwwp///www./ooo aclweb. httpllewwww..reeeee.e.e.eooeeeee arg//anthllogg/P17--11
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. November 2020. Association for Computational Linguistics. doi: 10. 18653/v 1 h/2/2/.emmlpmmainnn...
URL
http://aaaaattooggyyyoo.o 2020. cammppmmann...m.mm 550
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL
Attp::/arxiv: org/ wwwooogg////baeeeeeeeee awsp////.... 6980
Mojtaba Komeili, Kurt Shuster, and Jason Weston. Internet-auumentte dialogue generation. CoRR, abs/2107.07566, 2021.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slay Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019. doi: 10. a URL https://acllanthology
Julian Labeit, Julian Shun, and Guy E. Blelloch. Parallel lightweight wavelet tree, suffix array and fm-index construction. Journal of Discrete Algorithmss, 43:2-17, 2017. ISSN 1570-8667. doi: 1tts://ooi.oog//...0.16.i..11111
articiee/pi//S115776677777870000 http::ttts:///w. http://www.cciinnciirrr.... com/science.
12

---

Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, and Nikolai Grigorev. Internet- augmented language models through few-shot prompting for open-domain question answering. CoRR, abs/2203.051 15, 2022.
Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, and Danqi Chen. Learning dense representations of phrases at scale. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1hth International Joint Conference on Natural Language Processing ( (Volume I: Long Pupers), pages 6647,, Online, August 2021a. a. Association for Computational Linguistics. URL https: https://aclantthology org/ 2021 acl- acl-long
doi: 10. 10.18866/1112222.acclllooog
518
Jinhyuk Lee, Alexander Wettig, and Danqi Chen. Phrase retrieval learns passage retrieval, too. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cessing, pages 3661-3672, Online and Punta Cana, Dominican Republic, November 2021b. Association for Computational Linguistics. doi: 10. 10.18655//112221..co http://022....mmmmmmm.999999 URL wwn.p-main.2mm2.mm.mmmmmmmmmmmm.
httpps//acllamthooog..ooo
Oleg Lesota, Navid Rekabsaz, Daniel Cohen, Klaus Antonius Grasserbauer, Carsten Eickhoff, and Markus Schedl. A modern perspective on query likelihood with deep generative retrieval models. In Faegheh Hasibi, Yi Fang, and Akiko Aizawa, editors, ICTIR '21: The 2021 ACM SIGIR International Conference on the Theory of Information Retrieval, Virtual Event, Canada, doi: 10.1145/34711558447222
July 11, 2021, pages 185-195, ACM, 2021.
URL https
//doi.org//iii
org. /1/ 1wt.11111///11883353 3472229
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. CoNLL, 2017.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising seuuennceto-sequenee pre-training for natural language generation, translation, and comprehension. CoRR, abs/1910. 134611,, 2019.
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks, 2020.
Patrick Lewis, Barlas Oğuz, Wenhan Xiong, Fabio Petroni, Wen tau Yih, and Sebastian Riedel. Boosted dense retriever. CoRR, abs/2112.007711 2021a.
Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. Question and answer test-train overlap in open-domain question answering datasets. In Proceedings of the 16th Conference the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1000-1008, Online, April 2021b. Association for Computational Linguisties. doi: 10.18653/v12021..eee www.eeaacc....e..eee
URL hatp:///alaathooogyy org/ 2021. wwc..eeallmmma wwcl-main.....a.m
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. PAQ: 65 million probably-asked questions and what you can do with them. Transactions of the Association for Computational Linguistics, 1:159--1115, 2021c. doi: 10.1 10.1162/tac__ a 00415, (tRhhttps:/// ptattps://aacaanhoooogg. org.olllooooooooooooooooooo org/ 2021/2221120 tac1-1.65
Jimmy Lin and Xueguang Ma. few brief notes deepimpact, coil, and a conceptual framework frameeewwrrrrmoorr for information retrieval techniques. CoRR, abs/2106. 14807, 2021. URL https:///axxx org/ abs/2106. 14807
Jean Maillard, Vladimir Karpukhin, Fabio Petroni, ten-tau Yih, Barlas Oguz, Veselin Stoyanov, the Annual Meeting the Association for Computational Linguisties and the 11th International Joint
and Gargi Ghosh. Multi-task retrieval for bhowledge-intensive tasks. In Proceedings of the 59th
Conference Natural Laguage Processing (Volume I: Long Papers), pages 1098-111 1111, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18833/v1/022 long.ong.89.
URL htttp::/aclaatthol.ggg www.llloooooggg..oo wttp://g.g/........... acl- 89
Antonio Mallia, Omar Khattab, Torsten Suel, and Nicola Tonellotto. Learning passage impacts for inverted indexes. In Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai, editors, SIGIR '21: The 44th International ACM SIGIR Conference on Research and
13

---

Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 1723-1727. ACM. 2021. doi: 10.114573400885534463003 URL https://doi. org/ 10. 1145/3404888 3463030
Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 4089-4100, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021 lon-- 316. httttttsps//aaaaaaataaaa
Chen. Generation -ugmented retrieval for
Generation-augmented
.rg 2021 acl- 1ocl-loong.33 .16..
Luca Massarelli, Fabio Petroni, Aleksandra Piktus, Myle Ott, Tim Rocktäschel, Vassilis Plachouras. Fabrizio Silvestri, and Sebastian Riedel. How decoding strategies affect the verifiability of generated text. CoRR, abs/1911.00 1.05587, 2019.
Donald Metzler, Yi Tay, Dara Bahri, and Mare Najork. Rethinking search: Making domain experts out of dilettantes, SIGIR Forum, 55(1), jul 2021. ISSN 0163-5840. doi: 10.1 10.11455/476415534776652 https://:///di..... org/10 10. 1145/347641111 3476428
Reiichiro Nakano, Jacob Hilton, S. Arun Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answering with human feedback. CoRR, abs/2112.09332, 2021.
Rodrigo Nogueira and Jimmy Lin. From doc2query to doctttitquery. 2021. URL https://cs uwaterloo.co httpa///iwwiiiiiiiallaatttiioiii
1alii__019__oooe d pdf
Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. Document expansion by query prediction. CoRR, abs//194.08875. 2019. Utttttp:://arii http://aaxxv...gabb/./.// wrgg/bs//////..... 08375
Cicero Nogueira dos Santos, Xiaofei Ma, Ramesh Nallapati, Zhiheng Huang, and Bing Xiang. Beyond [CLS] through ranking by generation. In Proceedings of the 2020 Conference on Empirical
Methods in Natural Language Processing (EMNLP), pages 1727, Online, November 2020. Association for Computational Linguistics. doi: 10. htt533//1/222.mmmlpppp main. 134. URL https //aclanthology. org/ 2020. conlp-main. 134
Barlas Oguz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Piktus, Xilun Chen, Sebastian Riedel, Wen tau Yih, Sonal Gupta, and Yashar Mehdad. Nomain-matchnn pre-training tasks for dense retrieval. CoRR, abs/2107. 13602, 2021.
Ashwin Paranjape, Omar Khattab, Christopher Potts, Matei Zaharia, and Christopher D. Manning. Hindsight: Posterior-guided training of retrievers for improved open-ended generation. CoRR, abs/21 10.07752, 2021. URL https://arxiv.orrt h1tp:////aaa11110715...
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? EMNLP, 2019.
Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. How context affects language models' factual predictions. AKBC, 2020.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 2523-2544, Online, June 2021. Association for Computational Linguistics. doi: (0.18653/v1/20/) .naacl 200. URL
https://aclanthollgg org 2021 mwwcaamainnnn.aaaaaaaaaaaaaaaaam 200
Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Dmytro Okhonko, Samuel Broscheit, Gautier Izacard, Patrick Lewis, Barlas Oguz, Edouard Grave, Wen-tau Yih, and Sebastian Riedel. The web your oyster - knoeleeggeintensivee NI.P against very large web corpus. CoRR, abs/21 12.09924.
2021. URL https::ss: // arxiv. org/ /tabs/2112. 09924
14

---

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners OpenAl blog, 1(8):9, 2019.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring. Susamnah Young, al. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs/21 12.1122.1444. 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683, 2019.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of language model? CoRR, abs/2002.08910, 2020.
Stephen E. Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr. 3(4):333-389, 2009. doi: 10.1551//100000009,, URL https
//doi.org/10 /551/1500000019
Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen tau Yih, Joelle Pineau, and Luke Zettlemoyer. Improving passage retrieval with zero-shot question generation. CoRR, abs/2204.07496, 2022. https::/: org.:/iiwiirrr.oooooo..o /tt//////b//24/049996
M. Salson, T. Lecroq. M. Léonard, and L. Mouchard. A four-stage algorithm for updating a Burrows- Wheeler Transform. Theoretical Computer Science, 410(43):4350 2009. doi: 10.101 tcs.2009.07.016. URL articcess/inall Larrsioo,_aurenntouuchaaaa _CS___d________ Sttppaccccccccc.oSeeeeeaaaaaaaaa MaximeCrochemore. pdf
Kurt Shuster, Mojtaba Komeili, Leonard Adolphs, Stephen Roller, Arthur D. Szlam, and Jason Weston. Language models that seek for knowledge: Modular search & generation for dialogue and prompt completion. CoRR, abs/2203. 13224, 2022.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deep- speed and megatron to train megatron-turing nlg 530b, large-scale generative language model. abs/2201.11990, CoRR, 2022.
Yi Tay, Vinh Quang Tran, Mostafa Dehghani. Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen, and Donald Metzler. Transformer memory differentiable search index. CoRR, abs/2202.06991, 2022.
James Thome,, Andreas Vlachos, Christos Christodoulopoulls, and Arpit Mittal. FEVER: a large- scale dataset for fact extraction and verification. In NAACL-HLT, 2018.
Ellen M Voorhees et The trec-8 question answering track report. In Trec, volume 99, pages 77-82, 1999.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201. 1100, 2022. /ttptttts///axx..... org aba.aaa//.222 11903
Ikuya Yamada, Akari Asai, and Hannaneh Hajishirzi. Efficient passage retrieval with hashing for open-domain question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 Ithh International Joint Conference on Natural Language Processing (Volume 2: Short Papers). pages 979-986, Online, August 2021. Association for Computational Linguistics. doi: 10. 16653//1/222..allll 123. URL https: httpt:ttts///aaaateet httpe////llaathhoggooooo
a.c.222..ccl-soorr... 123
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question
answering. Proceedings of the 2018 Conference Empirical Methods in Natural Language
Processing, pages 2369,, Oetober-November 2018. doi: 10.18653/v1/D18 1259. URL.
ftttss//wee aclweb. org/ http/aattooogg/DD-1155555 1259
15

---

Tiancheng Zhao, Xiaopeng Lu, and Kyusong Lee. SPARTA: Efficient open open-domain question an- swering via sparse transformer matching retrieval. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 565-575, Online, June 2021. Association for Computational Linguisties. LingguttisiaLLnnuutitssssss doi: 10.18653/v1/2022 wnwccllaanaanllllleee maiac44.. URL http:///aaaaaaooooooyooooo org/
org/2021. 2021 .naac-lmamm. .47
Shengyao Zhuang and Guido Zuccon. TILDE: term independent likelihood model for passage re-ranking. In Fernando Diaz, Chirag Shab, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai, editors, SIGIR '21: The 44th International ACM SIGIR Conference Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 1483-1492. ACM, 2021. doi: 10.1 10.1145/340483533662222
URL https: /ttp://oiiiooo org/ 10.
3462922
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Discussion
in the Appendix ($9)
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [Yes] The prediction code is available in the supplementary materials. For training, fairseq, library. We have reported hyperparameters in $5.2 and in the Appendix ( ($A)
used the popular
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're
using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "qhApTs72KJP", "reviewer": "Reviewer_wK2G", "summary": "This paper proposed to use autoregressive language models as search engines. Previous work in this direction has explored ways to do such retrieval by generating some unique identifiers of the documents. The authors proposed an even simpler approach -- using all ngrams in a document as possible identifiers and directly generating them using an autoregressive language model. This process was done through constrained decoding with the help of FM-index. They conducted experiments on the Natural Questions and the KILT benchmark, and the experimental results demonstrated that they could achieve comparable performance with the SOTA models while using a considerably lighter memory footprint.\n", "questions": "- By comparing Table 2 in this paper with Table 3 (Tay et al., 2022), I saw that the performance varies a lot on NQ320K even for BM25. Also, they basically achieved better performance on hits@1 while you achieved better performance on hits@10. Can you elaborate on this issue a bit more? This makes me hard to judge where this work sits in the literature.\n- In Table 5, I noticed that the performance of **SEAL (LM+FM, intersective)** outperforms **SEAL (LM+FM)** by a large margin (almost doubled), whereas it is not the case on NQ. Do you know why this happens?\n\n\n", "limitations": "The authors sufficiently addressed the limitations in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Originality: Although the idea of generating unique document identifiers using autoregressive models is not quite new, I do like the idea of generating the ngrams in a passage as possible identifiers. Also, I appreciate the efforts the authors put to make this simple idea work, as can be seen from the results, one needs to combine equations (2), (3), and (4) to make it really work.\n\n- Quality: The proposed approach is technically sound and interesting. As seen from the experimental results, they outperformed the prior autoregressive baselines on NQ and KILT. They also achieved comparable performance with DPR on NQ and outperformed DPR by a large margin on KILT. Note that the authors also claimed they achieved SOTA performance on 4 out of 7 datasets on KILT. Although this does not hold by now, I don't believe it diminishes the quality of this work.  \nOther comments I have: (1) From my perspective, the ngram length $k$ and the number of k-grams selected during training are very important hyperparameters (set to 10 in the experiments) but lacked detailed discussion over in the paper. I would like to see an experiment regarding the performance change over a different length of ngrams. It could just be a small-scale experiment but I think having those numbers would help consolidate the claims made in the paper. (2) Correct me if I am wrong but I do not find the details of how to select a fixed number of k-grams from a total of $|d| - k$ k-grams. How important is the selecting strategy? It would be better if the authors can establish a baseline using a random selection of the k-grams.\n\n- Clarity: In general this paper is well-organized and easy to follow. However, I think having a more formal definition of some notations in the paper could further improve the clarity. For example:   \nLine 142, does ngrams(K) denote $K$ ngrams or a set of k-grams?  \nLine 147, how is $P(n|q)$ computed?\n\n- Significance: I think this paper studied a very interesting problem and proposed a promising approach. Although the experiments are not comprehensive for the readers to understand every aspect of the system, I think it still brings many benefits to the research community. I believe it would lead to some follow-ups in this direction.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "p6Zj8JJ2YRp", "reviewer": "Reviewer_tLF8", "summary": "This work proposes SEAL, which  trains a language model to perform retrieval tasks leveraging constrained decoding over a FM-index of ngrams in a corpus. More specifically BART is finetuned to generate sampled 10 ngrams from each ground truth document biased in favor of ngrams with high character overlap with the query.\n\nSeveral scoring functions are explored including:\n(LM scoring) the score P(n|q) of the most probable  fixed-length ngram.\n(LM+FM scoring) the pointwise mutual information between query and ngram computed from P(n|q) and P(n) \n(LM+FM intersective scoring) aggregates the contribution of multiple ngrams\n\nFor page level task NQ320k, SEAL improve over the quality of a much larger retrieval solution (GENRE).  For passage level task (NQ), SEAL significantly improve over DSI, which suffers capacity losses when memorizing document ids. SEAL produces comparable results to DPR and GAR (especially for A@100 which is the most important for QA purposes). More detailed analysis shows that SEAL is a lot better on novel questions or answers. \nFor KILT passage retrieval task, SEAL improved the state of the art result significantly.\nFor KILT downstream tasks, SEAL improves state-of-the-art for 4 out of 7 tasks.\n\nThis work represents a significant progress for learned indexing structures -- combining LM with multi-point indexing and scoring.\n", "questions": "how is the ngram sampling done to  bias towards overlap with queries? there seem to be an lack of details.\n\nwhy only train from the ground-truth doc? why not leverage unsupervized training for retrieval?\n\nit seems that the model is never trained to optimize ranking quality directly. Is there any possibility for learning to rank?\n\n", "limitations": "see the summary", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "see the summary", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "iXxGqu3a6x", "reviewer": "Reviewer_EwCK", "summary": "This paper proposes a novel scheme to apply autoregressive language models decoding to retrieval tasks, in which documents are  represented using all constituent n-grams as possible identifiers. The key idea is to use an FM-Index to prevent the generative model from producing document identifiers with text outside of any of the indexed documents. The appeal of the approach is that large language models can be adapted (fine-tuned) for this task without major architectural changes. Empirically, the approach performs competitively with a selection of recent baselines, in some cases outperforming them.\n\n", "questions": "* What are the implications of the assumption that documents are indexed by exact n-grams? For example, it would be good to provide intuition for why this does (or does not) preclude matching documents via synonyms or related words / expressions.", "limitations": "* I would have liked to see more discussion of inference-time latency.\n", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper does a good job explaining current drawbacks of autoregressive models in retrieval tasks (related work) and the empirical results demonstrate that the proposed scheme does improve the quality of retrieval results.\n\nMy main concern is that there may be better ways to apply large language models for retrieval, such as the re-ranking approach that is mentioned in the introduction; this wouldn't require complex generation involving indices and constrained decoding. Indeed, re-ranking shares the purported benefits of the proposed approach, namely straightforward application of large language models without architectural changes, with the key advantage that it only requires *scoring* a (query, target) pair. Autoregressive models, by definition, suffer from slow decoding speed, and therefore I'm skeptical the latency would be low enough for this approach to be practical in many settings.\n\nRegarding the experiments, not enough motivation is given for why the baselines were selected, and it's not clear whether these truly represent SoTA results. For example, do you compare to a re-ranking baseline? There are many approaches from the IR community such as ColBERT that seem relevant but are not discussed.\n\nRegarding the selected baselines, it doesn't look like the baselines leverage any sort of constrained decoding. It would be interesting to constrain decoding using simpler alternatives to fully understand the impact of the FM index and proposed weighting scheme.\n\nThe paper is readable and well-written. Additionally, the authors detail their experiments well and share their code which bodes well for reproducibility.\n\nL37: extend->extent\nL230: \"our the size of our index is\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "X9X0xDxuSH", "reviewer": "Reviewer_L2Y4", "summary": "- The paper presents an approach (called as SEAL) for document retrieval where a language model conditioned on a question generates n-gram tokens to identify the relevant documents from the evidence documents (or passages). To enable this functionality, the method trains BART using question and n-gram pairs, where the n-grams are sampled from the gold passages. To constrain generation to output valid n-grams such that they correspond to some documents, the approach indexes the evidence documents with an efficient datastructure called FM index. The paper presents several ways to score the generated n-grams, one where the n-gram corpus level frequency is considered and another one where  they try to score multiple n-grams using intersection of tokens. They experiment with Natural Questions (NQ) Open dataset and the KILT tasks to showcase the competitiveness of their proposed approach.", "questions": "Please see the points under the Weaknesses section.\n\n** EDIT **\nIncreasing the rating to 6.", "limitations": "Yes, the authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n - The biggest strength of the paper is the novelty of the proposed approach. While the dominant classes of retrieval methods consist of training dual encoder models with pre-computed evidence indexes, this paper presents another method with which retrieval can be performed. \n\n- The paper is well-written, organized, sound experiments, and related work section is good. The authors have submitted the code-base which will make the work reproducible. \n\n\n**Weaknesses and Suggestions for Improvements**\n\n- Apart from NQ, the paper does not perform evaluation on popular QA datasets such as MSMARCO, TriviaQA, SQuAD, WebQuestions, and Entity Questions where the performance of baseline models are well established. These are the datasets where a large fraction of the retrieval papers report results and benchmark their models. SEAL should also train a common model on all these datasets in the DPR-Multi style to assess the generalization of their approach.\n\n- Performance comparison with DPR and other strong dual encoder models. From the results in Table 3, SEAL falls behind DPR on Accuracy@ 5 and Accuracy@20, where its performance is considerably low. I suggest it’s important to highlight this limitation and probe further the reasons. There should be a more rigorous performance comparison with improved dual-encoder training approaches which obtain much better results. For example, the model ANCI (https://openreview.net/forum?id=zeFrfgyZln) and ICT-DPR and MSS-DPR in (https://arxiv.org/abs/2101.00408) obtains better performance numbers than DPR and these results should be reported in Table 3.\n\n- To understand the dependence of SEAL model on number of training examples, it will be useful to compare the sample efficiency of SEAL algorithm with that of DPR on the NQ training examples.\n\n- Considering the number of trainable parameters, where DPR trains 220M parameters while SEAL trains ~400 M parameters, the direct comparison with DPR is not fair. This should be clearly highlighted in the paper. Are the benefits of SEAL model on KILT tasks mainly due to the trainable parameter size? It would be good to finetune smaller and larger generator models such as different configurations of T5 (or T5 lm adapted models) and then study the correlation of performance vs model size.\n\n- The writing in the method section seems a bit dull, especially the paragraphs of “factoring in FM-index frequencies” and “intersective scoring for multiple n-grams”. Using visual illustrations and diagrams to convey the information would help the reader understand the importance of these ideas more.\n\n- Probably a minor point, I  feel that the training / inference time of SEAL / DPR should be calculated using the same compute hardware and then compared. Also, the 64 GB of DPR index size in Table 2 is when using 32 bit representation for floats. It has been shown that 16 bit or more optimized representation of passage embeddings also works just as well (https://arxiv.org/abs/2106.05346). Similarly, GPU is not always required to perform fast search. Toolkits such as ScaNN (https://arxiv.org/abs/1908.10396) also works well on CPU.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept