1-PAGER: One Pass Answer Generation and Evidence Retrieval
Authors: Palak Jain
Livio Baldini Soares
Tom Kwiatkowski
Venue: EMNLP 2023 Findings (2023)
Keywords: retrieval
openbook qa
generative retrieval
TL;DR: 1-Pager is the first system that answers a question and retrieves evidence with a single Transformer model and decoding pass.

Abstract: We present 1-Pager the first system that answers a question and retrieves evidence using a single Transformer-based model and decoding process.
1-Pager incrementally partitions the retrieval corpus using constrained decoding to select a document and answer string, and we show that this is competitive with comparable retrieve-and-read alternatives according to both retrieval and answer accuracy metrics. 1-Pager also outperforms the equivalent `closed-book' question answering model, by grounding predictions in an evidence corpus.
While 1-Pager is not yet on-par with more expensive systems that read many more documents before generating an answer, we argue that it provides an important step toward attributed generation by folding retrieval into the sequence-to-sequence paradigm that is currently dominant in NLP.
We also show that the search paths used to partition the corpus are easy to read and understand, paving a way forward for interpretable neural retrieval.

1-PAGER: One Pass Answer Generation and Evidence Retrieval
Palak Jain"
Livio Baldini Soares
Tom Kwiatkowski
Google Research (palakj, liviobs, Sankkwat)@@ooglle.
Google Deepmind
Abstract
Q: what the share of agriculture in indian economyy?
We present I-PAGER the first system that an- swers a question and retrieves evidence using a single Transformer-based model and decod- ing process. 1-PAGER incrementally partitions the retrieval corpus using construined decod- ing select document and answer string, and we show that this is competitive with com- parable retrieve-and-ree and-read alternatives accord- ing to both retrieval and answer accuracy met- rics. 1-PAGER also outperforms the equiva- lent 'closed-book' question answering model, by grounding predictions in evidence cor- pus. While 1-PAGER is not yet on-par with more expensive systems that read many more documents before generating an amswer, we ar- gue that it provides an important step toward attributed generation by folding retrieval into the sequence-to-ee to-sequene paradigm that is cur- rently dominant in NLP. We also show that the search paths used to partition the corpus are easy to read and understand, paving a way for- ward for interpretable neural retrieval.
Search path
Answer
India
Apricul
Matched docs out of 21M
200 deox
B doos
Plaraac Conscery of ledia (.ddi renios secaadd worrdeess in harm output Agricultare
employs 45% oiia tera voritaie 1n 20 Agfis Bune accountio unted for (39% of G.0 and
Antwwrr and evidence passage
by
Figure I: Example IP output that iteratively partitions the corpus sub-sets containing the generated n- grams. The last n-gram is taken as the answer.
Motivated by the observation that language model decoders already perform search over pos- sible sequences (Graves, 2012), and that evidence documents themselves are simply sequences of tokens, present alternative approach that relies on single Transformer model. In this approach, which we name 1-PAGER (One Pass Answer Generation and Evidence Retrieval) or sim- ply IP, the decoder iteratively partitions a corpus of evidence documents by generating search path consisting of a set of keywords that identify rele- vant documents and an answer string that is con- tained in at least one of these documents. With IP, aim to explore the spectrum between CBQA, where the answer is generated without reference an evidence corpus, and pipelined approaches that feed retrieved documents into the transformer.
1 Introduction
recent times, there has been a push to re- formulate a wide variety of tasks from NLP and other domains into the sequence-to-seequence paradigm, to make use of large pre-trained Trans- former networks (Vaswani al., 2017). However, despite evidence that large language models can of- ten answer questions (Roberts al., 2020), predict identifiers of documents that support those answers (Tay et al., 2022), or generate text that contains and explains an answer (Yu et al., 2022) the dominant paradigm in question answering is still the retrieve- and-read approach that pipelines separate retrieval and answer generation modules. This approach has the benefit that it can provide direct and targeted paragraph-Ievel attribution for the generated an- swers (Bohnet et al., 2022). However, also relies on a heterogenous mix of models that are hard to train in concert (Metzler et al., 2021).
Figure 1 illustrates an example in which the cor- pus is iteratively partitioned into documents that contain the string 'Economy of India', then those that also contain the string 'Agriculture', and finally those that also contain the answer string '23%'.
IP output sequences are guaranteed match at least one document in the evidence corpus. This is enforced via a constrained decoder that has ac-

---

cess an FMM-index representation of the evidence corpus contents (Ferragina and Manzini, 2000) and we evaluate IP's ability to correctly answer open- domain questions while also retrieving passages that provide support for those answers (Bohnet et al., 2022). Since IP is the first model that can do both of these tasks, we compare to pipelined systems that first retrieve single passage and then generate answer based on this evidence pas- sage. IP competitive as a passage retriever, per- forming similarly widely used dense retriever (Karpukhin et al., 2020) and outperforming the SEAL system which independently generates key- words rather than a search path (Bevilacqua et al.. 2022). IP also outperforms an equivalent closed- book question answering (CBQA) model (Roberts et al., 2020) according answer accuracy. Part of this improvement comes from the prediction of search paths themselves, reminiscent of chain-of- thought reasoning (Wei et al., 2022), and part from 1 P's constrained decoder, which forces the model to generate answers from passages that con- tain the keywords.
ing hard negative training (Xiong et al., 2020), late interaction (Khattab and Zaharia, 2020; Santhanam et al., 2022), few-shot learning (Izacard et al., 2022), joint retriever and reader training (Jiang et al., 2022).
A particular variant of interest is the Iterative Retrieval process where the query is reformulated incrementally (Das et al., 2019; Lee et al., 2022) leading to an interactive search process (Jiang et al.. 2023; Adolphs et al., 2021). This query augmenta- tion scheme has similarities with our use of search paths. However, we use the paths to iteratively par- tition the corpus while prior works have used it for refining the query.
To perform well, retrivee-and-eeal systems will typically retrieve 10s to 100s of passages that must be processed by a language model. In constrast, IP retrieves and extracts an answer in single pass of language model generation.
Closed Book Question Answering With data and parameter scale, LLMs in a closed-book setting (CBQA) have shown competitive perfor- mance (OpenAI, 2023; Anil et al., 2023; Yu et al., 2023) to retrieval pipelines (ODQA), however with- out producing any attributed passages (Rashkin et al., 2021; Bohnet et al., 2022). An extension of CBQA post-hoc retrieval where large language model LLM) is first used to generate an answer and then evidence for the question-answer pair is fetched by retriever (Gao et al., 2023a; Bohnet et al., 2022). While post-hoc retrieval serves the same goal as IP, it still uses pipeline of LLM and retriever to do so.
While IP does not yet perform as well as the very best retrieval or open-domain question an- swering systems in terms of accuracy, the fact that it is competitive with pipelined systems that are trained with the same data and which use simi- lar amounts of inference-time compute suggests a promising path ahead. Unlike those systems, IP can be trained end-to-end along with any other task that fits into the sequence-to-seve se-queece paradigm. Additionally, IP search paths are inherently inter- pretable, unlike embeddings used in dense retrieval.
2 Related Work
Generative Retrieval Recently, generative re- trieval has emerged as an alternative to the conven- tional "retrieve-and-read pipeline (Metzler et al.. 2021). Genre (De Cao et al., 2021) performed generative entity linking by constraining model's decoding to set of entities. DSI (Tay al., 2022) showed one of the first proof of LLM's ability to memorize docids in the corpus. However, atomic ids or hierarchical clusters, as used in DSI, are opaque identifiers and capture limited information. Works such as SE.A (Bevilacqua et al., 2022) and Ultron (Zhou et al., 2022) use semantically richer representation: keywords in the document. In par- ticular, SEAL constrains the generation only keywords in the corpus using the FM-index (Fer- ragina and Manzini, 2000), a key data structure we borrow in this work.
"Retrieve-and-read" Question Answering Question answering approaches in NLP dominated by the "retrieve-and-ren" where retriever first fetches hundreds of rele- vant documents from corpus, followed by a language model that reranks and extracts the answer (Harabagiu et al., 2003; Chen et al., 2017: Zhu et al., 2021). Sparse retrievers such BM25 (Robertson et al., 2009) build a high- dimensional lexical index over text corpus. Dense retrievers (Karpukhin et al., 2020) use dual encoder architecture to embed the query and document and perform an approximate nearest neighbor search. Various modifications to dense retrieval have been proposed over the years includ-
paradigm

---

Q
LLM
A
P
Reader
Retriever
P
Reader
D
Figure 2: System illustration different QA systems. From left to right: CBQA, 1-PAGER, SEAL, Retrieve- and-Read system. C denotes the retrieval corpus, retrieved passage, the input question and the generated answer. IP is closest to CBQA (only single model used) but it also outputs passage retrieved from C.
IP represents docids as keyword paths, which are arguably more interpretable, and learns soft partition over the corpus instead of the hard parti- tion imposed by DSI's clustering.
D by predicting n-gram keywords. An n-gram of tokens said to be contained in a document d, denoted by < d, when is a sub-sequence of d. We define a keyword corpus partitioning function
Another crucial distinction is 1P's ability to both retrieve and generate an answer while prior works rely external re-ranker/reader for the same. A high-level view of various question-answening systems presented in Figure 2.
F(D,k) = (dk
D}
that selects only those documents that contain k. 1 I-PAGER iteratively partitions the corpus by gen- crating a sequence of n n-grrams that we refer to as Search Path p: = [h, h Ai]. Each prefix of this search path defines subset of D via the search path corpus partitioning function
Attributed Question Answering Standard met- rics for open-domain question answering, such exact match token-based F1, have received criticism for being imprecise and/or insufficient. Several efforts have proposed augmenting an- swers with textual evidence, via retrieval cita- tions (Bohnet et al., 2022; Menick et al., 2022; Gao et al., 2023b). While this work does not directly evaluate the quality of retrieved answer evidence, our proposed model inherently produces passage to support the final answer, along with a search path of keywords, which could be used to provide users with answer evidence.
P(D.P) Dp. = (((e)L.FF(DDA
and each subsequent keyword kx+1 narrows down Dp: into further sub-spaces such that Dp S Dp--
A Answer prediction is treated in exactly the same way keyword selection and in IP the last key- word from p is taken as the answer.
4 Constrained Decoding and FM-Index
3 Iterative Corpus Partitioning and Answer Prediction
To avoid generating empty partitions, we constrain I-PAGER to only decode search paths that match at least one document. We modify the decoder's ss beam- searhsearch strategy to only allow keyword contin- uations that are contained in the current partition.
We focus on the problem of learning a mapping f(g. D) + (a, da) from a question q and corpus documents to an answer and supporting doc- ument (a, d. The predicted document d, is re- trieved from D and the answer a is a sub-string of dg. The document da should be relevant to the question and provide evidence for answer.
Given document subset Dpp, which could be the full corpus D at the start of decoding (i 0), and a keyword prefix k, which could be empty, the of all valid continuation tokens is defined as,
The goal of this puper is to model the function using a single ssquence-to-seqeene model, rather than a pipeline which first retrieves d, and then feeds into an answer generation module. To achieve our goal, we recast retrieval as an iterative corpus partitioning process illustrated in Figure 3.
C(k. Dr.) {ar k|| d, d Dr.))
where is any vocabulary token and || indicates concatenation of two token sequences. As special case, when & and O. all tokens in D are valid continuations. 1P separates keywords in pr with a special separator token - and marks the end the sequence with an EOS token. These two tokens are always valid continuations.
Iterative corpus partitioning adopts the LM de- coder's autoregressive search process to partition

---

Q: location of the ten commandments the bible?
Token Probe.8
books of Exodus...
Story of Exodus
Bible
locate
Ten Comma ndmmet
Ten Iiicc in
Doc Constraints
21M doca
doc
books of Bible
Story of Exodu
locate
Ten Commandments
twice in the Hebrew Bible
books of Exodus
Title: Tan Commerdfments The Ten Commandments (. "Aseret he a''Dibrot"), also known the Dacalague, are of biblical principles relating to othics and worship, which play appeer lwice The Habrew Bidee in the books of Exodus and Deuleronomy. Modemn..
Passage:
role
Christianlly...
Commandments
Answer: books of Exodus
Figure 3: Illustration of the IP decoding process. A keyword can only be generated from the documents matching previously generated keywords. Right panel shows a magnified view applying constraints to a decoding step. Any keyword not present in the documents is masked out.
Consider Figure 3. The three keywords corre- spond to the decoded token sequence [Ten, Com- mandments, -, twice, in, the, Hebrew, Bible, -, books, of. Exodus, LOS]. At the start of decoding, any token in D is allowed. After decoding Ten, only those tokens that follow Ten an n-gram in are allowed, along with the default separa- tors. After decoding [Ten, Commandments, +] we are ready to start new keyword, but only to- kens from documents that contain the keyword Ten Commandments are allowed. Decoding continues in this manner until BOS is generated.
Training data generation
For training 1P, we produce dataset with exam- ples of queries and search paths as described above. At a high-level, we generate search paths by itera- tively selecting n-grams from an answer passage, and simulating, using the FM-Index of the retrieval corpus, the partitioning of the corpus after selecting each keyword, until only a few documents remain. Finally, the answer span is appended to the search path. Each example produced be serialized lequence-to-sequeee pair of inputs and targets as:
To efficiently implement these constraints, we need structurruuuu that can quickly determine both Clk, Dp). ). the continuation tokens given a doc- ument set and P(pp,k k). the subset of documents that contain a given path.
inputs: Generate keywords for: <q>? targets:
K_SEP Ko K_SEP bn ... K_SEP A_P EOS
5.1 Keyword Selection
A good keyword should have a) high relevance to the query and b) effectively narrow down the search space. To identify relevant keywords, we restrict to only the gold document g. All ngrams in g of length up to five are extracted. Irrelevant keywords are filtered out such as those starting or ending with stop words. Similarly, keywords that are too rare in the corpus, e.g., "Philippines at Luzon" or too frequent, e.g., "part" are excluded based on a threshold their count in corpus. The remaining keywords are scored with combinations of heuris- tics, mainly Rouge-1 similarity with the query (Lin, 2004) along with minor award for keywords con- taining entities and penalty for keywords highly frequent in the corpus.
For this, we extend the usage of FM- index (Ferragina and Manzini, 2000) described in (Bevilacqua al., 2022). The FM-index compressed loken-based index over corpus Do with a few important properties for our usage: (1) it can efficiently list possible token continuations for a sequence prefix that occur in Do i.e., C(k, Do). (2) can list the set of documents in the corpus that match n-gram i.c., F F(Do, k), and (3) it supports search over arbitrary n-grams that occur within doc- uments. Note that the F-Mindex operations are optimized for Do. the original corpus is built over. We extend these to an arbitrary Dp C Do at additional cost deseribed in Appendix A.1.

---

This scoring mechanism often misses on key- words that are semantically relevant, but do not lexically overlap with the query. boost the rele- vance of our keyword set, we re-score the top hun- dred keywords using a language model. A T5-XXL. model is finetuned with the imput the query q and target as either the title or a heuristically sampled keyword in similar fashion to Bevilacqua et al. (2022). The heuristically sampled keywords re-scored using this model obtain refined LM- scored set. Two other special types of keywords are awarded high scores: Title of the gold passage and the keyword containing the answer string a.
For our experiments, we use the T5X (Roberts et al., 2022) framework. A T5-XXL 1.1 (Raffel et al., 2020) model is finetuned with a batch size of 256 and dropout of 0.1. No additional hyperparam- eter tuning is performed. We format search paths using the reserved tokens K_SEP = "<extra id 0>" and A_SEP = "<extra id_
6.3 Inference
Our best model employs beam decoding with a beam of 5. Even when the beam is greater than one, only the top-beam result is used for retrieval. We discuss the effeet of beam size in depth in Section 7. Given the top generated path p. Dp corresponds to the retrieved documents. In case [Dpl > 1, document is sampled arbitrarily for evaluation.
5.2 Search Paths
The first keyword in a search path needs to effec- tively partition the corpus. We experiment with ei- ther the title or the highest scored keyword from the gold passage as the first keyword in the path. The next keywords are sampled based on their score, given they do not overlap with any of the existing keywords in the path. We continue augmenting path p with keywords until most passages in the corpus match i.c., |ppl 10. The answer keyword is then appended to the path. Our train paths (including the answer) contain a median of three keywords and one matching document.
6.4 Baselines
We compare to a closed-book question answering (CBQA) system that generates answers, but does not ground these in evidence corpus, as well retrieve-ands aandvve- systems that combine a variety of retrievers with Transformer-basee answerer module. Both the CBQA baseline and the answerer module are derived from the same T5-XXL 1.1 pretrained model as IP.
6.4.1 T5-CBQA
A T5-XXL 1.1 model fine-tuned predict an- swers from the DPR training for 10,000 steps with a batch size of 128. Note that it is possible to achieve a higher closed-book performance on NQ using the full Open-NQ training split instead of the subset included in the DPR training set (Roberts et al., 2020). However, to enable meaningful com- parison we restrict the CBQA baseline to the same training examples used to train IP.
6 Experimental Setup
6.1 Datasets
We use Open-NQ (Kwiatkowski et al., 2019; Lee al., 2019) the question-ansserrigg dataset for training. For evaluation, besides Open-NQ, We- bQuestions (Berant et al., 2013) and CuratedTREC (Baudis and Šedivý, 2015) are used to measure out-of-domain performance. The FM-Index cor- pus for constrained decoding is built over DPR Wikipedia corpus with woodword splits (Karpukhin et al., 2020). The positive gold passages from DPR are used for sampling training paths. This setup is chosen mirror SEAL and also permits fair comparison against DPR.
6.4.2 Retrieve-and-Read
The retrieve-and-read baselines first retrieve a sin- gle passage from the evidence corpus, and then feed this passage and the question into the answer generation module2. We report retrieval accuracy for the retrieved passage and answer accuracy for the generated answer.
6.2 Training
T5-Reader We tune T5-XXL 1.1 model to gen- erate answers from (question, evidence passage)
IP's training dataset contains 310k paths corre- sponding 55k queries from Open-NQ. Majority of the training paths begin with the title, with small fraction starting with other keywords (12%). All keywords, except the title, are scored using the LM-scoring technique described above.
Phttps://gov http://oooolletttttteeeoe sleett-cchcckoints
"This differs from ODQA evaluations that do not include evidence retrieval as first-class task, where many retrieved passages are fed into a reader that generates an answer without attribution to any single piece of text.

---

pairs. This is the same base model used by 1P and we train on the (question, passage, answer) triples in the DPR training split to ensure fair comparison.
7 Results
Retrieval Hits @
Answer
Retriever
Answerer
DPR-Retriever We compare against vanilla DPR finetuned on NQ without hard negatives (Karpukhin et al., 2020) using the pre-computed index available on DPR's repository'. We note that our ODQA setup differs from the one used by Karpukhin et al. in that choose the highest scoring retrieval evidence for answer generation, instead of generating from the top-100 passages without attribution.
EM
F1
T5 CBQA
26.8
34.0
BM25
T5 Reader
23.6
17.9
24.0
SEAL
T5 Reader
37.9
29.4
35.8
DPR
T3 Reader
46.5
35.6
42,4
IP
T5 Reader
46.3
34.2
41,4
IP
Unconstrained
29.3
29.3
36.1
IP
46.3
31.7
38.0
BM25-Retriever We use Pyserini toolkit (Lin et al., 2021) with default configurations, retrieving the top-1 passage.
Table Comparison different Retriever and An- swerer combinations the NQ-Open test set. In retrieve-and-read setups, answers are generated from the top-1 retrieved passage. IP combines passage re- trieval and answer generation in single prediction.
SEAL-Retriever SEAL (Bevilacqua et al., 2022) is a generative retrieval system that gener- ates a set of keywords constrained on the corpus. In terms of technique, IP borrows inspiration from SEAL's 's use of the FM-Index as well as keywords- as-identifiers. However, the two setups have sub- stantial differences that we highlight Section We run SEAL with default configuration and beam of 5 using the publicly released checkpoint based on Bart-large (Lewis et al., 2020). All out- puts from the beam are used for retrieval.
WebQuestions
TREC
System
Hits @
EM
Hits @
EM
BM25 Rdr
19.7
14.2
35.2
29.1
DPR
Rdr
32.0
17.3
51.6
35.0
IP
+ Rdr
38.0
20.4
63.8
38.5
IP
38.0
20.5
63.8
36.4
Table 2: Comparison of different Retriever and An- swerer combinations on Out-of-domain datasets. Both the Retriever and Answerer (Rdr) trained on only Open-NQ. In retrieve-and-read setups, answers are gen- erated from the top-1 retrieved passage.
6.5 Evaluation
We evaluate in-domain performance on the Open- NQ test split and out-of-domann performance WebQuestions (WQ) and CuratedTREC (TREC) following the setup from Karpukhin et (2020). Passage retrieval performance is measured with Hits@I using Pyserini evaluation scripts".
We compare to the baselines described in Sec- tion 6.4 on Open-NQ using both retrieval and an- swer accuracy metrics in Table 1. Answers are generated based on the top retrieved document in systems that separate retrieval from answer gener- ation, to provide a clean comparison between sys- tems that return (answer, evidence passage) pairs. Table reports the out-of-domain performance of various systems on WQ and TREC.
6.6 IP configurations
We experiment with three configurations: a) IP: Our primary setup that uses both training and con- strained decoding procedures described above, pro- ducing retrieved passage as well as answer. b) P-Unnonstrriined: Only the training technique described in Section 5 is adopted, with standard unconstrained decoding. Since generation is uncon- strained, it possible that no passage gets retrieved for given path. c) IP + Reader: Here, we take the top retrieved passage from IP and input it to the Reader model (Section 6.4) to extract the answer.
IP outperforms CBQA in question answering and beats the retrieve-and-reat systems, BM25 and SEAL. On the passage retrieval task, it signifi- cantly improves over BM25 and SEAL. For in- domain setting, IP is competitive with DPR on retrieval task, but lags behind the QA pipeline that uses DPR. However, this appears to be more due to the reader rather than the retriever as discussed in Section 8. It is worth noting that 1P general-
t/gitp:://ithubb... http://shhhtacmccfaaceeooreeeeee /tttts://github.. http:tttww..oooccstttrnn///esses

---

izes significantly better out-of-domain compared other systems.
carded. Note that unconstrained decoding does not gain from planning. In the final setup in Table 1, we use a beam of for both IP and SEAL. Un- like IP, SEAL uses all the outputs from the larger beam for retrieval.
Utility of Search Paths IP- IP-Unconstrained can be viewed an extended version of CBQA that generates a search path before predicting the an- swer. Thus, improvement of i--nnoonstriined over CBQA can be attributed to this path- conditioned answer generation process, analogous to chain-of-though reasoning (Wei et al., 2022; Lampinen et al., 2022).
Discussion and Ablations
Generating Answers While IP is capable of generating answers, Table 1 highlights that falls behind the 1P+Reader. The reason seems be clear: the Reader has visibility into the full passage context while IP limited to the decoded search path and the constrained index which only ensures that generations are grounded in the corpus. Since IP does retrieve passages, it would be possible to pull in the corresponding text input for answer generation. We leave this future work.
Constrained Decoding
Beam
System
5
CBQA
No
26.7
26.8
IP Unconst.
No
29.0
29.3
SEAL + Reader
Yes
28.5
29.4
1P
Yes
28.7
31.7
Comparison to SEAL While IP takes inspira- tion from SEAL, in practice, there are few key differences between the two systems aside from IP's answer generation.
Table 3: EM for various decoding setups with differ- ent beam sizes on Open-NQ. Only top-beam result used for evaluation, except in SEAL which uses all beam outputs. IP constrained decoding benefits the most from a large beam whereas Unconstrained setups have only slight effect.
SEAL generates a large set of keywords (Ta- ble 4) using many separate decodes and heuristic guidance (Appendix A.3). In contrast, IP decodes single sequence of about three keywords.
Effect of Constrained Decoding The purpose of constrained decoding is to ground the answer in an evidence retrieved from the corpus. As ex- pected, the constrained setup enables IP to achieve higher Hits@ than |P-unconstrained. Surpris- ingly, when decoding with beam of one, we ob- serve small drop in answer accuracy for 1P com- pared to 1P-Unconstrained (Table 3). Inspecting the losses, two dominant reasons surface. Firstly, As DPR passages are chunked into 100-words (Karpukhin et al., 2020), some queries may be- come unanswerable given single passage due to missing context. This is disadvantagcous when the model has memorized the answer but there is no single passage to attribute it to.
SEAL
1P
Median keywords
32
Median does retrieved
500
Generates answer
X
Table 4: Key differences between SEAL and 1P mea- sured over Open-NQ test split with beam of 1.
The SEAL keywords are a set, decoded inde- pendently of each other and re-scored using so- phisticated techniques to retrieve large number of documents. For instance, the default config- uration in SEAL retrieves up 500 documents. This makes SEAL suitable be employed in con- junction with re-ranker. In contrast, search path's 's map directly to a single (or few) relevant documents (Appendix A.6).
Secondly, during constrained decoding, after generating the initial keywords, the search space may soon become sparse with no good candidates to pick from. Could a larger room for planning its actions help the model here? Indeed, increasing the beam size to 5 improves performance by 3% (Table 3), even when only the top-beam is used for retrieval. We refer to this as Planning, since the larger beam only enables the model to plan better and the remaining beam outputs are otherwise dis-
We acknowledge the model-size variation be- tween SEAL and IP in the reported experiments, however preferred using the publicly avail- able SEAL checkpoint. Given the discrepancies with larger beam-size, multiple decodes and use of Reader model, difficult to have an apples to apples comparison between the two systems.

---

Path vs Keyword set We qualitatively observe that keywords in a 1P path, owing sequential generation, are distinct and add new information as compared to the SEAL output set where over- lapping keywords are common (Appendix A.3). Thus, paths are advantageous for precisely narrow- ing down to single relevant document while key- word sets are effective for retrieving a large number of documents that can later be reranked. This is corroborated by the fact that IP is better Hits@ I while SEAL is better Hits@5 (Appendix A.4).
returns no match. As a result, constrained decoding drops the predicted keyword altogether, resorting to lower ranked keywords in the beam. We par- tially fix the issue by modifying the answer in a fraction of the training data to include surrounding punctuation tokens based on how they appear in the FM-index. For instance, the keyword "I Ran ..." would update to "(I Ran ... This simple change leads jump in answer accuracy from 26.4% to 28. 7%. However, much more work needed make IP robust variations in tokenization.
See Appendix A.2 for analysis of training data size and Appendix A.5 for masking logits vs log- probs.
Qualitative Analysis Table 5 illustrates patterns of Search Paths generated by IP. We note some of the common path patterns here:
1) First keywords are entities in the query, fol- lowed by query predicates that itcratively narrow down towards an answer. This is the most common type of path observed and can be attributed to the dominant presence of title in the training data. 2) Rewrites of the original query related pred- icates such as "seasons consists of", "appeared on...... ...". Such paths are more prevalent where there is no canonical entity in the query or no entity can be determined with high confidence.
Conclusion
We introduce 1-PAGER, the first system to perform question answering and passage retrieval in one pass with single language model, using con- strained decoder to iteratively partition the retrieval corpus and then generate an answer. We show com- petitive or improved performance over a variety of comparable baselines and carefully analyze the results, ablating both training strategies and decod- ing style. We also provide a qualitative analysis of predictions to illustrate the system's capabilities. Challenges with constrained decoding are surfaced including poor search spaces and sensitivity to tok- enization and mitigation strategies are presented.
3) Answer is directly generated followed by sup- porting keywords that guide towards an attributed passage. This happens in small fraction of cases, likely where the pretrained model has memorized answer with high confidence.
Overall, we find the generated search paths to be fairly meaningful and interpretable.
We hope that IP adds value in demonstrating how a single transformer model can be harnessed to do both retrieval and answering and pave the path for further progress in the generative retrieval domain.
Sampling Search Paths for Training Table 6 highlights that high quality keywords crucial to performance. The LM re-scored set of keywords result in significant accuracy gain over heuristically sampled keywords. Paths with first keyword as Title boost performance further. Mixing in a small fraction of paths starting with non-title keywords encourages the model to generate predicates where no entity can be determined, giving us the best results.
Limitations
P is geared towards identifying a concise, small set of documents and generating answer in sin- gle go. While this makes the architecture simpler, also adds certain weaknesses. IP is not effec- tive for retrieving a large number of documents and falls behind pipelined systems that combinc retrieval with re-ranking. Even for single passage, it lags behind state-of-the-art dense-retrieval tech- niques. IP's method of answer generation is also not competitive with the use of a reader, due to lack of passage context.
Sensitivity to tokenization We find that con- strained decoding is highly sensitive to rare tok- enization or punctuation formatting in the corpus. Consider the query "who sang i ran all the way home" with the gold document title "Sorry (I Ran All the Way Home)". In the unconstrained setup, the model's top prediction starts with Ran All the Way Home". However, "(I" is tokenized dif- ferently from "I" and searching over the FM-Index
Our training strategy relies heavily on titles or entities and it's generalization on corpora without rich structure or on queries without central entities, remains to be studied.

---

Query (Q) and Generated Search Path (SP)
Comment
Correctly altributed passages and answers
Q: how many episodes of greys anatomy season 14
Query entity resolved first, followed by query predicates
SP: Grey's 's Anatomy (scason 14) > season consists of 24 episodes 24
Q: when they start adding zinc to pennies
Query entity resolved itcratively
SP: Penny (United States coin) > zinc Lincoln cent > 1943
Q: who executed for being american spy during the revolutionary
Answer generated first
SP: Nathan Hale Army during the American Revolutionary Nathan Hale
Q: who was the grandfather on the cosby show
Query rewrites
SP: appeared on "The Cosby Earle Hyman
Incorrect Passage or Answer
Q: who decides the number of judges in the high court
A: President of India
Path correctly resolved,
SP: judge is appointed High Court Chief Justiee of India
Failed answer
Q: when did the isle of wight hecome an island
A: During the last Iec Age
Query entity resolved,
Isle of Wight 1890 > 1890
Failed on supporting keywords
Q: love yourself by justin bieher about who
A: Rihana
Failed resolve
SP: Love Yourself: Her > music video > Her
query entity
Table Example IP Search Paths (SP) on Open-NQ test set, The last keyword in is the predicted answer. Gold answers indicated by A.
Ethical Considerations
Search Path
Hits @
EM
Hcuristic LM-scored
34.5
22.6
While Large Language Models can solve a wide range of tasks effectively, they also suffer from bi- ases across axis such as gender, race, region (Chan, 2023). LLMs are also prone to generating toxic content, especially when probed about it. Although, our task grounds the model's generations on cor- pus, some of the biases pre-trained LLMs, may seep 1-PAGER.
40.0
27.2
Title LM-scored
41.9
28.0
Title LM-scored +
42.9
28.7
LM-scored (7+1)
Table 6: Comparison of Training Search Paths on Open- NQ. Here LM-scored denotes re-scoring by LM on heuristic set. All results are with beam of one. ">" indicates keyword separator and "+" mixture of path types in the give ratio.
Building the FM-index and constrained decod- ing is a compute- compute-intensive affair. We have exper- imented over a single dataset, Natural Questions, involving only knowledge-seeking queries, and sin- gle model family, T5. It possible that some of our findings may not hold over other datasets or model families. Finally, our experiments are limited to En- glish corpus and queries. The proposed approaches are resource-intensive and may not be accessible or valid for several low-resourced languages.
Constrained decoding also comes with its own challenges. Constrained beam outputs often lack diversity, so that even with a larger beam one may still end up in poor search spaces. Computing document-level constraints across the corpus is ex- pensive as it may require scanning a large num- ber of rows in the index. Further, communication between FM-Index and Transformer model slows down inference.
References
Leonard Adolphs, Benjamin Boerschinger, Christian Buck, Michelle Chen Huebscher, Massimiliano Cia- ramita, Lasse Espeholt, Thomas Hofmann, Yannic Kilcher, Sascha Rothe, Pier Giuseppe Sessa, et al. 2021. Boosting search engines with interactive agents. arXiv preprint
Acknowledgement
We thank Don Metzler, Nicholas FitzGerald, Partha Talukdar, Srini Narayanan, as well as our anony- mous reviewers, for their thoughful comments and valuable feedback
Rohan Anil, Andrew M Dai. Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak

---

Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint arXiv:2305. 10403.
Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023b. Enabling large language models to generate text with citations.
Petr Baudiš and Jan Sedivy. 2015. Modeling of the question answering task the yodaqa system. In Experimental IR Meets Mulrilingaality, Multimodal- ity, and Interaction: 6th International Conference of the CLEF Association, CLFFF'5, Toulouse, France, September 8-11, 2015, Proceedings 6, pages 222- 228. Springer.
Alex Graves, 2012. Sequence transduction with recur- rent neural networks.
Sanda M Harabagiu, Steven J Maiorano, and Mar- ius A Paşca. 2003. Open-domain textual question answering techniques. Natural Language Engineer- ing. 9(3):231-267.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural lan- guage processing, pages 1533-1544.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot Learning with Retrieval Augmented Language Models.
Michele Bevilacqua, Giuseppe Ottaviano, Patrick Lewis. Wen tau Yih, Sebastian Riedel, and Fabio Petroni. 2022. Autoregressive search engines: Gen- crating substrings as document identifiers. arXiv pre-print 2204.10628.
Zhengbao Jiang. Luyu Gao, Jun Araki, Haibo Ding, Zhiruo Wang. Jamie Callan, and Graham Neubig. 2022. Retrieval as attention: End-to-end learning of retrieval and reading within a single transformer. arXiv preprint 2rXi::2212.02027.
Bernd Bohnet. Vinh Q Tran, Pat Verga, Roce Aharoni, Daniel Andor, Livio Baldini Soares, Jacob Eisen- stein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, al. 2022. Auributed question answering: Evalua- tion and modeling for attributed large language mod- els. arXiv preprint arXiv:2212.0800
Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun. Qian Liu, Jane Dwivedi- Yu. Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Ac- tive retrieval augmented generation. arXiv preprint arXiv:2305.0663
Anastasia Chan. 2023. Gpt-3 and instructgpt: techno- logical dystopianism, utopianism. and "contextual" perspectives in ai ethics and industry. Al and Ethics, 3(1):53-64.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 6769- 6781, Online. Association for Computational Lin- guistics.
Danqi Chen, Adam Fisch, Jason Weston, and An- toine Bordes. 2017. Reading wikipedia to an- swer open-domain questions. arXiv preprint arXiv:1704.00051.
Omar Khattab and Matei Zaharia. 2020. Colbert: Ef- ficient and effective passage search via contextual- ized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference research and development in Information Retrieval, pages 39-48.
Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, and Andrew McCallum. 2019. Multi-step retriever- reader interaction for scalable open-domain question answering. In International Conference on Learn- ing Representations.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In 9th International Conference Learning Repre- sentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-We Wei Chang. Andrew Dai, Jakob Uszkoreit, Quee Le, and Slav Petrov. 2019. Natu- ral questions: benchmark for question answering research. Transactions of the Association of Compu- tational Linguistics.
P. Ferragina and G. Manzini. 2000. Opportunistic data structures with applications. In Proceedings 41st An- nual Symposium Foundations of Computer Sei- ence, pages 390-398.
Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty. Yicheng Fan, Vin- cent Y. Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2023a. Rarr: Rescarching and re- vising what language models say, using language models.
Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler. Antonia Creswell, James McClelland, Jane X Wang, and Felix Hill. 2022. Can language models learn from explanations in context? arXiv preprint orXiv:2204.02339.

---

Hyunji Lee, Sohee Yang, Hanseok Oh, and Minjoon Seo. 2022. Generative multi-hop retrieval. In Pro- ceedings the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 1417- 1436.
Adam Roberts, Hyung Won Chung, Anselm Levskaya. Gaurav Mishra, James Bradbury, Daniel Andor, Sha- ran Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowyez, Alex Salcianu, Mare van Zee, Jacob Austin, Sebas- tian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jas- mijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark. Stephan Lec, Dan Garrette. James Lee- LeorTh Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepard, Noah Fiedel, Mark Omernick, Brennan Sacta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. 2022. Scaling up models and data with t5x and seqio.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 1086-6096, Florence, Italy, Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Mar- jan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising eqquence-t--sqqueme pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 7871-7888 Online. Association for Computational Linguistics.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the pa- rameters of language model? arXiv preprint arXiv:2002.08911
Stephen Robertson, Hugo 7aragoza, et al. 2009. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends@ in Information Re- trieval, 334:3333-38.
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pon- tus Stenetorp, and Sehastian Riedel. 2021. Paq: 65 million probably-asked questions and what you can do with them.
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022, Col- bertv2: Effective and efficient retrieval via lightweight late interaction.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74 81.
Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, et al. 2022. Transformer memory as a differentiable search index. Advances Neural In- formation Processing Systems, 35:21831-21844
Jimmy Lin, Xueguang Ma, Sheng-Chich Lin, Jheng- Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: An easy-to-use python toolkit to support replicable ir research with sparse and dense representations. arXiv preprint arXiv:2102. 10073.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information process- ing systems, 30.
Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese Susannah Young. Lucy Campbell- Gillingham, Geoffrey Irving. and Nat McAleese. 2022. Teaching language models to support answers with verified quotes.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903
Donald Metzler, Yi Tay, Dara Bahri, and Marc Najork. 2021. Rethinking search: making domain experts out of dilettantes. ACM SIGIR Forum, 15(1):1-27.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok- Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020. Approximate nearest neighbor neg- ative contrastive learning for dense text retrieval. arXiv preprint 2007::0007000800
OpenAI. 2023. Gpt-4 technical report.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter Liu, et al. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21(140):1-67,
Wenhao Yu, Dan Iter, Shuohang Wang. Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2022. Gen- erate rather than retrieve: Large language mod- els are strong context generators. arXiv preprint 2rXiv:2209.10063
Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das. Slav Petrov, Gaurav Singh Tomar, Iulia Ture, and David Reitter. 2021. Measuring attribution in natu- ral language generation models. arXiv preprint arXiv:21 12870.
Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023. Generate

---

rather than retrieve: Large language models are strong context generators. In The Eleventh Intema- tional Conference Learning Representations.
Yujia Zhou, Jing Yao, Zhicheng Dou, Ledell Wu, Peitian Zhang, and Ji-Rong Wen. 2022. Ultron: An ultimate retriever on corpus with model-based in- dexer. arXiv preprint 09.257220..09257.
Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021. Retrieving and reading: A comprehensive survey on open-domain question amswering.

---

due higher diversity in training. However, this method of dataset expansion limited by the num- ber of relevant paths we could extract for query. A Appendix
A.1 Constrain Computation
IP relies on two key operations for constrain com- putation:
We also experiment with increasing the query set manifold by mixing in unsupervised datasets. A total of 9M QA pairs are sampled from PAQ (Lewis et al., 2021), synthetic QA dataset, and search paths extracted with heuristic scoring described in Section 5. The original P training dataset is mixed in 1:1 ratio. This further boosts performance, but not proportionally to the amount of data added, indicating diminishing returns from silver datasets.
a) F(D,k) Documents that contain keyword k
b) C(k, D): Next tokens for keyword in arbi- trary document set D
F(D,k) is preprocessed and cached to allow for quick computation. Ckk, D) is trickier to compute. When D represents the full corpus, FM-index can
fetch the next tokens in O( ([V]log((V))) where V is the token vocabulary and independent of [D]. However, arbitrary D requires traversal over all documents and can be very expensive. In practise.
A.3 SEAL keywords
SEAL generates a set of document substrings con- strained on the corpus, that are combined to form document identifiers. Besides using LM to gener- ate keywords, SEAL utilizes several other mecha- nisms for extracting keywords. This includes par- tial beam sequences, heuristically adding query n- grams, sampling the top-k tokens from the logprobs of the first decoding step, force decoding title etc. The keywords are re-scored using the LM as well as FM-index count and all keyword combinations are retrieved. Table 8 illustrates keywords generated by both the systems. Note that SEAL keywords can be repetitive and therefore require large num- ber of keywords to narrow down to meaningful documents. This also makes SEAL suitable for re- trieving a much larger set of documents that can be re-ranked later. The maximum number of retrieved documents for SEAL are capped by hyperparam- eter with default value of 500. In contrast, IP is geared towards retrieving only the top-document.
the LLM training guides to generate effective keywords such that |D] is small.
We also apply certain other optimizations to re- duce the compute cost:
Constrains are computed lazily over a decod- ing pass.
Several computations are cached, eg: keyword to document id mapping
To cap the cost of constraints at each decoding step, we allow for unconstrained generation in rare scenarios, when the estimated cost is too high. If the generated path is absent in the corpus (<1 % examples), these can be filtered out later.
Despite these optimizations, inference continues to be expensive and we perhaps need special data structure for next token look-up.
A.4 Hits@5
SEAL does significantly better than IP for Hits @5 (Table 9). We attribute this to the large set of key- words generated by SEAL as explained in the Ap- pendix A.3.
A.2
Training data size
Dataset
Queries
Paths
Hits @
EM
Open-NQ
55k
55k
41.9
28.1
A.5 Normalizing sequence likelihood over constrained space
Open-NQ
55k
310k
42.9
28.7
Open-NQ
55k
310k
During constrained decoding a sequence X, we need to choose the next token from C(X, D) and not the entire vocabulary space V. Should the se- quence likelihood be re-normalized over this con- strained space? We find that re-normalizing the probabilities results in inflated likelihoods, making hard for the model to back-track.
43.6
29.5
PAQ
+ 9M
+ 9M
Table 7: Comparison of different dataset sizes for queries and paths
In Table 7, we observe the effect of dataset size on performance. Increasing the numbers of paths sampled per query improves performance, perhaps
Consider the query, "where did the butchers the slaughterhouse cases live" to which our model

---

System
Question or Search Path
Answer
who has the most catches nfl history
Jenry Rice
IP
2,000-yard club Barry Sanders
Barry Sanders
SEAL

 Michael Irvin @ 60, yards per catch, caught his, touchdown, record
TJJ. Houshmandzadch
when was harry potter and the philosophers stone published
1997
IP
Harry Potter and the Philosopher's s Stone first published in the United 1997
1997
SEAL

 Harry Potter and Philosopher's Stone @ @ "Harry Potter. Potter and the
1999
Philosopher's Stone is, Potter and the Philosopher's s Stone Harry, novel
what is the meaning of the harp in ircland
the arms of Ireland national symbol of Ireland aristocracy
IP
Harp > national symbol of Ireland national symbol Ireland 
 Harp @ Irish harp.. harp is, harp was, harp
SEAL
was the president pakistan during 1971 war Indo-Pakistuni War of 1971 Prime Minister of Pakistan Zulfikar Ali Bhutto 
 Indo-Pakistani War of 1971 @ @ East Pakistan, Pakistami, Pakistan Army, Pakistan's
Yahya Khan
1P
Zulfikar Ali Bhutto
SEAL
Muhammad Yaqub Khan
when do you declare honors in contract hridge
any time after the auction end of the hand bidding
IP
Contract bridge declaring end the hand
SEAL

 Contract bridge @ declarer, bidding, honors, hands
Table 8: Comparison of keywords generated by SEAL and IP for randomly sampled exampled from Open-NQ test set. For IP, we show the full search path separated by ">" with the last keyword as the answer. For SEAL, we illustrate the top-5 keywords along with the answer from Reader model. "
" and "@ @ are special tokens used by SEAL for identifying start passage and title marker respectively. The Answer next to the question is the gold answer while others predictions from corresponding systems.
System
Beam
@5
1750
SEAL
1
59.7
1500
Number examples
SEAL
5
62.8
1250
IP
1
46.5
1000
IP
5
50.8
750
Table 9: Hits@ on Open-NQ test. SEAL achieves a much higher score than owning the larger number of documents matched and re-scored. Note that only top- beam result used for 1P while SEAL uses all beam outputs.
Matched documents for path
Figure 4: Number of matching documents in the corpus for 1P generated path in the set. About half the examples match only single path.
predicts an irrelevant search path [Slaughterhouse Five, but, EoS]. What's going on under the hood? The first keyword incorrect lending the model into a poor search space. With the second key- word, the model is possibly looking to generate "butcher" but there's no such keyword in the con- strained set. Ideally, the model should backtrack at this point to other candidates in the beam. How- ever, since the of continuations is small, re- normalizing inflates the probablities of all tokens in including EoS, even though the true likelihood such sequence very low. Indeed, using the language model's scores directly without any re-
normalization cures this issue yielding (Slaughter- house cases, Butcher, EoS]. and this is the strategy opt for in all our experiments.
A.6 Number of matching documents
P generated paths effectively narrow down the corpus, generally matching only a few documents in the corpus illustrated Figure 4. Note that small fraction of paths match documents due to pruning optimizations applied during inference

---

time detailed in Appendix A.


Reviews:
[{"review_id": "bO7KXwkQPb", "reviewer": "Reviewer_RLpb", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful reading of our paper and valuable feedback. Below we have provided some responses to your two criticisms of the current paper.\n\n1.  > “1-pager is only evaluated on Open-NQ”\n\nThis is a valid criticism also raised by other reviewers. To understand this better, we also tested 1P on WebQuestions and CuratedTrec. Both of these datasets, like Open-NQ, are suitable for joint retrieval + QA experiments. \n\nIn the below experiments, all systems: 1P, DPR, Reader are trained only on Open-NQ. We find that 1P performs better than DPR + Reader on both the retrieval and answer accuracy metrics even in this out-of-domain setting.\n\n**WebQuestions**\n```\n\t              Hits@1  ||    EM   ||   F1\nDPR + Reader :    32.0    ||  17.3   ||  30.2  \n1P           :    38.0    ||  20.5   ||  32.7\n```\n\n**CuratedTrec**\n```\n\t              Hits@1  ||    EM   ||   F1\nDPR + Reader :    51.6    ||  35.0   ||   -\n1P           :    63.8    ||  36.4   ||   -\n```\n\nWe will include these results along with associated analysis in the final version of the paper. \n\n2. > “ Although one of the selling point of 1-pager is to generate the document keyword and answer at the same time, which is different from the retrieve-and-read paradigm, it seems to me that 1-pager with the reader makes more sense. As discussed in the paper, if 1-pager does not use retrieve-and-read, the answer generation will be only conditioned on the generated keywords, instead of the retrieved document. This will lose some important information in the answer generation process. Therefore 1-pager with reader yields better performance than the no reader version of 1-pager.”\n\nIn this paper, we aim to explore the spectrum between CBQA, where the answer is generated without reference to an evidence corpus, and pipelined approaches that feed retrieved documents into the transformer.\n\nWe agree that the pipelined approach is still most performant and do not yet argue for 1P as the best solution for anyone aiming to build the most accurate QA system possible. However, we believe that 1P's improvement over the equivalent CBQA model is significant and adds value in demonstrating how a single transformer model can be harnessed to do both retrieval and  answering, which has not been done before.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Although the idea of using generative language model to generate the keywords of documents is explored in SEAL, the authors of 1-pager change the way the keywords are generated by scoring the keywords in the training data. Ablation experiments show that the scoring of keywords during training is important to the later keyword generation.\n - The paper is well presented and easy to follow. Most of the details are included.\n - 1-pager yields pretty good results on the Open-NQ dataset that are competitive to DPR.", "reasons_to_reject": "- 1-pager is only evaluated on Open-NQ. It is not clear whether the improvement of retrieval performance is associated with some particular artifacts of the dataset. \n - Although one of the selling point of 1-pager is to generate the document keyword and answer at the same time, which is different from the retrieve-and-read paradigm, it seems to me that 1-pager with the reader makes more sense. As discussed in the paper, if 1-pager does not use retrieve-and-read, the answer generation will be only conditioned on the generated keywords, instead of the retrieved document. This will lose some important information in the answer generation process. Therefore 1-pager with reader yields better performance than the no reader version of 1-pager.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces an approach to handle open-domain QA problems using generative language models, i.e., the 1-pager approach. Instead of using tf-idf/BM25/dense retrieval, the authors propose to use a single generative language model to generate both the keywords of the relevant document and the final answer in only one forward inference process. The authors also make sure the generated keywords of the relevant documents indeed appear in some documents of the knowledge base by introducing constraints in the decoding process of the generative language model. \n\nThe proposed approached is realized with a T5 model, evaluated on Open-NQ dataset. Results show that the proposed approach yields competitive performance to the SOTA retrieve-and-read approach (such as DPR + T5 reader). A few other experiments are conducted to investigate a few different variations of the proposed approach. \n\nAlthough the generation-with-constraint method is very similar to SEAL, 1-pager uses different heuristics to generate the keywords of the relevant documents. Experiments show that these modifications lead to some improvements to the retrieval performance compared with SEAL.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AqGWm3S3JZ", "reviewer": "Reviewer_waPh", "summary": "", "questions": "- Do you think planning in the form of MCTS (actual lookahead) could be helpful? \n- Could you extend this method to retrieve multiple documents effectively?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough reading and valuable feedback. Below we have provided some responses to your criticisms of the current paper.\n\n\n1. > “Evaluations are performed only on a single dataset (Open-NQ).”  \n\nThis is a valid criticism also raised by other reviewers. To understand this better, we tested 1P on WebQuestions and CuratedTrec. Both of these datasets, like Open-NQ, are suitable for joint retrieval + QA experiments. \n\nIn the below experiments, all systems: 1P, DPR, Reader are trained only on Open-NQ. We find that 1P performs better than DPR + Reader on both the retrieval and answer accuracy metrics even in this out-of-domain setting.\n\n**WebQuestions**\n```\n\t              Hits@1  ||    EM   ||   F1\nDPR + Reader :    32.0    ||  17.3   ||  30.2  \n1P           :    38.0    ||  20.5   ||  32.7\n```\n\n**CuratedTrec**\n```\n\t              Hits@1  ||    EM   ||   F1\nDPR + Reader :    51.6    ||  35.0   ||   -\n1P           :    63.8    ||  36.4   ||   -\n```\n\nWe will include these results along with associated analysis in the final version of the paper. \n\n\n2. > “The authors clearly differentiate the work from prior methods that retrieve multiple documents (passages) and perform unconstrained generation. However, only supporting the retrieval of a single document seems like a serious limitation. It is often the case the answer is not contained in a single passage or document and can only be derived by combining information from multiple sources and potentially reasoning over that information. In this case, 1P will not be effective.”\n\nWe agree that question answering can often require multi document reasoning but also point out that most datasets commonly used for open-domain QA generally have the presence of an answer in a single supporting document. This is a prerequisite for answers in NQ, in particular.\n\nWhen reasoning over multiple retrievals for NQ and associated datasets, the standard retrieve-then-read methods do not generally perform multi-hop reasoning but are instead relying on the reader to identify which of 100 retrievals are most relevant, similar to re-rankers used in the IR literature. For this reason, we feel it is reasonable to compare 1P to top-1 retrievals to get a clear picture of the model's performance at the pure retrieval task. Note that 1P is capable of retrieving multiple documents as depicted in Appendix A7.\n\nWe have not addressed the more challenging task of multi-document reasoning but believe this could be a good next area of focus for 1P. Pipelined systems need to chain together multiple steps of document retrieval and document reading for true multi-hop tasks. By integrating retrieval into the decoding process directly, 1P could simplify this process and support better end-to-end learning of multi-hop tasks.\n\n\n\n3. > “The generation of training data seems quite complex and full of heuristics. Will the model just learn to emulate these heuristics?”\n\n The heuristics used to kick-start training are not elegant but we have evidence that 1P is not simply emulating these at test time. \n\n* The heuristics themselves are not sufficient---1P also needs LM scoring to find good search paths. (Table 5)\n\n* Our experiments with Unconstrained-1P show that simply replicating the training distribution is insufficient, the constraints introduced in 1P are essential to ensuring that the model can navigate the evidence corpus effectively. \n\n* The qualitative analysis presented in Table 4 also illustrates diverse patterns of search paths generated in different scenarios that are more effective and context dependent than the heuristically generated paths used during training.\n\n* Finally, the out of domain results on WebQuestions and CuratedTrec presented above show that the choice of  training-time heuristics does not hurt generalization to new tasks. \n\nFuture work should explore even more general methods of training.\n\n\n\n4. > “However, 1P+Reader outperforms 1P alone, which is somewhat disappointing / calls into question the viability of this method.”\n\nIn this paper, we aim to explore the spectrum between CBQA, where the answer is generated without reference to an evidence corpus, and pipelined approaches that feed retrieved documents into the transformer.\n\nWe agree that the pipelined approach is still most performant and do not yet argue for 1P as the best solution for anyone aiming to build the most accurate QA system possible. However, we believe that 1P's improvement over the equivalent CBQA model is significant and we also believe that 1P adds value in demonstrating how a single transformer model can be harnessed to do both retrieval and  answering, which has not been done before.\n\n\n5. > “Do you think planning in the form of MCTS (actual lookahead) could be helpful?”\n\nThank you for the suggestion. We agree that better search algorithms would be extremely beneficial. While we have initially focused on a proof-of-concept with the beam search algorithm that is commonly used for Transformer decoding we agree that MCTS would likely be a better fit to help 1P recover from poor choices. We refer to Table 2 for an initial investigation of how beam search improves 1Ps performance over greedy decoding.\n\n\n6. > “Brittleness with respect to tokenization / exact match”\n\nWe acknowledge that 1P is sensitive to exact matches. This is a general limitation with lexicalized retrieval but is countered by the model's ability to directly match rare named entities (e.g. EMNLP 2023) and generate interpretable search paths. We believe that there is value in exploring lexicalized approaches that can both generalize over variations in surface form, via paraphrasing, and also capture highly discriminative n-grams. However, we also agree that there is more to be done to ensure that 1P is more robust to minor variations in tokenization.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ The paper is well-written. It is very clear what the authors did and why they did it, which I certainly appreciate (as will future readers). \n+ The constrained decoding method is clever and seems like an interesting direction to explore for greater factuality / grounding.\n+ Desirable to have a single model that does both retrieval and answer generation.\n+ Greater interpretability via keywords seems desirable as well.", "reasons_to_reject": "- The authors clearly differentiate the work from prior methods that retrieve multiple documents (passages) and perform unconstrained generation. However, only supporting the retrieval of a single document seems like a serious limitation. It is often the case the answer is not contained in a single passage or document and can only be derived by combining information from multiple sources and potentially reasoning over that information. In this case, 1P will not be effective. 1P Unconstrained may be able to overcome this issue, but it doesn't seem particularly competitive with other approaches.\n- Evaluations are performed only on a single dataset (Open-NQ).\n- The generation of training data seems quite complex and full of heuristics. Will the model just learn to emulate these heuristics? \n- This method is presented as desirable because it can retrieve and generate in a single pass with a single model. However, 1P+Reader outperforms 1P alone, which is somewhat disappointing / calls into question the viability of this method. \n- Brittleness with respect to tokenization / exact match seems like a serious concern, though the authors have worked toward mitigating this.", "questions_to_authors": "- Do you think planning in the form of MCTS (actual lookahead) could be helpful? \n- Could you extend this method to retrieve multiple documents effectively?", "paper_topic_and_main_contributions": "This paper describes 1-PAGER (or 1P), a method in which a single model both retrieves a document and generates an answer to a given question. More specifically, 1P stores documents in an FM-Index and performs constrained (to text present in the corpus) decoding to generate keywords that iteratively narrow the space of documents until a single passage is returned. The last generated keyword is treated as the model's answer. Although 1P outperforms closed QA systems, it (unsurprisingly) underperforms SOTA retrieval-augmented models which retrieve more than one document.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper is overall very well-written, so there are unusually few typos / grammatical errors, which is appreciated.\n\n\"Each prefix of this search path defines subset of D via the search path corpus partitioning function.\" -> \"a subset\"\n\"Answer prediction is treated in exactly the same way as keyword selection and, in 1P...\" -> remove comma"}}, {"review_id": "lGgN1Buk7q", "reviewer": "Reviewer_Sf64", "summary": "", "questions": "1. What are the run time for inference? How is it compared to other method?\n2. What are the exact solution for the constrained decoding with FM-index? The explanation in appendix is unclear and difficult to reproduce.\n", "limitations": "", "rating": 3, "confidence": 1, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful reading and insightful critique. We have provided some responses to your feedback below: \n\n1. > “The comparison with SEAL is not fair, as the model of SEAL (bart-large) is much smaller than the one used in the current work (t5-xxl).”\n\nWe agree that the model size difference between SEAL and 1P is not desirable, however we preferred using the publicly available SEAL checkpoint (Bart-large) instead of training our own. Note that there are several other discrepancies between SEAL and 1P: \n\na) *Inference time FLOPS*: SEAL uses significantly higher accelerator FLOPS than 1P. SEAL runs about 3 decoding passes. Besides, SEAL also re-scores all its keywords (30 on average), which is equivalent in cost to decoding. In contrast, 1P runs a single decoding pass of ~3 keywords.\n\nb) *Beam-size*: The reported results used a beam=5 for SEAL retrievals while 1P uses only top-beam for retrieval.\n\nc) *Heuristic keyword generation*: As discussed in Appendix A4, besides LM keyword generation, SEAL uses a variety of other heuristics such as extracting ngrams from the query, force decoding the title, fetching all high scoring next-tokens from the model given the query, partial beam matches etc. \n\nGiven these discrepancies, it is difficult to have an apples-to-apples comparison.\n\n2. > “The decision to use a single model for both retrieval and answer generation seems forced and counter-productive. As shown in the experiment section, using a separate reader is superior to the proposed single model solution”\n\nIn this paper, we aim to explore the spectrum between CBQA, where the answer is generated without reference to an evidence corpus, and pipelined approaches that feed retrieved documents into the transformer.\n\nWe agree that the pipelined approach is still most performant and do not yet argue for 1P as the best solution for anyone aiming to build the most accurate QA system possible. However, we believe that 1P's improvement over the equivalent CBQA model is significant and we also believe that 1P adds value in demonstrating how a single transformer model can be harnessed to do both retrieval and  answering, which has not been done before.\n\n\n\n3. > “What are the exact solution for the constrained decoding with FM-index? The explanation in appendix is unclear and difficult to reproduce.”\n\nThank you for flagging this, we have presented a summary below and would add a more detailed explanation to the Appendix. 1P’s Constrain decoding has 3 key functions to compute: \n\na) Next tokens following the keyword k \n   - How: FM-index contains auxiliary data structures to compute this efficiently. \n   -  Time: O(Vocab)\n\nb) Documents that contain keyword k\n   -  How: All occurrences of k can be located by the FM-index in constant time. We iterate through each occurance and a hashmap provides the corresponding document id.   \n   -  Time: O(count-of-k)\n\nc) Tokens in a document set D\n-  How: We iterate over each doc in D, perform an FM-Index operation to get the token position and check if it lies in the given range. \n-  Time: O(number of documents). In practise, the model is trained to generate highly specific initial keywords that result in a small number of documents.\n\nA combination of these yields all the constraints needed. We also apply a few optimizations on top of this, two of which are highlighted below: \n\n* Several computations are cached across a decoding pass, eg: position of k in index.\n* To cap the cost of computing constraints at each decoding step, we allow for unconstrained generation in rare scenarios, when the estimated cost is too high. If the generated path is absent in the corpus (<1% examples), they are awarded a 0 score.\n\n\n4.  > “What are the run time for inference? How is it compared to other method?”\n\nOur current implementation is not optimized for inference time efficiency. Currently 1P pays a significant overhead for communication between the accelerator used for transformer decoding, and the separately stored index managed by CPU. Subsequently, the full runtime is significantly slower than well optimized retrieve-then-read methods. However, we believe this should be addressable by architectural improvements like moving the FM-index to the accelerator, implementing parallel calls to the index, caching documents.\n\n5. > Additional Experiments\n\nTo better understand whether 1P generalizes beyond the NQ task, we also tested it on WebQuestions and CuratedTrec. Both of these datasets support retrieval and answer accuracy metrics. \n\nIn the below experiments, all systems, 1P, DPR and Reader are trained only on Open-NQ. We find that 1P performs better than DPR + Reader according to both the retrieval and answer accuracy metrics in this out-of-domain setting.\n\n**WebQuestions**\n```\n\t              Hits@1  ||    EM   ||   F1\nDPR + Reader :    32.0    ||  17.3   ||  30.2  \n1P           :    38.0    ||  20.5   ||  32.7\n```\n\n**CuratedTrec**\n```\n\t              Hits@1  ||    EM   ||   F1\nDPR + Reader :    51.6    ||  35.0   ||   -\n1P           :    63.8    ||  36.4   ||   -\n```\n\nWe will include these results along with associated analysis in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Generating a sequence of keywords to partition the corpus is a new technique in generative retrieval.\n2. A single model for both passage retrieval and answer generation.", "reasons_to_reject": "1. The decision to use a single model for both retrieval and answer generation seems forced and counter-productive. As shown in the experiment section, using a separate reader is superior to the proposed single model solution.\n2. As pointed out by the paper itself, constrained decoding with FM-index is very expensive for keywords after the first one, which limits the value of the work in practice.\n3. The comparison with SEAL is not fair, as the model of SEAL (bart-large) is much smaller than the one used in the current work (t5-xxl).", "questions_to_authors": "1. What are the run time for inference? How is it compared to other method?\n2. What are the exact solution for the constrained decoding with FM-index? The explanation in appendix is unclear and difficult to reproduce.\n", "paper_topic_and_main_contributions": "This paper proposes a single language model for both retrieval and answer generation for open-domain QA. Like previous work SEAL, the proposed method generate keywords from query to retrieve documents containing the generated keywords with the help of FM-index. Different from SEAL, this work generates a sequence of keywords to gradually narrow down to a subset of the corpus, and then directly generate the final answer after keywords generation. The experimental result shows the method performs worse than baselines.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In the abstract, \"While 1-PAGER is not yet on-par with more expensive systems that read many more documents before generating an answer...\" is misleading. As shown in your experiment, DPR retriever + reader outperforms the proposed method, but is also much cheaper and operates on a single document."}}]

Author rebuttal:

Meta-review: The paper proposes using a single LLM to, given a query, generate keywords for retrieval and then produce the final answer in one pass. Constrained decoding is adeptly implemented, using the FM-index, to ensure that the documents contain the specified keywords. Experiments validate the efficacy of the proposed method on Open-NQ (as presented in the paper), as well as on WebQuestions and CuratedTrec in a zero-shot manner, as recommended by the reviewers (details reported in the rebuttal). The paper is excellently presented with clear motivation. We strongly urge the authors to address and refine the areas pinpointed by the three reviewers.
Decision: Accept-Findings