Contextualized Generative Retrieval
Authors: Hyunji Lee
JaeYoung Kim
Hoyeon Chang
Hanseok Oh
Sohee Yang
vladimir karpukhin
Yi Lu
Minjoon Seo
Venue: Submitted to ICLR 2023 (2023)
Keywords: NLP
Information Retrieval

Abstract: The text retrieval task is mainly performed in two ways: the bi-encoder approach and the generative approach. The bi-encoder approach maps the document and query embeddings to common vector space and performs a nearest neighbor search. It stably shows high performance and efficiency across different domains but has an embedding space bottleneck as it interacts in L2 or inner product space. The generative retrieval model retrieves by generating a target sequence and overcomes the embedding space bottleneck by interacting in the parametric space. However, it fails to retrieve the information it has not seen during the training process as it depends solely on the information encoded in its own model parameters. To leverage the advantages of both approaches, we propose Contextualized Generative Retrieval model, which uses contextualized embeddings (output embeddings of a language model encoder) as vocab embeddings at the decoding step of generative retrieval. The model uses information encoded in both the non-parametric space of contextualized token embeddings and the parametric space of the generative retrieval model. Our approach of generative retrieval with contextualized vocab embeddings shows higher performance than generative retrieval with only vanilla vocab embeddings in the document retrieval task, an average of 6% higher performance in KILT (NQ, TQA) and 2X higher in NQ-320k, suggesting the benefits of using contextualized embedding in generative retrieval models.

Under review conference paper at ICLR 2023
CONTEXTUALIZED GENERATIVE RETRIEVAL
Anonymous authors Paper under double-blind review
ABSTRACT
The text retrieval task is mainly performed in two ways: the bi-encoder approach and the generative approach. The bi-encoder approach maps the document and query embeddings to common vector space and performs nearest neighbor search. It stably shows high per- formance and efficiency across different domains but has an embedding space bottleneck as it interacts in L2 or inner product space. The generative retrieval model retrieves by gen- erating a target sequence and overcomes the embedding space bottleneck by interacting in the parametric space. However, it fails retrieve the information has not seen during the training process as it depends solely on the information encoded in its own model parameters. To leverage the advantages of both approaches, we propose Contextualized Generative Retrieval model, which uses contextualized embeddings (output embeddings of a language model encoder) as vocab embeddings at the decoding step of generative retrieaal. The model uses information encoded in both the non- non-parametric space of con- textualized token embeddings and the parametric space of the generative retrieval model. Our approach of generative retrieval with contextualized vocab embeddings shows higher performance than generative retrieval with only vanilla vocab embeddings in the docu- ment retrieval task, an average of 6% and 18% (25%) higher performance in KILT (NQ. TQA) R-precision and NQ-320k Hits @ (@ 10), respectively, suggesting the benefits of using contextualized embedding generative retrieval models
INTRODUCTION
Text retrieval is often formulated as finding the most relevant items from a large corpus given an input query. The bi-encoder approach of using encoder to map the documents and the query to common vector space and performing a nearest neighbor search has been a common practice in text retricval tasks (Karpukhin et al.,
2020
Wu et al. 2020 Ni et al. 202T Despite its high performance and popularity, it has embedding space bottleneck (Luan et al. 2022 Lee et al. 2022 Cao et al.) (021). The performance decreases as the document length increases due to the limited expressiveness of fixed-size document embeddings. Also, it misses the fine-grained interaction between the query and the document as they interact in L2 or inner product space. The bi-encoder approach also requires large storage space save all document embeddings.
A recently-proposed alternative to the bi-encoder approach is using generative retrieval model (Cao et al.) 2021 Tay et al.) 2022; Bevilacqua al., 2022 Lee et al., 2022) which retrieves the most relevant sequence by generating the item token-by-token where the item is the identifier of the target sequence or the sequence itself (e.g., title, passage, document ID). They show high performance while using low storage footprint by overcoming the embedding space bottleneck. These models interact in the parametric space of the language model rather than just in the inner product space. However, as existing generative retrieval models rely solely on the information encoded in their own parameters, the model cannot retrieve the correct target sequence if it has not seen such information during the training process.
We will make our code publicly available.

---

Under review conference paper at ICLR 2023
To this end, we propose contextualized generative retrieval model (CGR), a retrieval model that overcomes the aforementioned limitations of existing generative retrieval models by leveraging contextualized vocab embeddings (output embeddings of language model encoder) to make use of non-parametric information from the context surrounding the vocab tokens. It uses not only the parametric space of the model as in generative retrieval models but also the non-parametric space of contextualized target embeddings (exter- nal memory) as in bi-encoder models. As in Figure [ the model has two submodules: (1) an EMBedding model (EMB), which is an encoder model that outputs contextualized embeddings, and (2) a RETrieval model (RET), which is encoder-decodee model that retrieves a target sequence when given an input query. The model first constructs the contextualized embedding matrix with the output embeddings of EMB uses the matrix the decoder vocab embeddings when training RET. By utilizing the contextualized embedding matrix rather than the vanilla embedding matrix while generating target sequence, RET uses both informa- tion encoded in its own parameters as existing generative retrieval models and information encoded in the contextualized embeddings. Also, as RET uses the contextualized embeddings during both the training and inference step, RET is optimized to utilize the information encoded in the contextualized embeddings.
We show the importance of using external memory noon-paraeetric space) of contextualized target emn bedding in generative retrieval models by comparing the performance between CGR and GENRE (Cao) et al. 202T), generative retrieval model which only operates on the parametric space. CGR shows an av- erage 6% increment in Natural Questions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017) in KILT (Petroni et al.) [222T) and 18% (25%) higher performance in Hit@| (@ 10) of NQ-320k. We also compare the results with different baselines for a comprehensive understanding of the model performance.
The main contributions of our paper are as follows:
We present Contextualized Generative Retrieval (CGR), a generative retrieval model which uses the contextualized embedding matrix while generating a target sequence. It shows an average of 6% and 18% (25%) higher performance in KILT (NQ, TQA) R-precision and NQ-320k Hits@) (@ 10), respec- tively, compared to GENRE in the same setting.
We show that using contrastive learning as intermediate training further increases the performance of the contextualized generative retrieval model by a large margin.
We perform extensive ablation studies and analysis over several variants of contextualized generative retrieval models for a comprehensive understanding of how to use contextualized embeddings and why using contextualized embeddings is better than using vanilla vocab embeddings.
2 RELATED WORK
Existing generative retrieval models retrieve relevant items by generating either the identifiers or entire sequences of the items. Cao al. (2021) propose GENRE (Generative ENtity Rttreeaal), which retrieves a document by generating the titles with constrained beam search. TTy et al.) (2022) propose DSI (Differentiable Search Index), which assigns unique ID each item in the corpus and trains the model to encode all information of the document and the ID in the model parameters. During the inference step, DSI generates the ID of the most relevant document. Waan et al. (2022) propose NCI (Neural Corpus Indexer), which also retrieves by generating the document ID as in DSI, but improves performance by query gener- ation and prefix-awan weight- adaptive decoder. [Bevilaquua et al. (2022) propose SEAL (Search Engines with Autoregressive LMs), which can retrieve any span from any position in the corpus by using the com- pressed full-text substring index (FM-Index). this work, we propose Contextualized Generative Retrieval which generates the target sequence by utilizing the contextualized embedding matrix rather than the vanilla vocab embedding matrix as in the aforementioned generative retricval models. Therefore, the model utilizes both the parametric space of the generative retrieval and the non-parametric space of contextualized token embeddings. To the best of our knowledge, we are the first to utilize the contextualized token embeddings on generative retrieval models.
Generative Retrieval
2

---

Under review conference paper at ICLR 2023
Embedding Model (EMB)
Retrieval Model (RET)
Cape A cape a dirting Foessoy p.
Copete
Ciinate of
South Africa
Coeds:
Cass Town: Cass Town one
Tuee
Softmax
T5 Enc
of South Afric deseminee 2-3
ba
Contextualized Embedding Space
(terate aol corpun)
Contextualized Embedding Mairi (elspe:
Swit
Enbedddin
Sree
During which season does town recuive rainfall
Trainabie
Charro
Prozes
Embedding
Vanilla Embedding Space
Figure 1: The side shows the model architecture CGR the right side shows the difference between contex- tualized embedding space and vanilla embedding space which is constructed with contextualized embedding matrix and vanilla embedding matrix, respectively. Contextualized embedding matrix is constructed by the output embeddings from EMB and is used as the decoder vocab embeddings in RET; existing generative retrieval utilizes the vanilla embedding matrix when generating the target sequence, RET utilizes the contextualized embedding matrix. In the document retrieval task with a title the target sequence, the title of document and its corresponding content from document given as input to EMB and we only save the output embeddings of the title.
Retrieval Models with Contextualized Token Embedding ME-BERT (Luan et al., 2021), Col-- BERT (Khattab & Zaharia 2020) and COIL (Gao al., 2021a retrieval models which retrieve the target sequence by utilizing the multiple contextualized token embeddings. It has shown high performance by leveraging the benefits of the cross-encoder architecture in bi-encoder architecture. CGR also utilizes the contextualized token embeddings, but differs from the three models in that while they interact in the inner product space, CGR has the benefit of interacting in the parametric space.
Semi-Parametric Models KNN-LM (Khandelwal et al., 2020), RAG (Lewis et al., 2020 and RETRO (Borgeaud et al. 2022) are semi aimi-prrmmtric models which use both the parametric space of the model and the non non-parametric space. KNN-LM improves the LM performance by generating the next token during the inference step by interpolating between the nearest neighbor distribution (distance in the contex- tualized embedding space) and the model vocab distribution. RAG and RETRO are semi-paramen cani-parametric models that first retrieve relevant texts with the retriever in parammaetteic manner and generate the output based on the retrieved texts. CGR also utilizes both the parametric and non-paran parameeriiic space as the three models do. However, it differs from KNN-LM in that it is trainable and from RAG and RETRO in that CGR uses the non parameerii space for the decoder vocab embeddings.
3 CONTEXTUALIZEI GENERATIVE RETRIEVAL
Generative retrieval is the task of retrieving the most relevant retrieval target (e.g., title, passage, document identifier) by generating the retrieval target token-by-toke when given input query. The training objective of the generative retrieval model is to maximize
P((t, ,tt::,tt,)))))))))))))) & II P(t. (g
am
where t,, denote the tokens of the retrieval target. Such approach has shown high performance while using a low storage footprint (Cao et al. 2021] Tay et al. [2022] /Bevilacqua et al. 2022 Lee et al.) (022). However, it has limitations in that the model depends solely on the information encoded in its own parameters. Thus,
3

---

Under review conference paper at ICLR 2023
the model fails to retrieve the correct target sequence if it has not seen the information during the training process.
To overcome the limitations, we propose Contextualized Generative Retrieval model (CGR), generative retrieval model which uses not only the parametric space of the model but also the non- non-parametric space (external memory) of contextualized token embeddingg to leverage the benefits of the bi-encoder model and combine the advantages of the two models. CGR (Figure T utilizes the contextualized embeddings (output embeddings of the language model encoder) rather than the vanilla vocab embeddings while generating the retrieval target. Therefore, the model does not depend only on the information encoded in its own parameters, but can also take advantage of con-paraettric information during generation as CGR encodes the document content into contextualized vocab embeddings. Also, by allowing single token to have multiple token embeddings, the model can learn about the different meanings of the token and the different contexts in which it is used. Therefore, the embedding space constructed with multiple contextualized token embeddings (contextualized embedding space) will become more expressive and fine-grained than the embedding space constructed with model vocab embeddings (vanilla embedding space).
CGR differs from existing generative retrieval models by which token embedding matrix is used for the de- coder model. Generative retrieval models such as GENRE (Cao et al. 202I) utilize the pre-trained language model architecture as-is: both the encoder and the decoder model share the same vanilla vocab embedding matrix of shape V x D where V is the vocab size, and D is vocab embedding dimension. CGR whereas uses different vocab embedding matrices for the encoder and decoder model: the encoder model uses the vanilla vocab embedding matrix (V D) as in existing generative retrieval models, but the decoder model uses the contextualized embedding matrix of shape C' x D where C' is the number of contextualized embeddings. C' is larger than V as a token is matched with multiple contextualized embeddings in most cases (c.g., in Figure same token "Cape" has two different contextualized embeddings which name Cape(1) and Cape(2) to differentiate the two.), but C' can be reduced with practical tactics (Section 4.4
CGR is composed of two submodules: (1) EMBedding model (EMB), which is an encoder model that outputs meaningful contextualized embeddings, and (2) a RETrieval model (RET) which is an encoder- decoder model that retrieves a target sequence by generating the sequence while utilizing the information encoded in the contextualized embedding matrix. For example, when the retrieval target sequence is the title of document, we pass the concatenation of the title and its corresponding document content as the input EMB and save the pair of input tokens and their output embeddings. By passing not only the title but also its corresponding document content to the model, the output embeddings contain both the information of the title and the document content. In practice, for efficiency, we only sample few embeddings that are deemed be the most informative and representative of the target sequence, or simply the first few embeddings. The extracted contextualized embeddings are then used to form the contextualized embedding matrix, which serves as the decoder vocab embeddings of RET. As the vocab embedding matrix the encoder and decoder model have different shapes, we assign different token IDs for the decoder; a unique token is paired with multiple token IDs where each ID indicates different contextualized embedding in the decoder.
4 MODEL DETAILS
we show the overall architecture of CGR that can be applied to general tasks. In this section, we present details of how we design CGR for practical usage in document retrieval tasks with the document title as the retrieval target. The ideal design of CGR is to use the encoder of RET as EMB for every gra- dient update during the training step to ensure the high coherency between the contextualized embeddings and RET. However, such a method requires high computational cost as it needs to construct contextualized embedding matrix at every step. Therefore, we present practical models; the base architecture of contextu- alized generative retrieval (CGR and two improvements, CGR and CGR Also, we show how
In Section 3
In this paper, contextualized embeddings refer to the output of language model encoder, which can incorporate information from the nearby context.
4

---

Under review conference paper at ICLR 2023
we reduce the number of contextualized embeddings, We add the figure of each model and more details in Appendix A
4.1
CGRBASE
Base CGR (CGR in Figure is the most basic contextualized generative retrieval model among the ones we propose. It uses the pre-trained T5 encoder as EMB and the T5 encoder-decoder as RET. EMB is frozen during the training step, and only RET is trainable.
4.2 CGRASY
Asynchronous CGR (CGRAsyne) is model where EMB is asynchronously replaced by the encoder of RET for every N epochs. When the model parameters of EMB are replaced, we construct a new contextualized embedding matrix with the replaced EMB and resume the training. As the decoder vocab embeddings of RET are updated every N epoch, EMB and RET of CGRAsyne would have more coherency between each other compared to BGR We keep N 20 for all experiments. See Appendix C.T details on how N affects the performance.
4.3 CGRCONTRA
Bi-encoder retrieval models with contrastive loss have shown high performance, the model learns and constructs well-structured global embedding space and regularizes the space to be uniform (Ni et al.] [222]] Gao al. 2021b Gao & Callan 2022 Izacard et al., 2022). CGR with Contrastive Learning (CGR designed to leverage such benefits of contrastive learning in contextualized generative retrieval model; the model first trained with contrastive loss and then on the generative retrieval objective, i.c., retrieving the most relevant sequence by generating the sequence token-by-token.
Step Token-level Contrastive Learning We first train RET with token-level contrastive loss to allow the model to learn the overall search space of token embeddings in the target corpus. Given training dataset of pairs {(g.1) f)} where q is the query text, and f is the retrieval target (title of the document to retrieve) composed of multiple tokens f, S i < l where k is the length of the target), k separate pairs {(g, 4 where f, is a token of f to construct the training dataset of query-token. With the query-token dataset, we train the first output token embedding from the decoder of RET be close to all token embeddings in T when given query q as an input to RET. T = {t
split the dataset into
1 (k |T+ () is a set of positive token embeddings (tokens that make up one title), and T .t..... .t....t--]1 1 is the set of negative token embeddings, which are all other token embeddings in contextualized embedding matrix. The objective is to minimize the contrastive loss to make the query text embedding q closer to all positive token embeddings in T
EVeT e I(q.ti t + .tr--) - llog ΣLee eggtt> Σx-Er- <<a
where ) the inner product value between the two embeddings. As we have the whole set con- textualized embeddings of the corpus (contextualized embedding matrix), it is possible to consider all other embeddings as negative. See Section C.2 for more details and analysis of different contrastive loss and model architecture.
Step 2. Generative Retrieval We further train RET from stepl with generative retrieval objective, which is the same as in CGR To be specific, we use the encoder of RET in stepl as EMB of step2 to extract the contextualized embeddings, and use the RET in stepl the initial parameter of RET in step2.
5

---

Under review conference paper at ICLR 2023
Training Dataset
Single (g 3%)
NQ+TQA (S 5%)
All KILT (100%)
Model
NQ
TQA
NQ
TQA
NQ
TQA
GENRE
$1.8*
65.0*
52.7*
64.8*
60.3
69.2
CGRavv
59.0
68.2
59.4
68.7
CGR Asyne
59.2
68.4
59,8
68.7
CGRcutta
59.8
68.6
60.3
68.9
BM25
23.1!
25.2!
23.41
25.2!
23.4!
25.2!
DPR
60.1
63.91
59.5
62.91
61.5
SEAL
63.2
68.4
Table I: R-precision(%) (%) for document retrieval task on NQ and TQA test dataset in KILT. Results except CGR are from the KILT leaderboard. The column of the table is divided by how many training datasets used. Numbers in the bracket are the rate of the number of training datasets over the number of training datasets when using all KILT datasets. Results with GENRE are from GENRE* Underlined model is direct baseline of CGR Results with i BM25 and DPR are trained in same setting as CGR (Appendix B.2) Best in bold.
4.4 CLUSTERING
To construct the contextualized embedding matrix to be used as the vocab embedding matrix of the decoder of RET, we first extract all contextualized embeddings of each target token with EMB. As it requires large storage footprint to save all the embeddings, we reduce the number of embeddings by using clustering and saving only the representative embeddings of each cluster. To be specific, we perform k-means clustering over the contextualized embeddings of the same token (which might have different surrounding contexts) and leave only the & centroid embedding as the decoder vocab embeddings of the token. We keep = 5 for all experiments. When k = 5. it only requires 0.3% of storage footprint compared all contextualized token embedding. Also, it requires 0.34GB more storage compared to the vanilla vocab embeddings (k 1) which is marginal compared the storage footprint to save the model parameters (3GB). See Appendix C.3 for examples, and how l affects the performance and the storage footprint.
when saving
5 EXPERIMENTS
In Section
5.1 we describe the baselines, datasets, and basic setup used in our experiment. In Section 5.2 we show both qualitative and quantitative effectiveness of using contextualized embeddings by comparing CGR with baseline models. We also compare the performance and characteristics among the variants of CGR.
5.1 SETUP
We compare CGR with six baselines widely used in document retrieval task: BM25, GENRE, DSL, NCL, SEAL, and DPR. BM25 (Robertson & Zaragoza 2009) is a term-matching model relying on an efficient algorithm. DPR (Karpukhin et al., (020)) is bi bi-encoder retrieval model which retrieves the most relevant document by mapping query and documents to a common vector space and performing a nearest neighbor search. See Section 2 for descriptions of GENRE, DSI, NCI, and SEAL. We compare all models using document retrieval task, where the input is query, and the output is a sequence related to relevant Wikipedia documents (e.g., title, document ID). See Appendix B for more details.
KILT (NQ, TQA) We train CGR with two datasets, Natural Questions (NQ) (Kwiatkowski et al. 2019) and TriviaQA (TQA) (Joshi al. 2017) from the KILT dataset (Petroni al. 2022)) benchmark for knowledge-intensss language tasks with eleven different datasets spanning five different tasks (fact check- ing, question answering, entity linking, dialogue, and slot filling). It gathers data different formats into a
'When the number of extracted contextualized embeddings of token is smaller than k we do not perform k-means clustering but use its own contextualized embedding. Also, we single non-contextualized embedding for special tokens such the EOS token PAD token.
6

---

Under review conference paper at ICLR 2023
Table 2: Hits@1, Hits 10 in NQ-320k. Results of BM25&DSI are from Tay et al. 2022) and NCI&NCI- are from Wang et al. 2022). NCI- is NCI without query generation to match the number of training datasets with other models. All models based on T5-large. Underlined model direct baseline of BGe Best in bold.
Table 3: R-precision($$ (%) for the document re- trieval task on NQ and TQA test dataset in KILT. We compare the results of GENRE* BGR title-only and CGe where the models are trained with NQ+TQA (Sectiion5.2) The results show the importance of extracting contextual- ized embeddings with not only the title but also corresponding document content.
Training Dataset
Model
Hits0e1
Hits@9 10
BM25
11.6
34.4
DSI
35.6
62.6
NQ-320k
NCI-
53.6
67.8
GENRE
-idlo--itl--nnly
CGRau
GENRE
53.7
64.7
CGR
62.2
78.8
NQ Trivia
52.7
58.4
59.4
Custa
63.4
81.1
64.8
68.2
68.7
NQ 320k additional datasets
NCI
88.7
95.8
common format, and the corresponding datasets share the same snapshot of Wikipedia as the corpus. DPR and GENRE are trained with all nine datasets in KILT Contextualized embeddings of the target retrieval sequence (title of the page) are the output embeddings from EMB when the title and its corresponding docu- ment content are given as the input. We evaluate all results with R- R-precision, metric widely used to evaluate retrieval performance in KII.T is calculated as & where R is the number of Wikipedia documents in each provenance set, and r is the number of related documents among the top-R retrieved documents.
NQ-320k To compare with Tay et al. (2022). we experiment on NQ-320k, a restricted setting from the official NQ dataset; it uses about 4% of Wikipedia corpus the corpus se We construct the contextualized embedding matrix with the title the document and its corresponding content as the input to EMB. The results are evaluated using Hits@N (N=(1, 10)), which shows the proportion of the correct documents ranked in the top N predictions.
5.2 RESULTS
KILT (NQ, TQA) Results in Table I show that CGRcomia outperforms GENRE* by 6% which demon- strates the effectiveness of contextualized embeddings. For both cases where the model is trained over a single dataset and over NQ and TQA together (NQ+TQA), all CGR variants show higher performance over GENRE* All CGR variants trained jointly on NQ and TQA show higher performance than those trained on a single dataset (NQ or TQA). Such results suggest that CGR tends to improve the performance when trained with more datasets. Note that due to limited available resources, we did not train CGR with the full KILT dataset (ALL KILT) as in G DPR, or SEAL. However, CGR trained on less than 5% of the training dataset from the full KILT dataset show higher comparable performance those models. See Appendix C.7 for results in the KILT dev set.
NQ-320k We report the results on NQ320k in Table 2 which uses about 4% of Wikipedia corpus as the corpus set, to compare CGR with DSI and NCI which only experiment over NQ-320k. We compare the results between CGR CGR and baselines (BM25, DSI, NCI, GENRE*). CGRcontr shows the highest performance when trained on the same number of datasets; 18% and 25% higher performance to GENRE @its@I and Hits@ 10, respectively. We also compare the result of CGR Base with DSI as a direct baseline in Appendix C.8
*Due to limited resources, we did not train CGR with full KILT datasets as GENRE, which used 128 V100 GPUs with 32GB of memory for about 33 hours. For a fair comparison with CGR, we train GENRE", GENRE trained with the same resource, same pre- ve-trrined model (T5), hyperparameter, and dataset as CGR.
'The corpus set is the union train/dev/test target sequences. As exact splits, document ID, preprocessing code used by TTy et al. (2022) are not released, we tried to replicate the setting as closely as possible when constructing the NQ-320k dataset to train CGR
GENNE uses 128 V100 GPUs with 32GB of memory for about 33 hours.
7

---

Under review conference paper at ICLR 2023
Table 4: R-precision(%) for the test sets of document retrieval tasks KILT. Both GENRE* and CGR Bass are trained
with NQ + TQA; other datasets are not seen during the training time. Best in Bold.
In- In-Domain Datasets
Out-of-Domuin Datasets (Inference Only, Zero-Shot)
NQ
TQA
FEVER
AY2
WnWi
WnCw
T-REX
2=RE
HoPo
ELI5
WoW
GENRE"
52.7
64.8
64.2
9.1
2.8
3.4
53.9
76.1
34.3
11.2
48.9
CGR Base
59.4
68.7
67.0
10.3
5,4
59.1
79.2
37.5
12.5
GENRE* vs. BGR in Zero-Shot Setting Table 4 shows that CGR is stronger than GENRE in the zero-shot setting where the models are trained on NQ and TQA and are evaluated on the other 9 datasets in KILT that are not used during the training step. CGR shows an average of 3% improvement from GENRE BGe shows high performance on information unseen during the training step as it does not solely rely the information encoded in the parametric space (information that the model sees during the training step) but also on the non- non-parametric space of the contextualized embeddings.
Differences among CGR CGRAsyic- and CGRcontra In Table we can see that CGR shows consistently higher performance than CGR and Asyne We hypothesize two factors for such improve- ments. First, as the model is trained on contrastive learning before training on generative retrieval task, CGRcomta can leverage the benefits of contrastive learning where it learns and constructs well-structured overall embedding space and regularizes the space to be uniform (Ni et al.) [022]] Gao et al. 2021a b Izac-
ard al., 2022). We check the quality of the embedding space with Luniformity proposed in Wang & Isola (2020), where the numbers represent how uniform the embedding space is. CGRcomta (-19.7) shows a lower
number than BG (-18.2) where the lower the better. Second, as EMB is initialized with the encoder of RET, there is high coherency between EMB and RET. The importance of having high coherency between EMB and RET can also be seen from the performance gain from CGR to CGRAsync: CGR Base uses the initial EMB without any replacement, but CGRAsyne replaces EMB with the encoder of RET every N epochs and shows higher performance as N decreases. i.e., the update is more frequent. More details in Appendix C.4
Importance of Having Contextualized Embeddings with Document Content In Table 3 we compare the results between CGR -itt---lll--nly a model trained with contextualized embeddings extracted with only the title as the input to EMB, and CGR a model trained with contextualized embeddings extracted with both the title and corresponding document content as input to EMB. an intermediate model between CGRBase and GENRE* as uses the non non-parametric space but is constructed with limited information (only with the title, without the entire document content). CaB shows the highest performance, GENRE* shows the lowest performance, and the performance of -Gee is in-between the two models, suggesting that there is correlation between the performance and how much contextual information is in the non-parametric space.
-ssc--itle-only can be considered as
GENRE*, which uses vanilla vocab embedding as the target embedding, has to depend solely on the in- formation encoded in its own parameters (the parametric space of the generative retrieval model). On the other hand, CGR and CGRaae-title--nn retrieval model as GENRE* does but also the non- parametric space of corpus information embedded in the contextualized target embedding. By utilizing the contextualized target embedding. the model can know in which context the token is used and discern documents with different contexts.
can depend on not only the parametric space of the generative
Although both CGR and ized target embedding of CGRass-title-nnn tille-only utilize contextualized target embeddings, the contextual- contains constrained information compared to that of CGR Therefore, CGRnae-tille-nnis title-only fails on cases where the document content is necessary to retrieve the target sequence successfully. Table shows examples where there is no direct relationship between the query and the target sequence such as lexical overlap or semantic similarity. It is difficult for the model to predict the target without the help of the document content about what information is in the document what relation- ship exists between the query and the target sequence. We can see from the table (Table (5) that CGR successfully retrieves such information is embedded in the contextualized target embeddings whereas
8

---

Under review conference paper at ICLR 2023
Table 5: Top-3 prediction results of CGRiue. CoR and GENRE* on NQ dev set in KILT. Highlights on the correct target sequence.
Query
Prediction Results
CGR
Therefore sign
Infinity symbol, Equation
what the dots mean in math
-itle-ooy
Slashed zero, Homo sapiens, Equation
GENRE
Ellipsis, Infinity symbol, Homo sapiens
CGR
Evolution of the horse
Equis, Eurydice
when did equus first appear in fossil record
CGR Equidae, Equis, Euclid
GENRE
Equidae, Equis, Equinox
CGRyu
Noli Me Tängere (novel) Noli Me Tangere (opera). Noli Me Tangere (Bemini)
rizal finished all the chapters of the novel noli tangere in
tilleony me tangere,
Tängere (novel) Noli Me Tangere (opera)
GENRE
Noli me tangere. Non è I'infemo, Noli Me Tangere (opera)
CGRBae-title-one title-only fails does not contain the information in its embeddings. See Appendix C.5 for more details about the contextualized embeddings of CGR
Lexical overlap between query and retrieval target To see whether the main performance improvement of CGR over GENRE* comes from CGR leveraging the information contained in the contextualized token embedding, we check the performance of CGR -ittl---ill,,,,,y, and GENRE* on queries that need document content find the answer. We first run TF-IDF over all the queries of NQ dev set in KILT and divide the queries into two sets: low-overlap and high-overlap. Low-overlap is set of queries with TF-IDF score lower than average, and high-overlap the rest of the queries. For queries in the high-overlap se all three models show high performance as it is easy to infer the correct retrieval target from the query alone even the model does not know the document content. On the other hand, while all models show rela- tively lower performance for queries in the low-overlap sets la the context information is required to infer the relationship between the query and the retrieval targe CGR shows the most strong performance. CGR shows about 7% higher performance
the low-overlap set and 5% higher performance on the high-overlap than GENRE" by leveraging the contextualized information encoded in the token embedding (Table 9 in Appendix). See Appendix C.6 see more detailed examples of the prediction result of GENRE* and CGR
6 CONCLUSION
In this paper, we propose Contextualized Generative Retrieval (CGR), generative retrieval model that uti- lizes contextualized embeddings (output embeddings of the language model encoder) rather than vanilla vocab embeddings while generating the target sequence. This way, the model does not rely only on the in- formation encoded in its own model parameters but also on the information encoded in the contextualized embeddings. Experimental results show that CGR achieves significantly higher performance than vanilla generative retrieval, demonstrating the effectiveness utilizing such non-parametric external memory dur- ing decoding. We also perform extensive ablation studies and analysis on several variants of contextualized generative retrieval models to better understand how they work.
ec.g. Q: where was the world economic forum held this year / Target Document: World Economic Forum *c.g.. Q: During which scason does cape town receive rainfall Target Document: Climate of South Africa "Among the queries that all three models successfully retrieved the right retrieval target, 61% % of queries are in the high-overlap set. Also, among the queries that all three models failed, 74% of queries are in the low-overlap set. Such rates show that queries in low-overlap relatively difficult.
9

---

Under review conference paper at ICLR 2023
REFERENCES
Michele Bevilaequa, Giuseppe Ottaviano, Patrick Lewis, Wan-Tau Yih, Sebastian Riedel, and Fabio Petroni. Autoregressive search engines: Generating substrings as document identifiers. arXiv preprint arXiv:2204. 10628, 2022.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George den Driessche, Jean- Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring. T. W. Hennigan, Saffron Huang, Lorenzo Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and L.. Sifre. Improving language models by retrieving from trillions of tokens. In ICML, 2022.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. Autoregressive entity retrieval. In ICLR, 2021.
Luyu Gao and Jamie Callan. Unsupervised corpus aware language model pre-training for dense passage retrieval. In ACI,, 2022.
Luyu Gao, Zhuyun Dai, and Jamie Callan, Coil: Revisit exact lexical match in information retrieval with contextualized inverted list. In NAACL, 2021a.
Tianyu Gao,, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. In EMNLP, 2021b.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. TMLR, 2022.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Duta, 2021.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly super- vised challenge dataset for reading comprehension. In ACL, 2017.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP, 2020.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In ICLR, 2020.
O. Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. 2020.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural ques- tions: A benchmark for question answering research. TACL, 2019.
Hyunji Lee. Sohee Yang. Hanseok Oh, and Minjoon Seo. Generative multi-hop retrieval. arXiv preprint arXiv:2204.13596, 2022.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Regrened generation for knowledge-intensinn nlp tasks. In NeurlPS, 2020.
10

---

Under review conference paper at ICLR 2023
Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. Sparse, dense, and attentional repre- sentations for text retrieval. TACL, 2021.
Jean Maillard, Vladimir Karpukhin, Fabio Petroni, Wen tau Yih, Barlas Oğuz, Veselin Stoyanov, and Gargi Ghosh. Multi-task retrieval for knoeledge-ittensive tasks. In ACL, 2021.
Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma. Keith B Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. CoRR, 2021.
Fabio Petroni, Aleksandra Piktus. Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorn,, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: benchmark for knowledge intensive language tasks. In NAACI, 2021.
Stephen E. Robertson and Hugo Zaragoza. Found. Trends Inf. Retr. 3:333-388 389, 2009.
The probabilistic relevance framework: Bm25 and beyond.
Yi Tay, Vinh Quang Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen, and Donald Metzler. Transformer memory as a differentiable search index. arXiv preprint arXiv:2202.06999) 2022.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity the hypersphere. In PMLR, 2020.
Yujing Wang, Ying Hou, Hong Wang, Ziming Miao, Shibin Wu, Hao Sun, Qi Chen, Yuqing Xia, Chengmin Chi, Guoshuai Zhao, Zheng Liu, Xing Xie, Hao Sun, Weiwei Deng, Qi Zhang, and Mao Yang. neural corpus indexer for document retrieval. arXiv preprint arXiv:2206.02743 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Transformers: State-of-the-art natural language processing. In EMNLP, 2020.
Ledell Yu Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. Scalable zero-shot entity linking with dense entity retrieval. In EMNLP, 2020.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. ICLR, 2021.
A MODEL DETAILS
See Figure 2a Figure Figure 2b and Figure 2c for figures of generative retrieval model, CGR CGRAsyna Asynes and CGR respectively.
A.1 INFERENCE STEP
We perform constrained beam search with prefix tree (Cao et al. 2021 Lee et al. 2022 during the in- ference step to assure that all generated sequences are in the corpus. The prefix tree constructed with the tokenization result of the corpus, and we perform constrained beam search by masking out the tokens that do not create sub-string of the text in the corpus. We find the next tokens from the top-k of the unmasked ones. While token ID was used as the node of the prefix tree in previous works since each token to a unique token ID, we construct a prefix tree with the text of the token as the node, because CGR contains multiple token IDs for single token. Therefore, rather than unmasking only single token ID, we unmask all token IDs that correspond to the text in order to unmask a token. We keep the beam size to 10 for all experiments following Cao et al. (202T).
mapped
11

---

Under review conference paper at ICLR 2023
Climate
South Africa
Sottmax
Libear
T5 Dec
T5 Enc
Embedding
Vanilia Embedding Matrix (shape: Vx
During which sesson does capa town receive raintall
(a) Model architecture of Generative Retrieval Model. uses vanilla embedding matrix while generating the retrieval target.
Embedding Model (EMB)
Retrieval Model (RET)
Climate
South Africs
sdeeeias panantt..
Enc
Linwa
Emtiedding
aptape: x
Enc
Embeediing
Baring which sasson does cape town recdire raintall
Trainable
:Prosan
(b) Model architecture of CGRAsync- It differs from CGRBuse in that EMB is replaced by the encoder of RET every N epochs.
Embedding Model (EMB)
Retrieval Model (RET)
Guea: may auee deevente cali
Helll Alte to caat ni-T
12 14
Enc
(beaae for coppes)
(Fape Cx D)
Embadding
Dusing which sasson doon town reodive ratrial
Ernbadding
(c) Model architecture of CGRoom stepl (contrastive learning). The first output token embedding from the decoder of RET is trained to be close to all contextualized token embeddings of the retrieval target (positive pairs) but far from other contextualized token embeddings (negative pairs). Note that the model architecture of CGRcomm step2 (generative retrieval task) the same CGRaue (Figure
12

---

Under review conference paper at ICLR 2023
Table R-precision((6 (6) for the document retrieval task on NQ and TQA test dataset in KILT. See Appendix C.2 for dctails about how the loss term differs. The loss term is used while training CGRconta in contrastive learning (step of training CGRconta..
Positive
Negative
NQ
TQA
Single Token Finh
In-Batch Negatives
60.0
68.9
Single Token Eimh Contextualized Finhedding Matrix
58.9
68.4
Multiple Token Emb
Contextualized Embedding Matrix
60.3
68.9
B EXPERIMENTAL SETUP
B.I CGR & GENRE*
We train all models using a pre-trained T5-large checkpoint from Wolf et al. (2020) as the initial checkpoint. GENRE* and CGR trained with the same hyperparameter setting for fair comparison. We experiment over 32GB V100 GPUs or 48GB A6000 GPUs. We train using Adafactor with a learning rate le-4 with linear warm-up for the first 10% of training and then linear decay with batch size 512 till maximum of 150 epochs.
B.2 BM25 & DPR
Unlike Maillard et al. (021), which performs document retrieval tasks by training the model on passage- level tasks and considers retrieval successful when it retrieves the document that contains the passage, match the setting (dataset) similar to CGR, we train the model in document retrieval task. We consider the first five paragraphs as the content of the document and train the model so that the query embedding gets close to not the paragraph embedding but the document embedding. The number of the corpus in the document retrieval tasks are the same as the number page in the KILT dataset.
For BM25, the corpus is the same in DPR where each item in the corpus is the first five paragraphs of individual documents in the KILT corpus.
C EXPERIMENTAL RESULTS
C.I UPDATE FREQUENCY IN CGRASYe-TIILLE-ON
We analyzed how the performance changes according how often the replacement of EMB by the encoder of RET occurs (replacement for every N epoch) with CGR When comparing the performance with N {10, 20, 50). -Gyeesye-tttlee -itle-onlly shows the highest performance at N 10, and the perfor- mance tends to deteriorate as N becomes larger. Also, all CGR CGGRssye-iitle-onn show higher performance than CGR title-only (CGR-tike -ittle-olly without any replacement). Results show that although the model requires high computation cost and longer training time N gets smaller, it is important to have high coherency be- tween the contextualized embeddings (output embeddings of EMB) RET by frequent replacement.
C.2 DIFFERENT CONTRASTIVE Loss IN CGR
We experiment with three different types of contrastive loss when training CGRcomtaa In this section, we show the losses and how the results differ by such methods.
13

---

Under review conference paper at ICLR 2023
Given a training dataset of pairs {(q,)} where q the query text, and is the retrieval target (title of the document to retrieve) composed of multiple tokens I; (1 < í S h where k is the length of the target), we split all tokens into separate pairs {(g.4, construct the training dataset of query-token The three loss differs in what the model considers as negative set and a positive set.
Loss 1: Neg: In-Batch Negatives / Pos: Single Token Embedding With the query-token dataset, we train RET'S first output token representation from the decoder to be close all t+ E {L...... , 42 (embedding of any token the retrieval target f) given the query as an input to RET. The objective is to minimize the contrastive loss to make the query text embedding q be closer to positive token embedding t+:
e<qtt>>
L(q, t+, ,t ( - log e<att> + -E-ET--E0 Exq.t~>
- is the set of where (, ) is the inner product value between the two embeddings, and T ( negative token embeddings. which are other token embeddings in the training batch that are not paired with q (in-batch negatives (Karpukhin et al. 2020)).
Loss Neg: Contextualized Embedding Matrix Pos: Single Token Embedding The loss differs from the upper loss in that it considers all embeddings in contextualized embedding matrix except the single positive embedding as negative rather than performing the in-batch negatives which consider the subset of contextualized embedding matrix negatives. The equation is same as Equation 3 but elements in T are all other token embeddings in contextualized embedding matrix.
Loss 3: Neg:
Contextualized Embedding Matrix Pos: Multiple Token Embedding The loss differs from the upper loss in that it considers all token embeddings in the title as positive embeddings; for each query q. there are more than one positive contextualized token embeddings.
With the query-token dataset, where T+ = [t first output token representation from the decoder be close to all token embeddings in T given the query q as an input to RET. The objective is to minimize the contrastive loss to make the query text embedding q be closer to all positive token embedding in T
t }. set of positive token embeddings, we train RET's
EIEE++ e<q
L(q. tt. 17+ 1 - -log EXtET+ e > Ex-er- e<qtt>
(4)
where < is the inner product value between the two embeddings, and T .t.,.... .,7 } is the set of negative token embeddings, which are all other token embeddings in contextualized embedding matrix.
Results Tablo the performance of CGRcouta with different contrastive loss by what it considered as the positive pair and the negative pair. Multiple Token Emb considers all token embeddings in the same target sequence as positive pairs, and Single Token Emb considers all token embeddings separately thus only one of the token embedding from the title token embeddings is considered as positive pair. In-Batch Negatives considers all embeddings in a batch except for the positive embedding as negative pairs, and Contextualized Embedding Matrix considers all embeddings in the contextualized embedding matrix (a matrix constructed with the contextualized token embeddings) except for the positive embeddings as negative pairs.
The model trained on contrastive loss with multiple token embeddings as positive pairs, and all other em- beddings in contextualized embedding matrix as negative pairs (Loss3) show the highest performance. The model trained the same negative but with single token embedding as positive (Loss2) shows the lowest performance. The model with single token embedding as positive and bn-chtch negatives as negative pairs (Lossl) shows the performance in-between.
14

---

Under review conference paper at ICLR 2023
As in Xiong al. (2021), ). the model with Loss2 and Loss3 has the benefits of looking at the global embedding space by considering the contextualized embedding matrix as the negative pair, unlike Loss which only considers embeddings in the same batch as negatives (in-batch negatives). However, Loss2 show lower performance than Loss as in the case where the model considers single token embedding as a positive pair, the model considers the rest of the token embeddings in the same title as the negative pair. As the token embeddings in the same title are matched with the same query, such a training method seems to make the model confused and leads to bad performance. Thus when considering a single token embedding as positive pair (Loss] or Loss2), it is better to consider only the embeddings in the same batch as negatives (in-batch negatives) rather than on all the token embeddings (Contextualized Embedding Matrix) as there is a low possibility of the model to have two different token embeddings of the same title in batch.
C.3 CLUSTERING
Example of Clustering When token "the" appears in the corpus 100 times, 100 different contextualized embeddings of "the" are extracted by the encoder model at first. Then, we perform k-means clustering on the 100 contextualized embeddings cluster them into at most & clusters and save all centroid embeddings. We leave only the k centroid embeddings as the decoder vocab embeddings of the token "the" and assign a new decoder token ID for each contextualized embedding by the cluster belongs to. By repeating the process over all the tokens, each token has a number of contextualized embeddings less or equal to k. As there are multiple contextualized token embeddings for single token, we replace the ground-truth target token IDs with the newly constructed decoder token IDs to speciffy which contextualized token embedding the ground-truth target token ID is referring to.
Performance by Number of Clusters and Storage Footprint As saving all contextualized token em- beddings to use the vocab embedding matrix requires large storage footprint (~ 148GB), we reduce the number of token embeddings by clustering and saving only the & centroid embeddings for each token (Section (.44). Figure [ shows the effect of the maximum number of clusters for each token (k) on the per- formance. Models with a k = 5 (maximum of five different contextualized token embeddings for each token) show the highest performance and having k smaller or larger than five decreases the performance. We hypothesize that performance of models with k < 5 degrades because the number of the embeddings is too small contain all different contextual meanings of the token and thus will be closer vanilla token embedding. In contrast, the performance of models with k >> 5 decreases because the search space of each generation step is too large and the parametric space of the model becomes too fine-grained which might distract the model. When k= 5, the number of embeddings is 980 times less than using all the contextual- ized embeddings of KILT corpus as the vocab embeddings and 3.7 times larger than using the vanilla vocab embeddings of T5 (k = 1). Therefore, when A = 5, it needs 0.47GB of storage footprint to save all the vocab embeddings, whereas the vanilla vocab embeddings (k- = 1) need 0. The increase in the storage footprint of vocab embeddings (0.34GB) is marginal compared the storage footprint to save the model parameters (3GB).
C.4 CHARACTERISTICH OF CONTEXTUALIZED EMBEDDINGS IN VARIANTS OF CGR
We compare the contextualized token embeddings of CGR Base. CGRAsyne- and CGR embeddings, we check the rate of the same token among the top-5 embeddings similar to the corresponding embedding. CGRBas shows the lowest rate of 50%. CGR Asyn: and CGRc show similarly high rate of 70%. The rate tends to increase as N increases in CGRAsy Such results suggest that as same token has similar lexical meaning, it is better to have a relatively similar meaning, However, as the performance increases as a single token are matched to multiple token embeddings till = 5 (Appendix C,3), is also
To For 1000 cluster
We analyze the EMB of step2 in CGR and last replace EMB for CGRAs
15

---

Under review conference paper at ICLR 2023
important to have slightly different meanings depending on the surrounding context. When checking which corpus bundles are bound to the same cluster, all three tend to depend on which position of text the token is placed on and the meaning of surrounding tokens. See Appendix C.5 for more details.
C.5 CLUSTERING OVER TOTAL EMBEDDINGS
To understand the spatial properties of the contextualized embeddings, we conducted qualitative analysis on the embeddings, by performing k-means clustering over the total contextualized token embeddings of Bae (EMB is the encoder of largee. Specificaly, we clustered 36 million token embeddings, obtained from EMB, into 11 117,508 clusters using the FAISS k-means library (Johnson et al. 2021).
First, we randomly sampled 100 tokens, and for each token, we calculated the portion of the contextualized embeddings that belong to the top 10% of the clusters which contain the most embeddings of the token. As result, on average 67.6% of the embeddings of a token are contained in the 10% of the clusters which contain the token, with standard deviation of 22.7. This indicates that most of the tokens are concentrated in a few spatial regions, while the others are spread over many different areas.
To get deeper insight into the spatial properties of the embeddings, picked two tokens, "Lincoln" and "Squad" and visualized some of the clusters that contain the tokens(Table 77. For each cluster, the tokens belonging to the cluster and their corresponding document names shown. In (Table 7 at most 20 documents are shown for each token and only 4 tokens are shown in cluster 3 and 4 for simplicity. The first and second examples show the case that cluster is composed of only single token, mentioned above. Interestingly, all of the corresponding documents of the first cluster are related to Lincolnshire, a county of England. Similarly, the tokens the second cluster are related to the documents about sports (usually football) squads. On the other hand, the third and fourth examples show the other case that cluster contains only few tokens that we are interested in. The members of the third cluster are related the middle names, and a few embeddings of the token "Lincoln" is contained in this cluster since there are some Wikipedia documents of the people whose middle name is Lincoln. Likewise, the fourth cluster consists of the embeddings which are related to the name of music albums(usually hip-hop and rock), where some of them are produced by the group named "Blazin" squad", for example. These examples show how expressive the contextualized embeddings be compared to the vanilla token embeddings: in this case, it is hard to expect that these various context-dependent information of token can be sufficiently encoded into a single token embedding.
summary, the results show that the contextualized embeddings corresponding to the same token are mapped many different regions of the embedding space, depending on its context. This implies that the contextualized embeddings successfully acquired the contextual information of the corresponding docu- ments, highlighting the effectiveness of utilizing contextualized embeddings for generative retrieval.
C.6 LEXICAL OVERLAP BETWEEN QUERY AND ANSWER
CGR show especially strong performance queries in the low-overlap set; queries that in most cases need the context information unless the model saw the information during the training step (Section 5.2). We check four sets:
1. GENRE+/CGR +: queries where both CGR and GENRE successfully retrieved
2. GENRE+/CGR- queries where GENRE* successfully retrieved and CGR failed
3. GENRE-/CGR +: queries where GENRE* failed and CGR succeed
4. GENRE-/CGR=: queries where GENRE and CGR both failed.
"The number of the clusters is same the number of the tokens in contextualized embedding matrix, henee same the number of the clusters used in 4
16

---

Under review conference paper at ICLR 2023
GENRE CGR
GENRE-
CGR
50
30
10
10
30
Low-Overlap (NQ)
High-Overlap (NQ)
Figure 3: Red bar indicates the high-rate and the blue bar indicates the low-rate. The rate is measured by NQ dev in KILT. Details about high-rate and low-rate is in Appendix C.6
GENRE+
GENRE-
CGR+
CGR
-70
-50
-30
-10
100
30
70
Low-Overlap
High-Overlap (TQA)
Figure 4: Red bar indicates the high-rate and the blue bar indicates the low-rate. The rate is measured by TQA dev set in KILT. Details about high-rate and low-rate is in Appendix C.6
70
NQ
TQA
64-
58
10
cluster
Figure 5: Effect of number cluster in Base CGR. Results of zero number clusters are from GENRE
17

---

Under review conference paper at ICLR 2023
Table 7: Examples of clusters when clustering over total contextualized token embeddings when EMB is the encoder of Ia-larg.
Cluster
Token
Documents
Moulton, Lincolnshire Belton, North Lincolnshire Walcon, Lincolnshire Wrangle, Lincolnshire Swineshead, Lincolnshire Leverron, Lincolnshire Kirton, Lincolnshire Benington, Lincolnshire / Bicker, I. Lincolnshire/ Dyke, Lincolnshire Hilldyke, Lingolnshire Waltham, I Lingolnshire / Reepham, Lincolnshire / Bradley. Lincolashire Allington. Lincolnshire Donington. Lincolashire ettleton. Lin- colnshire Panton. Lincolnshire Beekingham. Lincolashire/ / Bigby. Lincolashire
I Lincoln
Field hockey at the 2000 Summer Olympies Men's teum squads Field hockey the 2004 Summer Olympics Men's team squads Field bockey the 1996 Summer Olympics Men's team squads Football at the 2000 Summer Olympics Men's team squads / Football at the 1996 Summer Olympies Men's team squads List of Queensland rugby league team squads Football at the 2006 Lusophony Games - Men's 's team squads List of current AFL team squads/ Football at the 1912 Summer Olympies squads / Football at the 1996 Summer Olympics Women's 's team squads Football at the 1988 Summer Olympics MMnn' team squads Foothall at the 1984 Summer Olympics Men's 's team squads / Football at the 1976 Summer Olympics Men's 's team squads / Football at the 1900 Summer Olympics Men's team squads / Football at the 1904 Summer Olympics Men's 's team squads/ Foothall the 1908 Summer Olympics Men's team squads Football the 1992 Summer Olympics- Men's 's team squads Football the 1980 Summer Olympics Men's 's team squads Football at the 1972 Summer Olympics -Men's 's team squads /...
Men's squads / List of New South Wales rugby league
2
Squad
William Lincoln Garver / Albent Lincoln Washbum Charles Lincoln Edwards TThomas Lincoln Casey St. / James Lincoln Collier/ / Abraham Lincoln Lewis/ Earl Lincoln Poole / George Lincola Goodale George Lincola Burr / Abraham Lincolu Keister / Elmer Lincola Irey / Walter Lincoln Hawkins / Abram Lincoln Harris/ / Abraham Lincoln DeMond / Thomas Lincoln Tally / Abraham Lincoln Filene/ / Mary Lincoln Beckwith / Frederick Lincoln Emory/ Howard Lincoln Hodgkins / Oliver Lincoln Lundquist ...
Lincoln
John Levi Marti / John Levi Sheppurd / Moses Levi Ehrenreich / Nathaniel Levi Gaines/ Harry Levi Hollingworth/ Thomas Levi Whittle Austin Levi Fraser George Levi Crane/ Olin Levi Wamer
3
Levi
Milledge Luke Bonham / Henry Luke Orombi Henry Luke White Vincent Luke Palmisano George
Luke
Luke Smith Henry Luke Bolley/ Mary Luke Tobin/ / James Luke Prendergast / John Luke Lowther Thomas Luke Msusa Robert Lake Deakin Juseph Luke Cecchini
Jerry Luke LeBlanc/
Laae
Carroll Lane Fenton
Squad
True Story (Teroo Squad album) / The Album (Terror Squad album)
Covenant (Morbid Angel album) / Domination (Morbid Angel alhum) / The Art of Dying (Death Angel album) / Act III (Death Angel alhum)/ / Heretic (Morbid Angel alhum)
Angel
Butterlly
Heavy (Iron Butterfly alhum)/ / Metamorphosis (Iron Butterfly album) / Ball (Iron Butterfly album)
Flip-flop (electmoncs)) Flipper (anatomy) Respect (Láll Flip album) Leprechaun (l.ll Flip album)
Flip
Note that CGR is BGR in this section. Figure 3 and Figure 4 show the low-rate (blue) and high-rate
(red). Low-rate of each case is calculated as (Qnz) where Q is set of queries in each case and L is set of queries in low-overlap set. High-rate of each case is calculated as (Q2H) where H is set of queries in high-overlap set.
Q
18

---

Under review conference paper at ICLR 2023
Table 8: Top-3 prediction result of CGRaave. and GENRE
Query
Prediction Result
CGRyas
Therefore sign
Infinity symbol, Equation
what do the dots mean in math
GENRE
Ellipsis, Infinity symbol, Homo sapiens
CGR
Parable of the Pearl Mitzvah, Pearl of Wisdom
what does the pearl symbolize in the bible
GENRE
Pearl of Great Price, Perlin, Promised Land
CGR
Archie Marries Veronica/Archies Marries Betty List of Riverdale characters,
does archic end up with betty or veronica in riverdale
Archic Buchanan
GENRE
Riverdale (2017 TV series), List of Riverdale characters, Archie Mitchell
CGRBaw
Jesse Williams (actor) Jesse Williams, Jesse Spencer
actor who plays di avery on grey's 's anatomy
GENRE
Mare Alaimo, Patrick Warburton, Jeffrey Dean Morgan
CGRpus
Evolution of the horse
Equis, Eurydice
when did equus first appear fossil record
GENRE
Equidae, Equis, Equinox
CGRBus
Indian High Courts Act 1861 High Court Australia, Supreme Court of India
who decides the number of judges in the high court
GENRE
Supreme Court of the United Kingdom, Supreme Court of India, High Court of
Australia
when's the last time philadelphia eagles played the new england patriots
CGR
Super Bowl XXXIX New England Patriots, Super Bowl XXXVIII
GENRE
New Englland Patriots, Philadelphia Eagles, History of the Philadelphia Fagles
CGRgase
Noli Me Tángere (novel) Noli Me Tangere (opera), Noli Me Tangere (Bemini)
rizal finished all the chapters of the novel noli tangere
GENRE
Noli tangere, Non è I'inferno, Noli Me Tangere (opera)
during which season does cape town receive rainfall
CGR
Climate of South Africa City of Cape Towa, Cape Town water crisis
GENRE
Cape Town, City of Cape Town, Cape Town water crisis
Table R-precision( %) for document retrieval task on NQ dev dataset in KILT, See details about Low and High Overlap in Section 5.2
GENRE
CGi fitle-only
CGR
Low-Overlap
45.8
51.6
52.7
Hveh-Overlap
71.3
75.3
75.8
Total
58.3
63.2
64.0
For both figures, GENRE-ICGR + shows a higher number in low-rate, which indicates that CGR tend to suc- cessfully predict queries in the low-overlap set compared to GENRE* Also, for both figures, GENRE+/CGR + shows a high number in low-rate and GENRE-/ours: shows a high number of high-rate, which indicates that queries in the high-overlap set tend to be easy questions for both GENRE and CGR whereas queries in the low-overlap set are difficult for both models.
Also, Table g shows samples of the top-5 prediction results of CGR and GENRE* where CGR Base suc- cessfully retrieved the correct item and GENRE* failed. Such queries tend to be in the low-overlap set. Such results suggest that CGR (CGR Base) is robust queries in the low-overlap set compared to GENRE*
19

---

Under review conference paper at ICLR 2023
No Truining
Single (< 3%)
All KILT (1005)
Dataset
BM25
DPR
GENRE
CGRave
AGR
CGRcana
DPR
GENRE
SEAL,
SEAL,
NQ
25.8
63.2
59.9
63.4
63.7
63.9
62.3
64.3
64.2
TQA
29.4
65.1
64.2
67.5
67.9
68.7
62.0
71.1
68.3
68.3
Table 10: R-precision($8 () for document retrieval task on NQ and TQA dev dataset in KILT. Results of BM25 and DPR are from Maillard et al. (,02T), results of SEAL are provided by the authors of [evilacqua et al. (022),, and results of GENRE from the released pre-trained models. SEAL, is SEAL (LM+FM) and SEAL; is SEAL (LM+FM, intersective) Bevilacqua al. (2022). Underlined model the direct baseline of CGR. Best of each section (same training dataset) in bold.
Table 11: Hits@1 and Hits@ 10 in NQ-320k. Results of BM25, DSI-Naive, and DSI-Semantic from Tay al. (2022). CeRawe-Naive is CGRase with document ID as retrieval target, CGe Title CGRaas with title the document as a retrieval target. Underlined model is direct baseline of CGRRae-Naive Best from document ID as a target sequence in bold.
BM25
DSI-Naive
DSI-Semantic
CGR Naive
CGR Title
Hits@1
Hits 10
34.4
33.6
62.6
73.1
78.8
C.7 KILT DEV RESULTS
Table 10 shows the result of five models (BM25, GENRE, DPR, SEAL, and CGR) in the NQ and TQA of the KILT dev set. Note that the DPR and BM25 models in the single setup 12 are different from Table (Section B.2). As in results with KILT test datasets (Table D. results with KILT dev datasets (Table 10) show similar trends. CGR shows an average of 7% higher performance compared to its direct baseline model, GENRE* CGRcomta shows the highest performance over three variants of CGR. When comparing the results with other retrieval models, CGR shows the highest performance in the single setup and shows comparable performance to models trained with all KILT datasets although CGR is trained with only 3% of the training dataset.
C.8 CONTEXTUALIZED EMBEDDINGS WITH DOCUMENT ID AS RETRIEVAL TARGET
To show that CGR is not restricted to the title but is generalizable to various retrieval targets, we experiment CGR (CGR Base -Naive) when the retrieval target is random document ID (Naively Structured String Identi- fiers in DSI TTy et al. )022)) in NQ-320k. In the case, direct baseline model to CGR -Naiee is DSI-Naive (DSI with Naively Structured String Indentifiers the document ID) not GENRE From Table 11 we can see that CGR shows more than four times higher performance in Hits @ compared to DSI-Naive.
Also, CGR shows higher performance than DSI-Semantic where the document ID of DSI- Semantic is built by the hierarchical semantic meaning inside the document content. The results show the importance of using the contextualized embeddings as the decoder vocab embeddings; rather than building the document ID using the document content, using a random document ID but with the document content inside the token embedding of each document ID shows higher performance.
Single setup setup where models are trained with single dataset; NQ only or TQA only. Note that the document ID of CGR -Givv NNive and DSI-Naive are not exactly the same as the document ID of DSI is not relcased. However, both the document ID of CGR Naive and DSI-Naive are the same in that they are constructed randomly. We plan to update the numbers when the official repo is open
20

---

Under review conference paper at ICLR 2023
Moreover, from the results between BGR and BaReeeeTitle, we can see that the retrieval target affects the performance. We assume the difference comes from whether the retrieval target is natural language text or not; CGR can leverage the benefits of pre-training step more when the target sequence is natural language (aGR Title). However, the difference between R and Base is marginal compared the performance difference between DSI and GENERE where the two models both used vanilla model embeddings, which suggests that CGR is generalizable to various retrieval targets. We leave other target sequences apart from the title and the document ID as future work.
When compared to the performance of DSI T5-XXL with Semantic String Docid (the best performing model Tay et al. (2022) (40.4), BGee (58.7) 7) shows about 1.5x higher performance in NQ-320k Hits Note that the model has 14 times more parameters than CGR
D LIMITATIONS & FUTURE WORKS
As CGR uses k-means clustering to reduce the number of contextualized embeddings, the performance may change by how the contextualized embeddings are clustered. Generative retrieval models show low performance to unseen cases due their dependency 2022]; the model is likely to retrieve sequences it has seen during the training step better as the information is saved in the parameters. Therefore, CGR also leverages the benefit of the parametric space, we leave the direction of adapting the model to new corpus as future work.
the parametric space of the model (Lee al.
21


Reviews:
[{"review_id": "JTRfbcB2Doq", "reviewer": "Reviewer_ZLXX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a contextualized generative retrieval model.  The generative retrieval model usually performs worse on unseen data and the traditional KNN retrieval model has the advantage of good generalization but also fails on documents with long sequences. The authors propose a new architecture to overcome these problems.  The proposed model uses the output of a language model encoder as vocab embeddings. The retrieval model then retrieves a target sequence by using the vocab embeddings as the decoder embeddings. Experiments show that the proposed model achieves promising retrieval performance. ", "summary_of_the_review": "This paper proposes a new generative retrieval model and the proposed model shows promising results. ", "strength_and_weaknesses": "Strength:\n\n1. The proposed architecture is simple and elegant.\n\n2. The proposed approach shows promising results on downstream tasks. \n\n\nWeaknesses:\n\n1. The proposed model follows the framework of generative retrieval models but applies contextualized token embeddings.  Such contextualized embedding methods have been applied in KNN-LMs. The novelty contribution is somehow limited. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8suLjYKfxP", "reviewer": "Reviewer_UYMW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper regards generative retrieval that given a query generates important information about text documents from the parameter space of a trained model instead of relying on fixed-sized document embeddings which might be characterized by limited expressiveness. Such a generative model interacts with the parameters of the trained model.  The authors propose the use of a contextualized embedding space. The contextualized embedding space is created using a language model.\n", "summary_of_the_review": "Information retrieval is characterized by dynamically changing document collections and their content (e.g., novel named entities, events at new locations, etc,). It is not clear how the model guarantees a correct generation of key descriptors of a document, if the generative retrieval model and the contextual language model were not trained with it. This relates to the core claim of the paper. Please clarify. \n\nThe initial representations used for training the retrieval model and the representations obtained by the language model and their postprocessing (e.g., clustering of embeddings) play a role with respect to the content that is finally retained from each document. What if these representations were changed by using alternative technologies to build them? The representations are still built with neural nets that are currently only able to precisely capture a limited context and not the content of a large document. \n\nCould the proposed generative model be combined with older generative retrieval models that operate on bag-of-words or n-gram representations (e.g., Robertson-Sparck-Jones model, language retrieval model) in order to deal with specific content that should be generated? Overall references to older generative retrieval models could be added in related research. \n\nThe paper could be strengthened by analyzing the limitations of the work. \n\n", "strength_and_weaknesses": "Strengths\n-\tImportant timely topic in information retrieval.\n\n\nWeaknesses\n- Evaluation on datasets that are characterized by title, document ID or datasets for language tasks, and not on large-scale document retrieval collections where the specific content of the documents is important to satisfy the query, and where expressiveness and discriminative power of the document representations play an important role. \n- Limited technological contribution.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "86TXR72eSTO", "reviewer": "Reviewer_DJS8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The proposed model targets a Wikipedia page-level retrieval task: given a query, return the page (or in their case, the Wikipedia title following GENRE). Like GENRE, the authors use a sequence-to-sequence model, take the query text as input, and directly generate the title in an autoregressive manner. The innovation is instead of using static token embeddings in autoregressive decoding, the proposed model uses contextual embeddings, which are acquired by encoding all tokens in possible titles (with the Wikipedia content as context). The embeddings naturally contain both the token information like the static word embeddings and the passage information through the encoder. Arguably, the new model has both the power of autoregressive generation and the external knowledge of Wikipedia through contextual embeddings.\n\nThe authors explored several variants. First, to compress the huge number of contextual embeddings, they conduct clustering for each token’s corresponding embeddings and keep k=5 clusters. The authors also have different versions where the contextual embeddings are fixed at the beginning (CGR-base), the embeddings are updated by the updated seq2seq model (CGR-async), and the embeddings are jointly trained through contrastive learning (CGR-contra), though the differences are small. \n\nThe authors mainly conduct experiments on KILT and NQ320K, and mainly compare their model to the direct baseline, GENRE. The results show that CGR (the proposed model) significantly outperformed GENRE on KILT (especially in-domain datasets NQ and TriviaQA), and outperformed a series of other autoregressive retrieval models on NQ320K. However, one questionable point: GENRE uses BART and CGR uses T5, but the authors only compared to the original GENRE numbers in KILT experiments instead of reproducing GENRE with the same setup, which might lead to unfair comparison. There should also be other reproduced baselines like DPR (or any dense retrieval methods) and SEAL in KILT for a more comprehensive understanding. \n", "summary_of_the_review": "The proposed method -- using contextual embeddings instead of static token embeddings in decoder for autoregressive retrieval -- is novel, intuitive, and inspiring. However, the experiment has significant flaws (for the most direct and important baseline, the authors took numbers from the original implementation that uses even a different pre-trained model, which leads to unfair comparison). I am leaning toward rejection. However, if the authors can update the results with the reproduced GENRE baseline, I am willing to raise my score.", "strength_and_weaknesses": "# Strength\n\nThe proposed idea is novel: it combines the advantage of both autoregressive generation (GENRE, DSI, SEAL) and contextual embeddings of databases (DPR). The use of token contextual embeddings is also smart since it can be compressed by the clustering algorithm, without too much loss in information. The study of different variants of training is comprehensive. Experiment results (if trustworthy) are strong.\n\n# Weakness\n\nThe main weakness comes from the experiment setup. The main KILT results only have the original GENRE (using a different pre-trained model, BART) and CGR (even though the authors listed DPR and SEAL, the numbers are not in the same comparable settings). The authors should provide a reproduced GENRE with the same pre-trained model and the same training recipe, as well as other baselines like BM25, a dense retrieval method (like DPR), and, if possible, SEAL.\n\nI have some other concerns in Table 2 and Table 3. In Table 2, GENRE, DSI, and SEAL’s performance is so low on NQ-320K. I would appreciate it if the authors could clarify the setting (sorry if I missed it) or maybe retrain the GENRE on NQ-320K in the same setting as CGR (it shouldn’t be difficult since the code should be similar to CGR). ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5P-H4M10Fh", "reviewer": "Reviewer_rcjJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to pre-compute contextual token embeddings and use them in the decoder of generative retrievers (as a way to enable them to use external knowledge). The model performs favorably on the KILT retrieval tasks when trained on a subset of KILT. ", "summary_of_the_review": "The paper proposes a reasonable way to use external knowledge in generative retrievers, but the quality of innovation, experiments, and writing seems to fall below the standard of ICLR. ", "strength_and_weaknesses": "Strengths: \n- It is a reasonable approach to injecting external knowledge in generative retrievers. \n\nWeaknesses: \n- Unclear writing: Figure 1 and a verbal description in Section 3 is all we have to decipher what the paper does. I can guess what it does, but it's never fully clear.\n- The setup is questionable. GENRE uses BART-large whereas this paper uses T5, so the focus on comparing directly with GENRE seems a bit misguided.\n- Limited experiments (not training on all of KILT). ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject