Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The proposed model targets a Wikipedia page-level retrieval task: given a query, return the page (or in their case, the Wikipedia title following GENRE). Like GENRE, the authors use a sequence-to-sequence model, take the query text as input, and directly generate the title in an autoregressive manner. The innovation is instead of using static token embeddings in autoregressive decoding, the proposed model uses contextual embeddings, which are acquired by encoding all tokens in possible titles (with the Wikipedia content as context). The embeddings naturally contain both the token information like the static word embeddings and the passage information through the encoder. Arguably, the new model has both the power of autoregressive generation and the external knowledge of Wikipedia through contextual embeddings.

The authors explored several variants. First, to compress the huge number of contextual embeddings, they conduct clustering for each token’s corresponding embeddings and keep k=5 clusters. The authors also have different versions where the contextual embeddings are fixed at the beginning (CGR-base), the embeddings are updated by the updated seq2seq model (CGR-async), and the embeddings are jointly trained through contrastive learning (CGR-contra), though the differences are small. 

The authors mainly conduct experiments on KILT and NQ320K, and mainly compare their model to the direct baseline, GENRE. The results show that CGR (the proposed model) significantly outperformed GENRE on KILT (especially in-domain datasets NQ and TriviaQA), and outperformed a series of other autoregressive retrieval models on NQ320K. However, one questionable point: GENRE uses BART and CGR uses T5, but the authors only compared to the original GENRE numbers in KILT experiments instead of reproducing GENRE with the same setup, which might lead to unfair comparison. There should also be other reproduced baselines like DPR (or any dense retrieval methods) and SEAL in KILT for a more comprehensive understanding. 

strength_and_weaknesses: # Strength

The proposed idea is novel: it combines the advantage of both autoregressive generation (GENRE, DSI, SEAL) and contextual embeddings of databases (DPR). The use of token contextual embeddings is also smart since it can be compressed by the clustering algorithm, without too much loss in information. The study of different variants of training is comprehensive. Experiment results (if trustworthy) are strong.

# Weakness

The main weakness comes from the experiment setup. The main KILT results only have the original GENRE (using a different pre-trained model, BART) and CGR (even though the authors listed DPR and SEAL, the numbers are not in the same comparable settings). The authors should provide a reproduced GENRE with the same pre-trained model and the same training recipe, as well as other baselines like BM25, a dense retrieval method (like DPR), and, if possible, SEAL.

I have some other concerns in Table 2 and Table 3. In Table 2, GENRE, DSI, and SEAL’s performance is so low on NQ-320K. I would appreciate it if the authors could clarify the setting (sorry if I missed it) or maybe retrain the GENRE on NQ-320K in the same setting as CGR (it shouldn’t be difficult since the code should be similar to CGR). 
clarity,_quality,_novelty_and_reproducibility: The proposed method is novel and intuitive. The authors could improve the clarity — e.g., the authors can make it clearer what text they put into the encoders for the contextual embeddings (in the paper they only vaguely mention that they are titles). Figure 1 is a bit confusing to me (especially Cape(1), Cape(2)). The experiment setup can be significantly improved. From the current experiments, it is hard to draw a clear conclusion.

summary_of_the_review: The proposed method -- using contextual embeddings instead of static token embeddings in decoder for autoregressive retrieval -- is novel, intuitive, and inspiring. However, the experiment has significant flaws (for the most direct and important baseline, the authors took numbers from the original implementation that uses even a different pre-trained model, which leads to unfair comparison). I am leaning toward rejection. However, if the authors can update the results with the reproduced GENRE baseline, I am willing to raise my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a contextualized generative retrieval model.  The generative retrieval model usually performs worse on unseen data and the traditional KNN retrieval model has the advantage of good generalization but also fails on documents with long sequences. The authors propose a new architecture to overcome these problems.  The proposed model uses the output of a language model encoder as vocab embeddings. The retrieval model then retrieves a target sequence by using the vocab embeddings as the decoder embeddings. Experiments show that the proposed model achieves promising retrieval performance. 
strength_and_weaknesses: Strength:

1. The proposed architecture is simple and elegant.

2. The proposed approach shows promising results on downstream tasks. 


Weaknesses:

1. The proposed model follows the framework of generative retrieval models but applies contextualized token embeddings.  Such contextualized embedding methods have been applied in KNN-LMs. The novelty contribution is somehow limited. 
clarity,_quality,_novelty_and_reproducibility: The paper is well-written. 
summary_of_the_review: This paper proposes a new generative retrieval model and the proposed model shows promising results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper regards generative retrieval that given a query generates important information about text documents from the parameter space of a trained model instead of relying on fixed-sized document embeddings which might be characterized by limited expressiveness. Such a generative model interacts with the parameters of the trained model.  The authors propose the use of a contextualized embedding space. The contextualized embedding space is created using a language model.

strength_and_weaknesses: Strengths
-	Important timely topic in information retrieval.


Weaknesses
- Evaluation on datasets that are characterized by title, document ID or datasets for language tasks, and not on large-scale document retrieval collections where the specific content of the documents is important to satisfy the query, and where expressiveness and discriminative power of the document representations play an important role. 
- Limited technological contribution.



clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. The proposed method makes sense but the contribution of adding a contextualized embedding space might be considered as light.
summary_of_the_review: Information retrieval is characterized by dynamically changing document collections and their content (e.g., novel named entities, events at new locations, etc,). It is not clear how the model guarantees a correct generation of key descriptors of a document, if the generative retrieval model and the contextual language model were not trained with it. This relates to the core claim of the paper. Please clarify. 

The initial representations used for training the retrieval model and the representations obtained by the language model and their postprocessing (e.g., clustering of embeddings) play a role with respect to the content that is finally retained from each document. What if these representations were changed by using alternative technologies to build them? The representations are still built with neural nets that are currently only able to precisely capture a limited context and not the content of a large document. 

Could the proposed generative model be combined with older generative retrieval models that operate on bag-of-words or n-gram representations (e.g., Robertson-Sparck-Jones model, language retrieval model) in order to deal with specific content that should be generated? Overall references to older generative retrieval models could be added in related research. 

The paper could be strengthened by analyzing the limitations of the work.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes to pre-compute contextual token embeddings and use them in the decoder of generative retrievers (as a way to enable them to use external knowledge). The model performs favorably on the KILT retrieval tasks when trained on a subset of KILT. 
strength_and_weaknesses: Strengths: 
- It is a reasonable approach to injecting external knowledge in generative retrievers. 

Weaknesses: 
- Unclear writing: Figure 1 and a verbal description in Section 3 is all we have to decipher what the paper does. I can guess what it does, but it's never fully clear.
- The setup is questionable. GENRE uses BART-large whereas this paper uses T5, so the focus on comparing directly with GENRE seems a bit misguided.
- Limited experiments (not training on all of KILT). 
clarity,_quality,_novelty_and_reproducibility: The paper is unclear on the model definition, making it hard to judge the quality. Based on my inference, the approach is reasonable but not terribly surprising. 
summary_of_the_review: The paper proposes a reasonable way to use external knowledge in generative retrievers, but the quality of innovation, experiments, and writing seems to fall below the standard of ICLR.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a semi-parametric approach to generation based retrieval. While generating a target sequence, the decoder accesses a memory bank of pre-computed token embeddings. Consider that similar techniques have been extensively studied in language modeling (KNN-LM, Yogatama et al), translation (Khandelwal et al, ICLR'21) and other generation applications including dialog (Fan et al), the novelty of the approach is somewhat limited. Also, reviewers had further concerns about the experiments (DJS8, rcjJ), presentation/writing (rcjJ). Considering all these, I am in favour of rejecting the paper in its current form.   
justification_for_why_not_higher_score: Not good enough
Decision: Reject