Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a novel scheme to apply autoregressive language models decoding to retrieval tasks, in which documents are  represented using all constituent n-grams as possible identifiers. The key idea is to use an FM-Index to prevent the generative model from producing document identifiers with text outside of any of the indexed documents. The appeal of the approach is that large language models can be adapted (fine-tuned) for this task without major architectural changes. Empirically, the approach performs competitively with a selection of recent baselines, in some cases outperforming them.


strengths_and_weaknesses: This paper does a good job explaining current drawbacks of autoregressive models in retrieval tasks (related work) and the empirical results demonstrate that the proposed scheme does improve the quality of retrieval results.

My main concern is that there may be better ways to apply large language models for retrieval, such as the re-ranking approach that is mentioned in the introduction; this wouldn't require complex generation involving indices and constrained decoding. Indeed, re-ranking shares the purported benefits of the proposed approach, namely straightforward application of large language models without architectural changes, with the key advantage that it only requires *scoring* a (query, target) pair. Autoregressive models, by definition, suffer from slow decoding speed, and therefore I'm skeptical the latency would be low enough for this approach to be practical in many settings.

Regarding the experiments, not enough motivation is given for why the baselines were selected, and it's not clear whether these truly represent SoTA results. For example, do you compare to a re-ranking baseline? There are many approaches from the IR community such as ColBERT that seem relevant but are not discussed.

Regarding the selected baselines, it doesn't look like the baselines leverage any sort of constrained decoding. It would be interesting to constrain decoding using simpler alternatives to fully understand the impact of the FM index and proposed weighting scheme.

The paper is readable and well-written. Additionally, the authors detail their experiments well and share their code which bodes well for reproducibility.

L37: extend->extent
L230: "our the size of our index is"
questions: * What are the implications of the assumption that documents are indexed by exact n-grams? For example, it would be good to provide intuition for why this does (or does not) preclude matching documents via synonyms or related words / expressions.
limitations: * I would have liked to see more discussion of inference-time latency.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: - The paper presents an approach (called as SEAL) for document retrieval where a language model conditioned on a question generates n-gram tokens to identify the relevant documents from the evidence documents (or passages). To enable this functionality, the method trains BART using question and n-gram pairs, where the n-grams are sampled from the gold passages. To constrain generation to output valid n-grams such that they correspond to some documents, the approach indexes the evidence documents with an efficient datastructure called FM index. The paper presents several ways to score the generated n-grams, one where the n-gram corpus level frequency is considered and another one where  they try to score multiple n-grams using intersection of tokens. They experiment with Natural Questions (NQ) Open dataset and the KILT tasks to showcase the competitiveness of their proposed approach.
strengths_and_weaknesses: **Strengths**
 - The biggest strength of the paper is the novelty of the proposed approach. While the dominant classes of retrieval methods consist of training dual encoder models with pre-computed evidence indexes, this paper presents another method with which retrieval can be performed. 

- The paper is well-written, organized, sound experiments, and related work section is good. The authors have submitted the code-base which will make the work reproducible. 


**Weaknesses and Suggestions for Improvements**

- Apart from NQ, the paper does not perform evaluation on popular QA datasets such as MSMARCO, TriviaQA, SQuAD, WebQuestions, and Entity Questions where the performance of baseline models are well established. These are the datasets where a large fraction of the retrieval papers report results and benchmark their models. SEAL should also train a common model on all these datasets in the DPR-Multi style to assess the generalization of their approach.

- Performance comparison with DPR and other strong dual encoder models. From the results in Table 3, SEAL falls behind DPR on Accuracy@ 5 and Accuracy@20, where its performance is considerably low. I suggest it’s important to highlight this limitation and probe further the reasons. There should be a more rigorous performance comparison with improved dual-encoder training approaches which obtain much better results. For example, the model ANCI (https://openreview.net/forum?id=zeFrfgyZln) and ICT-DPR and MSS-DPR in (https://arxiv.org/abs/2101.00408) obtains better performance numbers than DPR and these results should be reported in Table 3.

- To understand the dependence of SEAL model on number of training examples, it will be useful to compare the sample efficiency of SEAL algorithm with that of DPR on the NQ training examples.

- Considering the number of trainable parameters, where DPR trains 220M parameters while SEAL trains ~400 M parameters, the direct comparison with DPR is not fair. This should be clearly highlighted in the paper. Are the benefits of SEAL model on KILT tasks mainly due to the trainable parameter size? It would be good to finetune smaller and larger generator models such as different configurations of T5 (or T5 lm adapted models) and then study the correlation of performance vs model size.

- The writing in the method section seems a bit dull, especially the paragraphs of “factoring in FM-index frequencies” and “intersective scoring for multiple n-grams”. Using visual illustrations and diagrams to convey the information would help the reader understand the importance of these ideas more.

- Probably a minor point, I  feel that the training / inference time of SEAL / DPR should be calculated using the same compute hardware and then compared. Also, the 64 GB of DPR index size in Table 2 is when using 32 bit representation for floats. It has been shown that 16 bit or more optimized representation of passage embeddings also works just as well (https://arxiv.org/abs/2106.05346). Similarly, GPU is not always required to perform fast search. Toolkits such as ScaNN (https://arxiv.org/abs/1908.10396) also works well on CPU.

questions: Please see the points under the Weaknesses section.

** EDIT **
Increasing the rating to 6.
limitations: Yes, the authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work proposes SEAL, which  trains a language model to perform retrieval tasks leveraging constrained decoding over a FM-index of ngrams in a corpus. More specifically BART is finetuned to generate sampled 10 ngrams from each ground truth document biased in favor of ngrams with high character overlap with the query.

Several scoring functions are explored including:
(LM scoring) the score P(n|q) of the most probable  fixed-length ngram.
(LM+FM scoring) the pointwise mutual information between query and ngram computed from P(n|q) and P(n) 
(LM+FM intersective scoring) aggregates the contribution of multiple ngrams

For page level task NQ320k, SEAL improve over the quality of a much larger retrieval solution (GENRE).  For passage level task (NQ), SEAL significantly improve over DSI, which suffers capacity losses when memorizing document ids. SEAL produces comparable results to DPR and GAR (especially for A@100 which is the most important for QA purposes). More detailed analysis shows that SEAL is a lot better on novel questions or answers. 
For KILT passage retrieval task, SEAL improved the state of the art result significantly.
For KILT downstream tasks, SEAL improves state-of-the-art for 4 out of 7 tasks.

This work represents a significant progress for learned indexing structures -- combining LM with multi-point indexing and scoring.

strengths_and_weaknesses: see the summary
questions: how is the ngram sampling done to  bias towards overlap with queries? there seem to be an lack of details.

why only train from the ground-truth doc? why not leverage unsupervized training for retrieval?

it seems that the model is never trained to optimize ranking quality directly. Is there any possibility for learning to rank?


limitations: see the summary
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed to use autoregressive language models as search engines. Previous work in this direction has explored ways to do such retrieval by generating some unique identifiers of the documents. The authors proposed an even simpler approach -- using all ngrams in a document as possible identifiers and directly generating them using an autoregressive language model. This process was done through constrained decoding with the help of FM-index. They conducted experiments on the Natural Questions and the KILT benchmark, and the experimental results demonstrated that they could achieve comparable performance with the SOTA models while using a considerably lighter memory footprint.

strengths_and_weaknesses: - Originality: Although the idea of generating unique document identifiers using autoregressive models is not quite new, I do like the idea of generating the ngrams in a passage as possible identifiers. Also, I appreciate the efforts the authors put to make this simple idea work, as can be seen from the results, one needs to combine equations (2), (3), and (4) to make it really work.

- Quality: The proposed approach is technically sound and interesting. As seen from the experimental results, they outperformed the prior autoregressive baselines on NQ and KILT. They also achieved comparable performance with DPR on NQ and outperformed DPR by a large margin on KILT. Note that the authors also claimed they achieved SOTA performance on 4 out of 7 datasets on KILT. Although this does not hold by now, I don't believe it diminishes the quality of this work.  
Other comments I have: (1) From my perspective, the ngram length $k$ and the number of k-grams selected during training are very important hyperparameters (set to 10 in the experiments) but lacked detailed discussion over in the paper. I would like to see an experiment regarding the performance change over a different length of ngrams. It could just be a small-scale experiment but I think having those numbers would help consolidate the claims made in the paper. (2) Correct me if I am wrong but I do not find the details of how to select a fixed number of k-grams from a total of $|d| - k$ k-grams. How important is the selecting strategy? It would be better if the authors can establish a baseline using a random selection of the k-grams.

- Clarity: In general this paper is well-organized and easy to follow. However, I think having a more formal definition of some notations in the paper could further improve the clarity. For example:   
Line 142, does ngrams(K) denote $K$ ngrams or a set of k-grams?  
Line 147, how is $P(n|q)$ computed?

- Significance: I think this paper studied a very interesting problem and proposed a promising approach. Although the experiments are not comprehensive for the readers to understand every aspect of the system, I think it still brings many benefits to the research community. I believe it would lead to some follow-ups in this direction.


questions: - By comparing Table 2 in this paper with Table 3 (Tay et al., 2022), I saw that the performance varies a lot on NQ320K even for BM25. Also, they basically achieved better performance on hits@1 while you achieved better performance on hits@10. Can you elaborate on this issue a bit more? This makes me hard to judge where this work sits in the literature.
- In Table 5, I noticed that the performance of **SEAL (LM+FM, intersective)** outperforms **SEAL (LM+FM)** by a large margin (almost doubled), whereas it is not the case on NQ. Do you know why this happens?



limitations: The authors sufficiently addressed the limitations in the paper.
review_title: null

Meta-review: metareview: This paper proposes a method (SEAL) for document retrieval where a language model (LM) conditioned on a question generates n-grams as document identifiers. This is done by training BART on question and n-gram pairs, where the n-grams are sampled from the gold passages, and at test time constraining generation to output valid n-grams that correspond to document identifiers. Experiments on Natural Questions (NQ) Open dataset and the KILT tasks obtain strong results.

Overall, all reviewers agree that this is a strong paper that proposes a simple but effective approach. I agree with their assessments and recommend acceptance. However, a weakness that has been pointed out is that the paper does not perform evaluation on other common QA benchmarks (MSMARCO, TriviaQA, SQuAD, WebQuestions, and Entity Questions) where the performance of baseline models are well established. I strongly encourage the authors to train SEAL on at least some of those datasets and compare with stronger baselines.
confidence: Certain
Decision: Accept