Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: While the traditional document retrieval problem is formulated as mapping both the document and the query to the same vector space and then performing nearest neighbor search to find the closest document fore each query, the paper instead proposes to formulate the task as a document identifier generation problem, where the input is the query and the output is the target document's id. This is not entirely novel with respect to DSI (Tay et al., 2022) but given that DSI was available on the archive only three months before NeurIPS deadline, one might be able to consider this paper to be a concurrent work. The proposed model by this paper, Neural Corpus Indexer, also introduces a few techniques: (1) query generation that augments the query-to-doc data for training (equivalent to document indexing), (2) prefix-aware weight-adaptive decoder that gives different symbols to the same tokens in different positions of the ids, (3) semantic document id so that similar documents share similar ids, and (4) regularization technique to prevent overfitting. The results on Natural Questions 320k (a subset of NQ) are very promising; the proposed model not only outperforms other generative retrieval models such as DSI and SEAL by a big margin but also strongly outperforms competitive baselines such as ANCE and BM25. 
strengths_and_weaknesses: Strengths:
- The results are very strong. The baselines that the paper compares against, such as ANCE and BM25, are very strong ones and hard to beat. It is remarkable that a generative model can beat those baselines.
- Query generation helps a lot and this is a very useful information for people working in this field. In fact, as shown in Table 2, among other techniques introduced by the paper, QG is by far the biggest factor for the superior performance of NCI.

Weaknesses:
- If query generation is the dominant factor, then it seems to me that what the model is really doing is "query indexing" rather than "document indexing". This means the performance of the model will highly depend on generating a comprehensive range of queries which are likely to overlap with the test queries. 
- Constructing the semantic hierarchy of the document ids can be considered as constructing the inverted index in the canonical NNS retrieval setup. Then what the decoder is doing can be considered as locating the correct cluster in each level of the hierarchy. Then I am wondering if it is fair to say this model is really "generative retrieval" or it is more of using different embeddings for the nearest neighbor search in each layer of the hierarchy. 
- The paper only experiments on Natural Questions and only a subset of it. While NQ is a very representative dataset in this task, the paper would have looked much better if it was tested on different datasets and/or the full dataset of NQ. It is not clear why such results are not shown.
questions: - With regard to the weakness mentioned above, I am wondering how sensitive the model's performance is with respect to the quality of the generated queries.
- I don't think it is correct to say that "DSI hardly captures the hierarchical semantics of document identifiers" in L42. DSI indeed proposes a similar clustering mechanism to what this paper proposes.
- How is the documents clustered for semantic identifiers? Could you please give more description of Section 3.1.
- What is difference between p1 and p in L223? I think Section 3.4 is unclear.
limitations: Yes, the paper mentions crucial limitations such as its lack of scalability compared to canonical NNS retrieval system, which I agree as well.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposed a sequence-to-sequence model that generates the relevant document given an input query, as a retrieval method. The proposed method have a huge improvement on the NQ320k dataset (Natural Questions). The most improvement seems to be coming from using generated query (based on the document) paired with the document as augmented training data.
strengths_and_weaknesses: Originality:
The proposed data augmentation technique is interesting and demonstrate good performance. The other techniques are interesting and providing much smaller but also good gains.

Quality:
The author addressed their limitations and present the work in context. The experiments are well designed.

Clarity:
The paper is clearly written.

Significance:
Very good gains on NQ320k over previous best method.

questions: 1. Can we use the same k-means procedure to obtain an identifier for a new document? I guess new documents will have the same identifier to be meaningful to the model?
2. It seems that the query generations has most contribution to the improvement. Have you compared with other query generation models / techniques so that we can understand what contribute to the success of this method?
limitations: The authors acknowledge a set of limitations of the model, including scale of the index set, inference speed and updating the index. I'm most interested in the third one. The model can only work on a fixed set of documents (to retrieve from), or there is a cost to update the model when the index set changes. This is a limitation of the line of works that generating the document identifiers, not specific to this work. However, I'm also curious, given that the document identifier is generated using a k-means algorithm, how the model performs when there are new documents if we just assign an identifier to a new document using the same k-means algorithm.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper focuses on a promising direction to learn retrieval via generation, where the model will generate the relevant document IDs for given queries. It resembles the previous Differentiable Search Index work with several minor improvements.

strengths_and_weaknesses: Strengths:
They propose several techniques to improve the model such as prefix-aware weight-adaptive decoder, query generation, consistency-based regularization. They show excellent improvement in recall@1 and recall@10 on the NQ320k datasets, largely outperforming previous baselines.

Weaknesses:

NCI is quite similar to DSI - e.g., the encoder-decoder architecture, indexing training and the hierarchical clustering algorithm to generate semantic doc IDs. It introduces several small improvements such as using prefixLM, consistency-based regularization, query generation, etc. The overall contribution seems limited. 

Another weakness is that many of the gains come from using the additional query generation data. Interestingly, all other improvements seem to have very limited effect on the performance. This also raises a question about how much the model relies on the quality of the query generation model and how it can adapt to new domains that the QGEN model can’t generate good questions for the target document. 

questions: NA
limitations: NA
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: 
This paper proposes a new framework for neural IR: given query, directly predict a document ID. The document IDs are obtained by hierarchical clustering of documents beforehand. This is a novel formulation of the problem, and is very distinct from current two-stage methods that have a high-recall sparse retrieval stage, followed by a high-precision neural reranker, or approximate nearest neighbor methods that encode both documents and queries as vectors.  
strengths_and_weaknesses: 
Strength:
- The formulation is unique and intriguing. I really like this idea! Despite some limitations, the results are very promising and I imagine many follow-up work may come from this paper. 

Weakness:
- I think the method rests on one large assumption, that documents can be pre-clustered into some fixed set of IDs. All queries are being trained with the same fixed set of document IDs. This is a somewhat unconventional way to think about IR, as one usually does not assume document cluster structure, and ad hoc queries may flexibly retrieve any subset. I think some experiments that try different definitions of document IDs (e.g. random) would shed light on the assumption of having a strong document clustering prior. 
questions: - Can you provide some timing analysis for the baselines (e.g. in footnote), so Table 4 has some reference of comparison?
limitations: - The limitation section is reasonable. I think the index-update aspect may be one of the most important near-term challenges.
review_title: null

Meta-review: metareview: This paper  proposes a new framework for neural IR: given query, directly predict a document ID. The document IDs are obtained by hierarchical clustering of documents beforehand. This is a novel formulation of the problem, and is very distinct from current two-stage methods that have a high-recall sparse retrieval stage, followed by a high-precision neural reranker, or approximate nearest neighbor methods that encode both documents and queries as vectors. The paper is fairly well-written, the authors have addressed reviewers concerns with honest detailed feedback, and have made their code available to facilitate experimentation. The results are particularly strong compared to more traditional BM25-based models and competing neural approaches like DSI. I anticipate there will be much follow-on work and eventually a paradigm shift in neural IR. 
confidence: Certain
Decision: Accept