Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper explores how to leverage retrieved context for language modeling by decoupled encoder-decoder models, like Retro (Retrieval-Enhanced Transformer). The model mainly includes two components: 1) a retrieval that retrieves the k most relevant texts of input x via vector-based search; then 2) a transformer-based encoder-decoder model is used to predict the target based on both input x and the retrieved context. Compared with Retro, this integration of input and retrieved context is at decoder instead of encoder. The model is evaluated on auto-regressive language modeling on C4 datasets and Natural Question (a question asking task).
strengths_and_weaknesses: Strengths: 

The proposed approach is simple, and it seems more parameter efficient than Retro.

It provides extensive experiments especially on different scales.


Weaknesses:

1)Many experiments results are about bits-per-byte on C4 dev. I’m wondering how this metric correlated to end-task performance. 

2)On the only one real-world application, NQ, the performance is still far behind FiD which is very similar to the proposed approach in terms of model architectures. Nothing that FiD only has 770m parameters


questions: How the bootstrapped retriever training affects the NQ results? Do you have any ablation? 


limitations: This type of training should help more on knowledge intensive or grounded tasks, and it would be great to see more evaluation on them. 

The model may work on zero-/few shot transfer settings, and it would be great to have such experiments as well.

It is hard to understand the point of Figure 5, and how it is related to main story.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a method to augment the context based on retrieved relevant information, and keep the additional context representation separated from the history/prompt in an encoder-decoder setup, and thus decoder can be separately trained, and there’s latency benefit since the context and query don’t need to be re-encoded jointly after context is retrieved in inference time.  Authors evaluated their methods on language modeling and open domain question answering tasks and showed the computational benefit of the proposed method. 

strengths_and_weaknesses: The proposed method is intuitive. I feel people have done such evaluations but couldn't find an exact paper with such results. 
The paper presents experimental results showing the proposed method has good performance (comparable to other approaches while reducing computational cost). 

Authors did a good job and provide enough details to show the advantage of their method, mainly in terms of the computational efficiency. 

One weakness of the paper is the novelty. As mentioned above, the proposed method is straightforward, making people feel this is a standard practice to save computational cost.  On the other hand, I'm not aware of similar results in prior work, so it's good to see such quantitative comparisons.  

About the two sets of experimental results.
For the LMs, the authors showed perplexity results, and also the ablation study. The POS based analysis is good, providing some insights about the retrieved context and why it's helping the model performance. 
However, for the QA task, the analysis is not strong. The results are overall acceptable. But comparing the proposed method to FID and other similar ones, is the performance difference purely because of the encoding during inference (jointly encode the query and retrieved context)?  such understanding is important and very relevant to the proposed method.  
 

questions: Authors provided enough details about computing, which is great.  

Additional suggestions:
Qualitative analysis would also help.
The paper doesn’t provide any examples. Showing retrieved context and explaining the relevance of that is useful.  

limitations: The discussion is mostly focused on the computing aspect, which I think is appropriate given the paper's main focus.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a retrieval-augmented generator/language model, based on an encoder-decoder LM. 
Unlike other recent papers in the literature, the focus here is on a model that:
1) doesn't have a complex, custom architecture (unlike, e.g. RETRO), instead using the well-studied and popular seq2seq encoder-decoder transformer architecture.
2) has cheaper inference because context encoding can be done offline, and cached. 
The proposed model accomplishes this by sacrificing bidirectionality compared to the quite-closely related FID architecture. 
The result is a model that will have much cheaper and faster test-time runtimes than models that must process context on-the-fly. 
The model is trained by using the encoder part of the model for contexts only. The input is prefxed to the output in the decoder. The authors propose an interesting context-retrieval training setup, that I do not recall seeing this precise formulation before.
The model also compares favourably to RETRO both in term of language modelling and Open domain QA.
The authors go the extra mile by providing a series of interesting and thoughtful analyses. 


strengths_and_weaknesses: Strengths:

I really like this paper.

* the paper is well-written and easy to follow, with sensible notation and equations where needed.
* Identifying the cost of computation of retrieved items as an important problem is a good insight, and the proposed solution is elegant.
* the depth of the analysis is really excellent- I enjoyed reading these sections and learnt a lot. 
* the simplicity and elegance of the architecture, compared to e.g. RETRO is inspiring
* I am intrigued by paragraph 220-227, I find this an interesting and simple fix to get the context conditioning to work.
* Improved downstream results compared to RETRO on a "practical" NLP application (NQ), 


Weaknesses:
* the figures are quite complicated and bit hard to follow (esp fig 1).
* Not sure I fully buy the argument about the encoder not needing to contribute to the parameter count in - the retriever parameters should also count in my view (by the way)
* Add the parameter counts of other models to Table 3b - some of these models that perform better are significantly smaller than both RETRO and your model.
* Not a big weakness, but it is a shame the dense retriever isnt better than bm25, I would expect it to be better.
* Whilst the model is comparatively strong on NQ c.f. RETRO, I am still a bit disappointed by this number for a 3B param model 
* Also,  although the context can be decoupled with your architecture, the encoder is fine-tuned, so it needs a full index refresh over the retrieval corpus for the caching argument to work,  which isn't needed for models like FID (in the limit, this is will be more efficient, but we should be careful to be precise).
* Missing/relevant work to compare to, or at least mention in RW: DensPI/DensePhrases (https://arxiv.org/abs/1906.05807,https://arxiv.org/abs/2012.12624), MARGE (https://arxiv.org/abs/2006.15020), Memorizing Transformers (https://openreview.net/forum?id=TrjbxzRcnf-). In particular, I think the DensePhrases work is quite spiritually related. 
questions: Why mt5 vs other t5s?
Why doesn't initialisation from pretrained models work? - I'd like to see numbers or more detail here.

typos:
L1: context retriever -> a context retriever
L37: metrics -> metrics
L118: extra bracket
limitations: I think the authors do this well in section 6
review_title: null

Meta-review: metareview: The paper contributes a retrieval-augmented LM that offers some nice features compared to earlier work. First, it is based on a seq2seq architecture instead of a custom one (e.g., RETRO), which is a plus as it is simple, well studied, and the overall approach is quite elegant. Second, the model of the paper also offers cheaper inference compared to many prior models, and context encoding in the paper can be precomputed offline. 

While the reviewers agreed on these pros, the main concern during the reviewers’ discussion was in relation to the downstream evaluation. The paper presents evaluation on only one downstream task, which is on the Natural Question (NQ) QA task and results are much worse than FiD. However, as pointed out by the authors, the better performance of FiD isn’t so surprising (as the paper’s model encodes contexts independently of the questions, as this is done offline). By contrast, FID comes with much greater computational costs. Ultimately, we agree with the authors’ argument that the most comparable model is RETRO, and the paper does quite well in comparison. It is nice to see the paper’s model outperforming other strong baselines such as REALM, DPR, and RAG. To mitigate concerns about their model’s relative poor performance relative to FiD, the authors might want to bring up computational efficiency (e.g., concrete running time) earlier in the paper.
confidence: Less certain
Decision: Accept