Decoupled Context Processing for Context Augmented Language Modeling
Authors: Zonglin Li
Ruiqi Guo
Sanjiv Kumar
Venue: NeurIPS 2022 Accept (2022)
Keywords: Retrieval Augmentation
Encoder-Decoder
Language Modeling
Efficiency

Abstract: Language models can be augmented with context retriever to incorporate knowledge from large external databases. By leveraging retrieved context, the neural network does not have to memorize the massive amount of world knowledge within its internal parameters, leading to better parameter efficiency, interpretability and modularity. In this paper we examined a simple yet effective architecture for incorporating external context into language models based on decoupled $\texttt{Encoder-Decoder}$ architecture. We showed that such a simple architecture achieves competitive results on auto-regressive language modeling and open domain question answering tasks. We also analyzed the behavior of the proposed model which performs grounded context transfer. Finally we discussed the computational implications of such retrieval augmented models.

Decoupled Context Processing for Context Augmented Language Modeling
Zonglin Li
Ruiqi Guo Google Research, New York guorgggoogle.com
Sanjiv Kumar Google Research, New York sanjivk0googl com
Google Research, New York Lizonglin@google.coo
Abstract
Language models be augmented with context retriever to incorporate knowl- edge from large external databases. By leveraging retrieved context, the neural net- work does have memorize the massive amount world knowledge within its internal parameters, leading to better parameter efficiency, interpretability and mod- ularity. In this paper we examined simple yet effective architecture for incorporat- ing external context into language models based on decoupled Encoder-Decode architecture. We showed that such simple architecture achieves competitive results on auto-regressive language modeling and open domain question answer- ing tasks. We also analyzed the behavior of the proposed model which performs grounded context transfer. Finally we discussed the computational implications of such retrieval augmented models.
1 Introduction
Transformers have proven to be powerful language models that capture an impressive amount of
world knowledge in its internal parameters and generalize to variety of downstream tasks [39 31
Recently, there has been lot of success in improving language model quality by increasing the number of parameters in transformers, often the order of hundreds of billion [10] 30 38 6).
However, the scaling model size also contributes to the exponential rise of the computation costs, both in terms of the number of accelerators needed and energy consumption (29]
To overcome the exponential increase in the number of parameters, one natural idea to utilize information retrieved from an external source such as massive external database, therefore freeing the neural network from having to memorize world knowledge, To this end, researchers proposed multiple context augmented language model architectures [2]] [5]
26 Such architecture typically has two components: retriever that embeds the input sequence and retrieves relevant context from external source through vector similarity search; neural network that integrates both the input and retrieved external context into the prediction of target sequence, formally:
P x, C Retrieve(x, D): 2 P(y|x, 8')
Here, C {c} is a set of context retrieved from the external database D. 0' is a self-contained language model which predicts target sequence y based solely on the input whereas corresponds to the context augmented language model which incorporates both the input x and the retrieved context C.
One of the challenges for such context augmented language model is the computational cost of context retrieval and incorporation, especially when multiple pieces of context is present or the context sequence long. this paper, we propose a computationally efficient architecture for
Eqqual contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

incorporating context based on vanilla Encoder-Decodee which decouples the encoding of context and the prediction of target sequence. We show that the model with such simple architecture is competitive when compared with customized mechanisms such as Bruunedd-Cros-AArei on language modeling score (as measured by bits-per-byte, BPB), while being more efficient in terms of parameter count and computation cost. Then, we define metrics to measure the utility of the retrieved context and use it to guide the training of the retriever. We further show competitive results on downstream tasks of question answering. and demonstrate that the model takes advantage of the retrieved context without memorizing facts within its internal parameters. Finally, we study the implication of context retrieval in terms of retrieval latency, accuracy and computation cost.
To summarize the main contributions of this article:
Proposed a novel Seroder--eeoder based architecture for incorporating retrieved external context, which decouples context encoding from language model inference.
Demonstrated the competitive results of the proposed model on both the auto-regressive language modeling task and the open domain question answering task.
Analyzed model behavior by understanding how context improves language modeling pre- diction on tokens with different linguistic properties and how the model performs grounded context transfer.
Discussed computational cost and retrieval efficiency in context augmentation.
2
Related Works
Large language models, typically in the form of big neural networks, are trained with huge amount of training data rich in unstructured knowledge. Researchers have found that after model training, the neural networks often end up storing surprisingly large amount of memorized information within its weights 2 7 which are then leveraged as knowledge base. Multiple hypotheses have been developed on how components such as fully-connected layers [13] and attention layers (5) may be responsible for such memorization behavior. While the capability of storing world knowledge is desirable, memorization also contributes to huge model sizes and the lack of explicit control over knowledge base, such performing selection or updates.
An alternative strategy to enable language models to incorporate world knowledge in the form of retrieved context from external sources, instead of having to memorize them. Multiple works have proposed architectures that support external retrieval, usually composed of a context retriever that searches a large external key value store and a method of integrating retrieved information.
There are various ways to construct the key value store. The keys are primarily used for similarity matching, and they can be sparse vectors such as BM25 (34], or dense embeddings extracted from part of the model 21 25 40 or from pretrained embedders [12 28 or embedders trained for specific downstream tasks [5
20 41 35 [1] The values also have various different forms. For example, TOME [1] stores a dense embedding about the contextual information of an entity mention, while Realm [S) RAG [26). FID [1]] Retro 14. MARGE [25). DenSPI 335 and DensePhrases 24] store the raw text as the value. Works such as k-N-LMM 21 and Spalm 44 store one token as a value. Finally the key value store is searched over using vector similarity techniques, typically with some off the-hhh-hhfff nearest neighbor search implementations such as FAISS 19) ScaNN [4).
HNSW (27 or SPTAG (
There are also many different ways to integrate the retrieved context. A popular approach is to concatenate the retrieval results with the original input and jointly process them. It has been adopted by works such as Realm 115), RAG (26). and FiD [8] Other works utilize some forms of cross attention for the context integration, such as the Chunked-Cross-AAtteee
with input conditioning in Retro 4. ANN Attention in Memorizing Transformer 40), Memory Attention in TOME ( and Attenttinnn in MARGE (25). For token level integration, ANN-LM (21) uses simple linear interpolation while Spalm
44 uses learned gate based on the last layer embedding. There are also works that directly utilize the retrieval results without any integration, such as DenSPI [35 and
DensePhrases 24). Most of the works use retrieval as way to augment tasks such as language modeling or question answering, with the exception of MARGE [25] where retrieval is treated as an autoencoder bottleneck for multilingual pretraining, and not strictly necessary for inference. We compare representative previous works and contrast with our proposal in Table

---

Retrieval
Retrieval Encoding
Context Integration
Decoupled Context Encoding
Method
Tasks
Granularity
kNN-LM 21
Token
Last layer
Interpolation
Yes
LM
Spalm 44
Token
Last layer
Gating
Yes
LM
Realm [1]. RAG
Input
Trained
Concat
No
OpenQA
FID 18
Chunked- Cross-Attention
LM,
Retro
Chunk
Frozen
No
OpenQA
Frozen
Encoder-
LM.
Proposed
Chunk
trained
Decoder
Yes
OpenQA
Cross-Attention
Table 1: Architectural differences between previous retrieval augmented model and ours in (i) context
retrieval, (ii) context integration and (iii) targeted applications.
Server
Client
(oetriewal+Context Encoder)
(Context Augmented LM)
Decoder
Vilue: Mo.()
Output
X
Y,
Y2
Ya
Key Value Store
Batriewe
Fewar
Emb.lcl
Respond
Doc Retrieval Embedder Emb,
Corteot
Query: Emb_(4)
X, X2 X2
Y, Y2
H
Decoder Input
Context с
Query Retrieval Embedder Emb
Request
Input
Precompute
Auto-regressive Decoding
Figure 1: Architecture of context augmented language model. We opt use the standard Encoder-Decoder cross attention mechanism for context incorporation which allows us to de- couple context encoding from LM inference. C, x, y serve as context, decoder input and decoder target respectively. In other words, the client sends over input and the server conducts retrieval to find relevant context and returns the encoded representation HEre (c; Ozc)) The encoded representa- tion Hzac is pre-computed offline and returned as "metadata" of the retrieval. Note that training of the Encoder and Decoder joint, while they are decoupled inference time: the client does not need to store parameters or the run inference on the Encoder component.
3 Architecture
We use Encoder-Decodes Transformer architecture [39 to integrate language model input and retrieved context. We denote the context encoder and LM decoder as Enc and Dec respectively. Given an input token sequence x (æ F2, Fn.) the task to predict the next tokens y (y1 3/2 ..., y.).
Without external context, we concatenate x before and the task becomes a traditional auto-regressive language modeling predicts targets y following input (or "promopt") x. In this setting, only the decoder is involved (denoted as "No-retrieval"", To incorporate external context, we use x, y to serve encoder input, decoder input and decoder target respectively, We first use a retriever to identify the context c = (C1, c.,..., Cm. given input x, then fetch the the pre-computed encoder output of the corresponding context tokens HEse (c; fras) as output. In this setting, encoder output Hgne is directly used by the decoder through Encoder-Decoder cross- aoos-atttetionn to influence the final prediction. The decoder does not have know the exact tokens of that are retrieved.
No-retrieval: P(y Meci, 21, I2, Fni (pec)
Retrieval: P(y Uei 1 F2, Zn, (Henc (anee(c) (c)]: Opec)

---

Under this formulation, only decoder parameters fDec are required at inference time. The retriever retrieves indices of the relevant context and looks up their encodings. The context encodings are processed ahead of the time, and are completely decoupled from online operations. This is in contrast to previous works of Realm [5]. Rag (26 or Retro where the interaction between input x and context e. is bi-directional, which necessitates context encoding at inference time. In our model, information flows uni-directionally from с to x and y. and that the encoding of each context is processed independently. On one hand, this is more restrictive than bi-directional interaction; on the other hand, such a design ensures complete decoupling of context processing and the online language model inference. The exact mechanism is detailed in Figure. T
Conceptually the retriever can be an arbitrary blackbox. practice, we use a dual encoder formula- tion 88 [6). which first represents x as query embedding Embg(x) and performs vector similarity search over a database of to find the indices of documents whose document embedding has the highest inner products with the query embedding. We then look up the context encoder outputs that correspond to retrieved indices and return them as the retriever output.
["
arg max Embo (x)) v)); D (Embn(c);ce C} VED
H H [1]] Enc (C-- (gae)
In the case multiple supporting context, k-arg max is used instead of arg max. The encoder outputs of each supporting context then concatenated:
P(y ø1, 22, ................................ nn, Concat HEne [41,, Hane H2,. - HEac [+x]); Obcc):
Where Concat is simply vector concatenation:
Concat ((h, ha.... hn [gI, (2,,g,,..... g2, gm.)...) = [hi, h2, ... hm: g1 82g m. 8m ...
At training time, the encoder and decoder are jointly trained. We first perform offline retrieval to form triplets of (x, y, c). where e retrieved by some predefined retriever. The loss is masked and only defined on the targets y. Because the encoding of each context is independent and there is no interaction between context, the attention matrix of encoder block diagonal and we process them in linear loop over each diagonal block. Thus, the computation cost of both encoder and decoder at each step is linear the number of context.
For online language model inference, only the retriever and the decoder are involved. The retrieval embedding Embn (c) and the encoder output Haue(e) of the context both offline pre-processed and prepared into the retrieval database of such associated key-value pairs. When a new input sequence arrives, the retriever is only responsible for the approximate nearest neighbor search and lookup of the associated value that is the pre-computed encoder output. The decoder then takes in the input sequence and cross attends on the concatenation of pre-computed encoder output to generate the targeted tokens. Thanks to the decoupling, neither the retriever nor the decoder needs to store encoder parameters. Hence, such an approach is more parameter efficient compared to similar works such as Retro 4 by saving the storage and computation budget on encoder, which is helpful in "client-served scenario where the capacity of the "client" can be limited. When accounting the parameter count in comparison with other models, we only need to count the decoder and cross attention parameters. We also followed Retro's 's approach of excluding the embedding matrices from the parameter count.
Auto-regression Language Modeling
We experimented with the same "encoder-decoder" context incorporation mechanism for both auto- regressive language modeling and open domain question answering. The only difference is that regr-segressive language modeling processes input sequences in a sliding window fashion, while question answering task receives the full input sequence (the question) at once.
4.1 Experimental Setup
version 2. the same as Retro. We train the language model and prepare the retrieval database using the train split and evaluate the results using validation split. The language model target sequence is sliding window (chunk)
For auto- regressveee language modeling, we use English C4 32

---

Retrieval Database
LM Inference
Database Split
# Articles
# Entries
C4 Train
364.6M
3382M
C4 Val, unfiltered
0.3645M
3.369M
Val, filtered
2.868M
NQ-open Train
79k
NQ-open Dev
8.8k
q Embg(x)
My
1y
NQ-open Test
3.6k
Wiki Database
21M
D (- Emby(c):h-
Hane (c))
MMgemwww (y:x.h,
(b)
Figure 2: (a) Chunking scheme for language modeling training with C4. For each article in C4, we divide the tokenized text into non-overlapping blocks at most 64 tokens and use them as targets y. We use the preceding tokens of at most 448 in length as x. (b) The number of entries in the retrieval database of C4 auto-regressive LM and Natural Question QA task.
of 64 tokens, with at most n = 448 preceding tokens are used as input sequence. This setup is similar XLNet [3 and Retro [4. The target and input sequences that are smaller than the given window size (64 and 448, respectively) are padded with zeros.
To construct the retrieval database, the same sliding window processing is also used for the context sequences. The database is formed as associated pairs retrieval embedding and encoder output: (Embd(c) Ene [c)] where context
the sliding window of 512 tokens with stride of 64 to be comparable to Retro: chunk size s = 64 and input window size n 448 (smaller than 2048 that of Retro). This also implies that the number of entries in the database larger than the number of articles, but smaller than the number of tokens.
tokens. We choose our
Our training corpus is in the form of triplets (x, y. c). X and are acquired directly by applying sliding window on the train split C4. Then BM25 [4 is used a bootstrapping retriever to mine relevant context e to from the database. The first retrieval results with no more than 8 consecutive token overlap with the target is used as context. Figure 2a illustrate the sliding window construction of database well sequence served as input and targets. Table 2b gives the exact number of entries in the resulting database used as external context.
We use mT5 42 as the backbone architecture for our context augmented Encoder-Decodee and train our models from scratch. Train split used both for training and retrieval, while validation split is used for the fact that text are crawled from web sources, there can be a non-trivial overlap of tokens between the training and validation splits. In such cases, tokens are often "copy-pasted" from retrieved context into targets without changing. Such copying leads to near-zero perplexity on targets and has big effect on final bits- bits-per-byte measurement. Following the discussion on dataset leakage of Retro []] we filtered any example whose targets and context sequences have more than 8 common consecutive tokens (correspond to 12.5% filtering of Retro). We found 14.87% of the validation chunks are removed by the filtering longest common substring.
(Bpb) evaluation. In auto-regressive language model evaluation, due to
Figure 3a reports our auto-regressive language model results with different model sizes under the Byt-per-byte (bpb) values. Bpb tokenizer agnostic and is often used to compare models with different vocabulary. We use mT5 base, large and XL respectively without modification, and the results compare favorably to Retro models with similar sizes that uses customized chunked-cross- attention which couples context encoding and LM decoding. Our experiments demonstrate that context incorporation can be achieved with simple Encoder-Decoder cross attention, with the additional benefit of decoupled encoder processing.
4.2 Retriever Training
The goal of context retrieval to identify the context sequences c* that maximize the improvement of some utility function, such the log- likelloooood improvement target prediction. i.c.
5

---

LM BPB at 12.5% filtering
Retro: With Retrieval
Model Size
Exact Match
172M,
Propased: With Retrieval
Model
Accuracy
bptr 0.98
Realm [13
110M
40.4
5.95
425MM bpb: 0.92
DPR
110M
41.5
9itspen-byls
113M
opo: 0.947
Ours (Large)
409M
44.35
RAG 26
400M
44.5
Retro
7.5B
45.5
8.65
bpb: 0.884
409M,
ppt:
Ours (XL)
1.56B
47.95
FiD 18
770M
51.4
1560M
7500M bob: 0.79
EMDR 37
440M
52.5
FiD Distill 17
770M
54.4
250
1630
#arametters [M]
(b)
Figure 3: Performance of decoupled encoder-decoder auto-regressive language modeling (C4) and question answering tasks (Natural Question). (a) Comparing the proposed method and Retro on 14-en-2.22 2. validation split with 12.5% token overlap filtering. The y-axis measures bits-per-byr (bpb, lower is better), which is the perplexity normalized by token length. The x-axis shows the number of non-embedding parameters in log scale. (b) End-to-end result on Natural Question test split. Following previous works, we measure exact match (EM) accuracy on "short answer type" with most five tokens.
c² arg max yx,,
U y(),y,c) = Σ log P'ylleen x, 0) - loo P ((uluei,
However, is infeasible to evaluate the utility function Ux,, y, for all triplets (x, c). Therefore, we introduce a proxy to approximate U by computing the expectation U conditioned on the token y: and whether it appeared in the input x and context sequence
(()
Σ 1y, E xx C + L U (vile $ (X B:Ey.c
Intuitively, the "context utility" i is a weighted token overlap between context and target y. The weight is higher the token y; did not appear in input but is contained in context, and when the token is "sensitive" to context by showing larger loglikelihood change when the context is present. To train a retriever, we adopted the typical dual encoder formulation with in-batch softmax training similar to that of DPR 220 16]. We use training data bootstrapped from BM25 retrieval, where (1) the valid retrieval with highest context utility Ú is the positive; (2) the other top retrieval from BM25 that has less than 80% of context utility of the highest one is used as hard negative; (3) The rest of the in-batch samples are regarded random negatives. All retrievals subject the filtering criteria of no more than consecutive tokens.
We trained our retriever using T5X retrieval framework 28 based on m mn-5mmal1 with an embedding dimensionality of 128. The model trained for 100,000 iterations on batch size of 8, 8.192 on 64 TPUv3 chips. Unfortunately, using the dense trained retriever does not lead to visible improvement on the final C4 Bpb evaluation, possibly because both the dense and BM25 retriever end up retrieving similar context. We report results using trained dense retriever because of the better retrieval efficiency.
4.3 Ablation and Analysis
To analyze the effect the context augmentation, we show first in Figure 4a that the context augmentation in language model consistently helps across different model sizes in the proposed

---

Bpb under different model sizes
Token accuracy an validation set vs shops. mT5 large
without with retrieval
0.955
5,0
Accurary
2A3
Accuracy jump
2.41
Without context, from scratch
1.2
With context, worm started 100k steps
Base
Large
50030
100001
350000
200283
250000
Steps
(a)
(b)
Figure 4: (a) Bpb evaluation (lower better) comparing models of the same sizes trained from scratch, with and without context retrieval, respectively. (b) Comparing the accuracy history of Large model training with and without retrieval. Notice that the curve "jumps" when training with retrieval but without warm-starting.
Not in context, not input
Not in context. ingut
context, input
25
20
105
10.
MR55
833
P10
V122
VIG
V8N
VBl
V01
CD0
19075
ND
WD1
Figure 5: Percentage of improvement on target log-likelihood when there's context. We show the most frequent 20 POS tags, excluding punctuation. For the data on all the POS tags, see Appendix A.
method. While the absolute improvement on Bpb over "No-retrieval" may seem small when filtered to 12.5% in longest common sequence, note that the improvements were made on a limited set of important tokens. Intuitively, tokens of functional words such as "the", "a", "to" should not benefit much from external information and can be predicted with its local context. Yet functional words make up significant portion perplexity because of their high frequencies in text.
To this end, we consider the difference in log-likelihood of two language models of same size, trained with and without retrieved context. We evaluate the models on validation split of C4 and obtain breakdown using part-of-speech (POS) tagging. We used NLTK [3] to assign each target token to POS tag. Note that the tokenization boundaries NLTK and sentencepiece (22 could different, and the assignment is done by majority voting in that case.
From Figure 5 it is immediately clear that the impact of context retrieval on different types of tokens is not uniform. Noun (NN, NNP, NNS, NNPS), and number (CD) benefited the most from the context, followed by adjective (JJ) and verb (VB, VBZ, VBG, VBN, VBD, VBP). Tokens such as preposition (IN), coordinating conjunction (CC), etc. are less helped by context. We believe is because their prediction usually only depends on immediate local context. We also found that whether the target token appeared in the context sequence or input sequence has a large impact on the improvements, and included this in the breakdown. Intuitively, tokens that are grounded in the retrieved context but did not appear in the input x benefit the most from context retrieval.
4.3.1 Training Details
We trained the Incoder-Decodee LM model for total of 100, 000 steps with a batch size 512 and default learning rate schedule of square-root decay. This corresponds to 10, 000 warmup steps with fixed learning rate 0.01, followed by square-root decay for 990, 000 steps. We found training from scratch and fine-tuning pre-traine rreitraind models result in similar performance C4. Therefore
7

---

we opted to train from scratch. We also found it beneficial to include additional 100, 000 steps with lower learning rate of 10 after standard training schedule, which slightly improves the results. For base and large we used 64 TPUv3 chips whereas 128 TPUv3 chips for training XL. XL runs 1.5 training steps per second. XXL is left out due to the insufficient computation resources.
Comparing the training curve of model with and without retrieval, as depicted in Figure 4b we noticed a phenomenon where both models have roughly the same performance initially but the one with retrieval suddenly increases at some point and continues to improve afterwards. This is especially true in large and XL. We suspect that decoders of larger models have higher memorization capacity, and in the initial phase they can improve training objective without delegating memorization to context retrieval. This is a challenge to training stability and one way address this is by first "warm start" the model with 100, 000 steps of training on subset of training data with highest 10% of the context utility before proceed on the full dataset.
5
Open Domain Question Answering
5.1 Natural Question
Large language models are useful because they generalize downstream tasks, in addition to per- forming auto-regressive language generation. To demonstrate that the proposed context augmentation scheme effective on downstream applications we evaluated our proposed model of decoupled Encoder-Decoder on the OpenQA task of Natural Question 223
We use the same question and context processed by [8] where the context is re- trieved with DPR retriever [20]. We construct each context sequence in the format of "title: {title) source: {source}" and pad trim it to 256 sentencepiece tokens. The context sequences are then encoded by the encoder independently and simply concatenated, to ensure there is no interaction between them.
The decoder is trained to receive and respond on the sequence containing both the question and answer in format "question: {question} An answer: {answer]" similar to [4. where "question : {question) An answer: " is the input prompt (and there no loss defined on these tokens) and [answer} is the target sequence the decoder is expected to predict. Again, the decoder cross- cross-attends attends to the concatenated encoder output Hgne (c) from all the retrieved context and the decoder is completely decoupled from encoder. It does not even have to know the exact tokens of context sequence being returned by the retriever.
The model weights are initialized from 15.1.1 checkpoints 332 We jointly fine-tuned the encoder and decoder for 40, 000 steps with 20 context passages for each input of the train split, and validated on the dev split every 1,000 steps. We used batch size of 64, fixed learning rate of 10-4 and Adafactor optimizer (36). Finally, we selected the checkpoint with the best validation accuracy on the dev split and evaluated on the test split. Each model is trained on 64 TPUv3 chips, and the evaluation metrics is string exact match with SQuAD (33 normalization. We report our final results in Table 3b Overall, the decoupled Encoder -ecoder-Decoder produced competitive results on Natural Question task, albeit being a much simpler architecture and not specifically designed for QA tasks. We note that FiD [18 uses similar architecture. However, FiD concatenates the question with the retrieved context passages. Therefore the context encoding is dependent on the input sequence and needs to be computed at inference time.
5.2 Grounded Answer Generation
Context augmented models tend to generate answers by transferring tokens from context to the output. is interesting to quantify how often the model output comes from grounded transfer of context tokens and how often this leads to the correct answer. We use the same model trained in to (a) run language model decoding with the original set of retrieved context to obtain "original output"" y" = Decode(x, C; (); and (b) rerun model prediction but remove all context passages that contain the "original output": y = Dxcode(x, C' = C {c:y" S c)::8 8). In the case of grounded context transfer, we expect the "secondshot output" change, but still grounded in the remaining context.
We show grounded-ness analysis in Table 2 Overall, we found 76% (2,765 out of 3,354) of the cases are likely due to grounded context transfer, because the "sccondshot output" has changed to
8

---

Original output y" Decode (x. C,6)
Original ourput context
Original output out context
3354
256
Ouiput changed still context y' # y",y" .y' C' y' yy.. y ş C'
Outpui chunged out of context
Secondshot outpat y Decode(x, C C Clley' cc]
Output unchunged
Excluded
2765
261
291
37
Table 2: Analysis of grounded context transfer on Natural Question. First row shows the number of questions broken down by whether the original model output appeared in the retrieved passages. To demonstrate the model follows grounded context transfer, we removed context passages containing original output and reran inference the remaining context passages. 76% of output changed due to the removal of "original output" but the new prediction is still contained in remaining context.
something different from "original output", but still grounded in the remaining context passages. In addition, found that the accuracy is 51.5% when model output is from grounded transfer, higher than 35.4% for the rest. This implies that the model has higher accuracy when the output is grounded. We exclude 37 examples from the analysis because they have less than 20 neighbors after removing original output.
Computational Discussion
We evaluate the computation implications of our method around inference latency and space-time trade off. We estimate the latency based on benchmarks and publicly available packages and performance the 1.56B XL model in Table 33b] whose encoder
data. In the Natural Question experiment, output have dimension of 2048 and the activation data type is bfloat16 []. When profiled on a single TPUv3 core, takes 200ms encode all 20 retrieved context passages each containing 256 tokens (with paddings). This is equivalent to 10ms per context. Thanks to the decoupling of the context encoding, the context passages encoding is done offline, and simply looked up inference time. Inferencing the DPR query embedder model takes around 8ms unbatched on one TPUv2 chip. We use ScaNN [14] perform approximate similarity search on the DPR embeddings with dimensionality of 768, which takes roughly 12ms to retrieve 20 neighbors with recall @ 20=0.97 on single CPU core. For each passage, the uncompressed encoded embeddings have a total size of I MiB (s (sizeof (bfloat16) * 2048 256). When reading the encoding from an NVMe SSD with a read throughput of 660 MB/s, the lookup takes around 1.5ms and we use another 0.8ms to transmit of one encoding over network. End to end, it takes around 66ms ((1.5 51.5- 0.8) 20 + 12 8) to retrieve encoded passages. This is about third of time for running inference of the encoder, and everything except query embedder is conducted with generic hardware without accelerators which is cheaper and consumes less energy.
However, one limitation of this approach is the amount of disk space and network bandwidth needed store and transmit all the pre-computed encodings, which scales with the dimensionality encoder output. We leave it for future research to reduce the encoder output size while maintaining similar quality, which may be possible through compression, dimensionality reduction, or adding projection layer to the encoder.
7 Conclusion
In this paper, we advocate the design of context augmented language model based on Encoder-Decodee architecture. Encoder-Decodee models are simple, proven and enjoy the unique computational advantage that the context encoding and language generation are decoupled. That is, the context encoding can be produced with offline pre-computation and caching, while the decoder parameter efficient because it is agnostic to the encoder computation. We further demonstrated the effectiveness of this simple architecture by comparing with competitive baselines on common NLP tasks such as language modeling on C4 and question answering on Natural Questions. We also analyzed the model behavior and showed the context augmentation resulted in big improvements on content words and the model generates output grounded in retrieved context most of the time. Finally, estimated the latency improvement from caching enabled by the decoupled computation.

---

References
[1] leee standard for floating- point arithmetic. IEEE Std 754-2019 (Revision of IEEE 754-2008), pages 1-84, 2019.
[2] Devansh Arpit, Stanistaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization deep networks. In International conference on machine learning,
pages 233-242. PMLR, 2017.
[3] Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit. "O'Reilly Media, Inc.", 2009.
[4] Sehastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, al. Improving language models by retrieving from trillions of tokens. arXiv preprint arXiv:2112.04426 2021.
[5] Trenton Bricken and Cengiz Pehlevan. Attention approximates sparse distributed memory. Advances in Neural Information Processing Systems, 34, 2021.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877 2020.
[7] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633-2650, 2021.
[8] Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedde, Steve Yuan, Chris Tar, et al. Universal sentence encoder. arXiv preprint arXiv: 183. 2018.
[9] Qi Chen, Haidong Wang. Mingqin Li, Gang Ren, Scarlett Li, Jeffery Zhu, Jason Li, Chuanjie Liu, Lintao Zhang, and Jingdong Wang. SPTAG: A library for fast approximate nearest neighbor search, 2018.
[10] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.022. 02211111..022.2.22210 2022.
[11] Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Fei Sha, and William Cohen. Mention memory: incorporating textual knowledge into transformers through entity mention attention. arXiv preprint arXiv:2110.06172 2021.
[12] Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 2018.
[13] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012. 14913, 2020.
[14] Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. Accelerating large-scale inference with anisotropic vector quantization. In International Conference on Machine Learning, pages 3887-3896. PMLR, 2020.
[15] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wa Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020.
[16] Matthew Henderson, Rami AI-Rfou, Brian Strope, Yun-Hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. Efficient natural language response suggestion for smart reply. arXiv preprint arXiv:1705.00652. 2017.
10

---

[17] Gautier Izacard and Edouard Grave. Distilling knowledge from reader to retriever for question answering. arXiv preprint arXiv:2012.04584, 2020.
[18] Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering, arXiv preprint arXiv:2007.01282, 2020.
[19] Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 2019.
[20] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020.
[21] Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. General- ization through memorization: Nearest neighbor language models. ICLR 2020, 2019.
[22] Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv: 062266 2018.
[23] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Was Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.
[24] Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, and Danqi Chen. Learning dense representations of phrases at scale. arXiv preprint arXiv:2012. 12624, 2020.
[25] Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. Pre-training via paraphrasing. Advances in Neural Information Processing Systems, 33: 33:18470-18881, 2020.
[26] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Rerrieval-augmented generation for knowledge-inenniiee nlp tasks. Advances in Neural Information Processing Systems, 33:9459-9474, 2020.
[27] Yu A Malkov and Dmitry A Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence, 42(4))824-836, 2018.
[28] Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B Hall, Daniel Cer, and Yinfei Yang. Sentence-15: Scalable sentence encoders from pre-trained text-to-text models. arXiv preprini arXiv:2108.0880 2021.
[29] David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprini arXiv:2104. 10350, 2021.
[30] Jack Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:21 11446, 2021.
[31] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv preprint arXiv:1910.10683 2019.
[32] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine I Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020.
11

---

[33] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension text. arXiv preprint (ab.::600005500 2016.
[34] Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Now Publishers Inc, 2009.
[35] Minjoon Seo, Jinhyuk Lee, Tom Kwiatkowski, Ankur P Parikh, Ali Farhadi, and Hannaneh Hajishirzi. Real-time open-domain question answering with dense-sparse phrase index. arXiv preprint 1906:v990600580000 2019.
[36] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learming, pages 4596-4604, PMLR, 2018.
[37] Devendra Singh, Siva Reddy, Will Hamilton, Chris Dyer, and Dani Yogatama. End-to-end training of multi-document reader and retriever for open-domain question answering. Advances in Neural Information Processing Systems, 34, 2021.
[38] Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye. George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron train megatron-ture nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.1199 11990, 2022.
[39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[40] Yuhuai Wu, Markus N Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing trans- formers. arXiv preprint arXiv:2203. 0.8913, 2022.
[41] Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations (ICLR), April 2021.
[42] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 483-498, Online, June 2021. Association for Computational Linguistics.
[43] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
[44] Dani Yogatama, Cyprien de Masson d' Autume, and Lingpeng Kong. Adaptive semiparametric language models. Transactions of the Association for Computational Linguistics, 9:362-373, 2021.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes] We mentioned the limitations in
Section 6
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [No] We still need to clean up the code before it's ready.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4 and Section
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [No] It would take too long and too much resources to run each experiment multiple times.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] We reported the type of device used to train the model as well as the steps per second for mT5 XL model. For more information on time, see Appendix.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] Will be mentioned in Appendix.
(c) Did you include any assets either in the supplemental material
URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "iqTPOOg_Izl", "reviewer": "Reviewer_DW3N", "summary": "This paper presents a method to augment the context based on retrieved relevant information, and keep the additional context representation separated from the history/prompt in an encoder-decoder setup, and thus decoder can be separately trained, and there’s latency benefit since the context and query don’t need to be re-encoded jointly after context is retrieved in inference time.  Authors evaluated their methods on language modeling and open domain question answering tasks and showed the computational benefit of the proposed method. \n", "questions": "Authors provided enough details about computing, which is great.  \n\nAdditional suggestions:\nQualitative analysis would also help.\nThe paper doesn’t provide any examples. Showing retrieved context and explaining the relevance of that is useful.  \n", "limitations": "The discussion is mostly focused on the computing aspect, which I think is appropriate given the paper's main focus. \n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed method is intuitive. I feel people have done such evaluations but couldn't find an exact paper with such results. \nThe paper presents experimental results showing the proposed method has good performance (comparable to other approaches while reducing computational cost). \n\nAuthors did a good job and provide enough details to show the advantage of their method, mainly in terms of the computational efficiency. \n\nOne weakness of the paper is the novelty. As mentioned above, the proposed method is straightforward, making people feel this is a standard practice to save computational cost.  On the other hand, I'm not aware of similar results in prior work, so it's good to see such quantitative comparisons.  \n\nAbout the two sets of experimental results.\nFor the LMs, the authors showed perplexity results, and also the ablation study. The POS based analysis is good, providing some insights about the retrieved context and why it's helping the model performance. \nHowever, for the QA task, the analysis is not strong. The results are overall acceptable. But comparing the proposed method to FID and other similar ones, is the performance difference purely because of the encoding during inference (jointly encode the query and retrieved context)?  such understanding is important and very relevant to the proposed method.  \n \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "g_gfrtVPYCb", "reviewer": "Reviewer_PRTw", "summary": "This paper proposes a retrieval-augmented generator/language model, based on an encoder-decoder LM. \nUnlike other recent papers in the literature, the focus here is on a model that:\n1) doesn't have a complex, custom architecture (unlike, e.g. RETRO), instead using the well-studied and popular seq2seq encoder-decoder transformer architecture.\n2) has cheaper inference because context encoding can be done offline, and cached. \nThe proposed model accomplishes this by sacrificing bidirectionality compared to the quite-closely related FID architecture. \nThe result is a model that will have much cheaper and faster test-time runtimes than models that must process context on-the-fly. \nThe model is trained by using the encoder part of the model for contexts only. The input is prefxed to the output in the decoder. The authors propose an interesting context-retrieval training setup, that I do not recall seeing this precise formulation before.\nThe model also compares favourably to RETRO both in term of language modelling and Open domain QA.\nThe authors go the extra mile by providing a series of interesting and thoughtful analyses. \n\n", "questions": "Why mt5 vs other t5s?\nWhy doesn't initialisation from pretrained models work? - I'd like to see numbers or more detail here.\n\ntypos:\nL1: context retriever -> a context retriever\nL37: metrics -> metrics\nL118: extra bracket", "limitations": "I think the authors do this well in section 6", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nI really like this paper.\n\n* the paper is well-written and easy to follow, with sensible notation and equations where needed.\n* Identifying the cost of computation of retrieved items as an important problem is a good insight, and the proposed solution is elegant.\n* the depth of the analysis is really excellent- I enjoyed reading these sections and learnt a lot. \n* the simplicity and elegance of the architecture, compared to e.g. RETRO is inspiring\n* I am intrigued by paragraph 220-227, I find this an interesting and simple fix to get the context conditioning to work.\n* Improved downstream results compared to RETRO on a \"practical\" NLP application (NQ), \n\n\nWeaknesses:\n* the figures are quite complicated and bit hard to follow (esp fig 1).\n* Not sure I fully buy the argument about the encoder not needing to contribute to the parameter count in - the retriever parameters should also count in my view (by the way)\n* Add the parameter counts of other models to Table 3b - some of these models that perform better are significantly smaller than both RETRO and your model.\n* Not a big weakness, but it is a shame the dense retriever isnt better than bm25, I would expect it to be better.\n* Whilst the model is comparatively strong on NQ c.f. RETRO, I am still a bit disappointed by this number for a 3B param model \n* Also,  although the context can be decoupled with your architecture, the encoder is fine-tuned, so it needs a full index refresh over the retrieval corpus for the caching argument to work,  which isn't needed for models like FID (in the limit, this is will be more efficient, but we should be careful to be precise).\n* Missing/relevant work to compare to, or at least mention in RW: DensPI/DensePhrases (https://arxiv.org/abs/1906.05807,https://arxiv.org/abs/2012.12624), MARGE (https://arxiv.org/abs/2006.15020), Memorizing Transformers (https://openreview.net/forum?id=TrjbxzRcnf-). In particular, I think the DensePhrases work is quite spiritually related. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SHuiAlEO1K", "reviewer": "Reviewer_aAnM", "summary": "This paper explores how to leverage retrieved context for language modeling by decoupled encoder-decoder models, like Retro (Retrieval-Enhanced Transformer). The model mainly includes two components: 1) a retrieval that retrieves the k most relevant texts of input x via vector-based search; then 2) a transformer-based encoder-decoder model is used to predict the target based on both input x and the retrieved context. Compared with Retro, this integration of input and retrieved context is at decoder instead of encoder. The model is evaluated on auto-regressive language modeling on C4 datasets and Natural Question (a question asking task).", "questions": "How the bootstrapped retriever training affects the NQ results? Do you have any ablation? \n\n", "limitations": "This type of training should help more on knowledge intensive or grounded tasks, and it would be great to see more evaluation on them. \n\nThe model may work on zero-/few shot transfer settings, and it would be great to have such experiments as well.\n\nIt is hard to understand the point of Figure 5, and how it is related to main story. \n\n\n", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\nThe proposed approach is simple, and it seems more parameter efficient than Retro.\n\nIt provides extensive experiments especially on different scales.\n\n\nWeaknesses:\n\n1)Many experiments results are about bits-per-byte on C4 dev. I’m wondering how this metric correlated to end-task performance. \n\n2)On the only one real-world application, NQ, the performance is still far behind FiD which is very similar to the proposed approach in terms of model architectures. Nothing that FiD only has 770m parameters\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept