BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models
Authors: Qingqing Cao
Sewon Min
Yizhong Wang
Hannaneh Hajishirzi
Venue: ICLR 2024 spotlight (2024)
Keywords: language models
question answering
binary representations
retrieval-augmented language models
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We create cacheable binary token presentations for the passages in the reader of retrieval-augmented language models to improve inference efficiency

Abstract: Retrieval augmentation addresses many critical problems in large language models such as hallucination, staleness, and privacy leaks.
However, running retrieval-augmented language models (LMs) is slow and difficult to scale due to processing large amounts of retrieved text. 
We introduce binary token representations (BTR), which use 1-bit vectors to precompute every token in passages, significantly reducing computation during inference. 
Despite the potential loss of accuracy, our new calibration techniques and training objectives restore performance. Combined with offline and runtime compression, this only requires 127GB of disk space for encoding 3 billion tokens in Wikipedia.
Our experiments show that on five knowledge-intensive NLP tasks, BTR accelerates state-of-the-art inference by up to 4x and reduces storage by over 100x while maintaining over 95% task performance. Our code is publicly available at https://github.com/csarron/BTR.

Published as a conference paper at ICLR 2024
BTR: BINARY TOKEN REPRESENTATIONS FOR EFFI- CIENT RETRIEVAL-AUGMENTT
LANGUAGE MODELS
Qingging Cao, Sewon Min, Yizhong Wang. Hannaneh Hajishirzi Paul G. Allen School of Computer Science & Engineering
University of Washington aqicon,,eeon, {qicao, sewon, y y vizhongw, hannaneh)@cs hawhaaeheecs.wwssniggtooo wwshington. edu
ABSTRACT
Retrieval augmentation addresses many critical problems in large language models such as hallucination, staleness, and privacy leaks. However, running retrieval- augmented language models (LMs) is slow and difficult to scale due to processing large amounts of retrieved text. We introduce binary token representations (BTR), which use 1-bit vectors to precompute every token in passages, significantly ducing computation during inference. Despite the potential loss of accuracy, our new calibration techniques and training objectives restore performance. Com- bined with offline and runtime compression, this only requires 127GB of disk space for encoding billion tokens in Wikipedia. Our experiments show that on five -idteeeetnensive NLP tasks, BTR accelerates saute-of-the-al retrieval- augmented language model inference by up to and reduces storage by over 100x while maintaining over 95% task performance.
INTRODUCTION
Large language models (LLMs) (Brown al., 2020b; Touvron et al., 2023a), despite their widespread success, still suffer from issues such as hallucinations (Mallen et al., 2023; Mündler et al., 2023), staleness, and privacy leaks (Huang et al., 2022). Retrieval-augmented language models (e.g., (Lewis et al., 2020; Izacard et al., 2022b)) alleviate such problems via retrieve-and-reat approach (Figure 1), using a retrieval component find passages relevant the input query followed by a reader model (e.g., an LLM) that generates output given the query concatenated with retrieved passages. However, this approach slow (e.g.. only handles few queries for a powerful server GPU) at inference time mainly due to the reader model that needs to compute cross-attention between the input query and large number of passages. Existing solutions (Lee et al., 2021; Cao et al., 2020) improve the inference computation by decomposing query and passage encoding and precomputing passage representations, but they come with large storage costs (terabytes of storage; $4).
In this work, we improve the inference speed of the reader model in small storage footprint by introducing cacheable and calibrated Binary Token Representations (BTR). BTR (Figure 1) precomputes token representations for the retrieved passages in the reader model. The binary representations are 1-bit vectors for the tokens in a passage and are created from the hidden states in the reader encoder layers via calibrated binarization which are effective for downstream tasks such as question answering. BTR reduces the storage footprint and improves the runtime speed since the representations are 1-bit vectors, and the reader uses the cached representations. To avoid degradation of task accuracy caused by binary representations, we introduce two training objectives by adding (i) passage representation recovery objective that makes the binary representations preserve the passage semantics before the binarization; and (ii) a query-aware passage token distillation objective that compensates the information loss due to precompuation of passage representations independent of the query.
LM with
Furthermore, we observe significant redundancies in precomputed token representations in retrieved passages since they are relevant to the query and contain similar information. Removing such redun- dancies only causes minimal task accuracy loss (<0.5% in our experiments), but shows significant
Our code is publicly available at https: http:/gg.btt.ttt.///eeeeettb. http./ccc//carro////11

---

Published as a conference paper at ICLR 2024
Reader
Runtime
compression
Passagel
Query
Encoder
Query
Retriever
Passage2
Query
Encoder
Decoder
Passage3
Query
Encoder
Corpus Index
Decomposition and Calibrated Binarization
Cacheable Binary
Offline
Taken Representations
compression
Figure 1: Retrieval-augmenne models use retrieve-and-read pipeline. The reader can be either an encoder or an encoder-decoder model. BTR creates cacheable binary representations for the passages via decomposition and calibrated binarization to speed up reader inference. BTR further reduces storage by offline compression and improves inference speed by runtime compression.
benefits in storage and runtime inference. To address this, we further develop token compression over BTR by merging similar precomputed token vectors after training and merging similar concatenated query-passage representations during inference.
We evaluate BTR five knowledge-rich NLP tasks, including three open-domain question- answering tasks (NaturalQuestions (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), and WebQA (Berant al., 2013)), the FEVER (Thorne et al., 2018) fact-checking task, and the et al., 2020) reasoning benchmark. Compared to baseline systems, BTR reduces disk storage by up 101x and improves inference speed by 2-4x for the reader of two state-of-the-art retrieval-augmennedd language models. BTR also retains 90-95% of the original models' performance. Our analysis experiments show that binary token representations are effective and contribute most to improving the inference speed and reducing the storage costs of readers. The training regulariza- tion objectives help mitigate the task accuracy loss and the offline and runtime token compression techniques further bring down storage footprint and increase inference efficiency.
MMLU (Hendrycks
2 BACKGROUND AND RELATED WORK
We first describe the architecture of techniques designed for them. Next, summarize relevant methods improve model efficiency, including quantization and binary representations.
language models and then related efficiency
Retrieval-Augmenned Language Models. Retrieval- -atrevel- language models have shown strong performance in many tasks, including language modeling (Borgeaud et al., 2022; Min et al., 2023), open-domain question answering, and fact checking (Izacard et al., 2022b; Lewis et al., 2020; Guu et al., 2020; Shi et al., 2023).
As shown Figure 1, a retrieval- -ugmeettd language model works by first using a retriever to retrieve many passages relevant to the imput query and then using reader component extract generate the answers. While the retriever is often fast enough, the reader causes a speed bottleneck in retrieval augmentation because it requires computing cross-attention between the input query and many passages. For example, Figure illustrates a state-of-the--e encoder-decoder reader (Izacard et al., 2022b) architecture called Fusion-in- Decoder (FiD) (Izacard & Grave, 2021). FiD first concatenates each passage with the query and processes them in parallel (independently) in the encoder; then the decoder fuses information across all the concatenated passage-query representations and produces answers. In our experiments, we observe that the passage encoding takes over 60% of the reader's 's computation on commodity GPUs, and we save such computations by precomputing the passage representations, leading to significant speedup in the inference with reduced storage costs.
Efficient methods for reader models. DensePhrase (Lee et al., 2021) builds contextualized phrase representations for the passage corpus, completely removes the reader component, and uses phrase retrieval to produce answers for the query. Despite its high inference throughput, the accuracy is much lower than similar size BTR models. FiD-light (Hofstätter et al., 2022) and FiDO (de Jong et al., 2022) focus on improving the inference latency for FiD models on customized hardware like TPUs, they compress passage representations into fixed-size vectors reduce the decoder computation.
2

---

Published as a conference paper at ICLR 2024
But more popular hardware like GPUs, the passage encoder is the computation bottleneck, and their decoder-oriented optimizations are likely to be less effective in improving inference speed. LUMEN (de Jong et al., 2023) and DeFormer (Cao et al., 2020) precompute cacheable continuous passage representations to speed up reader inference, but they take up 100x more storage than BTR.
Model quantization and binarization. Another line of research focuses on model quantiza- tion (Dettmers et al., 2022; Xiao et al., 2023; Yuan et al., 2023; Frantar et al., 2022) or binariza- tion (Bai et al., 2021; Qin et al., 2022) for resource-constrineed settings. improves model efficiency by compressing model weights to lower precision to reduce runtime memory usage. However, quantization does not necessarily improve the inference speed of readers as we will empirically show in $4. This is because quantized models still need to process large amounts of input passage data for readers, which requires huge computation. Meanwhile, quantization methods are orthogonal to our BTR and can help deploy BTR for platforms with low resources.
Binary representations. Previous work has used binary representations for text retrieval (Yamada et al., 2021), semantic matching (Tissier et al., 2019; Shen et al., 2019), or image retrieval (Jain et al., 2017; Huang et al., 2019; Liu et al., 2016). Motivated by this line of work, we build binary token-level passage representations to improve reader efficiency which was unexplored before. The key difference is that existing passage-level (or image-level) binary representations provide coarse- grained semantic signals that work for retrieval or semantic matching. But token-level representations require storing more ine-grained information, which we effectively address by introducing new calibration techniques and training objectives.
3 METHOD
We present BTR, a fast and storage-efficient reader for retrieval-augmente language models. BTR creates cacheable calibrated binary representations for the retrieved passages to speed up inference and avoid large storage overhead. We describe BTR for encoder-decoder readers which are most widely used in the state-of-the the--rt models (Izacard et al., 2022b; Izacard Grave, 2021), and later show how to apply BTR for the encoder-only model ($4). The encoder and encoder-decoder readers both need to process many passages, which the computation bottleneck. Previous solutions (Cao et al., 2020; Lee et al., 2021) precompute the passage representations to avoid runtime computation but come with large amounts of storage costs (c.g. terabytes of storage for Wikipedia scale corpus). BTR tackles the challenge by building compact binary token representations for the passages.
Overview. Figure 2 summarizes the BTR reader architecture and how our techniques are applied to each component. The key idea in BTR is create cacheable binary token representations ($3.1) 1) of the retrieved passages such that the passage encoding can be precomputed offline and stored in compact format. We build on the previous technique (Cao et al., 2020) to decompose the passage and query computation in the lower layers of the reader encoder and jointly process the query and passage representations back in the upper layers. However, unlike them, we create calibrated binary passage representations after the decomposition to drastically reduce storage. We further develop offfine compression to reduce the storage of these precomputed binary token representations. Such binarization and decomposition incur task performance degradation, which we address by designing two regularization techniques during training ($3.2). For inference, we develop runtime compression techniques ($3.3) to further speed up the reader computation.
3.1 BINARY TOKEN REPRESENTATIONS FOR RETRIEVED PASSAGES
Our goal is to produce binary representations for the retrieved passages while maintaining downstream task accuracy. We binarize passage representations at the token level. Specifically, given a continuous passage token vector h4 [11 h2. ..., ha] (d is dimension) at the kth layer of the reader encoder, we hash he via the sign function to obtain the binary representation vector b = sign (h₂), where b, is 1 if h, > 0 and -1 otherwise. We use differentiable tanh function approximate the non-differentiable sign function (Cao et al., 2017).
Prior work (Yamada al., 2021) binarized passage representations for retrieval tasks through an annealing process, i.e., b tanh(3h), where 3 is scaling factor that anneals infinity to
3

---

Published as a conference paper at ICLR 2024
Output
Encoder
Decoder
Transformer
Transformer block
Layer
Runtime compression
Transformer encoder block
compression
Transformer
Transformer block
Add
Runtime compression
LayerNorm
Layer k+1
block
block
lookoup
Add
Runtime compression
01000110000001
Attention
Binarization
Transformer block
Binarization via hashing
Layer k
Precomputed offline
Transformer
LayerNorm
block
hg
Calibrated binarization
Offiine
valuua store of binary vectors and variance
Query
Passages
compression
Figure 2: BTR reader architecture, where light blue color indicates the model from prior work (based on T5 (Raffel er al., 2020) and FiD (Izacard Grave, 2021), and green indicates our methods. We create cacheable calibrated binary token representations for retrieved passages in the reader encoder to speed up inference. Additionally, we compress the precomputed binary token presentations offline to reduce storage costs. We further reduce inference computation by runtime compression for the encoder and decoder.
approximate the sign function. However, we find that directly converting the continuous float values into discrete binary values with annealing decreases the representation quality. The problem is that the float vector values often have large scales (c.g., ranging from -500 to +300) which causes the tanh function collapse into 11 and +1 quickly, this is partially due to the pre-layernorm (Xiong al., 2020) nature of the encoder-decoder architectures like T5. Most recent models like GPT-3 (Brown al., 2020a) and LLaMA (Touvron al., 2023b;a) also pre-layernorm. A naive fix introduce a normalization layer (e.g. layernorm) before the binarization, but then the scale of normalized values does not match the input passage presentations in the upper encoder layers and we empirically notice a larger drop in performance.
(c.g..ranging
Calibrated binarization. We introduce a calibration technique that makes two modifications to the pre-layernorm architecture. First, instead of applying the binarization after the Transformer encoder layer as in the previous work (Shen et al., 2019; Yamada et al., 2021), we insert the binarization after the layernorm layer (but before the multihead attention layer) inside the Transformer encoder layer as shown in Figure 2. We also save the variance of all passage token representations which comes with negligible storage costs?. Using the saved variance information along with the layernorm weights, we could recover the passage representations to their original scales (by dividing the product of variance and layernorm weights). Second, during training, instead of annealing the tanh function, we adopt a straight-through estimatior (Bengio et al., 2013) for the binarization, the forward pass of encoding still uses discrete binary values while the backward propagation uses the tanh function to provide the gradients. These two key changes make the binarization procedure better calibrated and help retain the representation quality after binarization.
Offline compression. The binary representations for the passage tokens are amenable to further compression due context redundancy. This redundancy originates from the passage corpus, where each token can occur millions of times in different contexts even though many of them have similar semantic information. This is especially the case for tokens that form stopwords. We design offline token compression that aims to reduce such redundancy such that we do not need to store its representations every time it appears in different context (in case, the context is retrieved
Not necessarily needed for models with post-layernorm such as BERT.
It requires 1/d storage of the passage vector. d usually several hundred thousand like 768 1024.
4

---

Published as a conference paper at ICLR 2024
passage). For a given token t in the reader model's vocabulary, find all binary representations from the corpus. If the token is a stopword t.. then we compute the mean vector values b, for all the tokens in T. Otherwise, we merge 7 of the binary token representations b, for the token by based on their semantic similarity. We use a bipartite merging (Bolya et al., 2022) algorithm and measure the semantic similarity using the Hamming distance. We find this more efficient than clustering method as they require more iterations and do not easily scale to billions of tokens. Details are provided in Algorithm I in Appendix A.I. Once we compress the binary token representations, we save them in key-value data store for fast runtime lookup.
3.2 BTR TRAINING
Binarization and decomposition provide efficiency benefits but come with performance degradation for downstream tasks. We describe how train BTR in three steps to mitigate accuracy drops.
Step 1.
We first train a reader model without any decomposition (precomputation) or binariza- tion. The training objective is the task loss Lusk using cross entropy between prediction logits and task labels.
Step 2.
We then train decomposed (no binarization) model with the reader model from step 1 as the teacher using query-aware passage token distillation loss Laissil and the original task loss Last (described below).
Step 3. Finally, we train a decomposed reader model by applying BTR binarization in the reader encoder, where the model weights are initialized from the distilled decomposed reader model in the second step. We add a passage representation recovery loss Crecovery to the original task loss Lusk (described below).
Query-Aware passage token distillation. Decomposing passages encoding from the query in the lower layers of the reader encoder often causes task performance degradation. Previous work (Cao et al., 2020) distilled upper-layer passage representations the decomposed reader encoder from the original (not decomposed) model to reduce information loss due to decomposition. We empirically find that the direct application of this distillation does not improve the task performance much for a retrieval-augmen -aggmentedd LM. This is likely because DeFormer-style distillation applies to single passage scenarios where all passage tokens are important to the task, whereas in our setting the reader retrieves many passages that may contain more redundant passage token information. Distilling all token representations using Cao et al. (2020) from all retrieved passages distracts the reader encoder and makes it fail to pass useful passage token information for the task.
To address this issue, we design a query-aware passage token distillation loss to improve the decom- posed passage representation quality. We only distill the passage tokens that are semantically relevant (salient) to the query from the original (non- -onddecompo)d) model. This is because query-relevant passage tokens carry more fine-gradd ane-grained query-related information from the original (not decomposed) passage representations. We obtain top-r' passage tokens using the query to passage attention scores. decomposed is the decomposed passage representations. Laisun Ei= (h.- h; d,dcompoacd )2 where h,
The second step training loss is: Lass + Ldistll-
Passage representation recovery. The end-to-end training objectives do not explicitly encourage the binary tokens representations to maintain semantic information the continuous representations before binarization. We add representation recovery loss (Shen et al., 2019) to the second-stage training task objective. This objective directly provides supervision signals for the binary repre- sentations to retain the semantics of continuous representations. Specifically, we first add linear projection layer (parameterized by W proj and biaspraj) on top of the binary passage representations by: then we use the mean square error as the recovery loss between the projected representations and original passage representations hy before the binarization (kth layer in the reader encoder):
Lrecovery Ei 1!rri)a y², where boros = W proj b biasproj- As we will show in $4.4, passage recovery loss helps improve the quality of binary representations and reduce the accuracy gap. The third step training loss is Lusk + Lrecovery.
$r,% is 100% for stopword token representation since stopwords are less contextual than other tokens.
'We set 50% of total passage tokens by default.
5

---

Published as a conference paper at ICLR 2024
3.3 BTR INFERENCE
During inference, given an input query and retrieved passages, BTR reader first computes query representations (continuous float-point vectors) the fly in the encoder (layer 1 to k), then it looks up the precomputed binary token representations (from layer k) for the retrieved passages from the data store. Next, BTR reader converts the loaded binary token representations into continuous float-poin vectors and concatenates them with query representations before layer kt + 1. Then from layer + to layer n in the reader encoder, BTR applies runtime compression (described below) over representations to reduce the number of tokens and further speed up
the concatenated the encoder inference. Once the encoder finishes computation, the reader flattens and concatenates all the
representations, and then feeds them into the reader decoder. The reader decoder applies similar runtime compression operations to shorten the concatenated text sequence with query and all passage tokens.
Runtime compression. Runtime compression is possible because the retrieved passages are relevant to the query and contain similar information for different passages. The runtime token compression consists of antra-passage and passse compression steps. At the upper layers (k + 1 to n)) in the reader encoder, the query representations are concatenated with representations of each passage, and the passage representations are processed together. We compress the token representations of each quasy-passage pair and call this process intra-passage token compression since the encoder still computes each the query-passage representations are flattened and concatenated. In the reader decoder, we further compress the resulting representations across all passages (and the query) for every g layer" and call this cross-passage token compression. We use a similar compression algorithm as in the offline token compression, merging r % tokens based on their semantic similarity. The difference here is that the representations are continuous vectors in the upper encoder and the decoder, so we change the similarity metric to cosine distance accordingly. Details are in Algorithm 2 in Appendix A.
pair representation in parallel. After the encoder computation, all
EVALUATION
We first describe BTR and baselines, followed by five knowledge-rich NLP tasks to evaluate them. We then report the main results of BTR on task performance and efficiency gains. We also evaluate the proposed techniques in BTR and their impacts efficiency and performance trade-offs.
4.1 BASELINES
We apply BTR to Atlas (Izacard al., 2022b) base and large variants and call them BTR-Atlas base and Atlas large. Atlas is a recent state-of-the-ard eetreeva-augmmente language model. The Atlas reader uses the T5-based (Raffel et al., 2020) iisionnin-Decodee (FiD) (Izacard & Grave, 2021) architecture. We include more implementation details in Appendix A.2.
Baselines. To compare task performance and efficiency with BTR, we evaluate five representative baselines (implementation details are in Appendix A.3):
(1) Atlas (Izacard et al., 2022b) a state-of-the-art retrieval -augmeted language model for knowledge-rich NLP tasks. Due to computing budget constraints, we compare the base (with 220M parameters) and large (770M parameters) size variants.
(2) Atlas-Q is quantized version of the Atlas base model. We use GTPQ (Frantar et al., 2023) to quantize the fine-tuned Atlas baseline model in 4bit.
(3) DensePhrase (Lee et al., 2021) creates dense phrase representations of passages. It retrieves answers from the phrase index without using the retrieve-and-read pipeline and achieves high query processing throughput.
(4) DeFormer (Cao et al., 2020) speeds up the inference in encoder-only reader models by decompos- ing the passage and query encoding and caching the continuous passage representations. We apply DeFormer over Atlas-base and fine-tune the decomposed model.
6 vvalues are -1. and L.0.)
We set to be 3 as default. We do not merge in every layer as in the upper part of the encoder because the decoder has more layers, skipping compression for few layers provides balance between efficiency and accuracy.

---

Published as a conference paper at ICLR 2024
Table 1: Task performance and inference efficiency comparison between the baseline models and BTR-Atlas over five evaluation tasks. 'Sp' denotes the speed up over the Atlas base model, and the subscript percentage numbers in brackets are relative performance compared to the Atlas base model. The speed up and percentage for BTR-Atlas large is over the Atlas large model. BTR achieves 2-3x inference speedup while maintaining > 95% of the original models' performance.
NQ
TQA
WQ
FEVER
MMLU
Sp
EM
Sp
EM
Sp
Acc
Sp
Acc
Sp
Atlas base
52.1
1.0
69.3
1.0
46.4
1.0
72.9
1.0
38.6
1.0
Atlas-Q
(1.8()0%)
1.2
68.5 (895))
1.3
45. (97%))
1.1
70.4(97%))
1.1
(988)
1.2
DeFormer
51.4(00%)
2.2
68, (880))
2.0
(4.8
2.0
(188()))))
2.3
33.9 (88%)
1.8
DensePhrase
40.9 (79%)
4.9
(77%)
5.8
(813))
5.1
LLAMA2-7B
(7.8(029)
47.8 (022%)
0.1
74.3 (108%)
0.1
51.2 (112(1))9))
0.1
76.3 (105%)
0.1
(1339)
0.1
base
(0.3))
3.1
66.7 (96%)
2.5
(348))
2.6
70.2 (06%)
3.1
(5.4
2.6
Atlas large
58.3
1.0
73.6
1.0
51.5
1.0
78.2
1.0
41.1
1.0
BTR-Atlas large
56. (06%))
4.0
(0.8))6%)
3.9
49. (058))
3.6
75.9
4.1
(962))
2.5
Afre
Allea-3
DaFomer
MA
60
LLLMA2-78
Acouray 04
TTR- ARt-- base
70
DaFarmer
Albot
LLaM22.7
Pareiiquue
Acoureyy
AEas
DaFomer
16
Storage (GB)
Storage (GB)
Parslo curee
Stor age (GB)
BTR Nlss base
0
Pardlo catee
127
127
127
DensePhrase
65
Danas uole
DensePhrace
20
30
40
50
20
40
50
10
20
30
40
50
(FF)))
Throughout (GPS)
Throughput (OPS)
(a) NaturalQuestions
(b) TriviaQA
(c) WebQuestions
Figure 3: Task performance and inference efficiency (throughput and storage) visualization across the baselines and BTR fine-tuned models on three open-domain QA datasets. The area the circle denotes storage size (we scale the size of BTR-base and DensePhrase by 10 for clear display). A top-right smaller circle is better.
(5) LLaMA2 (Touvron et al., 2023b) consists of open-source large language models that achieve top performance on many NLP benchmarks including the knowledge tasks we study. Since the few-shot versions still lag behind fine-tuned models, finetune the LLaMA2-7B model using the LoRA (Hu et al., 2021) adapter and report its results on the evaluation tasks.
4.2 TASKS AND METRICS
We evaluate BTR and baselines on three open-domain QA tasks: NaturalQuestions (NQ, Kwiatkowski et al. (2019), TriviaQA (TQA, Joshi et al. (2017)). WebQuestions (WQ, Berant et al. (2013)); one fact-checking task: FEVER (Thoree et al., 2018), and one knowledge-intensive reasoning benchmark: the mass-multitask language understanding (MMLU) dataset (Hendrycks et al., 2020). Dataset details in Appendix A.4. We use the same Wikipedia corpus (dumped from December 2018) for all tasks and compared systems for consistency. The corpus contains 32.1 million passages, roughly over 3.2 billion tokens assuming 100 tokens per passage.
Evaluation metrics. We the exact match score (EM) the metric for the open-domain QA tasks. We measure the accuracy of the fact checking task and the MMLU benchmark. For all tasks and models, choose the model checkpoint based on the development set and report the metries on the test set. All the numbers are average values across 5 different runs (with different random seeds).
Efficiency measurements. For each task and model, we randomly sample 100 queries from the test dataset and measure the real inference throughput the number of queries processed per second (QPS) of the compared models on the A100 GPU hardware. Actual throughput reflects a more realistic comparison of efficiency across different systems.

---

Published as a conference paper at ICLR 2024
Table 2: Inference throughput and accuracy for applying binary representations to BERT-base reader model on three open QA datasets.
NQ
TQA
WQ
EM
Sp
EM
Sp
EM
BERT base
42.9
1.0
57.1
1.0
43.1
1.0
BTR- RERBERT base
39.8
(93%)
54.1
(95%)
3.9
39.7
(92%)
3.2
Table 3: Ablation analysis for each component in BTR on the NaturalQuestions dataset.
Accuracy (%)
Throughput
Storage
(QPS)
(GB)
Atlas base
52.1
10.2
BTR-Atle base
49.5
31.5
127
w/o binary passage representations
50.8 (+1.3)
32.3
12,804
w/o offline compression
49.8 (+0.3)
31.5
159
efficiency
w/o runtime intra-passage compression
50.0.50055
28.1
127
w/o runtime cross- poossppssaage compression
(+0.4)
127
w/o passage representation recovery
47.4
31.5
127
accuracy
w/o query-aware passage token distillation
(-1.3)
31.5
127
4.3
MAIN RESULTS
BTRachieves favorable accuracy and efficiency tradeofTs. Table presents the accuracy, storage versus inference throughput trade-offs for BTR, DeFormer, DensePhrase, Atlas-Q, and LLaMA2-7B models on the evaluation datasets. We define the better efficiency trade offs under certain accuracy target (e.g, afford relatively 5% accuracy drop). A model has better trade-off if the model more efficient (less storage and faster speed) when satisfying the accuracy budget. Atlas-Q and LLaMA-7B provide high accuracy but with very low inference throughput. DensePhrase achieves the highest throughput, but comes with much lower accuracy numbers (> 6% gap); DeFormer has similar inference throughput and slightly higher accuracy than BTR but its storage overhead two orders of magnitude larger. Figure visualizes the efficiency versus accuracy trade-off' for the compared models on the three open QA datasets.
Furthermore, BTR is more scalable than DeFormer and DensePhrase. We estimate that using much larger corpus like CCNet (Wenzek et al., 2020) with over 350 million passages will take up 1389 GB using similar compression ratios for the Wikipedia corpus. However, for baseline models, the DeFormer will spend over 100TB of storage and the DensePhrase will occupy over 3TB of storage.
BTR remains effective for encoder-only models. We study binary passage token representations for encoder-basee models and validate their effectiveness and inference benefits. Specifically, we apply the token-level binarization technique in $3.1 for the BERT-base model with DPR (Karpukhin et al., 2020) as the passage retriever. We do not apply scale-aware modifications since the BERT model is post-layernorm architecture. Note that encoder models expect the answer predictions to be exact spans from the retrieved passages, therefore, our runtime token compression techniques do apply since we cannot recover the original exact spans once the tokens compressed. Offline binary token compression remains applicable and reduces storage footprint. Implementation details are Appendix A.2. Table shows the results of the BERT reader model on three open QA datasets, applying binary representations also effectively improves the inference throughput by over 3x while maintaining over 92% accuracy.
4.4 ABLATION STUDY
Each component in BTR provides either efficiency or accuracy gains. We study the effects of each component in BTR on the inference efficiency and accuracy using the NaturalQuestions dataset. BTR consists of four efficiency components that improve either inference throughput or storage footprint and two training components that improve accuracy. The efficiency components include (i) binary passage representations; (ii) offline token merging; (ii) runtime intra-passage token compression; and (iv) runtime cross cross-passee passage token compression. The accuracy components are (v) passage representation recovery and (vi) query-aware passage token distillation.
8

---

Published as a conference paper at ICLR 2024
BTR
BTR
48
Accuracy (%)
Accuracy (%6
44
40
35
40
80
100
120 Storage (GB)
140
28
30
32
36
38
40
Throughput (QPS)
(a) Offline token compression
(b) Runtime token compression
Figure 4: Accuracy, storage, and throughput comparisons for different two-stage token compression ratios on the NaturalQuestions dataset. To achieve a good balance between aceuracy, storage, and throughput in BTR, we choose compression ratio of 0.2 for both online and offline token compression. Detailed numbers are in Table 9 in Appendix A.6.
Table shows the efficiency techniques in BTR come with the price of task accuracy degradation, however the two accuracy components during training effectively mitigate such accuracy drop. Specifically, the binary passage representations dramatically reduce the storage by over 100x, and offline token compression further reduces the footprint by 20%. On the other hand, the two online token compression techniques improve the inference throughput by over 30% without sacrificing too much accuracy drop. Overall, the techniques in BTR collectively provide efficiency gains without compromising task accuracy too much.
Effects token compression ratios in BTR. Figure shows the effects of the two-stage token compression technique in BTR under different compression ratios. We choose the compression ratio from 5% to 50% with gap of 5% and report the accuracy and storage or throughput numbers for the NaturalQuestions dataset. For offline token compression, with bigger compression ratios, BTR achieves linear storage cost reduction but the accuracy does not drop significantly until the ratio bigger than 0.2 (default for BTR). We also observe that for online token compression, the inference throughput consistently improves with higher compression ratios, while the accuracy degradation is relatively small until the ratio is larger than 0.2. By configuring different compression ratios, BTR allows flexible inference throughput, storage, and accuracy trade-offs during inference without additional training efforts.
CONCLUSION AND FUTURE WORK
netreneal-augmented language models present viable solutions address issues like hallucinations in large language models, but their inference slow because the reader component needs to process many passages. We introduce BTR, system that creates cacheable binary token representations to improve the inference and storage efficiency for the reader in retrieval-augmmnned language models. Together with training regularization techniques and token compression methods, BTR effectively reduces the storage costs by 100x, improves inference throughput by 2~4xx, and maintains performance for a wide range of knowledge-intensive tasks.
We discuss directions for future work. 1) Extending BTR to decoder-only readers is non-trivia because decoder models compute passage representations together with the query sequential manner, making it challenging to break computational dependencies and cache passage representations. Moreover, the KV-Caches in decoder models speed up the inference decoding, but storing their binary representation causes much more storage than encoder models. 2) Improving BTR for extremely long input queries remains challenging and requires other orthogonal efficient methods. BTR can speed up the inference when queries are longer, but the speed-up is relatively smaller than shorter queries. 3) Scaling BTR for larger models with bigger representation sizes another important research topic. Potential solutions might include using autoencoders to compress the dimension of representations. 4) It will also be interesting to apply binary token representations to the retriever and incorporate BTR into model pretraining for building better and faster aatrival-augmenter language models.
Compression ratios bigger than 50% cause sigmificant accuracy drops.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This research was supported partly by NSF IIS-2044660, an Allen Investigator Distinguished award. We thank the members of the UW NLP group for their comments and feedback on this paper.
REFERENCES
Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jin Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. BinaryBERT: Pushing the Limit of BERT Quantization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4334 4348, Online, August 2021. Association for Computational Linguistics. doi: 10. 1186633/11222111al1llolao URL pettps://aclanthollee .rg. 2021.2221..g acl-long. 334. (page 3)
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation, August 2013. URL http://arxiv. org/abs/1 3432. (page 4)
Jonathan Berant, Andrew Chou, Roy Frostig, and Perey Liang. Semantic Parsing on Freebase from Question-Answer Pairs. In Proceedings of the 2013 Conference Empirical Methods in Natural Language Processing, pp. 1533-1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL httttp::/aacaanthollle/ org/D13- 3-600 (page 2, 7, 16)
Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token Merging: Your VIT But Faster. In The Eleventh International Conference Learning Representations, September 2022. URL https: https:://operreview wea.ie..nn/ffrummiiee JroZRaRw7Eu. (page 5)
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. Improving Language Models by Retrieving from Trillions of Tokens. Proceedings of the 39th International Conference on Machine Learning, pp. 2206-2240. PMLR, June 2022. URL https: http::::/p/cceedin//ee wwweeediinnsmmm......mmmm. mwt.smesss.pss///oooepm..mmmmm 2162/borgeaud22a. html. (page 2)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877- 1901. Curran Associates, Inc., 2020a. URL https://papets cecpapprrsniiis...e httppccccpppee/22//// c4576666bcbb49774 cb49544188bbbbaccc11
Alsc-aa-bbtract..... html. (page 4)
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter. Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. arXiv:2005. 14165 [cs], July 2020b. URL http: wwtp:/iwww..oga//////ea/ 14165. (page 1)
Qingqing Cao, Harsh Trivedi, Aruna Balasubramanian and Niranjan Balasubramanian DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering. In Proceedings of the 58th Annual Meeting the Association for Computational Linguistics, pp. 4487-4497 Online, July 2020. Association for Computational Linguistics. doi: 10, 10.18653/v1/200 acl- acl-mai41- main.411. URL https: //ww:///wwww//// aclweb org www.or/g/nngggg///noooo acl-main. 411. (page 1, 6, 16)
10

---

Published as a conference paper at ICLR 2024
Zhangjie Cao, Mingsheng Long, Jianmin Wang. and Philip S. Yu. HashNet: Deep Learning to Hash by Continuation. In 2017 IEEE Interaational Conference on Computer Vision (ICCV), pp. 5609-5618, October 2017. doi: 10.1 10.1109//CCC22217...588 (page 3)
Michiel de Jong, Yury Zemlyanskiy, Joshua Ainslie, Nicholas FitzGerald, Sumit Sanghai, Fei Sha, and William Cohen. FiDO: Fusion-in-Decodee optimized for stronger performance and faster inference, December 2022. URL http://arxiv.oom htt//ab/2220 2212. 08153. (page 2)
Michiel de Jong. Yury Zemlyanskiy, Nicholas FitzGerald, Joshua Ainslie, Sumit Sanghai, Fei Sha, and William Cohen. Pre-computed memory or on-the-fly encoding? hybrid approach to retrieval augmentation makes the most of your compute, January 2023. URL http: orgootgg//o.. org/abs/ 2301. 10448. (page 3)
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, August 2022. URL http:://axxiv.org/abb//22 07339. (page 3)
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs, May 2023. URL http: /ttt:i/
http::/arxiv..rrg/bbb///// wtgg//b/////...../o 14314. (page 16)
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, October 2022. URL http://arxiv.orr abs/2210. 17323. (page 3)
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, March 2023. URL http::/arxiv..o abs/2210. 17323. (page 6, 16)
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval Augmented Language Model Pre-Training In Proceedings of the 37th International Conference on Machine Learning, pp. 3929-3938. PMLR, November 2020, URL https: //poce///ggsec/ggss... mmr...mm........... mwr.press) v119/guu20a. html. (page
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding. In International Confer- ence on Learning Representations, October 2020. URL https://openreviien htt.:ppenrevieeeeee/oourro netp://rmffffi (7KBjmI3GmM) (page 2, 7, 17)
Sebastian Hofstätter, Jiecao Chen, Karthik Raman, and Hamed Zamani. FiD-Light: Efficient and Effective Strrivall-Augmente Text Generation, September 2022. URt:ttp: http::////// wwwpii.oorg//b/eee 2209. 14290. (page 2)
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations, October 2021. URL httpttts///ppnnreieee...... wwwperrvviieeeeeeorrrnnn net/forum? id=nZeVKeeFYF9, (page
Jie Huang. Hanyin Shao, and Kevin Chen- Chang. Are Large Pre-Trained Language Models Leaking Your Personal Information?, October 2022. URL httt:://axiiii .org/ orggabs//2aal 12288. (page 1)
Long- Kai Huang, Jianda Chen, and Sinno Pan. Accelerate Learning of Deep Hashing With Gradient Attention. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5270- 5279, October 2019. doi: 10. 1011111C.22009000000 (page 3)
Gautier Izacard and Edouard Grave. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 874-880, Online, April 2021. Association for Computational Linguistics. doi: 10,18653/v 1/2021 wea.l- 74, URL https://aclanttholgg..ook org/2// eacl-main. 74. (page 3,4,6 4)
11

---

Published as a conference paper at ICLR 2024
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised Dense Information Retrieval with Contrastive Learning. Transactions on Machine Learning Research, May 2022a. ISSN 2835-8856. URL https://
openreview.net/forume wwt.///oooooriiiiiiiiiii pXi7b0. (page 15)
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot Learning with Retrieval Augmented Language Models, November 2022b. URL httt:::/arxxi..rr/bbb.ss.. 03299. (page 1, 2, 6, 15)
Himalaya Jain, Joaquin Zepeda, Patrick Perez, and Remi Gribonval. SUBIC: A Supervised, Structured Binary Code for Image Search. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 833-842, 2017. URL ht /tpp//ccppcccccccccp thecvf. coon/content_ccom 201 paperr html. (page 3)
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1601-161 Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/ P17-1147. URL https: /aw.aahooaannn wrg/oogg 1147. (page 2, 7, 16)
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lunguage Processing (EMNLP), pp. 6769-6781, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/20222 11133//12222.ccmmm wet.:0mmmmmpmmaaeeee m50n.550. URL https://aclaatholoog// wwg.gg....r//// 202.0000..00....//// emnlp-main. 550. (page 8, 15)
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Men Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural Questions: A Benchmark for Question Answering Research. Transactions of the Association for Computational Linguistics, 7:453-466, August 2019. doi: 11116//acl_a0002100
URL https:// org/10.../... 2162/taccl. a_076_____2666_ (page 2, 7, 16)
Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, and Danqi Chen. Learning Dense Representations of Phrases at Scale. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 6634-6647, Online, August 2021. Association for Computational Linguisties. doi: 10. 10.18633/1120211 haw./22022...looon511111111 URL https: htttps://cclattool/// (page 1, 2, 6, 16)
org/2021.12.22 .cw. llng. 518.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sehastian Riedel, and Douwe Kiela. Retrieval-Augmented Generation for Knowledge-Intensing In Advances in Neural Information Processing Systems, volume 33, pp. 9459-9474. Cur- ran Associates, Inc., 2020. URL httpt::s///rroceeiiing neur wtwpppppppppe///pppppppppppppppp htlppppppr/22222 2200//ash/ 05 httm (page 2)
NL. Tasks.
Haomiao Liu, Ruiping Wang, Shiguang Shan, and Xilin Chen. Deep Supervised Hashing for Fast Image Retrieval. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition (CVPR), pp. 2064-2072, 2016. URL https: //tppsss///pencceesss thecvf. com//ootent_ wtt.22006/hmm/Liiii Sew.iueeeeeeSeeeiisseeeeeeeeeeee wwwpieerviiee_assinnn_.ooo
html. (page 3)
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Ha- jishirzi. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories, May 2023. URL http:://arriv.rrg/aaa//2222 10511. (page 1)
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, and Sayak Paul. Peft: State-of-the-a parameter-efficiie fine-tuning methods. Hugging Face, September 2022. URL thub.
https://ggituub..mm//uggggnnn/ www.huggiigaaaeeeeeeeoooooooeeee (page 16)
12

---

Published as a conference paper at ICLR 2024
Sewon Min, Weijia Shi, Mike Lewis, Xilun Chen, Wen-tau Yih, Hannaneh Hajishirzi, and Luke Zettlemoyer. Nonparametric Masked Language Modeling, May 2023. URL http://arxiv.orrrt 0bs/2212.01344 (page 2)
Niels Mündler, Jingxuan He, Slobodan Jenko, and Martin Vechev. Self-contradictrry Hallucinations of Large Language Models: Evaluation, Detection and Mitigation, May 2023. URL http: //arxiv.org abs//bbs/555 15852. (page 1)
Haotong Qin, Yifu Ding, Mingyuan Zhang, Qinghua Yan, Aishan Liu, Qingqing Dang, Ziwei Liu, and Xianglong Liu. BiBERT: Accurate Fully Binarized BERT. In International Conference on Learning Representations, March 2022. URL https:
wwt. www.eeenneettoooiid-//eEE.
5FAJ. (page 3)
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to- Text Transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. ISSN 1533-7928.
URL http:/mrrrrrrr/e/////e //1/s///22////
/ 200 074. html. (page 4, 6)
Dinghan Shen, Pengyu Cheng, Dhanasekar Sundararaman, Xinyuan Zhang, Qian Yang, Meng Tang. Asli Celikyilmaz, and Lawrence Carin. Learning Compressed Sentence Representations for On-Device Text Processing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 107-116, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1000 1011. URL https: / /ttp::s:s///llanhhoooo//// org /Ptg//19-111111 (page 3, 5)
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen-tau Yih. REPLUG: Retrieval- Augmented Black-Bor Language Models, May 2023. URL
.1....265555 12652. (page 2)
James Thorne, Andreas Vlachos, Christos Christodoullpoulls,, and Arpit Mittal. FEVER: a Large-scale Dataset for Fact Extraction and VERification. In Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- Language Technologies, Volume I (Long Papers), pp. 809-819, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18. 1074. URL https:///clanttholog. www.ethowooooo.o.o//..eoo oot.///-//////// (page 2, 7, 16)
Julien Tissier, Christophe Gravier, and Amaury Habrard. Ncar-lossless binarization of word embed- dings. In Proceedings of the T'irttyThirr AAAI Conference on Artificial Intelligence and Thirty- First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI' 1T/TATTT///AAA 19, pp. 7104-7111, Honolulu, Hawaii, USA, January 2019. AAAI Press. ISBN v33101.33017104, URL https: //dl. acm. www.:org//oo//i/e /609//aai.... v33101. 33017104. (page 3)
doi: 10.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models, February 2023a. URL http:://tt///ttt wwt. 2b5// 13971. (page 4, 16)
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi. Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami. Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Tuned Chat Models, July 2023b. URL http: // /ttxxv.xivvv////////ea//e/aa org/abs/ 09288. (page 4, 7, 16)
Stojnic,
13

---

Published as a conference paper at ICLR 2024
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 4003-4012, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-3444 URL
1ere222..eee11111 194.. (page 8)
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick Von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020, Association for Computational Linguistics. URL hwwph:ttp//t.. hwwp://wwwwwwooo aclweb or htt/aaaooolyy/2200 2020 httphhoooooo/222...mmmeeeeeeeee emnlp-demos. 6. (page 15)
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and Efficient Post- Dostnnrainnng Quantization for Large Language Models, February 2023. URL http: /ttp:p///aavv..o///eeeeeeeoee http::///axi..oo/b/////////
10438. (page 3)
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On Layer Normalization in the Transformer Architec- ture. In Proceedings of the 37th International Conference on Machine Learning, pp. 10524- 10533, PMLR, November 2020. URL https://proceedings. mlr wttpppsss////xiinnngeeeee html. (page 4)
Ikuya Yamada, Akari Asai, and Hannaneh Hajishirzi. Efficient Passage Retrieval with Hash- ing for Open-domain Question Answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). pp. 979-9869 Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v 12255///12221111 1 scll- URL
https://acllnnthooooo..orr////// acc..r//
acl--sorr. 123. (page 3, 4)
Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. RPTQ: Reorder-based Post- Post-training Quantization for Large Language Models, April 2023, URL http: http::///axii..o//ab//////// 2304. 01 089. (page 3)
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 TOKEN COMPRESSION ALGORITIM
Algorithm 1 Offline Compression for Binary Token Representations
Input: precomputed binary passage token representations B over corpus C, compression ratio To. the vocabulary of reader model
Output: compressed binary passage token vectors Be
for each stopword token t, in V. do
2: collect all binary representations B, in corpus C
3: compute the average across all binary representations: b,- = sign( B. BB,
4: end for
5: for each token tu in V that is non-stopword, do
6: collect all binary representations B, in corpus
7: merge B, using hamming distance 7:: B, = sign(bipartin merge (By, Fos H));
8: end for
9:
function BIPARTITE MERGE(input tokens: B, merge ratio: r. distance metric M)
11: divide tokens B into two sets of tokens B, and B, based on even and odd order for each token B, do create token edge to its most similar token in Bj using M
13:
14:
store token edge e into end for
15:
this is implemented as fast parallel operation Keep top-r' edges E,,, where =
17:
for each token edge in E,,, merge connected tokens in B, and into Be by computing
the average of token vectors, gather the rest unmerged tokens Brest and Brest return merged tokens B, gather(BB... Brest Breat)
end function
Algorithm 2 Runtime Compression
Input: query-passage token representations hg, compression ratio Tpp Output: merged text token representations ha
1: for each layer from layer & + to in the reader encoder, do
2: merge h, using cosine distance h = bipartite_mergeell Pp D);
3: end for
intra-passage token compression
4: for each layer from layer to n in the reader decoder, do
5: if d % 0 then continue end if
skip merging for g layers
6: merge h using cosine distance D:: h = bipartite_mergeeee rp, D);
7: end for
cross-passage token compression return H,
A.2 IMPLEMENTATION DETAILS
BTR Details. We develop BTR based on the Atlas codebase using PyTorch 1.13.1 and HuggingFace Transformers v4.18.0 (Wolf et al., 2020). We conducted training using 4 to 8 A40 A100 GPUs (depending on their availability on our cluster) with BF16 mixed precision. We list the training hyperparameters in Table 4. The decomposition layer & is for the BTR-Atlas base model, & is 20 for the large model, and is BTR-BERTS BERT. We implemented BTR-BERT using the original DPR (Karpukhin al., 2020) codebase.
A.3 BASELINE DETAILS.
A tlas (Izacard et al., 2022b) models obtain state-of-the-art performance on a wide range of knowledge- rich NLP tasks. Atlas model retrieves passages from the corpus using Contriever (Izacard et al., 2022a)- retriever, which a BERT-like architecture with 110 million parameters shared between the query and the passage encoder. Since our focus is the reader component, use the retriever in the Atlas IIB model to obtain query-relevant passages and fine-tune Atlas reader models with these retrieved passages. Due computing budget constraints, compare the base (with 220M
15

---

Published as a conference paper at ICLR 2024
Table 4: Training hyperparameters for BTR-Atlas.
NQ
TQA
WQ
Fever
MMLU
Hyperparameters
base
large
base
large
base
large
base
large
base
large
batch size
4
8
4
learning rate
6e-5
6e-5
4e-5
4e-5
8e-5
8e-5
6e-5
6e-5
5e-5
5e-6
training steps warmup steps weight decay
20000
20000
20000
20000
3000
3000
10000
10000
2000
2000
200
200
200
200
200
200
200
200
50
50
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
number of passages
40
40
40
40
40
max query length
40
40
64
64
40
40
40
40
256
256
max passage length
320
320
320
320
320
320
320
320
320
320
answer length
32
32
32
32
32
32
32
32
32
32
parameters) and large (770M parameters) size variants. We the original hyperparameters finetune Atlas.
Atlas-Q is quantized version of Atlas. We use GTPQ (Frantar et al., 2023) to quantize the fine- tuned Atlas baseline model in 4bit. We do not directly use quantization-wwan training such as QLoRA (Dettmers al., 2023) since the quantized model's inference is much slower.
DensePhrase (Lee et al., 2021) creates dense representations of phrases from the passages. It retrieves answers from the phrase index without using the eerieee-anndread pipeline and achieves high query processing throughput. We run the official DensePhrase implementation on an A A100 GPU and report the throughput numbers.
DeFormer (Cao et al., 2020) speeds up the inference in encoder-only reader models by decom- posing the passage and query encoding and caching the continuous passage representations. We apply DeFormer over Atlas and fine-tune the adapted model. In fact, we can view DeFormer simplified system from BTR without binary representations and the offline and runtime compression components.
LLaMA2 (Touvron et al., 2023a;b) consists of open-source large language models that achieve top performance on many NLP benchmarks including the knowledge tasks study. Since the few-shot versions still lag behind fine-tuned models, we finetune the LLaMA2-7B model using the LoRA adapter and report its results the evaluation tasks. Due memory constraints, limit the retrieved passages to We implemented this baseline using the PEFT library (Mangrulkar et al., 2022), the LoRA config is follows: r=8, alpha=32, dropout=0.05,, task tettttyy====sss. target_modee aageet_mdduls==*(kkktlllloggeee MglKiooolgggttlooollluul.n.a
bias="none"
A.4 DATASETS DETAILS
Table 5: Statistics of the number of examples for the evaluation datasets.
NQ
TQA
WQ
MMLU Fever
Train
79168
78785
3400
145449
95127
Validation
8757
8837
378
19998
1531
Test
3610
11313
2032
19998
14042
Open-domain QA. We use the original open-domain variants of NaturalQuestions (Kwiatkowski al., 2019). TriviaQA (Joshi al., 2017). WebQuestions (Berant et al., 2013). The NaturalQuestions (NQ) dataset consists of natural Google search questions that have answers from Wikipedia articles. The open-domain variant contains 91,55 ,535 question-answer pairs (79168 for training, 8757 for development, and 3610 for testing). TriviaQA (TQA) has 98,935 quiz-style trivia questions (78785 for training, 8837 for validation, 11313 for testing), its unfiltered version used for open-domain QA. The questions from WebQuestions (WQ) are constructed from Freebase KB and Google Suggest API and their answers are labeled by Amazon Mechanical Turk. It has 3,778 examples for training and 2,032 for testing. We reserve 10% of the training set for development and use the rest for training.
Fact Checking. We use the original FEVER (Thorne et al., 2018) dataset for the fact-checking task. FEVER contains 145k training examples, and both development and testing tests have 20k examples.
16

---

Published as a conference paper at ICLR 2024
Each example has claim sentence and 3-class (supported. refuted, not enough info) label. The task requires retrieving evidence from the Wikipedia corpus and making a judgment.
Table 6: Inference throughput numbers for the baselines and B TR-Attsss.
NQ
TQA
WQ
FEVER
MMLU
DeFormer
22.4
16.7
18.8
24.6
9.6
DensePhrase
50.4
48.5
48.1
Atlas-Qus alaas-uaant
12.5
13.2
11.2
11.8
6.4
LLaMA-7B
1.2
0.8
1.3
1.2
0.5
Atlas base
10.2
8.4
9.5
10.5
5.2
BTR-Atlas base
31.5
21.4
25.1
32.1
13.4
Atlas large
3.8
2.1
2.2
3.5
3.3
BTR-Atlas large
8.2
Knowledge-Intensin Reasoning. We use the mass-multitask language understanding (MMLU) benchmark (Hendrycks al., 2020) evaluate the knowledge reasoning ability of RaLMs. MMLU is a multiple-choice question-answering answering dataset that has 57 subdatasets covering various domains like history, math, law, etc. The benchmark tests models' world knowledge and problem-solving ability. The training set has 95127 examples, the development set has 1531 examples, and the test set has 14042 examples.
Table 5 summarizes the number of examples for each split in the datasets.
A.5 BASELINE RESULTS
Table 6 contains the inference throughput numbers of the baselines and BTR-Atlas models. Table presents the same inference and accuracy numbers in Table but with added standard deviation numbers.
A.6 ABLATION RESULTS
Table 8 presents the accuracy and speedup results of Atlas base and BTR-Atlas base models using different numbers of passages the NQ dataset.
Table 9 includes the detailed numbers of the compression ratio ablation experiments.
17

---

Published as a conference paper at ICLR 2024
Table 7: Task performance and inference efficiency comparison between the baseline models and BTR-Atlas over five evaluation tasks. '+' denotes the standard deviation accuracy or em scores across five with different random seeds. The speedup (Sp) deviations are quite small (<0.5%) which we omit here.
NQ
TQA
WQ
FEVER
MMLU
EM
Sp
EM
Sp
EM
Sp
Ace
Sp
Acc
Sp
Atlas base
52. 00.3
1.0
69.3
1.0
46.4 1.0
72.9 40.9
1.0
38.6
Atlas-Q Q
51.8-u.1
1.2
68.5
1.3
45.1 1.1
70.4 1.1
=7.8
1.2
DeFormer
51.4_0.2 2.2
68.0
2.0
44.820.5 2.0
71.8 2.3
33.9 1.8
DensePhrase
40.9_0.2 4.9
53.610.4
5.8
37.5 5.1
LLAMA2-7B
47.8_0.2 0.1
74.3
0.1
51.220.3 0.1
76.3 0.1
51.2 0.1
BTR-Atlas Atlas base
49.5_u.3
66.7+0.2
43.8 2.6
70.2
35.4 2.6
Atlas large
58.3_0.3 1.0
73.610.2
51.520.5 1.0
1.0 75.9
78.210.4
21.1
BTR-Atlas large
56.1 4.0
70.8
29.1 3.6
29.2 2.5
#of passages
Atlas base EM
BTR-Atlas base EM
Speedup
5
41.1
38.9
2.9
10
43.5
41.9
3.2
20
46.1
43.8
3.4
30
47.2
45.4
3.3
40
52.1
49.5
3.1
Table 8: Accuracy and speedup comparison between Atlas base and BTR-Atlas base for different numbers of retrieved passages on the NQ dataset.
Compression Ratio
Storage (GB)
Accuracy (%)
Compression Ratio
Accuracy (%)
Throughput (QPS)
0.05
151
50.2
0.05
50.1
28.5
0.1
143
49.8
0.1
49.8
29.2
0.15
135
49.6
0.15
49.6
30.4
49.5
49.5
31.5
0,25
119
48.2
0,25
48.7
32.2
0.3
111
47.1
0.3
47.6
34.4
0.35
103
46.2
0.35
45.2
35.6
43.5
43.2
37.5
0.45
87
38.4
0.45
40.6
38.9
0.5
80
32.1
0.5
38.1
41.2
(a) Accuracy and storage effects for different offline token compression ratios.
(b) Accuracy and throughput effects for different intra- and cross- passage token compression ratios.
Table 9: Accuracy, storage and throughput comparisons for different two-stage token compression ratios the NaturalQuestions dataset. To achieve good balance between accuracy, storage and throughput in BTR, we choose compression ratio 0.2 for both online and offline token compression.
18


Reviews:
[{"review_id": "LOpL1iFVYo", "reviewer": "Reviewer_X7Hc", "summary": "This paper presents a method to accelerate inference speed of retrieval augment language models, while reducing the required storage space. The authors use binary token representations and compression (for collapsing the embeddings of similar tokens) to increase speed. They show that their method maintains up to 95% task performance when compared with the base method.", "questions": "* Did you try different thresholds for r%? How much did the performance change?\n* Is it important to do runtime compression after every block?\n* How slow is the bipartite matching for compression?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths are as follows:\n* Retrieval augment language models are becoming increasingly popular. Increasing the inference speed and reducing the memory footprint of such methods will be quite useful. The authors demonstrate that they can achieve a 4x speedup in inference speed and a 100x reduction in storage space.\n* The authors present results across multiple datasets.\n* The authors ablate all of their modifications and show how each one affects performance, speed and memory.", "weaknesses": "The weaknesses are as follows:\n* Some things like passage representation regularization are mentioned in passing and it would be helpful if the authors added a couple sentences providing context and explaining what this is.\n* No motivation or explanation is given for why distillation is required/helpful. \n* It is not clear if the linear projection layer for passage representation recovery is a learnable layer. Also, the $L_{recovery}$ term is confusing; $b_i$ is the binary passage representation and $h_i$ is the original passage representation, where is the projection layer used?\n* No comparison to newer methods like Lumen.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9ZlXTDc5zn", "reviewer": "Reviewer_U3RY", "summary": "This paper proposes binary token representations for retrieval-augmented LMs.\nThe goal is to increase inference speed and reduce storage requirements.\nThe key idea to achieve this in BTR is to create cacheable binary token representations of the retrieved passages such that the passage encoding can be precomputed offline and stored in a compact format.\nEmpirical evaluation on knowledge-intensive NLP tasks shows the effectiveness of their approach w.r.t. corresponding non-optimized retrieval-augmented LM:\n* inference speed: 2-4x\n* storage: up to 100x reduction (e.g., 10TB --> 100GB)\n* task performance: retains 90-95% of original performance", "questions": "* The text accompanying Figure 3 says \"BTR presents better efficiency versus accuracy trade-offs by maintaining high\naccuracy and inference throughput with a smaller storage footprint.\": Could you please clarify what you mean with this? As for me, BTR is another point on the Pareto front.\n* A method that allows to choose an operating point on the Pareto front would be more useful in practice than a method for a single operating point. Can BTR be extended along this dimension?\n* Could you please say a few words on the complexity of the training pipeline?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* Importance and relevance of topic: LLMs are everywhere and retrieval augmentation LMs addresses critical problems in LLMs such as hallucination, staleness, and privacy leaks, but suffer from low inference speed and huge storage requirements.\n* BTR has much lower storage footprint than other approaches, is more scalable and has lower inference speed at the expense of a \"modest\" loss in performance.\n* The paper reports numbers for actual throughput for a more realistic comparison of efficiency across different systems.\n* The code will be publicly available.", "weaknesses": "* \"BTR is more accurate and efficient than other baseline methods.\": According to Figure 3, the proposed BTR appears to be on the Pareto front. I.e., it isn't substantially better than existing methods but provides a different tradeoff between speed and accuracy. This makes it hard to assess the merits of BTR. To make the result less sensitive to a specific operating point and better comparability, it would be interesting to show, for example, how the tradeoff changes with the resolution of the representation (1-bit vs b-bit representations). Furthermore, adding sub-optimal points to the plots would give a more comprehensive picture (for example, LLaMA at different size/speed/accuracy).\n* The benefits at runtime come with a clearly more complicated training pipeline and increased training time. How much?\n* Figure 3: Why do the plots include Atlas-Q but not the original model (Atlas)? Also, I can't see any small points for Atlas-Q and LLaMA2-7B.\n* Different spelling in title and text: \"retrieval augmented language model\" vs \"retrieval-augmented language model\".\n* Funsion-in-Decoder → Fusion-in-Decoder", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RWNoVwdv9g", "reviewer": "Reviewer_KKL3", "summary": "The authors introduce multiple compression techniques to speed up retrieval-augmented models collectively referred to as BTR. These techniques are specific to a certain class of models like BERT and Atlas. And it's not clear that they will generalize to other types of retrieval-augmented LMs. There are many steps in the compression that are all required for good speed and accuracy. These are covered well in the ablation table:\n\n* binary passage representation is needed to reduce storage (speed is almost the same when using the dense passages).\n* offline compression gives a minor improvement in storage and speed at cost of accuracy.\n* online compression gives big speed improvements.\n* passage token distillation improves representation of dense vectors, and is important for good accuracy. This filters for 50 percent of the most salient tokens according to attention, although it is not clear if this was necessary and maybe distillation of 50 percent random tokens would yield similar performance.\n* passage recovery loss for the binary vector retains semantic information of the dense vector, and is important for good accuracy.\n\nThe results empirically show the speed (2-4x) and storage benefits (100x) of BTR, albeit at about 5 percent drop in performance. The paper is clearly very informative with analysis and details, but sometimes some basic details are missing and the benefits of BTR are occasionally oversold.", "questions": "Q1: Why do we need Step 2 for the decomposed reader? Doesn't Step 1 already provide a reader?\n\nQ2: \"This is likely because prior work only considers a single passage where all tokens are important to the task\" What does this mean? Surely many prior works use more than one passage.\n\ntypo: Funsion-in-Decoder", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Storage savings of 100x. This will make retrieval more available to many, since storage can often become a bottleneck in retrieval-augmented ML.\n\n2. Speed savings of 2-4x. Although this is similar speed improvement as Deformer (Cao et al), it leads to 5 percent accuracy loss which is worse than Deformer's 1 percent accuracy loss.\n\n3. Extensive experiments and analysis. There are some confusing or missing details, but these can be probably be easily fixed.", "weaknesses": "1. There are many steps required to make BTR work well. Of course, that is also why this is a valuable paper since it outlines what these steps are.\n\n2. There is a substantial (about 5 percent) drop in model accuracy. It's not clear whether 2-4x speed boost is enough value to make up for this, although the storage improvements are definitely very valuable.\n\n3. The paper is not very self-contained. It seems like the reader is expected to have read Cao et al and Atlas papers very closely. This is not ideal. For example, it is hard to understand what it means by \"decomposed model\" or \"decomposed reader\" in sec 3.2. Also, the reader is left to infer that retrieval is done at the passage level, but passages are incorporated at the token-level.\n\n4. (minor) The paper is hard to read. In more than one instance, a method is used but is \"defined below\", so we must constantly revisit parts of the reading to get a full understanding.\n\n5. (medium) \"BTR is more accurate and efficient than other baseline methods\" This claim can easily be interpreted as \"all other baselines\", which is not accurate. There are baselines that are more accurate or more efficient. This should probably be revised to be more accurate/specific. Also, I am not sure why BTR base is constantly bolded in Sp column when it is not the best value.\n\n6. There is not a good breakdown of efficiency between retrieval + inference. Perhaps alternative methods akin to DistilBERT will give a substantial speed boost plus keep good performance.\n\n7. (minor) Similarly, given the emphasis on storage it would have been helpful to see some basic baselines to improve storage. Although I am not sure what else can easily achieve 100x savings without larger tradeoff in performance.\n\n8. (very minor) It was confusing whether the token distillation is taken directly from Cao et al, or is something new.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V3uNytBy7T", "reviewer": "Reviewer_PDbq", "summary": "This article introduces Binary Token Representation (BTR) to solve the problem that running retrieval-enhanced language models (LM) is slow and difficult to scale. Due to the potential loss of accuracy, the authors propose a passage representation recovery objective and a query-aware passage token distillation objective to restore performance. By combining offline and runtime compression, the proposed method speeds up state-of-the-art inference by 4x and reduces storage space by more than 100x in five knowledge-intensive NLP tasks, while maintaining over 95% of task performance.", "questions": "**A**: In addition to BERT, what other experiments have you conducted on the Encoder-Only model? For example, Deberta, COCO-LM, and so on.\n\n**B**: The author claims to have utilized different random number seeds for 5 runs. It would be beneficial to include the standard deviation of the results in Table 1.\n\n**C**: In the Calibrated binarization section (Section 3.1), the author asserts that employing a straight-through estimator (STE) yields superior results compared to combining the annealing method with the tanh function. Are there any analytical experiments conducted to substantiate this claim? Additionally, is it viable to utilize a linear layer and sigmoid function for binarization? I recommend the author to include additional binarization methods in the experimental section to leverage the benefits of STE.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.  For the first time, the authors construct binary token representations to improve the efficiency of retrieval augmentation models, an approach that has never been explored before.\n\n2. The paper is very well-written and well-organized and provides clear motivation, background, and some technical details for the proposed model, including model quantization and binarization.\n\n3.  Experiments are conducted on 5 datasets and provide meaningful comparisons with existing retrieval-enhanced language models such as Atlas and DensePhrase and large language models such as LLaMA2-7B. The results show that the proposed model can maintain 95% of the above task performance while improving the inference speed and greatly reducing the storage space. The paper also includes ablation studies to analyze the performance further.", "weaknesses": "1. As the paper mentions, BTR is difficult to apply to decoder-only models, and most current large language models utilize a decoder-only structure.\n\n2. The paper only discusses FiD, which is the SOTA model in the KiLT ranking. However, it is also important to discuss other commonly used retrieval-augmented structures, such as RAG, particularly when dealing with black-box LLM.\n\n3. For retrieval augmented language models, the number of retrieved passages is an important factor for performance and efficiency. While \nauthors only set this factor to a fixed number (40 or 30 for different datasets), thus additional experiments should be involved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)