Optimizing Retrieval-augmented Reader Models via Token Elimination
Authors: Moshe Berchansky
Peter Izsak
Avi Caciularu
Ido Dagan
Moshe Wasserblat
Venue: EMNLP 2023 Main (2023)
Keywords: fusion in decoder
efficiency
long-form question answering
TL;DR: We introduce a method for eliminating irrelevant tokens from retrieved information in long form question answering systems during decoding time.

Abstract: Fusion-in-Decoder (FiD) is an effective retrieval-augmented language model applied across a variety of open-domain tasks, such as
question answering, fact checking, etc. In FiD, supporting passages are first retrieved and then processed using a generative model (Reader),
which can cause a significant bottleneck in decoding time, particularly with long outputs. In this work, we analyze the contribution and necessity of all the retrieved passages to the performance of reader models, and propose eliminating some of the retrieved information, at the
token level, that might not contribute essential information to the answer generation process. We demonstrate that our method can reduce
run-time by up to 62.2%, with only a 2% reduction in performance, and in some cases, even improve the performance results.

Optimizing Retrieval-augmented Reader Models via Token Elimination
Moshe Berchansky
Peter Izsak
Avi Caciularu" 0
Ido Dagan
Moshe Wasserblat
Intel Labs, Israel
Bar-Ilan University, Israel *Google Research
(moshe. het.ebbechhhssskn peter. izsak, moshe. wasserblat)einter com
avica@google.co com, daganêcs wwu.
Abstract
Token Filtering
effective retrieval -aateeee-l language model applied across a variety of open-domain tasks, such as question answering. fact checking, etc. In FiD, supporting passages are first retrieved and then processed using generative model (Reader), which
Eusion-in-Decodee (FiD)
Decoder layer
Filter and reshape
Token Filtering
significant bottleneck in de- coding time, particularly with long outputs. In this work, we analyze contribution and ne- cessity of all the retrieved passages the per- formance of reader models, and propose elimi- nating some of the retrieved information, at the token level, that might not contribute essential information to the answer generation process. We demonstrate that our method can reduce run-time by up to 62.2%, with only 2% reduc- tion in performance, and in some cases, even improve the performance results.!
Compute cross attention
Decoder layer
Passage
Passage
Input passage keys/values
Token
An overview of our Token Filtering method. Here, two passages are considered as the input to the decoder module. The Token Filtering operation is per- formed when generating token between decoder layer I and 1 1. Using the representation of token t at layer 1, cross-attention computed for all the inpul tokens. Then, filter out the lowest ranked tokens from the input (marked in yellow), with only the highest ranking input tokens being used from the next generated token onward.
Figure I:
1 Introduction
The task of Open-Domain Question Answering (ODQA) (Voorhees, 1999) consists of answering questions using external knowledge, which used as a source relevant information that might be helpful for a model to extract or generate the right answer for question. The expected answer can be short and concise (Kwiatkowski et al., 2019), or long and detailed (Fan et al., 2019), in which it is called Long-Form Question Answering (LFQA). The retriever-reader architecture has been widely-used and adopted for ODQA tasks (Chen et al., 2017). The retriever fetches the most rele- vant passages using the question as query. Then, the reader extracts or generates an answer, using the question and the relevant passages. The explicit structure of the system, consisting of these two sub-modules. allows for decoupled optimization of either the retrieving or the reading process. In this work, we exclusively focus on the optimization
of the reading process. In order to assess ODQA methods, Petroni et al. (2021) presented a compre- hensive evaluation framework that examines these methods in various open-domain tasks. Our study specifically concentrates on the Long-Form Ques- tion Answering task, utilizing the ELI5 dataset foundation (Fan et al., 2019).
There has been rapid and remarkable progress in retriever-reader systems for solving ODQA tasks using generative approach (Sachan et al., 2021; Izacard et al., 2022). One such prominent approach is Eision-in-Decodee (FiD) (Izacard and Grave, 2021b) that utilizes generative text-to-text model to generate an answer. Despite the significant per- formance improvements, there are several compu- tational bottlenecks associated with FiD that have
We provide the source code for our work at https:// gi thub. wwwm///sheber/token.nnomm eliminationn

---

negative impact on efficiency. The most promi- nent ones are: (a) the need retrieve relatively large amount of documents to reach peak perfor- mance, and (b) an extensive cross- cross-attention opera- tion, caused by processing multiple concatenated retrieved passages, applied repeatedly for every generated token. These bottlenecks are negatively amplified in the case of Long-Form Question An- swering.
We show that models utilizing our approach can save up to 62.2% on the MS MARCO dataset, 54.9% on NQ. and 40.9% on ELI5, terms of the generation time, while incurring a drop of no more than 2% in performance.
Without computational restrictions, our method reaches state-of-the-art performance in KILT's ELI5 task.
Previous works have attempted to mitigate these bottlenecks, either by limiting the input to the reader or by directly optimizing it a variety of methods. Yu et al. (2021) included a passage re-ranker inside the reader which aimed to filter out the least relevant passages during encoding. de Jong et al. (2022) optimized the decoder module by pretraining a modified and optimized architec- ture, and Ainslie al. (2023) modified the attention operations performed to be less computationally in- tensive.
Preliminaries
In a retriever-reader system, the reader, which is typically language model, receives a query along with a collection of passages, where each passage often consists of title and a context. Additionally, we are provided with the ground truth, which can be expected answer or gold passage that is most relevant to the query. Since our main focus is generative models, used Fusion-in-Decoder (FiD) model (Izacard and Grave, 2021b), which is a cutting-edge encoder- decoder model based on the T5 model (Raffel et al., 2020). The encoder module of the FiD model pro- cesses the input passages in parallel, with each layer taking the output of the previous layer, and the final output of the encoder is the output of its last layer. Similarly, cach layer of the decoder processes its input by receiving the output of the preceding layer.
employ the widely-
In this work, we tackle the heavy cross-attention computation in the decoder by introducing Token Filtering, method that removes redundant tokens from imput passages during the decoding stage, by dynamically computing their salience during gen- eration. Using Token Filtering climinates uninfor- mative tokens from the cross- attention matrix, and prevents them from being utilized during answer generation, directly contributing to the reduction of the overall generation time. To further boost efficiency and reduce latency, combine our To- ken Filtering approach with dynamic decoder layer skipping introduced by Schuster et al. (2022). re- ferred to as CALM. By combining both approaches and by conducting experiments on three LFQA datasets, we find that this approach presents a bet- ter performance vs. efficiency trade-off than by using the methods separately, in most cases. Overall, our contributions
The decoder module then cross-attends to the large number of concatenated input representations and assimilates the information from the different passages to generate answer. At each decod- ing step, the decoder computes the attention scores based on the precomputed input tokens' representa- tions which serve as the query for the multi-headed attention operation, concurrently taking into ac- count the current decoded sequence.
follows:
Efficiency Analysis
We analyze the performance trade-off of the FiD model, in terms of la- tency, FLOPs and the salience of the input information within the reader model, during long-form generation.
efficiency
3.1 Encoder vs. Decoder Latency
There are multiple parts in a retriever-reader setup that have direct effect on the end-to-ent latency. One of them is potentially reducing the number of passages provided to the reader model. Izacard and Grave (2021c) evaluated the performance of FiD when decreasing the amount of passages provided to the reader, and found that the performance of the model drops as the number of input passages decreases. However, when excluding the bottom
We propose novel approach for improving the efficiency of FiD, with a combined ap- proach of Token Filtering and decoder layer reduction, which removes tokens and irrele- vant layers during the generation process of every token for long-form answers.

---

100
2022; Hofstätter et al., 2022).
of Decoder Runtime
90
3.2
Cross-Attention Scores Analysis
80
70
An additional bottleneck affecting the efficiency of FiD is the extended sequence created by concatenat- ing input passages, which the decoder focuses on during generation. Assuming the reader is supplied with an excessive amount of passages, our objec- tive to assess the importance of the input token representations. Essentially, our primary research question pertains to filtering out uninformative to- kens that have no impact on answer generation, without compromising performance. Inspired by previous works that have assessed the relevance of input decoders, focus the cross- Pross-attention scores. These scores have been recently demon- strated to serve metric of importance for the input token representations, particularly rela- tion to their impact on the accuracy of the answer (Caciularu et al., 2021; Izacard and Grave, 2021a; Izacard et al., 2022).
60
20
50
40
40
60
+ 80
30
100
125 Answer Length (in tokens)
50
75
100
150
The percentage of the time (latency) of the de- coder from the overall end-to-end latency, as a function of the number of generated tokens. Each color repre- sents different amounts of input passages to the reader.
Figure 2:
50% of the passages, the performance only drops by approximately 2%.
Naturally, the FiD latency could be reduced if we provide less input passages the reader. How- ever, it is unclear how much time is utilized by each of its sub-modules. de Jong et al. (2022) included a preliminary profiling of the execution time for the reader module of FiD. They show that even though the encoder is more expensive in terms of FLOPS computation, the decoder is more expen- sive in terms of actual latency.
In order to investigate the utility of cross- attention scores as meaningful indicator, we aim to verify their ability to focus on the important information within the input text. To accomplish this, we include the gold passage in a list of 100 retrieved passages (given specific question). To simplify the analysis, we position the gold passage rank the input matrix of the decoder's cross- attention matrix does not inherently incorporate any notion of order.
Thus, we undertake an additional analysis, to comprehend how the time (latency) is distributed between the FiD encoder and the decoder modules, depending on the number of input passages and the amount of generated tokens. Our findings are illustrated in Figure 2. We artificially modify Fil generate up fixed number of tokens. We observe that feeding greater number of passages results higher latency values for the encoder. However, as more output tokens are being gener- ated, the share of the decoder of the total run-time significantly increases. Particularly, in the cases that the answer is long, we observe that regard- less of the input number of passages provided to the reader, the majority of the time is spent in the decoder. Intriguingly, the ratio rapidly converges towards 100%, exceeding 50% after only 15 to- kens.
In order examine the input token scores throughout the entire generation process, cal- culate the average cross-attention scores for each decoder layer and every generated token index. With the aim of identifying and filtering out irrele- vant tokens, we select the top p% of tokens with the highest cross-attention scores and compute the pro- portion of the tokens that originate from the gold passage. Figure 3a demonstrates the outcomes of our investigation, where we selected p 10% of the input tokens. This analysis performed on of 1000 queries taken from the development set of ELI5, employing the FiD-Base model.
We observe that the decoder's initial layers (2nd and 3rd) exhibit the greatest proportion of tokens derived from the gold passage. This implies that these layers should be employed for calculating the input relevance scores. Additionally, we have no- ticed that, in most layers, the ratio reaches its peak around the th generated token and subsequently
Overall, in the specific case of long-answer tasks such LFQA, we can conclude that the decoder serves as the primary source of latency and com- putational load during inference. This finding is further supported by similar works (de Jong et al.,

---

3
3-
% of Gold from Chosen
2
% from Chosen Tokens
2.8
2.5
2.6
6
20
12
50
2.4
2-
2.2
100
1.5-
2
1.8
1.6
100
150
100
150
Answer Length (in tokens)
Answer Length (in tokens)
The percentage of gold tokens for several chosen decoder layers.
(b) The distribution of passages the chosen tokens, in the 2nd layer.
Figure 3: Cross attention score analysis when choosing p = 10% of the tokens, as a function of the generated answer length. Left: The ratio of tokens that were chosen from the gold passage, per decoder layer (1-12). Right: The percentage of tokens that were chosen from each passage (1-100). The gold passage (labeled as 1) is colored red.
declines during the generation process. This in- dicates that it is most advantageous utilize the cross-attention scores in the early stages of genera- tion.
4 Method
Following our analysis in Section 3.2, we turn to implementing method for filtering out the redun- dant information during the decoding stage. We aim to find subset of the input tokens that is the most relevant for generating the correct answer. As pointed out previously,
Next, proceed to examine the extent which the model attends tokens from the gold passage compared to other less informative tokens. The findings of this analysis are presented in Figure 3b, where we illustrate the number of selected tokens taken from each input passage specifically at the second layer. Our observations consistently indi- cate that the gold passage consistently maintains a notably higher proportion of selected tokens com- pared any other passage, throughout the entirety of the generation process. Conversely, most of the passages exhibit ratios that are lower than what would be expected from a uniform distribution. Interestingly, we also note that the top passages exhibit higher ratios compared to the bottom ones.
can utilize the cross- attention scores computed between the generated tokens and the passages as basic signal for filter- ing out irrelevant tokens, similarly to Goyal et al. (2020).
Thus, we suggest token filtering approach, ing the cross- cuts-attention scores computed at prede- termined layer and generated token index during inference. At that point, for each input token, we compute the average attention scores over all at- tention heads, similarly to Caciularu et al. (2021); Izacard and Grave (2021a). Once these scores are computed, we keep the top k6-scored input tokens, which will be the only tokens to be considered to- wards the next tokens' predictions. Formally, the cross-attention scores per input token are defined as follows:
In summary, we have demonstrated that the cross-attention scores possess the capability to pri- oritize the most pertinent information in the input, making them a reliable mechanism for selecting informative input tokens. Moreover, we have iden- tified the optimal layers and ranges of generated token indices generate these scores, ensuring the selection of the most appropriate input tokens. For a comprehensive examination of the cross-attention patterns, we encourage readers to refer to Appendix A for further details.
St, = Σ A
(1)
where / is the generated token index, / is the layer index, is the number of attention heads, and A represents the cross-attention scores index f, layer and head
We perform a descending argsort operation on

---

the scores above, and take the top p% from the sorted input token indices. Hence, we denote T as the total number of input tokens from all the passages, and T' as the amount of tokens we keep after filtering, which is p% from T:
(Campos et al., 2016) A collection of crowd sourced responses Bing queries. We use the Passage Ranking track, which consists of human generated natural and complete answers.
MS MARCO
NaturalQuestions (NQ) (Kwiatkowski et al., 2019) A large-scale dataset by Google designed for natural language understanding and question answering research, consisting of real user queries from Google Search paired with corresponding Wikipedia passages.
Sortedy, = argsort(SI.)
(2)
Top..1 = (Sorted,,
where [i] indicates accessing the vector at in- dex i. Finally, we keep only the tokens chosen in T T from the cross-attention past key-values states K Kpast, Vpast:
For all datasets, we use the validation as the test set and a subset of the training set for validation, as done by Lee et al. (2019). We note that EL15 is part of the KILT Benchmark?, and thus is additionally evaluated on held-out test set. We use the gold passages the answers in MS MARCO and NQ For full specification of the dataset sizes, we refer to Table 5 in the Appendix.
past = past [Tpp, Vpast
Vpast [T [Topt,]
(3)
where A[ selects the elements from whose indices appear in B. These new past key-value states are the only ones used for generating all subsequent tokens.
5.2 Baseline Readers
We specify the hyperparameters used for training on the various datasets in Table in the Appendix.
Since the filtering percentage, token index and layer can effect the quality of this mechanism, as inspected in Section 3.2, we obtain the optimal values for them by performing a hyperparameter- like search over their possible values, which is de- scribed in Section 5.4. We produce new past key and value representations for the input sequence (across all the decoder layers), containing only the selected tokens, resulting in more compact ten- sor attend to. We name the filtering mechanism Token Filtering, with an overview of the approach presented in Figure 1.
We base our models on the FiD generative reader (Izacard and Grave, 2021c), which uses pre- trained T5 models (Wolf et al., 2019). We used the official implementation" of FiD throughout all our experiments.
FiD
CALM While our Token Filtering approach pri- marily focuses on eliminating redundant input to- kens, it does not decrease the number of decoder layers responsible for processing them. To tackle this concern, also incorporate recent effec- tive early exiting method for the decoder mod- ule (Schuster al., 2022), known CALM. We thus implement CALM and compare it our method (see Schuster et al. (2022) for more in- formation on the training scheme employed to train the FiD model, including the confidence classifier stage). In addition to independently evaluating CALM, we combine it together with our Token Fil- tering approach, resulting in combined approach referred to Combined.
Note that we remove the irrelevant tokens from the keys and values of the encoder output sequence during inference time only once, hence reducing their dynamic dimension during computation for all the subsequent tokens. For additional details about the cross-attention scoring computation refer the reader Appendix B.
5 Experimental Setup
5.1 Datasets
Our experiments are conducted on commonly used datasets for LFQA.
5.3 Implementation Details
Retrieval We first create index for retrieval over Wikipedia dump", comprised of multiple
(Fan et al., 2019) A dataset created from Reddit forum named "Explain Like I'm Five". We use the train, validation and test sets as provided by the KILT benchmark (Petroni et al., 2020).
ELIS
1 http://tto.cccaaccooooreeeeee/e/ httpps/s//nggggnne..ooccaaaassss co/datasets/kilt.
wikipedia

---

passages. For all the evaluated datasets, we retrieve
100 passages for each question from the index, us-
Smoothing
ing combination of dense and sparse passage
Performance
Y3
rankers. We refer the reader to Appendix C for more details regarding the retrieval process.
Hardware We used 8 24GB NVIDIA RTX3090 for
training base-sized models, and 8 40GB A100 GPUs for training large-sized models. For inference and latency measurements we used a single accelerator.
X1
X2
X3
X4
X5
X6
Inference setup Throughout our latency mea- surements, we used batch size and averaged the latency over all queries. Decoding is done us- ing beam-search with 4 beams, and similarly as (Su et al., 2022) we limit the generated answer length to 300 tokens. We also control the minimal answer length per dataset, which specify in Table 4 in the Appendix.
Latency
Figure 4: For Performance (y-axis) vs. Latency (x-axis). we divide the x-axis into 5 intervals over all hyperpa- rameter combination results the development set, rep- resented as yellow dots. For each interval, we choose combination with the best performance (i.e. y-axis value), thus forming the Max Curve in blue, with its smoothed version in green.
5.4 Performance vs. Efficiency Evaluation Process
curve, we find the best hyperparameters for each method per interval, by selecting the hyperparame- ters of the representative point in the current inter- val. Finally, we take the best setting per interval, and run each one on the test set. For each of our methods, we produce smoothed version, the results not necessarily monotonically increas- ing, which is shown in Figure 4 as the green line. These smoothed curves are the ones showcased in the final results in Figure 5.
We use KILT's implementation of ROUGE-L and F1 for performance measurements.. We measure efficiency as end-to-end latency in seconds for generating an answer to a question. To evaluate each method, on each dataset, we focus on the per- formance vs. efficiency trade-off, particularly on ROUGE-L vs. latency. For the evaluation of a given method (for example Token Filtering), we perform hyperparameter search over multiple combinations on the development set, and end up with collection of 2-dimensional points, where the x-axis is the latency, and the y-axis the per- formance (ROUGE-L.). Each measurement is averaged across all questions the development set. Then, we take the maximum and minimum over the observed values of the x-axis, and divide the resulting range into equally-sized intervals. In our experiments, we use 30 distinct in- tervals. For each interval, find it's representative point, by taking the point with the maximum y-axis value from all the combinations in interval. Once all such points are determined per interval, they form a curve, which we name as the Max Curve of the method. We visualize the process in Figure 4, where the line in blue is the Max Curve.
When performing the search over the hyperpa- rameter space, we used grid search, with the hyper- parameters and their value ranges being specified in Table 7 in the Appendix. Other methods (such as random search, Bayesian Optimization (Snoek al., 2012), etc.) may be attempted just as well in futurr works.
6 Experimental Results
6.1 Main Trade-off Comparison
In Figure 5, we showcase the performance vs. effi- ciency trade-off in terms of ROUGE-L and latency on the test set of each dataset. These results are the ones obtained after performing the hyperparam- eter optimization procedure stated described Sec- tion 5.4. The methods shown are the standard FiD model, CALM, Token Filtering, and Combined. For the base-sized models (Figures 5a, 5b, 5c), we can observe all methods improve upon the base- line model, each one a different aspect. For CALM, the method is able to reach lower latency
Thus, for method to be better than another, the Max Curve for it should be above and the left of the curve of the other, meaning that reaches equivalent results for less resources. Using the

---

25
32
26.5
24.
31
Rouge-t
26
7%
23
30
2%
25
22
28
27
24.5
21-
26
24
25
2
2.5
3.5
4.5
5.5
0.6
0.8
1.2
1.4
1.6
1.8
2.2
1.5
2
3.5 Labency (seconds)
2.5
4
(seconds)
Latency (seconds)
(a) BL15 Base
(b) MARCO- Base
(c) NQ Base
26.5
31
26
Rouge-L
246
25.5
29
28
FiD
24.5
27
Token Filtering Combined
24
26
10
4 Latency (seconds)
10
12
Latency (seconds)
Latency (seconds)
(d) ELI5 Large
(c) MS MARCO Large
(f) NQ Large
Figure 5: ROUGE-L Performance results of the different methods on the test sets, plotted as smoothed Max Curves, function of latency (seconds), for Base (top) and Large (bottom) models. Overall, combined approach is able to reach a better trade-off than the regular FiD model, for most cases.
values, due skipping the redundant layer compu- tations. In the case of Token Filtering, it is also able to preserve and at times improve the performance of the model overall, while the latency improve- ment remains limited, since is still computing the remaining tokens across all decoder layers. The performance improvement is presumably due to the redundant tokens being removed early on during the generation process, hence allowing the model better attend the salient information in the
and FLOPS (MACs) analysis. For the large-sized models (Figures 5d, 5e, and 5f), observe similar patterns to those the base-sized variations, with the latency values being significantly larger. We note that the overall performance values for these models are not substantially different than those produced by the smaller versions, hence we do not focus as much on them.
6.2 Performance Comparison
input.
To asses the performance our Combined method further, we choose the best performing hyperpa- rameter setting for the FiD-Base model, and report the test set results for each dataset, with Table showing the results, compared approaches sug- gested in Su al. (2022). In particular, com- pare their implementation of FiD (named RBG FID) and their suggested system (named RBG), with the results of both taken from their published work. We note that both our models and the RBG models are of the same size. In our experiments, we denote the original FiD model we trained as FiD (ours). the FiD model with the Token Filter- ing method as FiD TF, and the Combined method as FiD Comb. On both datasets, FiD (ours), FiD TF and FiD Comb achieve state-of-the-art results, with Combined reaching the best overall perfor- mance terms ROUGE-L and Fl (aside from MS MARCO FI, where our approach is second).
When combining both methods, the performance enhancement of the Token Filtering and the latency reduction of CALM produce better curve than either method alone. In addition, we showcase the drop in 2% performance per dataset, show- ing that our method is able reduce the latency significantly more than the regular FiD, with the best reduction reached on the MS MARCO dataset for FiD- Base, saving 62.2% of the latency. In the NQ dataset however, for both the base-sized and large-sized models, while the CALM method does achieve proper latency reduction, the Token Filter- ing does not effect the results significaatly. Since we focus real-world scenarios, showcase the trade-off with the actual latency, instead of measurements such as FLOPS (MACs), as done by previous works (de Jong et al., 2022). For those, we refer to Figure 6 in the Appendix for the trade-off

---

BL15
MS MARCO
Model
ROUGE-L
F1
R-L
F1
R-L
F1
Krishna et al. (2021a)
23.36
23.14
RBG FiD
25.70
28.55
24.64
27.08
RBG
24.53
27.13
RBG
26.46
29.04
24.72
27.52
FiD TF
25.52
28.49
FiD (ours)
26.24
31.46
24.33
27.11
FiD Comb
25.61
29.99
FiD TF
26.65
30.32
24.75
27.26
FiD Comb 26.97
31.76
25.11
27.41
Table 3:
The published results on the official ELI5 test set, provided by the KILT leaderboard. FiD Comb runs with the same setting as best performing combination of the combined approach in Figure 5a.
Table I: A comparison of the performance of our model, in comparison with the RBG model, where FiD TF stands for FiD with Token Filtering. Highest perfor- mance values are marked in bold, where i-L ROUGE- We note that results for RBG's models were taken directly from their published paper.
achieve substantially better results (Izacard and Grave, 2021b). Subsequent works (Su et al., 2022; Krishna et al., 2021b) have adapted these genera- tive approaches to produce long-form answers well.
Model
EL15
MS MARCO
FiD (Ours)
83.68
85.07
FiD Comb
83.79
85.27
Encoder- Efficiency Due computa- tion bottlenecks in generative models, particularly in encoder-decoder models (Vaswani et al., 2017; Raffel et al., 2020), previous works attempt to mit- igate them. The encoder model can be used to filter out irrelevant passages during generation (Yu et al., 2021; de Jong et al., 2023). Nevertheless, the encoder's impact the model's latency in long- form scenarios is negligible, depicted in Figure 2. Consequently, our research centers on analyzing the computational aspects of the decoder instead.
Table 2: The BERTScore Fl results on the test set for both the original FiD model and our FiD Comb method, with each column indicating different dataset.
As an additional point of comparison, compute the BERTScore (Zhang et al., 2019) metric the original FiD model, and our FiD Comb method, and present our findings Table We observe that our model performs on par and even bit better overall than the original FiD implementation. We present a supplementary comparison of some of the answers generated by our method for the EL15 test set in Table 8 in the Appendix. In addition, at the time of writing this paper, Combined is ranked the #1 position in terms of ROUGE-L and Fl on the EL15 KILT leaderboard, with Table 3 containing all leaderboard results?.
In particular, the decoder model utilizes many redundant and heavy cross-attention operations, which can be removed replaced with simpler alternatives (de Jong et al., 2022; Ainslie et al., 2023).
Since encoder-decoder models perform compute heavy operations in multiple layers, previous works have proposed stopping the layer propagation dy- namically by assessing the model's confidence for prediction at certain layer (Teerapittayanon al., 2016; Schwartz et al., 2020). Other works have adapted this mechanism to decoder models as well (Schuster et al., 2022: Elbayad et al., 2019). How- ever, these studies fail to tackle the issue of input size during generation, thereby resulting in compu- tations being performed on irrelevant input to some degree, which we address through complementary token filtering.
7 Related Work
Open-Domain Question Answering Many pre- vious works utilized setups which are based on the retriever and the language model reader compo- nents (Chen et al., 2017; Guu et al., 2020; Lewis et al., 2020). The goal of the retriever is to fetch the most relevant passages to a given question (Karpukhin et al., 2020). The reader processes the question and the relevant passages to extract or generate the answer, with generative approaches
Data Reduction for an Efficient Computation The input to encoder models tends to become in- creasingly large, especially in ODQA settings with many input passages. Since not all spans of infor-
"Since the results in Table are on a hidden test set for the leaderboard, the results are different from those reported in Table 1.

---

mation are relevant to produce the correct answer, previous works propose eliminating the irrelevant tokens from the input to the encoder, by identify- ing the salient information during inference time (Goyal et al., 2020; Kim and Cho, 2020). Although these techniques effectively decrease computation at each layer, they are implemented in the encoder model rather than the decoder, which has been pre- viously determined have more significant in- fluence on latency. Our work leverages the cross- attention the decoder early on during generation, thus effectively filtering the input tokens.
Limitations
Regarding the retriever, mentioned in Section 5.2, we did not experiment with vast array of retrievers, due to the scope of the work being on the reader model.
Regarding the models for comparison, we pri- marily focused on the performance of the FiD model versus own approach, while testing them various datasets. Hence, we did not perform tensive reproductions of other methods, such other encoder-decoder models, but instead report their original results as they were published. We be- lieve that our results can be generalized to other architectures as well.
Qin and Durme (2023) suggested transform- ing language into representation, by selecting a dynamically determined subset of input tokens, with these "nuggets" being acquired through tasks such as machine translation. However, our method doesn't 't incorporate any learning, focusing on ama- lyzing the necessary input tokens for direct decod- ing instead.
In our hyperparameter search, chose a sub- space of all the possible values each parameter has, due limited amount of computation available. Our approximation of the space covers the main areas of relevance for our purposes.
Wingate al. (2022); Mu et al. (2023) proposed prompt compression techniques for minimizing the amount token vectors required to represent the same text. We note that our work does not discuss such aspects, given the context of questions and passages in our inputs.
References
Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontan' on, Siddhartha Brahma, Yury Zemlyanskiy, David C. Uthus, Mandy Guo, James LhorTh Yi Tay, Yun-Hsuan Sung, and Sumit K. Sanghai. 2023. Colt5: Faster long-range transformers with conditional computation. ArXiv, abs/2303.09755 099552.
8 Conclusions
Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew Pe- ters, Arie Cattan, and Ido Dagan. 2021. CDLM: Findings
Cross-document language modeling. of the Association for Computational Linguistics: EMNLP 2021, pages 2648 2662. Punta Cana, Do- minican Republic. Association for Computational Linguistics.
We analyze the precise performance vs. efficiency trade-off of the FiD's encoder and decoder in long-form settings, with an analysis the cross- attention operation the decoder model. We show that the decoder has more impact on the latency. particularly for long outputs, and that the decoder attends to more salient information early during generation. Hence, our proposed approach for ef- ficiency reduction, namely a combined approach of Token Filtering and CALM, removes irrelevant layers and tokens during the generation process, for every token produced. Our approach achieves a significant reduction in resources (up to 62.2%) while not sacrificing more than 2% of the perfor- mance, and is the current mate-of-the-an on the EL15 KILT leaderboard. Future work can further develop a more dynamic method for choosing the most relevant tokens from the input, instead of using predetermined hyperparameters, and train the Pross-attention patterns to better attend to the salient information during generation.
Daniel Fernando Campos, Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, Deng. and Bhaskar Mitra. 2016. Ms marco: A human generated machine reading compre- hension dataset. ArXiv, abs/1611 1.09268.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia answer open- domain questions. In ACL.
Michiel de Jong, Yury Zemlyanskiy, Joshua Ainslie, Nicholas FitzGerald, Sumit K. Sanghai, Fei Sha, and William Cohen. 2022. Fido: Fusion-in-decoder opti- mized for stronger performance and faster inference. ArXiv, abs/2212.08155.
Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGer- ald, Sumit Sanghai, William W. Cohen, and Joshua Ainslie. 2023. Glimmer: generalized inteiinteraction memory reranker.

---

Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. 2019. Depth-adaptive transformer. ArXiv, (bs//190.100033
Kalpesh Krishna, Aurko Roy, and Mohit lyyer. 2021b. Hurdles to progress in long- form question answering. In North American Chapter of the Association for Computational Linguistics.
Angela Fan, Yacine Jemite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. Eli5: Long form question answering. ArXiv, abs/1907.00 19900.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wo Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe V. Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics, 7:453 466.
Saurabh Goyal, Anamitra R. Choudhury, Saurabh Raje. Venkatesan T. Chakaravarthy, Yogish Sabharwal, and Ashish Verma. 2020. Power-hert: Accelerating bert inference via progressive vect-vector elimination. In International Conference on Machine Learning.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Ming-War Wei Chang. 2020. Realm: Retrieval- augmented language model pre-training. ArXiv, abs/2002.08909.
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. ArXiv, 0tss/196.0003
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Kuttler, Mike Lewis, Wen tau Yih, Tim Rock- täschel, Sebastian Riedel, and Douwe Kiela. 2020. -aggeeedl-ugmmented generation for knowledge- intensive nlp tasks. ArXiv, abs/2005. 1101..
Sebastian Hofstätter, Jiecao Chen, Karthik Raman, and Hamed Zamani. 2022. Fid-light: Efficient and ef- fective eetrieaal-ugmmeneed text generation. ArXiv, abs/2209.14299
Gautier Izacard and Edouard Grave. 2021a. Distilling knowledge from reader to retriever for question an- swering. ArXiv, abs/2012.04584.
Jesse MU, Xiang Lisa Li, and Noah D. Goodman. 2023. Learning to compress prompts with gist to- kens. ArXiv, abs/2304.00867.
Gautier Izacard and Edouard Grave. 2021b. Leveraging passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume, pages 874-880, Online. Association for Computa- tional Linguistics.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vassilis Plachouras, Tim Rocktaschel, and Sebastian Riedel. 2020. Kilt: a benchmark for knowledge intensive language tasks. In North Amer- ican Chapter of the Association for Computational Linguistics.
Gautier Izacard and Edouard Grave. 2021c. Leveraging passage retrieval with generative models for open domain question answering. In EACL.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 2523-254h Online. Association for Computational Linguistics.
Gautier Izacard, Patrick Lewis. Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane A. Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with retrieval augmented language models. ArXiv, abs/2208.03299.
Vladimir Karpukhin, Barlas Oguz. Sewon Min, Patrick Lewis, Ledell Yu Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020. Dense passage re- trieval for open-domain question answering. ArXiv, abs/2004.04906.
Guanghui Qin and Benjamin Van Durme. 2023. Nugget: Neural agglomerative embeddings of text. In Inter- national Conference on Machine Learning.
Colin Raffel, Noam M. Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu, 2020. Exploring the limits of transfer learning with unified text-to-text transformer. ArXiv, abs/ htss/191..100 10683.
Gyuwan Kim and Kyunghyun Cho. 2020. Length- adaptive transformer: Train once with length drop, anytime with search. In Annual Meeting of the Association for Computational Linguistics.
Kalpesh Krishna, Aurko Roy, and Mohit lyyer. 2021a la. Hurdles to progress in long-form question answering. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 3940-4957) Online, Association for Computa- tional Linguistics.
Nils Reimers and Iryna Gurevych. 2019, Sentence-bert: Sentence embeddings using siamese bert-networks. ArXiv, abs/1198..1084.
Devendra Singh Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping, William L. Hamilton. and Bryan Catanzaro. 2021. End-to-end training of

---

neural retrievers for open-domain question answering. In Annual Meeting of the Association for Computa- tional Linguistics.
A Cross Attention Pattern Analysis
In this section, we continue our discussion from section 3.2, regarding the analysis of the cross- attention scores.
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghami, Dara Bahri, Vinh Quang Tran, Yi Tay, and Donald Metzler. 2022. Confident adaptive language model- ing. ArXiv, abs/2207.0706
In Figure 7, present multiple versions of the plot in Figure 3a, with the rows indicating the dataset (ELI5, MS MARCO, NQ), and the columns representing a different percent of chosen tokens (10, 30, 50). For MS MARCO and NQ in 10%, the percentage of the gold passage tokens remains high for the lower layers, starting from token 10. The other layers do not reach the same percentage and degrade during the generation process. When increasing the percentage 30, and later 50, the percentage of the gold passage is getting reduced substantially.
Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, Jesse Dodge, and Noah A. Smith. 2020. The right tool for the job: Matching model and instance complexities. In Annual Meeting of the Association for Computational Linguistics.
Jasper Snoek, H. Larochelle, and Ryan Adams. 2012. Practical bayesian optimization of machine learning algorithms. ArXiv, abs/1206.22 120..444...44
Dan Su, Xiaoguang Li, Jindi Zhang, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. 2022. Read before generate! faithful long form question answering with machine reading. ArXiv, 003.0233...
In Figure 8, we showcase an extended version of Figure 3b, for the various datasets and chosen token percentages in Figure 7, with the rows and columns being similarly organized. For 10%, the gold passage gets the most tokens out of all the rest, for all datasets, with the lower passages getting less than 1%. However, for 30%, the gold passage is no longer the highest ranking for some of the datasets (MS MARCO, NQ), with the upper passages reaching higher, and the lower ones still being at the 1% mark. At 50%, the gold passage is no longer the most prominent, with it being nearly as insignificant as the lower passages the case of MSMARCO. This suggests that increasing the percentage of tokens taken introduces unnecessary noise to the selected tokens. thus forcing the model to receive input from lower ranked passages. For MS MARCO and NQ in 10%, the percentage of the gold passage tokens remains high for the lower layers, starting from token 10. While the results above were done using an FiD-Base model, similar patterns are present for FiD-Large models through all previously discussed aspects.
Surat Teerapittayanon, Bradley McDanel, and H. T. Kung. 2016. Branchynet: Fast inference via early exiting from deep neural networks. 2016 23rd Inter- national Conference on Putlern Recognition (ICPR), pages 2464-244444 2469.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Ellen M. Voorhees. 1999. The trec-8 question answer- ing track report. In Text Retrieval Conference.
David Wingate, Mohammad Shoeybi, and Taylor Sorensen. 2022. Prompt compression and contrastive conditioning for controllability and toxicity reduc- tion in language models. In Conference on Empirical Methods in Natural Language Processing.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistae, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's 's transformers: State-of-the-an natural language processing, ArXiv, abs/1910.03771.
Donghan Yu, Chenguang Zhu, Yuwei Fang, W. Yu. Shuohang Wang, Yichong Xu, Xiang Ren, Yim- ing Yang, and Michael Zeng. 2021. Kg-fid: Infus- ing knowledge graph in ausion-in-decoder for open- domain question answering. ArXiv, abs/21 10.04330.
B Attention Score Computation Extensions
In addition to the methods introduced in 4, the computation of the cross-attention scores can be further altered in a few key areas, which we tackle as well.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. ArXiv, abs/1904.09675.
Value Normalization. As mentioned in Izacard et al. (2022), the scores can benefit from scaling by the la normalized values tensor V. Thus, we can instead transform AL, into:

---

We note that while Schuster al. (2022) utilized complex threshold calibration system, instead showcase the effect of the various thresholding set- tings, once applied to the decoder model.
Al, 1r
(4)
where [n] is the nM row, in this case the n to- ken, and Un = [|V[n]]] is the norm of the th row (token) in V. Hence, we apply this normalization to the attention scoring operation.
For the Token Filtering, since we are discussing mainly Sncoder-Decodee architectures, apply the filtering by removing the redundant tokens from the past key and value states for the cross-attention. In addition, we also remove said tokens from the encoder hidden states, encoder attention mask, and the encoder-decoder position bias.
Mean over all decoder layers. Instead of taking the representation of the current decoder layer only, every layer before
we instead take the average the current one. Thus, we compute the attention scores I for the input tokens follows:
E Σ Rir S₁ th VE(LL))E[[L.
(5)
Dataset
Len. Chosen
ELI5
150
From our preliminary analysis, this mean op- eration does not effect the quality of the filtering method, and hence is not applied.
MS MARCO
50
NQ
50
Table 4: The chosen minimum answer length for during evaluation.
C Retrieval Details
Since our method primarily focuses on the reader model, we have implemented generalized ap- proach for creating ranked passage lists. Our doc- ument corpus is taken from a Wikipedia Dump, which has been split into 100-word-long passages, done in Karpukhin et al. (2020), including the then stored in
Dataset
Train
Dev
Test
KILT
ELI5
272634
3000
1507
600
article title. These documents Elasticsearch index. Given question from dataset, we use BM25 over the passage index, to retrieve 250 documents. Then, we re-rank the pas- sages using a sentence transformer' (Reimers and Gurevych, 2019) model that was finetuned on the dataset, and keep only the top 100 ranked docu- ments.
MS MARCO NQ
498000
3000
6980
55622
3000
6489
Table Sizes of the datasets, per train, dev and test respectively. We include the size of the KILT test set size, which we evaluate on separately.
D Method Implementation Details
For the CALM, we utilize beam search for long sequence generation. In the beam-search setting, we use ny beams, there which causes the issue of how to allow some tokens to cease computation at a certain level, while allowing the others to continue computation. For the scope of our work, we apply the hard assumption that the confidence value is the lowest one from all beams, hence exiting only if all the tokens in the beams have satisfied the iting condition. Formally, given confidence scores q = (et & ..., ci") at layer the confidence value used the layer will thus be a minje(l.ng) d.
Parameter
Value
Base Model
T5-Base, T5 Large
Optimizer
Adam
Max Seq, Length
235
LR
5c-5
LR Scheduler
Linear
Weight Decay
0.01
Precision
torch.bfloat1
Batch Size
64
Training Steps
60000
Warmup Steps
1000
Table 6: The training parameters used for training the FiD model on each dataset.
'https://www www.ww...wasccccooooo "multi bas

---

Parameter
Ranges
General Parameters
Input Psg.
[5, 100]
CALM
Confidence Thresh- [0.2, 0.9]
old
Threshold Coef.
{0.5, 0.7, 0.9)
Threshold Decay
{3,4, 5)
Token Filtering
% of Input Filtering Token
{10, 30, 50)
[1,20]
Filtering Layer [1,L]
Table 7:
The hyperparameter search space for the CALM and Token Filtering methods, including param- eters for all methods (Input Psg.), and the combined approach. {} indicate a set of possible values, while /min, max] correspond to range values from min max. As in the paper, L is the number of layers in the decoder.

---

27-
25.
32
26.5
24
31
30
Rouge-L
26
23
25.5
2%
22
27
25
21
26
20
25
1000
2000 FLOPS (MACs)
3000
1000
2000
3000
4000
1000
2000
3000
4000
FLOPS (MACs)
FLOPS (MACs)
(a) ELI5
(b) MS MARCO
(c) NQ
25
32
24
Rouget
23
29
256
25.5
2%
22
28
27
25
21
RID Token Filtering Combined
24.5-
20
20
40
80
100
20
40
60
80
100
20
60
80
100
Input Psgs
Input Psgs
Input Psgs
(d) ELI5
MS MARCO
NQ
Figure 6: The ROUGE-L performance on FiD- Base vs. the FLOPS (MACs) (First row), and vs. the input passage amount (Input Psg., second row). The results on the test for each dataset, for the various methods utilized. We observe that the trends of the FLOPS (MACs) and the Input Psg. very similar, since the passages effect the encoder the most, and the encoder has the most impact on FLOPS (MACs) (as stated by de Jong al. (2022)).

---

Question
Answers
How you know something done/said by Anonymous, when they are, by definition, anonymous? Can anyone stuif/say things on behalf of Abobynous, or do they somchow have recognized leadership who are not anonymous to each other/their mem- bers?
Any unknown person can do or say things the name of Anonymous. That's why you basically have to ignore everything Anonymous says they will do, and only look at what they actually do. There's no leadership. but there are believe certain core groups who know one another by persistent pseudonyms. and may possibly know one another afk. They are prohably disproportionatll influcntial, that they run widely-subseree role in getting messages out to the rest of Anonymous, but they aren't 't leadership in any classical sense of the word.
Gold:
youtubehwitterrwwee hatever accounts which play a significant
FiD (ours): I'm not sure what you mean by "anonymous". Anonymous is a group of people who are working together to solve problems. They aren't 't anonymous to each other, they're re just like any other group. There's 's no way to know if something is done by Anonymous or not, but you can be sure that it is. You can also be certain that someone is doing something that is not Anonymous, and that they are doing it for the benefit of others. If you's 'e ever heard of someone saying something like "I am an Anonymous member, I am not an anonymous member" then you know that you are an "Anon" member. Then you have to be able to prove that the person who made the post was Anonymous.
FiD Comb: I' I'm not sure what you mean by "anonymous" by definition. Anonymous is a loose hacktivist collective. They don't 't have an official leadership. but they do have some sort of "leadership" in place. _UR_____ It's not that they 'r able to do anything, they just use the name "Anon" to refer to the members of the group. and they can be viewed as "a group of people who believe in the idea of anonymity' or something like that. There is way to tell if someone is anonymous group. It possible for somcone to be identified own leader.
Anonymous member, even have their
why is cranberry juice good for your
Gold: It isn't, particularly. Some people use to think that the acid in cranberries stopped bacteria growing but thats been shown to be wrong. There is some vague evidence that cranberry juice can help "prevent* UTIs, though. ItS thought that cranberries might contain a chemical that stops bacteria sticking to the cells lining the walls of the urinary tract, but no-one knows for sure. Once you have a UTI, though, no juice will cure it. The real reason that cranberry juice. lemon barley water etc are good for your kidneys is that you need to flush them out. Drink at least 8 glasses of water a day (2-3 litres of fluid daily). THAT is what is good for your kidneys. My daughter gets LOT of UTIs, even with prophylactic antibiotics, and several members of my family have died from kidney discase. All ever hear from well-inter well-intentioned friends is "have you tried cranberry and/or barley water?". Sigh. FiD (ours): It isn't. There is evidence that cranberry juice is good for your kidneys. It's just that there is not enough evidence to suggest that it good thing for you. _URL____ 0_ TL:DR- if you have an infection. you should not drink it. If you're on dialysis, it lead urinary tract infections. which can be fatal. The reason for this is because the bacteria that cause the infection can get stuck in your urine, causing the urine become inflamed and irritated, leading pyuria and kidney stones. This why people who are in the throes of kidney problems should drink more than they normally do.
kidneys?
FiD Comb: isn't. There's no scientific evidence that cranberry juice is good your kidneys. However, there are studies that suggest that may be beneficial you if you have urinary tract infection. _URL__ TL.DR: Cranberries are incredibly acidic, so they can be sour and tart, which is why they' 're bad for the kidney. also has some anti-coagulants in that can help prevent the formation of bacterial plaques the urine, reducing the amount of urine that excreted from the bloodstream, and preventing the clotting of the bladder. This is also why some people who have kidney stones are more likely to get kidney stone formation.
Table 8: ELI5 test set answers from the standard FiD model, FiD (ours), and our method (FiD Comb), with the Gold Answer as a reference.

---

1.14-
of Gold from Chosen
% of Gold from Chosen
of Gold from Chosen
1.6
1.12-
2.8-
1.55
1.1
2.4
1.45-
1.00-
1.4
1.04
1.6
150
Answer Length (in tokens]
Answer Length (in tokens)
Answer Length (in tokens)
(a) ELIS 10%
(b) ELI5
30%
(c) ELI5 50%
3.4-
1.4
of from Chosen
%% of from Chosen
Gfld from Chosen
1.35
0.9
0.89-
2.8
12
1.25
0.87
0.85
1.2
0.85
1.15
0.84
0.83
20
50
Length tokens)
Length
Length
tokens)
(d) MS MARCO p= 10%
(c) MS MARCO p 30%
(f) MS MARCO
50%
% of Gold from Chosen
% of Gold from Chosen
% of Gold from Chosen
1.36
8.5
1.0-
1.3
1.7-
1.28
1.25-
40 60
100
20
40
60
80
100
20
60
100
Answer Length (in tokens)
Length (In tokens)
Answer Length (in tokens)
NQ p 10%
(h) NQ 30%
NQ- p 50%
Figure 7: The ratio tokens that were chosen from the gold passage, per decoder layer (1-12), for FiD- Base models. Each row represents different dataset, and every column represents different filtering percentage (10,30,50).

---

from Chosen Takens
166
11
% from Chosen Tokens
1.15
-
10
- 20
20
from Chose
50
1.05-
60
100
1.1
100
0.9
0.0
50
100
150
100
150
50
100
150
Answer Length (in tokens)
Answer Length (in tokens)
Answer Length (in tokens)
(a) ELI5 10%
(b) ELI5 30%
(c) ELI5 p 50%
3.5-
% from Chosen Tokens
% from Chosen Tokens
% from Chosen Tolcens
2.5
20
50
50
50
00
1.5-
0.9
0.8
0.5
0.8
0.6
10
30
10
30
Answer Length tokens)
Answer Length tokens)
Answer Length tokens)
(d) MS MARCO 10%
(c) MS MARCO p == 30%
(f) MS MARCO = 50%
3.5.
26 from Chosen Tokenss
%6 from Chosen Tökens
% from Chosen Tokcnss
1.8
10
10
1.2
80
1.4-
3.1-
80
1.2
1.5
0.8
0.5
40
60
100
20
40
60
80
100
20 40
80
100
Answer Length tokens)
Answer Length tokens)
Answer Length (in tokens)
(g) IQ p 10%
NQ
30%
(i) NQ P 50%
Figure 8: The percentage of tokens that were chosen from each passage, for FiD-Base models. The gold passage (labeled colored red. Each row represents different dataset, and every column represents different filtering percentage (10.30,50).

---

1.14
16 of Gold from Chosen
of Gold from Chosen
% of Gold from Chosen
1.55
1.12-
1.45
1.8
1.06
150
Answer Length (in tokens)
Answer Length (in tokens)
Answer Length (in takens)
(a) ELI5 10%
(b) ELI5
30%
(c) ELI5 50%
3.2
0.92
of Gold from Chosen
% Gfld from Chosen
1.35
of Gold from from Chosen
0.9
1.25-
0.00-
2.2
0.86
2
1.8-
1.15
1.1
20
Length tokens)
Length
Length tokens)
(d) MS MARCO p= 10%
(c) MS MARCO p= 30%
(f) MS MARCO pp 50%
2.1-
% of Gold from Chosen
% of Gold from from Chosen
% of Cold from Chosen
1.35-
3.5-
2.5-
1.7-
1.25
1.5
20
40
60
80
100
120
20
40
60
80
100
120
20
40
60
80
100
120
Answer Length (in tokens)
Length (in tokens)
Answer Length (In tokens)
NQ
10%
(h) NQ
30%
NQ 50%
Figure The ratio of tokens that were chosen from the gold passage, per decoder layer (1-24), for FiD-Large models. Each row represents different dataset, and every column represents different filtering percentage (10.30,50).

---

% from Chasen Takens
2.2-
% from Chosen Tokeri
1.5
+-2
% from Chosen Takens
1.15
10
-0
3.8-
20
-0
++80
1.05
100
1.2
-100
1.2
0.95
0.9
100
150
100
150
100
150
Answer Length (in tokens)
Answer Length (in tokens)
Answer Length (in takens)
(a) ELI5 10%
(b) ELI5 30%
(c) EL.5 p 50%
% from Chosen Tokcess
-1
% from Chosen Tokens
1.25-
1:15-
50
50
from Chosen
1.1-
50
1.05
-100
2.5
0.95
0.9-
0.5
10
30
20
Answer Length tokens)
Answer Length tokens)
Answer Length (in tokens)
(d) MS MARCO 10%
(c) MS MARCO p 30%
(f) MS MARCO p 50%
4.5
% from Chosen Tolkems
% from Chosen Tökenns
% from Chosen Tokens
3.5-
10
2.5-
1.4
80
2-
2.5
0.9
0.8
0.5
0.8
20
40
60
60
100
120
20
40
60
80
100
120
20
60
80
100
120
Answer Length (in tokens)
Answer Length (in takens)
Answer Length (in takens)
(g) NQ p 10%
NQ
30%
p 50%
Figure 10: The percentage of tokens that were chosen from each passage, for FiD-Large models. The gold passage (labeled colored red. Each row represents different dataset, and every column represents different filtering percentage (10.30,50).


Reviews:
[{"review_id": "95gZSygtWV", "reviewer": "Reviewer_cWny", "summary": "", "questions": "Question A:\nDid you do any qualitative analysis (either using human evaluators or scoring such as BertScore) of what is the impact of your approach in generation output? ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort in providing constructive feedback.\n\nQuestion A:\n\nWhile we conducted qualitative assessments (by ourselves) to ensure the coherence and accuracy of the long-form generations, we regrettably couldn't incorporate these findings into the paper due to space constraints. In addition, it is less customary to perform human evaluation for long-form question answering tasks, due to the length of the answers and the complexity of the annotation itself. However, we do intend to address this by including relevant examples upon the paper's acceptance. It's worth mentioning that automatic evaluation metrics like BertScore might not perform effectively on extended generations, as they are primarily designed for sentences or concise paragraphs. \n\n \nWe provide the results of the BertScore F1 comparison:\n\n| Model\\BertScore F1 | ELI5  | MS MARCO |\n|--------------------|-------|----------|\n| FiD (Ours)         | 83.68 | 85.07    |\n| FiD Comb           | 83.79 | 85.27    |\n\n\nThus, we observe that our model performs on par and even a bit better overall than the original FiD implementation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Very well written paper and contains through analysis. Improvement in latency is quite significant. Results are also comparable to SOTA.", "reasons_to_reject": "No major reason to reject.", "questions_to_authors": "Question A:\nDid you do any qualitative analysis (either using human evaluators or scoring such as BertScore) of what is the impact of your approach in generation output? ", "paper_topic_and_main_contributions": "This paper presents an approach how to improve latency without impacting performance for Fusion-in-Decoder (FiD) model. More specifically, the authors tackled the heavy cross-attention computation in the decoder by introducing token filtering and using dynamic decoder layer skipping (aka CALM). They did thorough empirical analysis and showed that decoder serves as the primary source of latency and computational load during inference. They also observed that observe that the decoder’s initial layers (2nd and 3rd) contains majority of the tokens derived from the gold passage. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bUujvJu19r", "reviewer": "Reviewer_g1LA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort in providing constructive feedback.\n\nReason to reject (1):\n\nWe thank the reviewer for pointing out these relevant references. [1] delves into a learned token filtering component, so it's important to clarify that our method doesn't incorporate any learning. Instead, we analyze the necessary input tokens for direct decoding. Furthermore, our work didn't encompass prompting techniques akin to those in [2] and [3], given the context of questions and passages in our inputs. We will certainly incorporate a comprehensive discussion of these aspects, along with the mentioned references and others, into our paper.\n\nReason to reject (2):\n\nRanking and filtering of input tokens constitute a thoroughly explored and significant area of research. As far as we are aware of, we are pioneering the utilization of cross-attention scores for this objective, conducting an in-depth examination of their impacts on extended content generation. While ATLAS (Izacard et al. 2022) did utilize comparable signals, it's worth noting that their focus was on enhancing retriever document filtering, diverging from our approach that centers around reader token filtering.\n\nWe plan to open source our code to reproduce our evaluations and datasets upon acceptance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper starts with a bunch of analysis justifying the motivations of filtering unimportant information per layer. The analysis has some insights on how decoders use information, which could benefit the community in future design of decoding algorithms. The proposed method is well-motivated and reasonably backed with experiment results. ", "reasons_to_reject": "1. The paper should have discussed a bit more prior work on token selection/filtering, such as [1], also work on prompt compression[2][3] (some might be contemporaneous, but also have similar work). \n\n2. The proposed idea is a bit incremental since a similar filtering mechanism has been widely adopted in tasks such as coreference resolution, where the difference is the item being filtered, are they tokens or other embeddings (mentions). Therefore, the contribution is more considered as transferring the rank and filter method to apply on token-level (combined with early exiting).", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper optimizes retrieval-augmented language models, specifically the FiD model, to reduce decoding time in open domain question answering (ODQA) tasks. It proposes Token Filtering, a token-level elimination method, to remove irrelevant information from retrieved passages. The method reduces runtime by up to 62.2% with minimal performance loss or improvement. Evaluation is done on the Long-Form Question Answering task using the ELI5 dataset.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Some of them might be contemporaneous and just for the authors' references.\n\n[1] Guanghui Qin, Benjamin Van Durme. (2023). Nugget: Neural Agglomerative Embeddings of Text. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:28337-28350 Available from https://proceedings.mlr.press/v202/qin23a.html.\n\n[2] David Wingate, Mohammad Shoeybi, and Taylor Sorensen. 2022. Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5621–5634, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.\n\n[3] Jesse Mu, Xiang Lisa Li, and Noah Goodman. \"Learning to compress prompts with gist tokens.\" arXiv preprint arXiv:2304.08467 (2023).\n"}}, {"review_id": "tYCJ8EZDtZ", "reviewer": "Reviewer_jDvi", "summary": "", "questions": "1) What is the detailed formula of the attention-based score which is used for token filtering? \n2) In Table 1-2, what are the results of only applying token filtering, without combining with CALM? What does mean FiD(ours), which are the reproduced results of FiD without applying the token filtering? \n3) When the token filtering is done at the decoder layer l, its result affects on filtering tokens at layer l+1? That is, the tokens removed at the previous layer are NOT anymore used after next layers? \n4) The token filtering is done per each decoding timestep or per a sequence? \n5) Comparing to FiD-light, how much the proposed method saves the latency? \n6) Hyperparameters need to be chosen for each latency? Then, in total, because there are 30 distinct latent intervals, we need to keep 30 sets of hyperparameters for enduing a FiD model? \n7) Without Max Curve, the methods are not fairly compared? The curves in Figure 5 are those with Max Curve?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort in providing constructive feedback, and we address your concern and questions below.\n\n\n**R1**: Reason to reject (1):\n\nWe would like to provide a clearer presentation of the token elimination method. The token filtering is performed only once during the decoding process. Before inference, we specify a specific generated token index $t$ and a specific layer $l$. When the decoding process reaches token $t$ and layer $l$, token filtering is done by taking the mean cross-attention scores over input tokens. This creates new filtered cross attention past key values states. Once these key and value states are computed, they are the only ones used for the rest of the decoding process. A more detailed description, including formal equations of the token filtering operation, are provided in the answers to the questions below. \n\nTo control the latency of the model (as done in Figure 5), we first run a search over the hyperparameter space on the development set, and get a large amount of combinations, each with a different ROUGE-L and latency value. \n\nFor example, we can consider a scenario where we have 200 combination results, with each combination having a latency value ranging from 1 to 5 seconds. Thus, we divide these combinations into, for example, 5 groups, each for a different latency range. According to this number of groups, the first group will have combinations in the range of 1 to 2 seconds, the second one with 2 to 3 seconds, and so on.\n\nFor each group, we select the combination with the best ROUGE-L score, ending up with 5 combinations in total, as in Figure 4.\n\nThus, during inference, we can choose a combination according to the latency of its group. If we wish for low latency, we choose the combination of the first group of 1 to 2 seconds. If we wish for better quality results, while sacrificing latency, we can choose the combination of the group corresponding to 4 to 5 seconds, and so on. \n\nRegarding the top-k selection for FiD, we do indeed perform the suggested operation, and provide FiD with an increasing number of passages (20, 40, 60, 80, 100 passages). We show the results of this input passage amount selection in Figure 5 as red lines, under the name FiD.We will add these exact passage amounts for the standard FiD evaluation for the camera ready version. A more detailed explanation is provided in the answers to questions 6 and 7 below.\n\n\n**R2**: Reason to reject (2):\n\nAs mentioned in our response to Reason to reject 1 (R1), we perform the suggested top-k operation, and show the results of this passage amount selection in Figure 5 as red lines, under the name FiD. By using the latency selection method we described in R1, we showcase the other curves, particularly our combined approach with the yellow color. Additionally, we also compare with the baselines of CALM (in green) and Token Filtering (in blue) separately in Figure 5.\n\nRegarding the comparison to FiD-light, we encountered challenges reproducing the FiD-light results, mainly due to the lack of open-sourced code. We note that our approach and FiD-light are complementary and have the potential to be integrated and employed collectively. We elaborate on that more in our answer to question 5, later on in this response.\n\nWe plan to open source our code to reproduce our evaluations and datasets upon acceptance.\n\nIn addition, we provide answers to your questions below:\n\n**Q1.** We provide the list of operations performed in the token filtering mechanism:\n\n1. Perform the mean operation over all attention heads.\n2. Rank the tokens according to their scores.\n3. Choose the top $p$\\% highest ranking tokens. \n4. Keep only the top tokens in the past cross-attention key value states.\n\nAs for a formal definition, the cross-attention scores per input token are defined as follows:\n\n$S_{t,l} = \\frac{1}{h} \\sum_{i=1}^{h} A_{t,l}^{i}$\n\nwhere $t$ is the generated token index, $l$ is the layer index, $h$ is the number of attention heads, and $A_{t,l}^{i}$ represents the cross-attention scores at index $t$, layer $l$ and head $i$.\n\nWe perform a descending $argsort$ operation on the scores above, and take the top $p$% from the sorted input token indices. Hence, we denote $T$ as the total number of input tokens from all the passages, and $T’$ as the amount of tokens we keep after filtering, which is $p$% from $T$:\n\n$Sorted_{t,l} = argsort(S_{t,l})$ \n\n$Top_{t,l} = (Sorted_{t,l} [i])_{i=1}^{T’}$ ,\n\nwhere $[i]$ indicates accessing the vector at index $i$.\n\nFinally, we keep only the tokens chosen in $Top_{t,l}$ from the cross-attention past key values states $K_{past}, V_{past}$:\n\n$K_{past} = K_{past}[Top_{t,l}], V_{past} = V_{past}[Top_{t,l}]$,\n\nWhere $A[B]$ selects the elements from $A$ whose indices appear in $B$.\nThese new past key value states are the only ones used for generating all subsequent tokens.\n\n**Q2.** The results for Table-1 for Token Filtering without CALM are presented below, with the reported results from the paper as well:\n\n|          | ELI5  |       | MS MARCO |       |\n|----------|-------|-------|----------|-------|\n|          | R-L   | F1    | R-L      | F1    |\n| FiD Token Filtering | 26.65 | 30.32 | 24.75    | 27.26 |\n| FiD Comb | 26.97 | 31.76 | 25.11 | 27.41 |\n\nAs shown above, the results of FiD Comb are better than those of FiD Token Filtering.\n\nSimilarly, the results for Table-2 for Token Filtering without CALM on the ELI5 dataset are:\n\n|  | ROUGE-L    |F1   |\n|---------|-------|---------|\n| FiD Token Filtering    | 25.78 | 29.49  |\n| FiD Comb | 25.61 | 29.99 |\n\nHere, we can see that the ROUGE-L is a bit higher, while the F1 score is a bit lower.\n\nFiD (ours) stands for the original FiD model (using T5-Base, and our implementation) and the results shown in Table 1 depict the results of the original FiD model with 100 passages. We will clarify the notation in the camera ready version.\n\n**Q3.** In our implementation, the tokens removed are still kept for layers $l+1,l+2$ and so on. The filtered past key value states will be used from the next generated token. \n\nWe performed a preliminary analysis of what happens if the filtering is applied right after layer $l$. \nBelow are the results on ELI5:\nWe compare our current results from Table 1 in the FiD Comb row, to filtering immediately after $l$, using the same hyperparameter configuration for both:\n\n| Dataset                     | F1    | ROUGE-L | Latency (sec) |\n|-----------------------------|-------|---------|---------------|\n| Filtering Immediately After | 31.88 | 26.9    | 3.653         |\n| FiD Comb                    | 31.76 | 26.97   | 3.833         |\n\nWe can observe that the differences between the two are not significant, though we leave further analysis of this comparison for future work.\n\n**Q4.** Per a sequence, the token filtering is performed only once during the decoding process. Before inference, we specify a specific generated token index $t$ and a specific layer $l$. When the decoding process reaches token $t$ and layer $l$, token filtering is performed, thus creating new filtered past key value states. Once these new states are computed, they are the only ones used for the rest of the decoding process.\n\n**Q5.** Although we encountered challenges reproducing the FiD-light results mostly since the lack of open-sourced code, we note that our approach and FiD-light are complementary and have the potential to be integrated and employed collectively. In particular, since FiD-Light utilizes removing input token vectors from the input before the decoder, we can combine their approach with ours. Also, their source-pointing usage can be incorporated as well.\n\n**Q6.** As mentioned in section 5.4 line 376, there are a lot of different hyperparameters to control for (the generated token to filter at, the layer to filter at, etc.), thus requiring a consistent system to choose the best hyperparameter combination for our quality vs. latency tradeoff.\nThus, we use the division of the combinations into groups, as described in R1 above, which allows for flexibility in choosing the optimal latency setting. Hence, since in the paper we use 30 groups, all the best combinations for every group are kept, but only a single chosen setting is used during inference.\n\n**Q7.** We do require the Max Curve method to compare the methods on the test set in the correct way, which incorporates the quality vs. latency tradeoff. As stated in question 6 and R1 above, each method has many combinations of hyperparameters over the development set. If we apply each one on the test set, and try to compare the methods, it will be difficult to compare them, since each one will be a cluster of points (x-axis is latency, y-axis is ROUGE-L). \n\nThus, instead of clusters of points, each method is described by a single line in the quality vs. latency plot, composed of the chosen points per group. This allows us to easily compare each method, as shown in Figure 5. \n\nWe note that the main focus of our work is on optimizing the quality vs. latency trade-off, hence the focus on the Max Curve method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed idea of eliminating irrelevant tokens based on the attention score is quite simple, interesting, and well-motivated. In the experiment, the proposed token filtering saves the latency and makes further improvements in terms of generation performance, on various datasets.", "reasons_to_reject": "- The presentation is not very clearly provided. In particular, the proposed token elimination method in Section 4 is largely unclear. The filtering is performed for each generated token at a specific layer? In Section 5.4, it is also unclear how to control the values of latency; For example, k in the top-k selection is the parameter to control the latency? In the case of the original FiD, how to control the latency in Figure 5? \n\n- Other baseline methods for saving latency need to be compared. For example, instead of using token-level filtering, top k passages can be provided to the FiD, instead of using a full set of input passages. Also, FiD-light could be considered as the baseline for comparing the latency.", "questions_to_authors": "1) What is the detailed formula of the attention-based score which is used for token filtering? \n2) In Table 1-2, what are the results of only applying token filtering, without combining with CALM? What does mean FiD(ours), which are the reproduced results of FiD without applying the token filtering? \n3) When the token filtering is done at the decoder layer l, its result affects on filtering tokens at layer l+1? That is, the tokens removed at the previous layer are NOT anymore used after next layers? \n4) The token filtering is done per each decoding timestep or per a sequence? \n5) Comparing to FiD-light, how much the proposed method saves the latency? \n6) Hyperparameters need to be chosen for each latency? Then, in total, because there are 30 distinct latent intervals, we need to keep 30 sets of hyperparameters for enduing a FiD model? \n7) Without Max Curve, the methods are not fairly compared? The curves in Figure 5 are those with Max Curve?", "paper_topic_and_main_contributions": "This paper proposes the token filtering to reduce the time complexity of FiD decoder, keeping only important input tokens which are selected based on the average attention scores, similar to Izacard and Grave (2021a). The proposed token filtering saves substantially the latency and leads to further improved performances, when combining with CALM method.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The method in Section 4 is not very clear. Full details need to be provided for clarity."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method of token filtering for reducing latency when using the Fusion-in-Decoder model for retrieval-based tasks (e.g. open domain QA). All reviewers see the problem as well-motivated and the method as useful, with good experimental justification and analysis. R2 notes that the method may be somewhat incremental but is mostly favorable towards it. R3 is mainly concerned with the presentation, and thinks details in the paper could be better clarified. Additionally, they think more baselines could be added. The authors have responded to this by more formally defining the method proposed in the paper and clarifying various aspects e.g. how the extensive number of hyperparameters is selected, how token filtering is performed, etc.. The overall perception is that the paper is sound.
Decision: Accept-Main