An Information Bottleneck Perspective for Effective Noise Filtering on Retrieval-Augmented Generation (2024)

Abstract: Retrieval-augmented generation integrates the capabilities of large language models with relevant information retrieved from an extensive corpus, yet encounters challenges when confronted with real-world noisy data. One recent solution is to train a filter module to find relevant content but only achieve suboptimal noise compression. In this paper, we propose to introduce the information bottleneck theory into retrieval-augmented generation. Our approach involves the filtration of noise by simultaneously maximizing the mutual information between compression and ground output, while minimizing the mutual information between compression and retrieved passage. In addition, we derive the formula of information bottleneck to facilitate its application in novel comprehensive evaluations, the selection of supervised fine-tuning data, and the construction of reinforcement learning rewards. Experimental results demonstrate that our approach achieves significant improvements across various question answering datasets, not only in terms of the correctness of answer generation but also in the conciseness with 2.5% compression rate.

heading: 1 Introduction
text: Large language models represent a significant advancement in natural language understanding and generation, with the capability to process and produce human-like language at an unprecedented scale and complexity (Achiam et al., 2023; Touvron et al., 2023; Team et al., 2023). Nonetheless, large language models have several drawbacks, such as hallucination (Huang et al., 2023) and lacking knowledge for specific domains or highly specialized queries (Kandpal et al., 2023). Retrievalaugmented generation (Lewis et al., 2020) has gained attention for its ability to incorporate information from external knowledge sources during the inference stage. By combining retrieval-based methods with generative models, this approach can
improve the relevance, coherence, and factual accuracy of text generation (Gao et al., 2023).
Retrieval-augmented generation also presents problems. On the one hand, the retriever’s efficacy may be suboptimal in practical use (Izacard et al., 2022; Shi et al., 2023b; Cheng et al., 2023; Lin et al., 2023). On the other hand, the internet data is often of low quality, with redundancy and noise. Indeed, the retrieved content can be completely irrelevant to the query, leading to the model producing incorrect results (Shi et al., 2023a). Recent solutions to mitigate noise in retrieval evidence often involve the adoption of a filtering module (Liu et al., 2023; Yang et al., 2023a; Xu et al., 2023). However, these methods encounter several issues: (1) The inability to ensure that the annotated filtering results can effectively support the generation model in accurately answering questions. (2) The difficulty in directing the filter to refrain from answering when confronted with retrieval evidence that does not support question resolution. (3) The lack of adaptation to the compression extent of the filtering results, impeding the achievement of an optimal solution in terms of cost performance.
1044
We observe that issues above originate from suboptimal objectives. As shown in Figure 1A, the intersection between retrieved passages X and outputs Y denotes the precise information in X which is useful for Y . The noise filter extracts compression X̃ from retrieved passages X , where the filter is optimized with log likelihood objective to output Y . The noise filter trained with this objective can obtain a compression X̃ containing the intersection X ∩ Y , but is incapable of realizing its exact area, which means the filter cannot in principle eliminate the interference of noise for subsequent generation. Therefore, we propose to utilize the information bottleneck theory (Tishby et al., 1999) to optimize the noise filter from a comprehensive perspective, via simultaneously maximizing the useful information while minimizing the noise, thus facilitating a precise delineation of the intersection X̃IB = X ∩ Y . Furthermore, in cases (Figure 1B) where retrieval is not necessitated for content generation or exhibits limited efficacy, the information bottleneck objective enables noise filters to compress the retrieved passages into empty X̃IB = ϕ.
Specifically, we consider information bottleneck as a principle for retrieval augmentation. We first theoretically derive the formula of information bottleneck for retrieval-augmented generation, which integrates large language models. Then we introduce information bottleneck as a new comprehensive evaluation metric for noise filtering, assessing both conciseness and correctness of compressed contents. Next we derive information bottleneck version of supervised fine-tuning and reinforcement learning objectives to train the noise filter.
We conduct experiments on the open-domain question answering datasets: Natural Questions (NQ) (Kwiatkowski et al., 2019), TRIVIAQA (Joshi et al., 2017), and a more complex multi-hop HOTPOTQA (Yang et al., 2018). Using LLAMA2 as filtering and generation model, our approach is proved to be effective compared with strong baseline models, including RANKGPT, LONGLLMLINGUA and LLLAMA2 on all three datasets. We achieve a 2.5% significant compression rate and a 3.2 improvement of exact answer match at most.
Our paper presents three main innovations:
• We are the first, to the best of our knowledge, to introduce the information bottleneck theory into retrieval-augmented generation, which illustrates the optimum of filtration.
• We propose to apply the information bottle-
neck on evaluation metrics, supervised finetuning objectives, and reinforcement learning rewards for retrieval-augmented generation.
• Experimental results reveal the effectiveness of our approach in terms of generation correctness and compression conciseness.

heading: 2 Related Work
text: Information Bottleneck The Information Bottleneck (IB) (Tishby et al., 1999; Fischer, 2020) is a rather simplistic concept: when facing a task, one should attempt to accomplish it using minimal information. The Information Bottleneck theory characterizes learning as a delicate balance between data compression and information retention. When applied to specific tasks, the idea is to extract all the essential informative features for the task while discarding redundant information (Shwartz-Ziv and LeCun, 2023). It has been applied in the study of representation learning (Wu et al., 2020; Federici et al., 2020; Lee et al., 2021), deep learning (Tishby and Zaslavsky, 2015; Saxe et al., 2019; Kawaguchi et al., 2023a), document clustering (Slonim and Tishby, 2000), speech recognition (Hecht et al., 2009), summarization (West et al., 2019), etc.
Noise Filtering Retrieval-augmented generation usually concatenates retrieved passages with their queries as the input of language models. However, this can potentially exceed the context window limit, introduce extra noise and redundancy, and increase computing resource requirements, leading to a decrease in model performance.
FLARE (Jiang et al., 2023c) and Self-RAG (Asai et al., 2023) are dedicated to training models to have the capability of actively retrieving and filtering retrieval content on their own. REPLUG (Shi et al., 2023b) improves the retriever by the KL divergence between the retriever and the LLM.
Post-processing techniques such as noise filtering can help alleviate these issues. (Bai et al., 2023) focuses on re-ranking retrieved articles to filter out noise. Some methods like Selective Context (Li, 2023) and LLMLINGUA (Jiang et al., 2023a) make use of small language models to measure prompt mutual information or perplexity, finding the highest-scoring elements.
There are also some methods employ summarization techniques to design compressors (Xu et al., 2023; Wang et al., 2023). TCRA-LLM (Liu et al., 2023) and LONGLLMLINGUA (Jiang et al., 2023b)
combine summarization and semantic compression techniques. PRCA (Yang et al., 2023b) also incorporates reinforcement learning algorithms in model training. However, these compression methods do not have a unified evaluation of the compression results. RECOMP (Xu et al., 2023) achieves a compression rate of 6%, but it comes at the cost of degraded performance. Our approach is to find the optimal balance between compression rate and performance with the information bottleneck theory.

heading: 3 Methodology
text: In this section, we first introduce the background of information bottleneck (§3.1), and then convert the information bottleneck into forms for noise filters of retrieval augmented generation (§3.2). Next, we provide details of information bottleneck objectives for retrieval augmented generation (§3.3).

heading: 3.1 Preliminary
text: The information bottleneck principle (Tishby et al., 1999) has been a great concept in finding a compression X̃ for signals X that preserves the maximum information relevant to signals Y . Given a joint probability distribution p(X,Y ) between a random variable X and an observed relevant variable Y
(with support sets x ∈ X and y ∈ Y), the amount of information about Y in compressed representation X̃ (x̃ ∈ X̃ ) is given by mutual information:
I(X̃;Y ) =
∫
X̃
∫
Y p(x̃, y) log
p(x̃, y)
p(x̃)p(y) dx̃dy,
(1) where I(X̃;Y ) ≤ I(X;Y ) since compressed data can not convey more information than original ones. The information bottleneck is obtained through
minLIB = I(X̃;X)− βI(X̃;Y ), (2)
where β is the Lagrange multiplier for the trade-off between preserving meaningful information and compressing at various resolutions.

heading: 3.2 Noise Filtering
text: Retrieval-augmented generation involves in generating contents conditioned on input queries q ∈ Q, where relevant passages x ∈ X are retrieved to advance the content generation. As illustrated in Figure 2A, recent noise filtering approaches for retrieval-augmented generation learn to compress retrieved passages x̃ ∈ X̃ ⊆ X for large language models with a log likelihood objective 9 log pLM(y|[q, x̃]) to ground outputs y ∈ Y (Liu
et al., 2023; Yang et al., 2023a; Xu et al., 2023; Wang et al., 2023), which are special cases of conditional mutual information I(X̃, Y |Q) and still unable to avoid irrelevant information.
Now we introduce our information bottleneck approach for retrieval-augmented generation and we derive the information bottleneck between retrieved passages X and ground outputs Y conditioned on queries Q given Equation 2. As demonstrated in Figure 2B, the noise filter is required to simultaneously maximizing the mutual information of its compression with ground outputs while minimizing the mutual information with retrieved passages:
minLIB = I(X̃,X|Q)︸ ︷︷ ︸ conciseness −β I(X̃;Y |Q)︸ ︷︷ ︸ correctness . (3)
The former term I(X̃;X|Q) serves not only to enhance efficiency, which is a common application, but also to promote conciseness by minimizing the inclusion of irrelevant information. As the filtered information becomes increasingly precise, language models are able to reduce the computational resources allocated to input extraction, thereby enhancing their capacity to concentrate on producing higher-quality contents. It’s worth noting that this term guarantees that the filtered information will be rendered null when the retrieved content is entirely irrelevant to the output.
Then we provide the details of each term in the information bottleneck. The conciseness is:
I(X̃;X|Q) =
Eq [ ∫
p(x̃, x|q) log p(x̃, x|q) p(x̃|q)p(x|q)dx̃dx
]
= Eq [ ∫ p(x|q)p(x̃|x, q)
log p(x|x̃, q)XXXp(x̃|q) p(x|q)XXXp(x̃|q) dx̃dx
]
≤ Eq [ ∫ p(x|q) log ∫ p(x|x̃, q)p(x̃|x, q)dx̃ p(x|q) dx ] = −Eq [ DKL [ p(x|q)||Ex̃∼p(x̃|x,q)p(x|x̃, q) ]] ,
(4) where we get an upper bound of conciseness based on Jensen’s inequality. Therefore, I(X̃;X,Q) can be converted to the form of the Kullback–Leibler divergence between the retrieval probability distribution p(x|q) and the expectation of the probability to recover retrieved passages from compression p(x|x̃, q), where x̃ is required to be integrated over the representation space of noise filter p(x̃|x, q).
In the scenario of offline retrievers, where the retrieved passages and queries are jointly sampled from training datasets {(q, x, y)}, p(x|q) becomes a constant number and we can simplify:
min I(X̃;X|Q) ≃ minE(q,x,x̃) [ p(x|x̃, q) ] . (5)
Hence, in cases where training the retriever jointly is not necessary, minimizing the conditional mutual information I(X̃;X|Q) can be elucidated as the process of selectively filtering out information to such an extent that it becomes unfeasible to reconstruct the original content, regardless of the strength of generative language models employed.
Next, the correctness is derived as:
I(X̃;Y |Q) = H(Y |Q)−H(Y |X,Q)
= − ∫
p(y, q) log p(y|q)dydq −H(Y |X,Q) (6)
where the former term E(q,y) [ log p(y|q) ] is considered as a constant independent of the noise filter. We can ignore this term and simplify as:
I(X̃;Y |Q) ≃ −H(Y |X̃,Q)
= ∫ p(y, x̃, q) log p(y|x̃, q)dydx̃dq = E(q,x̃,y) [ log p(y|x̃, q) ] .
(7)
When generative language models are fixed, queryanswer pairs {(q, y)} are sampled from datasets and x̃ is pre-obtained with noise filter. Therefore, maximizing I(X̃;Y |Q) approximates maximizing log likelihood log p(y|x̃, q), which is explained as the process of selectively retaining as much useful information as possible to enable the language model generate target outputs.
Besides, recent studies on the information bottleneck (Fischer, 2020; Federici et al., 2020; Lee et al., 2021; Kawaguchi et al., 2023b) suggest to replace I(X; X̃) with I(X; X̃|Y ), because I(X; X̃) can not be zero while maintaining the target-relevant information. Therefore, we follow Federici et al. (2020) and decompose I(X̃;X|Q) into two components by using the chain rule as:
I(X̃;X|Q) = I(X̃;X|Y ;Q) + I(X̃;Y |Q). (8) Finally, our information bottleneck for noise fil-
tering in retrieval-augmented generation is:
LIB = I(X̃;X|Y ;Q)− (β − 1)I(X̃;Y |Q) ≃ E(q,x,x̃,y) [ p(x|x̃, q, y) ]
− (β − 1) E(q,x̃,y) [ log p(y|x̃, q) ] ,
(9)
where the Lagrange multiplier β − 1 > 0.

heading: 3.3 Information Bottleneck as a Principle
text: The information bottleneck represents not merely a methodological approach, but a fundamental principle to be applied in retrieval-augmented generation. In this section, we will delineate three distinct applications of information bottlenecks, encompassing the establishment of an evaluation metric for noise filtering, the creation of supervised-finetuning (SFT) training datasets, and the formulation of reward functions in reinforcement learning.

heading: 3.3.1 Evaluation Metric
text: Prior to describing the method for training a noise filter, it is imperative to establish criteria assessing the efficacy of filtration outcomes, where the information bottleneck serves as an important evaluation metric. Given {(q, x, y)} from dataset and the compression generated by the noise filter p(x̃|x, q), based on Equation 9, we define the IB score as:
IB(x̃) = pLM(x|[q, x̃, y])−α pLM(y|[q, x̃]), (10)
where large language models are employed to estimate probability distributions, with the inputs to the language model comprising concatenated conditional variables. In addition, we balance the magnitude of values by implying logarithms into the hyperparameter α. The range of IB score is [9α, 1] and smaller IB means better filtration performance.

heading: 3.3.2 Supervised Fine-tuning
text: Training the noise filter from scratch is challenging since there is no ground truth compression of retrieved passages. Although Equation 9 provides a way for achieving the oracle compression, we have to search the optimal one x̃ from all potential subsequences of the retrieved passage x, which is to calculate the integral of x̃ over the language space:
E(q,x,y) [∫ IB(x̃) p(x̃|x, q)dx̃ ] . (11)
The integral is obviously intractable, but we can estimate it with the Monte Carlo sampling strategy. We utilize different existing compression or filtering approaches {pθ1(x̃|x, q), . . . , pθn(x̃|x, q)} to generate candidate compression outputs, as an approximation of sampling from p(x̃|x, q), and the candidate of the best IB score is considered as the pseudo x̃. As illustrated in the left part of Figure 2C, we collect pseudo x̃ over the retrieval-augment generation datasets and construct the {(q, x, x̃)} tuples as training data for supervised learning of
our noise filter pθ(x̃|x, q). In addition, since the noise filter is required to possess capabilities of input understanding and instruction following, we choose pretrained language models as backbones and fine-tune the model to a silver noise filter. The optimization objective is commonly used negative log likelihood LSFT = − ∑ (q,x,x̃) log pθ(x̃|x, q).
It’s worth noting that our approach shows a strong capability to handle the situation when retrieved passages X are irrelevant to ground outputs Y via minimizing I(X̃;X|Y ;Q) → 0, which usually makes x̃ → ϕ. Despite the fact that our information bottleneck objective inherently encompasses optimization objectives for addressing issues related to low-quality information, such as retrieval-free questions, noisy retrieval, and highloss compression, we demonstrate the incorporation of an additional predictive flag [IS_DISCARD] to determine the necessity of discarding the current filtering outcomes. When IB(ϕ) < IB(x̃), which means candidate compression contains too little useful information to assist in model generation, we will set [IS_DISCARD] = True and vice versa.

heading: 3.3.3 Reinforcement Learning
text: Through merely supervised fine-tuning, the efficacy of the noise filtering is suboptimal, with its performance being constrained by the quality of compression candidates. Drawing parallels with the reinforcement learning from human feedback (RLHF, Ouyang et al. (2022)) that aligns large language models with human preference, we propose leveraging reinforcement learning to enhance the noise filter by incorporating guidance from the information bottleneck. Our approach involves utilizing direct preference optimization (DPO, Rafailov et al. (2023)), a recent iteration of RLHF that offers ease of implementation and robust stability. We define the preference probability as:
p∗(x̃1 > x̃2|q, x, y) = σ(IB(x̃2)−IB(x̃1)), (12)
where our noise filter pθ(x̃|x, q) is initially the policy πref(x̃|x, q) that needs to generate two compression samples x̃1 and x̃2 before. Then the offline dataset of preferences D = {q, x, x̃w, x̃l} is constructed, where samples (x̃w wins and x̃l losses) are automatically labeled with the information bottleneck preferences p∗(x̃1 > x̃2|q, x, y). Hence, the objective for the noise filtering policy πθ(x̃|x, q),
which is initialized with πref(x̃|x, q), is as follows:
LDPO =
− E(q,x,x̃w,x̃l)∼D [ log σ ( γ log
πθ(x̃w|x, q) πref(x̃w|x, q)
− γ log πθ(x̃l|x, q) πref(x̃l|x, q)
)] ,
(13) where γ is a hyperparameter controlling the deviation from the base reference policy πref(x̃|x, q). As shown in the right part of Figure 2C, our information bottleneck can also provide the reward function RIB(x̃) = −IB(x̃) of online policies.

heading: 4 Experiments

heading: 4.1 Experimental Settings
text: Datasets and Retrieval Corpus We conduct experiments on three question answering benchmarks: Natural Questions (NQ) (Kwiatkowski et al., 2019), TRIVIAQA (Joshi et al., 2017) and HOTPOTQA (Yang et al., 2018), with results reported on development sets. We utilize the adversarial Dense Passage Retriever1 (DPR) (Karpukhin et al., 2020) to retrieve the top 5 passages from all Wikipedia passages for all datasets. The articles are truncated into non-overlapping documents of 100 words.
Implementation Details We use LLAMA2 (Touvron et al., 2023) as the backbone architecture of the large language model. We finetune the 7B model version with LORA (Hu et al., 2021) for noise filtering, where the optimizer is ADAMW with the learning rate of 5e-5 and the batch size of
1Lucene index of Wikipedia with DPR 100-word splits
Dataset Type Examples % of Recall
Top 5 Top 1
NQ Train 106926 36.5 18.9 Dev 2564 35.0 17.9 TRIVIAQA Train 87622 68.3 49.1 Dev 11313 68.5 49.4 HOTPOTQA Train 90447 51.5 35.1 Dev 7405 45.0 28.4
ness and correctness of generated contents on a semantic level compared to ground answers, leveraging the capabilities of advanced large language models. In addition, to evaluate the performance variations resulting from retrieval augmentation in detail, we use the flip rate of EM to evaluate the extent to which generated responses are influenced by retrieved context. The True-Flip-Rate (TFR) and False-Flip-Rate (FFR) are defined as follows:
TFR = p(EMpLM(y|[q,x]) = 0|EMpLM(y|q) = 1) FFR = p(EMpLM(y|[q,x]) = 1|EMpLM(y|q) = 0),
where TFR measures the degree of noise introduced by the retrieved information while FFR examines the amount of benefits from retrieval augmentation.
Baselines First we consider the results of the generator (LLAMA2-13B) without retrieval augmentation and with top-1 or top-5 retrieved passages. We next include two sets of filtering methods, reranking using RANKGPT (Sun et al., 2023) distilled from ChatGPT, and prompt compression with LONGLLMLINGUA (Jiang et al., 2023b). We also experiment with the base model LLAMA-7B doing summarization from top-5 passages.

heading: 4.2 Open-Domain Question Answering
text: Table 1 presents the experimental results on NQ and TRIVIAQA datasets. Initially, we demonstrate the performance of the generator (LLAMA2 13B) without retrieval, where a low EM score implies the limitation of large language models facing opendomain question answering task. When the generation is directly augmented with retrieved documents, the result is even worse with a drop of 9.2 EM score on TRIVIAQA at most, reflecting the presence of a large amount of noise in the retrieved content and the vulnerability of the generator to noise interference. Besides, the TFR reveals that noise in retrieved passages will cause over 50% of the generated answers wrong that could have been answered correctly on the NQ dataset.
Extractive noise filters including RANKGPT and LONGLLMLINGUA can compensate for the performance losses in EM, F1, and IB scores caused by retrieval, via noise filtering. However, they can hardly outperform the non-retrieval generator, because they enable the generator to answer more questions correctly with 10.3 of FFR, but also make more mistakes with 51.0 of TFR. Also as an extractive compression method, our approach significantly improves the EM scores on on NQ and
words EM↑ TFR↓ FFR↑ F1↑ IB↓ LLAMA2 - 16.2 - - 51.4 -4.46 Top1 103.6 13.4 56.8 7.7 51.0 -4.29 Top5 517.6 14.7 55.8 9.0 48.4 -4.21 I(X̃;Y |Q) 13.1 19.2 24.9 8.5 54.6 -4.58 IB 12.7 21.5 20.3 10.2 55.9 -4.78
Dataset Filtering Candidates HASANS EM F1 Words IB I(X̃;X|Y ;Q)

heading: 4.3 Multi-Hop Question Answering
text: Filtering models encounter more significant hurdles when dealing with multi-hop problems, as solving these requires not only auxiliary information but multiple rounds of reasoning and analysis. As demonstrated in Table 3, the results obtained are marginally improved compared to those achieved through supervised training alone.
4.4 Ablation Study for Conciseness We utilize ablation experiments to exhibit the significance of the conciseness term in the information bottleneck theory on NQ. As shown in Table 4, integrating the information bottleneck approach that combines both conciseness and correctness leads to superior outcomes compared to solely utilizing correctness. The filtered outcomes from the former approach are more concise and of superior quality.

heading: 5 Analysis

heading: 5.1 Silver Selection with IB
text: To visually demonstrate that applying the information bottleneck to select training data can enhance the upper bound, we conducted experiments on the validation sets of NQ and HOTPOTQA. Table 5 lists two basic filtering methods: exact search and greedy search. The application of IB selection is not limited to just two filtering methods and it can be generalized to other answer-guided silver mining strategies, such as the leave-one-out evidentiality mining strategy (Asai et al., 2022) and CXMI (Wang et al., 2023). Our approach is not fixated on finding the optimal solution in the initial stages, but rather focuses on gradually approaching the optimal filter model through iterative training. Here we choose the two easiest methods, which can greatly
reduce computational costs on the construction of training data
The goal of exact search is to find the paragraphs or sentences containing the ground answers. Greedy search (Nallapati et al., 2017) is one of the most popular heuristic method by far used in extractive summarization. This algorithm extracts oracle labels with the highest ROUGE (Lin, 2004) scores compared to human-annotated abstracts. We considered two silver summaries, one that concatenates the query and answer, and the other that focuses solely on the answer itself. The former can cover more information, while the latter focuses more on the answer itself. Specially, the answer in intermediate state, supporting facts, are incorporated for multi-hop questions.
By using information bottleneck for selecting among the four filtering results, the obtained outcome is superior to any of them, which reveals that existing simple annotation methods only yield solutions that are far from optimal.
5.2 Length of Summary and Conciseness Due to the adoption of extractive filtering method, the content retained after filtering is sourced from the original text. This leads to the hypothesis that there may be a correlation between the compression rate and the conciseness mutual information I(X̃;X|Y ;Q). We verify the relationship between compression rate and mutual information on NQ and HOTPOTQA with a toy experiment based on §5.1. For each query-answer pair with four filtering candidates, we calculate their corresponding length and conciseness I(X̃;X|Y ;Q). As the statistical information for different samples is independently and identically distributed, we convert the values of length and conciseness in each sam-
ple into their rankings among various compression methods. Then we calculate the Pearson correlation coefficient between length RankL and conciseness RankC , which is 0.953, indicating a significant correlation at the 0.01 level (two-tailed).

heading: 6 Conclusion
text: We apply the information bottleneck principle to noise filters in retrieval-augmented generation, balancing the trade-off between the conciseness and correctness. Not only as an evaluation method for noise filtering, we also apply it to select supervised fine-tuning training data and provide reward for reinforcement learning. Our two-stage optimization gradually approaches the oracle filtering objective. Experimental results demonstrate that our filter significantly outperforms baseline methods and achieves an impressive compression rate.

heading: Acknowledgements
text: Kun Zhu and Xiaocheng Feng contribute equally to to this work. Bing Qin is the corresponding author of this work. We thank the anonymous reviewers for their insightful comments. This work was supported by the National Key R&D Program of China via grant No. 2021ZD0112905, National Natural Science Foundation of China (NSFC) via grant 62276078 and U22B2059, the Key R&D Program of Heilongjiang via grant 2022ZX01A32, the International Cooperation Project of PCL, PCL2022D01 and the Fundamental Research Funds for the Central Universities (Grant No.HIT.OCEF.2023018).

heading: Limitations
text: Although our method has exhibited effectiveness in enhancing the performance of noise filtering task on retrieval-augmented generation, it does have limitations such as performance reliance on the generator and trade-off between True-Flip-Rate (TFR) and False-Flip-Rate (FFR). In order to analyze the mutual information between filtered content X̃ and retrieval content X , as well as between X̃ and Y , it is crucial to utilize a white-box generator equipped with robust capabilities. Furthermore, by introducing the additional predictive flag to assess the necessity of discarding the current filtering outcomes, we successfully decrease TFR, while at a cost of potentially decreasing FFR. We mitigate it by engaging in training iterations, which inevitably leads to an escalation in training cost.

heading: Ethics Statement
text: We are totally aware that text generation technology has a potential to be used maliciously to generate fake, toxic, or offensive content. If the retrieved content includes harmful or toxic information, it will influence the output of the generated content. Our approach is proposed to mitigate the influence of noise from retrieval, which includes toxic contents. However, there is no assurance that our approach will completely eliminate toxics.

heading: A Prompt

heading: A.1 Filter Prompt
text: We show prompts used to train the filter in Table 6.

heading: A.2 Generator Prompt
text: We show prompts used for inference in Table 7.

heading: B Experimental Configuration
text: We use LLAMA2 (Touvron et al., 2023) as the backbone architecture of the large language model. We fine-tune the 7B model version with LORA (Hu et al., 2021) for noise filtering, where the optimizer is ADAMW with the learning rate of 5e-5 and the batch size of 32. The 13B version is also employed as the generator without any adjustments.
For supervised fine-tuning, we set the coefficient of IB score α = 10 to balance the compression. As the value of α decreases, there is a greater tendency
to prioritize concise filtering results as the silver selection. Conversely, as the value of increases, there is a greater inclination to prioritize higher correctness. We balance the conciseness and correctness by the hyperparameter. In section 5.1, we conduct experiments on the validation sets of NQ and HotpotQA to visually demonstrate that applying the information bottleneck to select training data can enhance the upper bound. We also conduct experiments about the value of α here. As shown in the table 8 below, as long as the value of α is not zero, its value does not significantly impact the other performance metrics , such as HasAnswer, EM, and F1. α = 10 is a good empirical value that balance the conciseness and correctness.
Besides, filtering candidates are sampled from four different methods derived from traditional extractive summarization, with details in §5.1. For reinforcement learning, we utilize DPO and the hyperparameter γ = 0.1. We use one NVIDIAA100-40G to train 5 epochs given the question and 5 retrieval passages. For every additional 1W of data, the training time increases by 5.5 hours. We also use one NVIDIA-A100-80G to generate the answer given the question and compression results. We set a maximum length of 1024 tokens for all sequences during training and inference. The generator is configured to generate a maximum of 200 tokens and the decoding strategy is top-p sampling with p = 0.9.

heading: C Case Study
text: In this section, we present exxamples from each of the three datasets within two distinct scenarios. The first scenario is when the retrieved content fails to directly address the question and the compressed content should be empty. Alternatively, the second scenario involves instances where the retrieved content effectively addresses the research question, and the compressed content should be as short as possible.
Question: who won the 2017 sports personality of the year Standard Answer: Mo Farah Retrienval Passages "Newsfirst Platinum Awards" Angelo Mathews won the Most Popular Sports Person of The Year Award. The second Platinum ceremony launching started with January 2017, with the full support of the Ministry of Education. Nominations for Platinum Awards 2017 opened under 20 categories. The promotional campaign commenced at Thurstan College, Colombo on 30 January 2017. The campaign was held in every district in the island until the 28 February 2017. The Awards night was held on 31 March 2017 at Sirasa Stein Studios, Ratmalana. Five times world champion in Lamborghini motor racing, Dilantha Malagamuwa won the Most Popular Sports Person of The Year Award. "BBC Sports Personality Team of the Year Award" by Scottish teams; Celtic in 1967, after they became the first British football club to win the European Cup, and the 1990 Grand Slam winning Scotland rugby union squad. Football has had the highest representation among the winners, with 13 recipients. The most recent award was presented in 2017 to the England women’s cricket team. This table lists the total number of awards won by nations that the teams have represented. This table lists the total number of awards won by the teams sporting discipline. BBC Sports Personality Team of the Year Award The BBC Sports Personality Team of the "BBC Young Sports Personality of the Year" In 2001, the award was replaced by the Young Sports Personality of the Year, and sprinter Amy Spencer was the first recipient of that award. Scottish tennis player Andy Murray, who won in 2004, is the only non-English recipient of the award. The only person to win the award more than once is diver Tom Daley, who won the award three times, in 2007, 2009, and 2010, and was nominated to the ten-person shortlist in five successive years (2007-2011). The most recent award was presented in 2017 to Manchester City midfielder Phil Foden. <nowiki>*</nowiki> Including a Newcomer of the Year "Simone Biles" Foundation. She was also one of the finalists for ""Time magazine""’s 2016 Person of the Year. Biles was also nominated for a 2016 ESPY award for Best Female Athlete along with Elena Delle Donne, Katie Ledecky, and Breanna Stewart; Stewart won the award. In July 2017, Biles won the ESPY Award for Best Female Athlete. She is the second gymnast to win this award after Nastia Liukin won it in 2009. In 2017, Simone won the Shorty Awards for the best in sports. At the 2017 Teen Choice Awards, Simone won favorite female athlete, and won Laureus World Sports Award "BBC Sports Personality of the Year Award" (2008) and Ennis-Hill (2017), received the BBC Sports Personality of the Year Lifetime Achievement Award. Princess Anne (1971) and her daughter Zara Phillips (2006) are the only award-winners to be members of the same family. The oldest recipient of the award is Dai Rees, who won in 1957 aged 44. Ian Black, who won the following year, aged 17, is the youngest winner. Torvill and Dean, who won in 1984, are the only non-individual winners of the award, so in the 61 years of the award there have been 62 recipients. Of these 13 have been female. 17 sporting disciplines Without RAG answer: Mo Farah Top-1-passage RAG answer: Greg Rusedski Top-5-passage RAG answer: Simone Biles

heading: Method Summary words Answer
text: RankGPT "Newsfirst Platinum Awards"
Angelo Mathews won the Most Popular Sports Person of The Year Award. The second Platinum ceremony launching started with January 2017, with the full support of the Ministry of Education. Nominations for Platinum Awards 2017 opened under 20 categories. The promotional campaign commenced at Thurstan College, Colombo on 30 January 2017. The campaign was held in every district in the island until the 28 February 2017. The Awards night was held on 31 March 2017 at Sirasa Stein Studios, Ratmalana. Five times world champion in Lamborghini motor racing, Dilantha Malagamuwa won the Most Popular Sports Person of The Year Award.
103 Angelo Mathews
LONGLLMLINGUA 0] "BBC Sports Personality of the Award Document [3 "BBC Sports Person Team of the Angelo Matws won The Pl ceremony launch started with17 with the support of the Ministry of.ations01 2ionalenced Th3 The campaign in every district in the the7 The night117asa Stein Studiosana world in motor racing, D Malwa of The Year Award. Document [1] "BBC Young Sports Personality of the Year" Document [4] "Simone Biles" by Scottish teams; Celtic in 1967, after they became the first British football club to win the European Cup, and the 1990 Grand Slam winning Scotland rugby union squad. Football has had the highest representation among the winners, with 13 recipients. The most recent award was presented in 2017 to the England women’s cricket team. This table lists the total number of awards won by nations that the teams have represented. This table lists the total number of awards won by the teams sporting discipline. BBC Sports Personality Team of the Year Award The BBC Sports Personality Team of the who won the 2017 sports personality of the year 177 Simone Biles
LLAMA2 * Winner of the 2017 BBC Sports Personality of the Year: Simone Biles * Winner of the 2017 BBC Sports Personality Team of the Year: England women’s cricket team * Winner of the 2017 Platinum Awards Most Popular Sports Person of the Year: Angelo Mathe 45 Simone Biles
Question: who did the dominican republic gain its independence from Standard Answer: Haiti Retrienval Passages "Dominican Day Parade" leadership of General Gregorio Luperon, the war was ultimately won from Spain. In 1844, the Dominican Republic secured its independence from Haiti and became a sovereign state until 1861. Under the leadership of General Pedro Santana, segments of the Dominican population sought to annex the Republic back to Spain and did so during March 18, 1861. On August 16, 1863, the start of the war for the Restoration of the Dominican Republic under the command of General Luperon. The Dominican Republic originally declared its independence from Spain on December 1, 1821. Ultimately, the Dominican Republic was re-established, free from Spain,
"Colorism in the Caribbean" into preference for lighter skin. Ritualistic skin bleaching to lighten one2̆019s skin, brown paper bag tests to verify one’s skin tone, and degradation of darker-complected Haitians as ugly are contemporary manifestations of colorism in Haiti. After declaring its independence from Spanish rule in 1821, the Dominican Republic was overtaken by Haitian rule in 1822. The Dominican Republic did not achieve independence from Haiti until after their victory in the Dominican War of Independence in 1844. However, the country fell back under Spanish rule until it reclaimed its sovereignty after the Dominican War of Restoration of 1865. As the Dominican Republic
"La Trinitaria (Dominican Republic)" in August 1843 as a result of his dissident activities. La Trinitaria’s other members continued the fight in Duarte’s absence. One of them was Francisco del Rosario S0̆0e1nchez, who corresponded with Duarte during the latter’s exile in Venezuela, and Mat0̆0edas Ram0̆0f3n Mella, who along with Duarte and Sanchez became known as the founding fathers of the Dominican Republic. On January 16, 1844 La Trinitaria’s manifesto in favor of independence was released, and the fight for independence began to gain the necessary momentum. On the heels of La Trinitaria’s work, and after many battles and much bloodshed, the Dominican Republic was
"Dominican Navy" Dominican Navy The Navy of the Dominican Republic or Armada de Republica Dominicana (ARD), is one of the three branches of the Military of the Dominican Republic, together with the Army and the Air Force. After the Dominican Republic gained its independence from Haiti on February 27, 1844, there was a need to create a naval fleet. Three schooners were commissioned for this, (flagship), ""Mar0̆0eda Chica"" and ""Leonor"". These were the original three Dominican vessels which were incorporated in the newly created Dominican Navy as authorized by the Junta Central Gubernativa with the Naval Act of 1844 on April 23,
"Dominican Navy" Brigade, which was under Spanish command and operating in Southern Iraq. Raised in 2008 the MIC serves as the youngest arm of service in the Dominican Navy, which is tasked for amphibious and landing operations. One Marine Battalion is of active service as of the present. It also the youngest Marine unit in all of Latin America. Dominican Navy The Navy of the Dominican Republic or Armada de Republica Dominicana (ARD), is one of the three branches of the Military of the Dominican Republic, together with the Army and the Air Force. After the Dominican Republic gained its independence from Without RAG answer: Spain Top-1-passage RAG answer: Spain Top-5-passage RAG answer: Haiti

heading: Method Summary words Answer
text: RankGPT "Dominican Day Parade"
leadership of General Gregorio Luperon, the war was ultimately won from Spain. In 1844, the Dominican Republic secured its independence from Haiti and became a sovereign state until 1861. Under the leadership of General Pedro Santana, segments of the Dominican population sought to annex the Republic back to Spain and did so during March 18, 1861. On August 16, 1863, the start of the war for the Restoration of the Dominican Republic under the command of General Luperon. The Dominican Republic originally declared its independence from Spain on December 1, 1821. Ultimately, the Dominican Republic was re-established, free from Spain,
103 Spain
LONGLLMLINGUA 0] "Domin Day Par [4 "La Trinitaria (Dominican
Bade, was under Spanish command and operatingq. R0IC serves as theest arm of service in the Dominican, which task amphibious and operations. service of present also youngest in all of America.ican Navy The Navy of the Dominican Republic or Armada de), of Dominican Republic.ican Republic gained its [ " ican Navy [Color the Caribbean"
leadership of General Gregorio Luperon, the war was ultimately won from Spain. In 1844, the Dominican Republic secured its independence from Haiti and became a sovereign state until 1861. Under the leadership of General Pedro Santana, segments of the Dominican population sought to annex the Republic back to Spain and did so during March 18, 1861. On August 16, 1863, the start of the war for the Restoration of the Dominican Republic under the command of General Luperon. The Dominican Republic originally declared its independence from Spain on December 1, 1821. Ultimately, the Dominican Republic was re-established, free from Spain,
who did the dominican republic gain its independence from
171 The Dominican Republic gained its independence from Spain.
LLAMA2 The Dominican Republic gained its independence from Haiti on February 27, 1844, after the Dominican War of Independence. Prior to that, the country had gained independence from Spain in 1821, but was overtaken by Haitian rule in 1822. The Dominican R 41 Haiti
Question: What was Walter Matthau’s first movie? Standard Answer: The Kentuckian Retrienval Passages "Pete ’n’ Tillie" Walter Matthau received a Golden Globe nomination for Best Actor 2̆013 Motion Picture Musical or Comedy, and won the 1973 BAFTA Award for Best Actor in a Leading Role for his performance in this movie and for his performance in ""Charley Varrick"". Carol Burnett received a Golden Globe Award nomination for Best Actress - Motion Picture Musical or Comedy. Pete ’n’ Tillie Pete ’n’ Tillie is a 1972 American comedy-drama film directed by Martin Ritt and starring Walter Matthau and Carol Burnett. Its advertising tagline was: ""Honeymoon’s over. It’s time to get married."" Screenwriter Julius J. Epstein was nominated for
"Movers & Shakers" Movers & Shakers Movers & Shakers is a 1985 American comedy film distributed by MGM, starring Walter Matthau and directed by William Asher. The story follows the head of production at a Hollywood studio who wants to make a movie to fulfill a promise made to a dying friend. The film was written by Charles Grodin, who also appears in the movie. The cast includes Tyne Daly, Gilda Radner, and Vincent Gardenia. Steve Martin makes a cameo appearance as Fabio Longio. Hollywood studio mogul Joe Mulholland (Matthau) vows to produce the pet project of a dying acquaintance, who has been
"The Fortune Cookie" intention to sue the insurance company lawyers for invasion of privacy and report Purkey’s racist remarks to various organizations. Hinkle drives to the stadium, where he finds Boom-Boom ready to leave the team and perhaps become a wrestler named ""The Dark Angel"". Hinkle manages to snap Boom-Boom out of his funk, and the two run down the fields passing and lateraling a football back and forth between them. This was the first film to feature the movie partnership of Jack Lemmon and Walter Matthau, who were to appear together in ten films: 2̆020<br> Matthau and Lemmon also both had roles
"The Secret Life of an American Wife" a movie star who is known as an international sex symbol (Walter Matthau). Upon hearing that The Movie Star (the character is not given a name, and Matthau is credited as ""The Movie Star"" in the closing credits) indulges in the services of prostitutes, Victoria decides to pose surreptitiously as one in order to prove to herself that she is still sexually attractive. The Secret Life of an American Wife The Secret Life of an American Wife is a 1968 comedy film written and directed by George Axelrod. The film was released by 20th Century Fox in 1968, and was
"Charles Matthau" Charles Matthau Charles ""Charlie"" Matthau (born December 10, 1962) is a film and television director and actor and the son of actor Walter Matthau and actress/author Carol Saroyan. Born in New York City, he appeared as a child actor alongside his father in such films as ""Charley Varrick"" (1973), ""The Bad News Bears"" (1976) and ""House Calls"" (1978). Among his directorial projects have been ""The Grass Harp"", from a novella by Truman Capote, and the made-for-TV movie ""The Marriage Fool"", both of which starred his father. He also directed ""Doin’ Time on Planet Earth"" (1988), ""Her Minor Thing"" (2005), ""Baby-O""
Without RAG answer: The Kentuckian Top-1-passage RAG answer: The Taking of Pelham One Two Three Top-5-passage RAG answer: Pete ’n’ Tillie

heading: Method Summary words Answer
text: RankGPT "Pete ’n’ Tillie"
Walter Matthau received a Golden Globe nomination for Best Actor 2̆013 Motion Picture Musical or Comedy, and won the 1973 BAFTA Award for Best Actor in a Leading Role for his performance in this movie and for his performance in ""Charley Varrick"". Carol Burnett received a Golden Globe Award nomination for Best Actress - Motion Picture Musical or Comedy. Pete ’n’ Tillie Pete ’n’ Tillie is a 1972 American comedy-drama film directed by Martin Ritt and starring Walter Matthau and Carol Burnett. Its advertising tagline was: ""Honeymoon’s over. It’s time to get married."" Screenwriter Julius J. Epstein was nominated for
103 Pete ’n’ Tillie
LONGLLMLINGUA 1] "les Matthau Document [Pete ’n’ Till "The Fortune
a movie star is as (W). that character nameauited Star) the, pose is. Theife 9 film Fox ention and report Purkey’s racist remarks to various organizations. Hinkle drives to the stadium, where he finds Boom-Boom ready to leave the team and perhaps become a wrestler named ""The Dark Angel"". Hinkle manages to snap Boom-Boom out of his funk, and the two run down the fields passing and lateraling a football back and forth between them. This was the first film to feature the movie partnership of Jack Lemmon and Walter Matthau, who were to appear together in ten films: 2̆020<br> Matthau and Lemmon also both had roles
122 The Fortune
Question: Which was the only eastern bloc country to participate in the 1984 LA Olympics? Standard Answer: Romania

heading: Retrienval Passages
text: "1984 Summer Olympics boycott" the majority of Soviet Bloc countries will not participate in the Games, Ceau0̆219escu’s Romania is expected to attend. 1984 Summer Olympics boycott The boycott of the 1984 Summer Olympics in Los Angeles followed four years after the U.S.-led boycott of the 1980 Summer Olympics in Moscow. The boycott involved 14 Eastern Bloc countries and allies, led by the Soviet Union, which initiated the boycott on May 8, 1984. Boycotting countries organized another major event, called the Friendship Games, in July and August 1984. Although the boycott led by the Soviet Union affected a number of Olympic events that were normally
"Nicolae Ceaus0̆326escu" visit of Egyptian president Anwar Sadat to Israel in 1977. Also Romania was the only country in the world to maintain normal diplomatic relations with both Israel and the PLO. In 1980, Romania participated in the 1980 Summer Olympics in Moscow with its other Soviet bloc allies, but in 1984 was one of the few Communist countries to participate in the 1984 Summer Olympics in Los Angeles when most of the Eastern Bloc’s nations boycotted this event. In 1966, Ceau0̆219escu, in an attempt to boost the country’s population, made abortion illegal and introduced Decree 770 to reverse the low birth
"Summer Olympic Games" Eastern Bloc that did attend the 1984 Olympics. These games were perhaps the first games of a new era to make a profit. Although a boycott led by the Soviet Union depleted the field in certain sports, 140 National Olympic Committees took part, which was a record at the time. Again, without the participation of the Eastern European countries, the 1984 Games were dominated by their host country. The Games were also the first time mainland China (People’s Republic) participated. According to British journalist Andrew Jennings, a KGB colonel stated that the agency’s officers had posed as anti-doping authorities from
"Romania at the Olympics" Romania at the Olympics Romania first participated at the Olympic Games in 1900, with a single participant. The National Olympic Committee for Romania is the Romanian Olympic and Sports Committee, and was created and recognized in 1914. The nation first sent a team to compete at the Games in 1924, and has only missed two editions each of the Summer Olympic Games and Winter Olympic Games since then. Notably, Romania was the lone Eastern Bloc nation to participate at the 1984 Summer Olympics, which the other nations boycotted. That was also Romania’s most successful Olympic Games: they won 20 gold
"Craig Beardsley" medal in the 200-meter butterfly at the Pan American Games in Caracas, Venezuela. Beardsley failed to qualify for the U.S Olympic team in 1984, by placing third by 0.36 of a second in the U.S. Olympic Trials, and he retired from competitive swimming afterward. Taking advantage of the hard feelings created by the American-led boycott of the 1980 Moscow Olympics, the Soviet Union and other Eastern Bloc countries had pushed for and received Olympic rules changes limiting each country to only two participants in each swimming event. The United States, as the dominant swimming power, was hurt the most. Beardsley
Without RAG answer: East Germany Top-1-passage RAG answer: Romania Top-5-passage RAG answer: Romania

heading: Method Summary words Answer
text: RankGPT "1984 Summer Olympics boycott"
the majority of Soviet Bloc countries will not participate in the Games, Ceau0̆219escu’s Romania is expected to attend. 1984 Summer Olympics boycott The boycott of the 1984 Summer Olympics in Los Angeles followed four years after the U.S.-led boycott of the 1980 Summer Olympics in Moscow. The boycott involved 14 Eastern Bloc countries and allies, led by the Soviet Union, which initiated the boycott on May 8, 1984. Boycotting countries organized another major event, called the Friendship Games, in July and August 1984. Although the boycott led by the Soviet Union affected a number of Olympic events that were normally
104 Romania
LONGLLMLINGUA Document [1] "1984 Summer Olympics boycott
Document [] "Sum Olympic Games" [ "Romania
Eloc the4 These were first of a led the field was., the Eastern European4 by The also ( Republic participated.GB thes had- from will not participate in the Games, Ceau0̆219escu’s Romania is expected to attend. 1984 Summer Olympics boycott The boycott of the 1984 Summer Olympics in Los Angeles followed four years after the U.S.-led boycott of the 1980 Summer Olympics in Moscow. The boycott involved 14 Eastern Bloc countries and allies, led by the Soviet Union, which initiated the boycott on May 8, 1984. Boycotting countries organized another major event, called the Friendship Games, in July and August 1984. Although the boycott led by the Soviet Union affected a number of Olympic events that were normally
Which was the only eastern bloc country to participate in the 1984 LA Olympics?
144 Romania
LLAMA2 Romania was the only Eastern Bloc country to participate in the 1984 Los Angeles Olympics, and it was also the country’s most successful Olympic Games, winning 20 gold medals. 29 Romania
Question: What is the year of the event that occured first, Making Today a Perfect Day was produced, or Frozen was produced? Standard Answer: 2013

heading: Retrienval Passages
text: "Making Today a Perfect Day" Making Today a Perfect Day ""Making Today a Perfect Day"" is a song from the 2015 Walt Disney Animation Studios computer-animated short film ""Frozen Fever"", with music and lyrics by Kristen Anderson-Lopez and Robert Lopez and performed throughout most of the short. It was released as a single in the United States on March 12, 2015. On September 2, 2014, during the ABC airing of """", Walt Disney Animation Studios’ chief creative officer John Lasseter announced that a ""Frozen"" short film with a new song would be released in the future. On the same day, ""Variety"" announced that the short
"Making Today a Perfect Day" the lyrics off-by-heart. Kat Brown of ""The Daily Telegraph"" referred to the short film as a ""musical video"", due to such a large proportion of it being taken up by this song. ""Us Weekly"" negatively compared its catchiness to ""Let It Go"", though described the ditty as ""fresh"", ""bright"", and ""fun"". In a negative review, ""Slate"" felt that ""the song itself, while hummable, is fatally damaged by its need to do too much."" Making Today a Perfect Day ""Making Today a Perfect Day"" is a song from the 2015 Walt Disney Animation Studios computer-animated short film ""Frozen Fever"", with music
"Making Today a Perfect Day" would be released in early 2015 under the title ""Frozen Fever"", with Chris Buck and Jennifer Lee returning as co-directors, Peter Del Vecho returning as producer and a new song by Kristen Anderson-Lopez and Robert Lopez. In a mid-October interview, Idina Menzel revealed that the cast had already recorded their vocal tracks, stating ""We just worked on a short for ""Frozen""."" On December 3, 2014, it was announced that Aimee Scribner would be a co-producer and that ""Frozen Fever"" would debut in theaters alongside Walt Disney Pictures’ ""Cinderella"" on March 13, 2015. In late December, the co-directors told the Associated
"Making Today a Perfect Day" to start brainstorming possibilities. After early discussions about Olaf, head story artist Marc Smith pitched the idea of what might happen if Elsa had a cold, which became the basis for the short’s plot. The directors began working on the short in June and by August were back in the recording studio with the cast to lay down vocal tracks. The short features the song ""Making Today a Perfect Day"", by Anderson-Lopez and Lopez. At the premiere of ""Cinderella"" and ""Frozen Fever"" at the El Capitan Theatre in Hollywood, California, on March 1, 2015, Josh Gad told ""USA Today"", ""I
"Making Today a Perfect Day" want to apologize to parents everywhere for the fact that children are going to be singing a whole new ""Frozen"" song..."" Gad’s wife noticed he was still humming it two days after he recorded his lines. In ""Making Today a Perfect Day"" there is a lyrical reference to ""Let It Go""2̆014Elsa notes to Anna that ""a cold never bothered me anyway"", this time re-purposing the line by referring to an actual cold. ""Billboard"" suggests that the songwriting duo included this Easter egg because they ""know exactly what the fanbase wanted"". The beginning of the song also includes a passage to
Without RAG answer: 2013 Top-1-passage RAG answer: 2015 Top-5-passage RAG answer: Frozen was produced first.

heading: Method Summary words Answer
text: RankGPT "Making Today a Perfect Day"
Making Today a Perfect Day ""Making Today a Perfect Day"" is a song from the 2015 Walt Disney Animation Studios computer-animated short film ""Frozen Fever"", with music and lyrics by Kristen Anderson-Lopez and Robert Lopez and performed throughout most of the short. It was released as a single in the United States on March 12, 2015. On September 2, 2014, during the ABC airing of """", Walt Disney Animation Studios’ chief creative officer John Lasseter announced that a ""Frozen"" short film with a new song would be released in the future. On the same day, ""Variety"" announced that the short
105 2015
LONGLLMLINGUA Making Today a Perfect ""aking Today a Perfect Day is a song from the 20 Walt Disney Animation Studios-animated short ""Fro F"", music and lyrics Kristen Anderson-Lopez and Robert Lopez throughout most of short It was single in United States on March 12, 2015. On September 2, 2014, during ABC airing of Walt Disney Animation chief creative officer John Lass announced aFrozen"" short film with a new would be released future same day,Vari announced that the [Making Today a
Document [4M Day [fect Day" Frozen Fever"", with music
What is the year of the event that occured first, Making Today a Perfect Day was produced, or Frozen was produced?
109 2015
LLAMA2 The event "Making Today a Perfect Day" was produced first, in 2015. 12 2015
Question: Salisbury Woodland Gardens links a zoo with a park designed and built under the watchful eye of who? Standard Answer: Thomas Mawson

heading: Retrienval Passages
text: "Salisbury Woodland Gardens, Blackpool" enable the local community to get more involved in the sites management and interpretation. Salisbury Woodland Gardens, Blackpool Salisbury Woodland Gardens is an open space located in the east of Blackpool, flanked by East Park Drive and Woodside Drive and linking Blackpool Zoo with Stanley Park. Known simply as the ’Woodland Gardens’ to local people, the site was acquired in 1924 by Blackpool Corporation and was originally developed as a shelter belt for the adjacent Stanley Park Golf Course. The gardens were later developed in the 1940s as an arboretum and public open space for all to enjoy. It was
"Salisbury Woodland Gardens, Blackpool" Salisbury Woodland Gardens, Blackpool Salisbury Woodland Gardens is an open space located in the east of Blackpool, flanked by East Park Drive and Woodside Drive and linking Blackpool Zoo with Stanley Park. Known simply as the ’Woodland Gardens’ to local people, the site was acquired in 1924 by Blackpool Corporation and was originally developed as a shelter belt for the adjacent Stanley Park Golf Course. The gardens were later developed in the 1940s as an arboretum and public open space for all to enjoy. It was renovated in 1967 by Peter Perry and his ’Flying Squad (see below). Popular once
"Salisbury Woodland Gardens, Blackpool" as a wedding photograph location, the site went into decline during the 1990s. The Council’s Ranger Service manage and protect the gardens which they took over in September 2006 and have been funding and undertaking the restoration of the woodland. In 1967, Parks Director Norman Leach appointed gardener Pete Perry and his Flying Squad of gardeners to plant up the gardens. All plants, (primulas, meconopsis, etc.) were grown from seed in the greenhouses at Stanley Park, and planted ""en masse"". Extra shrubs, including azalea were also planted. The neighbouring Blackpool Zoo site was formerly Blackpool’s municipal airport. In 1927 the
"Stanley Park, Blackpool" Stanley Park, Blackpool Stanley Park is a public park in the town of Blackpool on the Fylde coast in Lancashire, England. It is the town’s primary park and covers an area of approximately . The park was designed to include significant sporting provisions, along with formal gardens, a boating lake and woodland area. It was designed and built in the 1920s, under the eye of Thomas Mawson. It is located in the Great Marton and Layton areas of the town. It is Grade II* listed and is on the Register of Historic Parks and Gardens of special historic interest in
"ZooTampa at Lowry Park" Larry Killmar, the zoo’s Director of Collections who had authorized many of Salisbury’s questionable animal transfers. Under Killmar, the zoo reorganized its internal policies over several months, and on March 27, 2009, the AZA reinstated the membership of both Lowry Park Zoo and its director of collections. The saga came to a close in August 2009 when Salisbury and the Lowry Park Zoo board agreed to a settlement in which Salisbury paid $2,200 and agreed to return all the structures, fencing, and equipment that the zoo had built at Safari Wild but did not admit to any wrongdoing. ZooTampa at
Without RAG answer: Capability Brown Top-1-passage RAG answer: Blackpool Corporation Top-5-passage RAG answer: Peter Perry.

heading: Method Summary words Answer
text: RankGPT "Salisbury Woodland Gardens, Blackpool"
enable the local community to get more involved in the sites management and interpretation. Salisbury Woodland Gardens, Blackpool Salisbury Woodland Gardens is an open space located in the east of Blackpool, flanked by East Park Drive and Woodside Drive and linking Blackpool Zoo with Stanley Park. Known simply as the ’Woodland Gardens’ to local people, the site was acquired in 1924 by Blackpool Corporation and was originally developed as a shelter belt for the adjacent Stanley Park Golf Course. The gardens were later developed in the 1940s as an arboretum and public open space for all to enjoy. It was
104 Blackpool Corporation
LONGLLMLINGUA Document [] "Salisbury Woodland Gard Blackpool
[1 "Salisbury Woodland Gard, Blackpool" ]Sbury,
Sburypool located eastpool by East Drive Drive and linkingpooloo withown asood local site acquired4pool Corporationt for Theens were0um and for. It97 by andFlying Squad (see below). Popular once
enable the local community to get more involved in the sites management and interpretation. Salisbury Woodland Gardens, Blackpool Salisbury Woodland Gardens is an open space located in the east of Blackpool, flanked by East Park Drive and Woodside Drive and linking Blackpool Zoo with Stanley Park. Known simply as the ’Woodland Gardens’ to local people, the site was acquired in 1924 by Blackpool Corporation and was originally developed as a shelter belt for the adjacent Stanley Park Golf Course. The gardens were later developed in the 1940s as an arboretum and public open space for all to enjoy. It was
Salisbury Woodland Gardens links a zoo with a park designed and built under the watchful eye of who?
158 Flying Squad
LLAMA2 Salisbury Woodland Gardens in Blackpool was designed and built under the watchful eye of Thomas Mawson in the 1920s, as part of Stanley Park. The gardens were later renovated in 1967 by Pete Perry and his ’Flying Squad’. The site is now managed by th 45 Thomas Mawson

heading: Future Work
text: On September 2, 2014, during the ABC airing of """", Walt Disney Animation Studios’ chief creative officer John Lasseter announced that a ""Frozen"" short film with a new song would be released in the future. On the same day, ""Variety"" announced that the short
105 2015
LONGLLMLINGUA Making Today a Perfect ""aking Today a Perfect Day is a song from the 20 Walt Disney Animation Studios-animated short ""Fro F"", music and lyrics Kristen Anderson-Lopez and Robert Lopez throughout most of short It was single in United States on March 12, 2015. On September 2, 2014, during ABC airing of Walt Disney Animation chief creative officer John Lass announced aFrozen"" short film with a new would be released future same day,Vari announced that the [Making Today a
Document [4M Day [fect Day" Frozen Fever"", with music
What is the year of the event that occured first, Making Today a Perfect Day was produced, or Frozen was produced?
109 2015
LLAMA2 The event "Making Today a Perfect Day" was produced first, in 2015. 12 2015
Question: Salisbury Woodland Gardens links a zoo with a park designed and built under the watchful eye of who? Standard Answer: Thomas Mawson