Venue: NAACL 2025

# K-comp: Retrieval-Augmented Medical Domain Question Answering

With Knowledge-Injected Compressor

 Jeonghun Cho

POSTECH GSAI

jeonghuncho@postech.ac.kr &Gary Geubae Lee

POSTECH GSAI

POSTECH CSE

gblee@postech.ac.kr

###### Abstract

Retrieval-augmented question answering (QA) integrates external information and thereby increases the QA accuracy of reader models that lack domain knowledge. However, documents retrieved for closed domains require high expertise, so the reader model may have difficulty fully comprehending the text. Moreover, the retrieved documents contain thousands of tokens, some unrelated to the question. As a result, the documents include some inaccurate information, which could lead the reader model to mistrust the passages and could result in hallucinations. To solve these problems, we propose **K-comp** (**Knowledge-injected COMPressor**) which provides the knowledge required to answer correctly. The compressor automatically generates the prior knowledge necessary to facilitate the answer process prior to compression of the retrieved passages. Subsequently, the passages are compressed autoregressively, with the generated knowledge being integrated into the compression process. This process ensures alignment between the question intent and the compressed context. By augmenting this prior knowledge and concise context, the reader models are guided toward relevant answers and trust the context.

## 1 Introduction

Retrieval-augmented question answering (QA) is a task where passages related to a question are appended to the prompt such that a reader model can reference them and infer the correct answer Ahmad et al. (2019); Guo et al. (2021). However, several limitations impede retrieval-augmented approaches in closed domains with large language models (LLMs) as readers. First, the documents retrieved for closed domains require domain expertise, so the reader may not trust the whole text. When faced with unfamiliar input, the model exhibits an availability bias toward commonly known knowledge, making it more willing to believe in information it can easily recall Jin et al. (2024). Also, retrieved passages contain thousands of tokens and are sometimes unrelated to the question. This can cause the language model to distrust the passages, perceive them as irrelevant noise, and generate answers that do not consider them. These problems lead to hallucinations Ji et al. (2023a), which result in the model generating inaccurate answers or inferring plausible but false responses. Lastly, LLMs are sensitive to the order of retrieved documents and the prompting method. Specifically, LLMs can have difficulty finding the necessary information within lengthy input prompts, especially when key information or correct answer clues are located in the middle of the prompt Liu et al. (2024); Xu et al. (2024b).

To address these issues, we propose K-comp (knowledge-injected compressor). We aim to use an autoregressive LLM as a compressor with the domain knowledge needed to answer the question, and increase the alignment of the retrieved passages with the question intent. Additionally, when the compressor is trained in domain-related terms and information, it becomes able to recognize the entities that occur in the question and provide descriptions for them. This process is significant for closed domains that require substantial prior knowledge. For retrieval augmentation, we use a large amount

Figure 1: K-comp helps the reader model infer accurate responses by using domain knowledge and compressed context aligned with the question.



of text from domain-specific sources, including Wikipedia. We exploit the advantages of domain relevance by efficiently reusing it when annotating prior knowledge, not just for retrieval. Furthermore, we use a causal masking objective Aghajanyan et al. (2022) during the training phase to inject domain knowledge into the compressor.

In summary, our contributions are as follows:

* We propose a novel approach to generate knowledge-injected summaries adapted for the medical domain. We incorporate causal masking to inject knowledge into the compressor without modifying its structure. This approach ensures that the summary is aligned with the question.
* Even without domain knowledge in the reader model, K-comp provides the description of the medical jargon to answer the question, thereby enabling LLMs with diverse backgrounds to handle medical questions more accurately.
* Experiments on three medical datasets show that K-comp improves performance over other query-based prompt compression methodologies and standard retrieval-augmented generation (RAG) without compression.
* K-comp has been shown to be effective when applied to previously unseen data, thereby presenting evidence that our method provides additional novel contributions in data-scarce closed domain environments.

## 2 Related Work

Text InfillingModels such as BERT Devlin et al. (2019), SpanBERT Joshi et al. (2020), T5 Raffel et al. (2020), and BART Lewis et al. (2020), are pre-trained using masked language modeling within a bidirectional encoder architecture. They have shown strong performance in infilling short and contiguous masked token spans. However, the bidirectional attention mechanism typically restricts the fillable span length to dimensions significantly shorter than a sentence.

In contrast, decoder-only models such as GLM Du et al. (2022), CM3 Aghajanyan et al. (2022), and InCoder Fried et al. (2023) operate by left-to-right generation. They can accommodate variable infill span lengths. Causal masking Aghajanyan et al. (2022) or fill-in-the-middle Bavarian et al. (2022) methods predict masked spans from the posterior context. These methods have their generative capabilities, which increase the length of infill spans. They can also exploit the advantages of considering contextual relationships that surround the masked span. The proposed method has the capability to fill the span by considering bidirectional context, as well as align the generated summary with the question by regressively encoding the infilled span.

Prompt CompressionSeveral studies have demonstrated that prompt augmentations effectively enhance the performance of LLMs across various tasks Liu et al. (2023a); Ram et al. (2023); Ryu et al. (2023); Wang et al. (2024c); Long et al. (2023); Yagnik et al. (2024). Yet, the relevance and reliability of the augmented passages are significant challenges in prompt augmentations. In order to address this issue, recent studies have attempted to extract content from ambiguous and lengthy passages directly. Kim et al. (2024) eliminates irrelevant information while maximizing the extraction of accurate information, whereas Yang et al. (2023) leverages the black-box LLMs by applying a reward-based method during compressor training to generate summaries. RECOMP Xu et al. (2024a) selects and augments the summary with the highest end-task performance by using prompts in which non-essential summaries are set to empty strings if necessary. LMLLingua Jiang et al. (2023a) dynamically assigns different compression rates to various components within the prompt. In contrast, K-comp focuses on the keywords needed to answer the question, emphasizing the alignment between the compressed context and the question.

## 3 Causal Knowledge Injection

Causal models trained using autoregressive language modeling rely exclusively on the context to the left of generated tokens to predict subsequent tokens Brown et al. (2020). This attribute confers an _advantage in causally generating entire documents_, such as text generation. However, these models show limited proficiency in tasks that require an understanding of post-positional relationships for span infilling. Conversely, masked language models excel at _predicting masked spans_ by referencing attention scores from tokens located 

both anteriorly and posteriorly_. Nonetheless, their training objective is limited to decoding only short segments of passages Devlin et al. (2019); Joshi et al. (2020).

We are inspired by causal masking Aghajanyan et al. (2022) that combines the advantages of both objectives. We focus on the _masked medical entities_ within the _question (prior context)_ and aim to predict them by considering the _retrieved snippets (subsequent context)_. Subsequently, by _auto-regressively compressing the retrieved snippets_, we can effectively leverage both advantages.

## 4 Methods

In this section, we report our proposed approach for knowledge-injected compression and retrieval augmentation. To retrieve passages similar to a question, we construct a retrieval pipeline composed of a large corpus (SS4.1). Next, we explain the data processing steps for training (SS4.2). Finally, we detail the training scheme for K-comp with the proposed objective and explain the inference phase for retrieval augmentation (SS4.3). Figure 1 shows an overview of the prompts that K-comp consists of.

### Retrieval Framework

Closed domain tasks have not been as thoroughly explored as open domain tasks, which have achieved notable performance enhancements using Wikipedia as a retrieval corpus Karpukhin et al. (2020). In contrast to open domains, the challenge in closed domains is that unified corpora have not been established. Research endeavors, such as Xiong et al. (2024); Wang et al. (2024b), are currently underway to address this gap. To ensure coverage of both general and domain knowledge, we adopt the MedCorp corpus Xiong et al. (2024) as our retrieval corpus. It combines Wikipedia, PubMed1, StatPearls2, and textbooks Jin et al. (2021). As our retriever, we employ embedding-based \(k\)-NN search Johnson et al. (2019) to mitigate bottlenecks and efficiently execute similarity searches on our large-scale corpus comprising four distinct text corpora3.

Footnote 1: [https://pubmed.ncbi.nlm.nih.gov/](https://pubmed.ncbi.nlm.nih.gov/)

Footnote 2: [https://www.statpearls.com/](https://www.statpearls.com/)

Footnote 3: We use Nomic Embed Nussbaum et al. (2024).

### Ground-Truth Data

Entity-DescriptionWe rely on off-the-shelf tools to perform named-entity recognition4, which identifies biomedical entities \({\cal E}=\left\{e_{i}\right\}\) in each question for masking. Retrieval corpus \(\mathbb{C}\) is constituted of title and text pairs, with the first sentence of each text assumed to be a short description of the title Xu et al. (2023). Subsequently, the pairs of titles and short descriptions are matched with the entities and their corresponding knowledge \(d_{i}\). We assume that the questions in the training dataset contain at least one entity. In the absence of an entity in a given question, the data are excluded. Similarly, instances lacking a corresponding title in the retrieval corpus are also filtered out of the training dataset (Table 15).

Footnote 4: We use ScispaCy Neumann et al. (2019) package.

However, in the test data, K-comp unveils a novel contribution by automatically generating domain-specific entity descriptions during inference even when no annotate exists for the entity in question. This obviates the need for costly and unnecessary tasks, such as searching for medical terms or finding definitions within the corpus.

SummaryTo synthesize gold summaries \({\cal S}\), GPT-4o-mini5 compresses the passages by considering \(\left\{{\cal P},{\cal E}\right\}\) input pairs, and the number of passages used for synthesis is set to five, i.e., \(|{\cal P}|=5\). Notably, we explicitly prohibit the inclusion of the question in the summary synthesis process. This is because incorporating the question into the input prompt for generating the summary may result in a focus shift from the generation of keyword-focused summaries to the formulation of a summary that is aimed at answering the question. Detailed instructions for the summary synthesis are provided in Table 8.

Footnote 5: We use gpt-4o-mini-2024-07-18 OpenAI (2024).

### K-comp

Preliminary\(q=\left[q^{1},q^{2},...,q^{N}\right]\), where \(q^{N}\) represents the \(N\)-th token in \(q\). We use the special token <ent> to mask each medical entity spans within \(q\), \(q_{m}=\left[q^{1},...,\texttt{<ent>},...,q^{N-l}\right]\). Also, a special <eod> token is appended at the end of the description of the corresponding entity, \(d_{i}=\left[d_{i}^{1},...,d_{i}^{M},\texttt{<eod>}\right]\). An example is provided as 

follows:

\[q_{m}=\text{What are the <ent> of <ent>?}\] \[d_{1}=\text{symptom: \{\{description\}\}$<eod>}\] \[d_{2}=\text{Down syndrome: \{\{description\}\}$<eod>}\]

By concatenating \(q_{m}\) and \(\mathcal{P}\) in the correct sequence, the masked spans can be predicted based on the preceding and subsequent context. We define the dataset for the compressor as \((q_{m}\oplus\mathcal{P},\mathcal{S},\mathcal{E},\mathcal{D})\), where \(\mathcal{D}=\{d_{i}\}\) and \(\mathcal{S}\) is a gold summary.

TrainingGiven an input query \(q_{m}\) and the set of retrieved passages \(\mathcal{P}=\{p_{1},p_{2},...,p_{5}\}\), K-comp aims to train a causal model \(f(q_{m}\oplus\mathcal{P})\) to generate \(\mathcal{E}\), \(\mathcal{D}\), and then \(\mathcal{S}\) auto-regressively.

The compressor is trained to encode \(q_{m}\oplus\mathcal{P}\) to generate <ent> tokens and their corresponding descriptions:

\[P_{\theta}(\mathcal{E},\mathcal{D}|q_{m}\oplus\mathcal{P})\\ =\prod_{i}\left(\prod_{\alpha,\beta}P_{\theta}(e^{\alpha}_{i}, d^{\beta}_{i}|e^{<\alpha}_{i},d^{<\beta}_{i},q_{m}\oplus\mathcal{P})\right)\]

where \(\theta\) represents the parameters of K-comp.

This approach facilitates the incorporation of descriptions into the prompt for the reader model and ensures that the generated entities and their descriptions are regressively encoded. As a result, a summary is generated in a causal manner with attention to the entities within the question and their related knowledge, thereby composing a summary centered on these domain entities.

\[P_{\theta}(s|\mathcal{E},\mathcal{D},q_{m}\oplus\mathcal{P})\\ =\prod_{\gamma}P_{\theta}(s^{\gamma}|s^{<\gamma},\mathcal{E}, \mathcal{D},q_{m}\oplus\mathcal{P})\]

We fine-tuned the compressor using the standard next token prediction with cross-entropy loss:

\[P_{\theta}(\mathcal{E},\mathcal{D},s|q_{m}\oplus\mathcal{P})\\ =P_{\theta}(\mathcal{E},\mathcal{D}|q_{m}\oplus\mathcal{P})\times P _{\theta}(s|\mathcal{E},\mathcal{D},q_{m}\oplus\mathcal{P})\\ \therefore L(\theta)=-\mathbb{E}(\log P_{\theta}(\mathcal{E}, \mathcal{D},s\mid q_{m}\oplus\mathcal{P}))\]

InferenceAt inference time, documents are retrieved in advance to construct the compressor input batch \(\{\mathcal{P},q_{m}\}\). This enables the sequential autoregressive generation of entities and descriptions from the question until the <eod> token is produced. The overall context, including entities and descriptions, is then considered, and a summary that aligns more closely with the question is generated. This process ultimately constructs the input prompt for the reader model, ensuring a reliable response to the question.

For all datasets, we use a 0-shot setting in our experiments. The prompt examples for the reader model can be found in Table 11.

## 5 Experiments

In this section, we evaluate K-comp trained by causal knowledge injection and the retrieval-augmented QA task. We report the datasets and settings used in the experiments (SS5.1) and discuss the main results (SS5.2).

### Settings

ModelsWe fine-tuned Gemma-2B Team et al. (2024) with our knowledge injection objective. Further details regarding the models and implementation can be found in Appendix A.

DatasetsTo reduce potential biases from fine-tuned medical LLMs Han et al. (2023); Chen et al. (2023), we conduct experiments using the medical QA datasets MedQuAD Ben Abacha and Demner-Fushman (2019), MASH-QA Zhu et al. (2020), and BioASQ Krithara et al. (2023), which were not directly used for training biomedical models. Although MASH-QA and BioASQ provide gold passages containing answers, our experiments do not utilize these gold passages. Instead, we rely on passages retrieved by our retrieval framework.

Evaluation MetricsSince all datasets consist of long-form answers, we use the trained model to evaluate the answers. We quantify the relevance of answers using BertScore Zhang* et al. (2020), which evaluates the similarity between two sentences by exploiting the contextual embeddings of the encoder. We also use UniEval Zhong et al. (2022), which is a multi-dimensional evaluation metric that has high correlation and similarity with human judgment. We explicitly assess the factual consistency between generated and gold answers.

### Results

BaselinesWe compare K-comp with standard RAG approach with top-1 and top-5 retrieved passages without applying prompt compression. We also compare with previous state-of-the-art prompt 

compression methods, including RECOMP [24] and LLMLingua [19]. Specifically, for implementing RECOMP, we use an abstractive compressor fine-tuned on our datasets, and for LLMLingua, we use Llamaz-2-7B [30] for compression. The prompts for synthesizing the summaries used in RECOMP are based on the paper and can be found in Table 9. Furthermore, the efficacy of causal knowledge injection is evaluated by comparing it to a model that has been fine-tuned (FineTune) using only the standard language modeling objective for summarization. FineTune fine-tuned with Gemma-2B, the same as K-comp.

Overall PerformanceTable 1 shows the main results of K-comp compared to the baselines across various reader LLMs. Overall, compression methods are effective. Chunking snippets for retrieval is inherently imperfect, making the Top-1 and Top-5 passages suboptimal. For MedAlpaca, which has the smallest context window size of 2048 among the reader models, the answer accuracy declines significantly with Top-5 passages input due to the limited window size. Consequently, a reprocessing stage, such as compression, is required to improve the quality of chunked text and enable the reader model to reference it appropriately. Among the baselines, LLMLingua lags behind other baselines trained in the medical domain due to its query-agnostic compression approach. We also observe different results depending on the model size. Larger models are less dynamic in their response, relying more on their internal knowledge and less on the prompt variations. As can be seen in the case study (SS6.3) and Table 14, larger models are capable of providing reasonable responses to questions even when presented with noisy input prompts. In other words, the result demonstrates that not only small models but also large models place greater trust in the description and concise text provided by K-comp than in other baselines.

To emphasize the importance of entity and description, we analyze a scenario where K-comp infers normally but only appends the summary to the reader prompt (Table 2). \(-Prior\) is comparable to the baseline fine-tuned for summarization tasks. Even so, it is clear that providing the reader model with prior knowledge significantly improves the accuracy of the final responses compared to FineTune. An interesting observation is that general-purpose models exhibit a more significant influence of information related to medical jargon compared to medical LLMs. Medical LLMs seem to treat knowledge about entities as noisy input, resulting in conflicts with their internal knowledge. Still, these analyses are confined to the QA accuracy of reader LLMs, as they are affected by changes in the components that make up the prompt. The following sections will discuss the relevance and alignment of the summary.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline  & \multicolumn{4}{c}{**General-purpose LIMs**} \\  & \multicolumn{2}{c}{**Licensema-3JB**} & \multicolumn{2}{c}{**Licensema-3JOB**} & \multicolumn{2}{c}{**Mixtral-NaTB**} & \multicolumn{2}{c}{**GPT-4o\({}^{*}\)**} & \multirow{2}{*}{**Medal-Apoca-1BB**} & \multirow{2}{*}{**Defuction-7bB**} \\  & \multicolumn{2}{c}{**BerScore**} & \multicolumn{2}{c}{**UnifEval**} & \multicolumn{2}{c}{**BerScore**} & \multicolumn{2}{c}{**UnifEval**} & \multicolumn{2}{c}{**BerScore**} & \multicolumn{2}{c}{**UnifEval**} & \multicolumn{2}{c}{**BertScore**} & \multicolumn{2}{c}{**UnifEval**} \\ \hline \multicolumn{13}{c}{**Without compressor**} \\ Top-1 document & 75.65 & 53.44 & 79.44 & 57.47 & 58.68 & 44.30 & 84.45 & 60.24 & 75.05 & 36.89 & 76.38 & 52.74 \\ Top-5 documents & 76.20 & 52.45 & 79.95 & 55.25 & 64.17 & 47.47 & 83.08 & **64.98** & 17.57 & 10.82 & 73.14 & 51.88 \\ \hline \multicolumn{13}{c}{**With compressor**} \\ RECOMP & 73.27 & 55.74 & 79.75 & **63.57** & 71.58 & 58.78 & 52.60 & 69.02 & 82.52 & 39.15 & 76.41 & 56.14 \\ LLMInguas & 76.45 & 54.65 & 78.78 & 57.00 & 59.79 & 45.34 & 84.46 & 53.04 & 83.10 & 44.17 & 74.44 & 54.18 \\ FireTime & 74.27 & 54.89 & 79.97 & 56.77 & 74.14 & 56.26 & 83.47 & 62.40 & 82.25 & 39.40 & 76.14 & 55.97 \\ K-compf & **83.79** & **64.05** & **82.06** & 62.91 & **80.64** & **63.07** & **85.49** & 63.16 & **85.75** & **49.39** & **78.38** & **58.95** \\ \hline \multicolumn{13}{c}{**Wishout compressor**} \\ Top-1 document & 77.87 & 54.08 & 82.65 & 56.48 & 72.63 & 51.88 & 83.39 & 60.66 & 76.95 & 43.41 & 80.45 & 55.31 \\ Top-5 documents & 77.59 & 52.94 & 83.07 & 57.20 & 73.99 & 53.18 & 83.12 & **64.81** & 31.91 & 20.07 & 79.31 & 55.85 \\ \hline \multicolumn{13}{c}{**With compressor**} \\ RECOMP & 80.20 & 58.80 & 82.27 & 60.50 & **82.12** & 62.35 & 83.82 & 62.22 & 78.51 & 34.83 & 81.54 & 60.09 \\ LLMInguas & 79.74 & 54.37 & 83.25 & 56.43 & 76.71 & 85.32 & 83.37 & 56.73 & 80.41 & 44.85 & 80.78 & 58.55 \\ FinTime & 77.30 & 55.21 & 82.90 & 60.66 & 78.74 & 60.62 & 83.88 & 62.82 & 79.96 & 44.02 & 81.70 & 60.06 \\ K-compf & **83.45** & **61.45** & **83.64** & **61.40** & 81.49 & **62.91** & **83.92** & 64.06 & **81.95** & **51.76** & **82.67** & **61.85** \\ \hline \multicolumn{13}{c}{**Without compressor**} \\ Top-1 document & 82.53 & 56.03 & 81.40 & 59.58 & 82.23 & 62.07 & 87.13 & 62.47 & 76.86 & 40.06 & 83.43 & 57.55 \\ Top-5 documents & 81.44 & 61.33 & 80.91 & 61.09 & 83.54 & 65.53 & 86.81 & **64.15** & 20.93 & 12.41 & 83.63 & 59.52 \\ \hline \multicolumn{13}{c}{**With compressor**} \\ RECOMP & 83.89 & 59.80 & 85.45 & 59.65 & **86.85** & 62.82 & 87.87 & 60.62 & 78.07 & 30.68 & 84.77 & 57.58 \\ LLMInguas & 81.70 & 53.62 & 86.53 & 61.53 & 82.54 & 62.24 & 86.85 & 51.11 & 81.19 & 42.97 & 81.51 & 55.26 \\ FireTime & 82.26 & 60.39 & 86.17 & 62.86 & 85.48 & 64.01 & **87.97** & 63.99 & 78.67 & 41.49 & 85.05 & 61.20 \\ K-compf & **86.61** & **62.28** & **87.03** & **62.99** & 86.52 & **66.31** & 87.77 & **64.38** & **84.88** & **45.47** & **86.70** & **62.90** \\ \hline \end{tabular}
\end{table}
Table 1: Main results. We report automatic evaluations for retrieval-augmented QA with and without compressors.

 



## 6 Analyses

We analyze the results from various perspectives (SS6.1, 6.2, 6.3). Finally, we appraise the outcomes focusing on the GPT-4o evaluation to highlight the advantages of K-comp through comparison with previous studies (SS6.4).

### Reranking Preference

In addition to evaluating the end-task performance, it is crucial in RAG to ensure that prompts are augmented to be pertinent to the question. Although human evaluation is valuable, it demands significant resources and domain expertise, which are not readily available in our case. Instead, we propose to employ a state-of-the-art reranker6 to measure the relevance between the context and the question. For each question \(q\), we execute K-comp to generate 10 contexts using a high temperature setting (temperature=1) based on \(q_{m}\oplus\mathcal{P}\). Next, we retrieve the top-10 passages related to \(q\). Thus, we gather a total of 20 passages to be fed to the reranker. By applying Recall@\(K\) to these 20 passages, we observe the \(K\) passages that are most similar to \(q\), and quantify the proportion of the compressor varied as \(K\) varied.

Footnote 6: We use BAAI/bge-reranker-large Xiao et al. (2024).

Figure 2 illustrates Recall@\(K\) across different values of \(K\). Specifically, we achieved Recall@1 scores of 77%, 73%, and 83% on MedQuAD, MASH-QA, and BioASQ, whereas the top-5 retrieved passages achieved 23%, 27%, and 17%. This comparison demonstrates that the reranker strongly prefers our compressed contexts across all three benchmarks.

### Inference Speed

In Table 3, we report the inference time and the number of tokens used in the prompt input as metrics for evaluating efficiency. Specifically, we employed Llama-3-70B as the reader model and measured the GPU runtime on MASH-QA test set. Both the compressor and reader are executed on a single NVIDIA A100 GPU with 80GB memory. Even when considering the time needed for the compressor inference, our method was able to double the throughput compared to prepending the top-5 passages, making it more efficient. Moreover, we note that inference speed is dependent on the implementation and size of the reader model. For

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{**Llama-3-SB**} & \multicolumn{2}{c}{**Llama-3-70B**} & \multicolumn{2}{c}{**MedAlpaca-13B**} & \multicolumn{2}{c}{**Meditron-70B**} \\  & **BertScore** & **UniEval** & **BertScore** & **UniEval** & **BertScore** & **UniEval** & **BertScore** & **UniEval** \\ \hline \multirow{2}{*}{K-comp} & \multirow{2}{*}{**83.79**} & \multicolumn{2}{c}{MedQuAD} & \multirow{2}{*}{**85.75**} & \multirow{2}{*}{**49.39**} & \multirow{2}{*}{78.88} & \multirow{2}{*}{58.95} \\  & & & & & & & & \\ \cline{1-1} \cline{6-10} \multirow{-2}{*}{\(-Prior\)} & \multirow{2}{*}{73.92} & \multirow{2}{*}{54.47} & \multirow{2}{*}{80.06} & \multirow{2}{*}{60.03} & \multirow{2}{*}{84.84} & \multirow{2}{*}{48.23} & \multirow{2}{*}{**79.80**} & \multirow{2}{*}{**59.00**} \\ FineTune & & & & & & & & \\ \hline \multirow{3}{*}{F-True} & \multirow{3}{*}{74.27} & \multirow{3}{*}{54.89} & \multirow{3}{*}{79.97} & \multirow{3}{*}{56.67} & \multirow{3}{*}{82.25} & \multirow{3}{*}{39.40} & \multirow{3}{*}{76.14} & \multirow{3}{*}{55.97} \\  & & & & & & & & \\ \hline \multirow{3}{*}{K-comp} & \multirow{3}{*}{**83.45**} & \multirow{3}{*}{**61.45**} & \multirow{3}{*}{**83.64**} & \multirow{3}{*}{61.40} & \multirow{3}{*}{81.95} & \multirow{3}{*}{51.76} & \multirow{3}{*}{**82.67**} & \multirow{3}{*}{**61.85**} \\ \cline{1-1} \cline{6-10} \multirow{-2}{*}{\(-Prior\)} & \multirow{2}{*}{78.63} & \multirow{2}{*}{57.80} & \multirow{2}{*}{83.16} & \multirow{2}{*}{60.88} & \multirow{2}{*}{**82.18**} & \multirow{2}{*}{**52.02**} & \multirow{2}{*}{81.93} & \multirow{2}{*}{60.42} \\ FineTune & & & & & & & & \\ \hline \multirow{3}{*}{F-Comp} & \multirow{3}{*}{**86.61**} & \multirow{3}{*}{**62.28**} & \multirow{3}{*}{**87.03**} & \multirow{3}{*}{**62.89**} & \multirow{3}{*}{**84.88**} & \multirow{3}{*}{45.47} & \multirow{3}{*}{**86.70**} & \multirow{3}{*}{**62.90**} \\ \cline{1-1} \cline{6-10} \multirow{-2}{*}{\(-Prior\)} & \multirow{2}{*}{80.75} & \multirow{2}{*}{55.06} & \multirow{2}{*}{86.22} & \multirow{2}{*}{62.40} & \multirow{2}{*}{84.53} & \multirow{2}{*}{**46.67**} & \multirow{2}{*}{85.23} & \multirow{2}{*}{59.73} \\ FineTune & & & & & & & & \\ \hline \hline \end{tabular}
\end{table}
Table 2: Ablation studies. \(-Prior\) denotes the scenario where K-comp does not provide prior knowledge to the reader LLMs.

\begin{table}
\begin{tabular}{l|r r r} \hline \hline

- & 497s \\ Total time & 9,798s & 20,193s & 10,274s \\ \hline \hline \end{tabular}
\end{table}
Table 3: Inference speed of Llama-3-70B on MASH-QA.

Figure 2: Percentage of Recall@\(K\) according to the variation of \(K\) for the retrieved passages and our compressed contexts, where Top-10 denotes the ten retrieved passages with the highest similarity scores.



instance, models with more parameters will suffer increased latency by increasing the number of input tokens. This phenomenon amplifies the speed advantage of K-comp.

### Case Study

Here, we report how K-comp generates medical knowledge. In Table 4, K-comp is able to address the practical utilization of X-rays in the diagnosis and monitoring of RA, and provides detailed explanations regarding their application, which offers a more transparent rationale for addressing the questions. By contrast, FineTune provides a more general context without focusing on the specific role of X-rays in RA diagnosis. Although it mentions X-rays along with other diagnostic techniques, its focus is on advancements in imaging methods such as MRI. FineTune merely summarises the passages retrieved based on semantic and overall lexical similarities to the question without considering the queried intent. This results in the reader model does not fully trusting the augmented passages, instead perceiving them as irrelevant noise and generating answers not based on the passages. This result can lead to inaccuracies and potential hallucinations.

### GPT-4o Evaluation

We additionally explore the reliability of the context. Given that GPT-4 has been demonstrated to correlate highly with human judgments Liu et al. (2023), even in the medical domain Nori et al. (2023), we employed GPT-4o7 to perform a comparative evaluation of summaries generated by baselines and K-comp. All prompts utilized in the evaluation were structured using the identical format as presented in Table 12. Finally, we report examples of results for all baselines in Table 18 for qualitative analysis.

Footnote 7: We use gpt-4o-2024-05-13 Hurst et al. (2024).

Query-AgnosticFigure 3 compares our approach to previous studies that compress prompts both with and without relying on the query. Intuitively, the baselines compared in query-agnostic section are compressed regardless of the question, which demonstrates that K-comp outperforms the other methods because it references the masked question. Detailed analyses are provided in Appendix B.

Query-BasedSPLADE Lassance and Clinchant (2022) is a lexical-based retriever that reweights each term by emphasizing important terms associated with the query. The most informative top-1 passages among the top-5 retrieved passages are extracted and assumed to be compressed context. GPT-4o-mini 5 is the result of a summary generated using prompt 9, which is the same prompt used when synthesizing the RECOMP train data. The rationale for K-comp being rated higher than RECOMP is that even when trained on that dataset, it produces a summary that lacks specialization, which is similar to the output of FineTune in case study. By using a seq2seq model rather than token-level pruning such as LLMLingua, it allows

\begin{table}
\begin{tabular}{l|p{65.4pt}|p{142.3pt}} \hline  & \multicolumn{2}{c|}{Question: How do **x-rays** diagnose rheumatoid arthritis (RA)?} \\ \cline{2-3}  & Augmented Prompt & Answer \\ \hline FineTune & **ffff** Passage** & X-rays are used to diagnose RA by detecting joint damage and erosion. \\  & Rheumatoid arthritis (RA) is a chronic systemic disease characterized by inflammation of the synovial tissue, affecting 0.5\%-1.0\% of the global population, with ... & X-rays are used to diagnose RA by joint damage and erosion. \\ \hline K-comp & **ffff** Entity & X-rays diagnose rheumatoid arthritis (RA) by revealing changes in the axial skeleton and joint space. As the disease progresses, early RA may not show significant changes on X-rays, but it can indicate osteopenia and joint space narrowing. Advances in imaging techniques like MRI and ultrasound enhance the ability to detect soft tissue changes and joint damage, which is crucial for accurate diagnosis ... \\ \hline \end{tabular}
\end{table}
Table 4: Case study. We provide the passages used to augment the reader’s prompt and the answers. **Red texts** highlight the medical jargon within the question. The complete prompt can be found in Table 14.

 

the delivery of complete sentences to the reader LLM in QA tasks. This prevents the LLM from perceiving excessive noise, thereby ensuring relatively strong end-task performance among the baselines (Table 1). However, due to the maximum token limitation, the generated text is shorter and contains less information in the context, which results in lower evaluation scores for summary quality.

K-comp demonstrates comparable performance to GPT-4o-mini, particularly exhibiting significant superiority on MedQuAD. In query-based comparisons, GPT-4o-mini generates a rationale that is highly effective for answering based on its demonstrated capabilities in text generation. Similarly, K-comp exhibits performance comparable to that of GPT-4o-mini, despite being composed of only 2B parameters.

Unseen EvaluationIn order to provide additional novelty to our approach, we evaluate baselines on data that was not used during training. Unlike query-based methods, such as RECOMP, which are trained to generate summaries optimized for responding to questions, our approach, based on the entities, demonstrates efficacy when applied to unseen data. The right-hand column of Figure 3 illustrates the results of applying each baseline to 140 test data from the MEDIQA Ben Abacha et al. (2019). As discussed in Appendix B, SelectiveContext Li et al. (2023) and LLMLingua generate incomplete sentences, which result in inferior performance despite their compression in a questionable fashion. In contrast, K-comp maintains a competitive performance on unseen data, comparable to the results evaluated on seen data.

A noteworthy point is the comparison result with query-based baselines. As can be seen in Table 8 and 9, the instructions used for synthesizing summaries in RECOMP and K-comp were as follows: "Compress ... used to answer the question" and "Extract the content about the entity", respectively. By focusing on the entities, the objective of our training approach is to provide a concise context of the medical terminology that has been requested in the question. As a consequence, K-comp achieves a win rate that is similar to, and even exceeds, the results obtained in the training datasets. This presents a limitation of previous methods, which are unable to maintain their performance levels due to their reliance on the distribution of training data. On the other side, our training approach focuses on medical jargon, which is advantageous because the medical terminology remains consistent even when the data changes. Therefore, our causal knowledge injection substantially contributes to improving performance in data-scarce, closed-domain settings.

Additional NLG EvaluationThe validity of our method has been established through rigorous empirical validation employing the BertScore

Figure 3: GPT-4o evaluation results with baselines. t-BioASQ indicates that MEDIQA was inferred using K-comp (or RECOMP) trained on BioASQ. For clarity, results in the 0% range are not indicated. Accordingly, we report all results in Table 17. (Abovey Query-agnostic prompt compression, (Below) Query-based prompt compression methods, (Left) Results from the seen data, (Right) Results from the unseen data.

 

and UniEval evaluation metrics. To further enhance methodological consistency and ensure a comprehensive evaluation, we have incorporated G-Eval Liu et al. (2023), a cutting-edge assessment metric, to perform an extensive supplementary analysis. This evaluation is conducted leveraging a highly advanced commercial large language model (LLM), thereby reinforcing the reliability and validity of the proposed approach. For G-Eval-4, GPT-4 is sampled 20 times, with the resulting average score used to minimize the impact of any potential variability. Given the broad scope of our methodological evaluation, which covers diverse datasets and a wide range of models, evaluating a full-scale analysis with G-Eval would be extremely computationally expensive. As a result, 1k data points per dataset were randomly selected as the test data. The results are reported in Table 5. Regarding performance, K-comp consistently exhibits superior performance compared to other baselines.

Interestingly, our results align closely with the trends observed in UniEval. According to the G-Eval study, UniEval exhibited a stronger correlation with human judgments in summary, dialogue generation, and consistency evaluation than all baselines except G-Eval-4. This trend is also reflected in our findings. In Table 1, when GPT-4o is used as the reader model, the responses generated by augmenting the top-5 documents achieve the highest scores, a pattern consistent with G-Eval. Additionally, on the MedQuAD dataset, when Llama-3-70B serves as the reader, UniEval shows a preference for RECOMP, which aligns with the corresponding G-Eval results. Similarly, in other scenarios, both metrics indicate a preference for K-comp. These observations suggest that UniEval effectively mirrors G-Eval despite being a relatively older metric. While our study does not include human evaluation, the strong alignment between UniEval and G-Eval suggests that our methodology is likely to correlate well with human judgments.

## 7 Conclusion

In this paper, we have proposed a novel method to improve retrieval-augmented QA by compressing retrieved documents focused on the questions. We have devised a comprehensive scheme for identifying medical entities and automatically generating prior knowledge. This is followed by the extension of training and inference methods, which enable the autoregressive generation of summaries that incorporate domain knowledge while considering the context causally. Furthermore, we proved that this approach is practical even when applied to unseen evaluation, which represents a novel contribution in closed domains where data is scarce.

## Acknowledgements

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.RS-2019-III91906, Artificial Intelligence Graduate School Program(POSTECH))

This research was supported by Smart HealthCare for Police Officers Program(www.kipot.or.kr) through the Korea Institutes of Police Technology(KIPoT) funded by the Korean National Police Agency(KNPA, Korea)(No. RS-2022-PT000186)

## Limitations

Our methodology is limited in scenarios where the NER tool is unable to automatically detect ambiguous keywords or entities that are absent from the questions. To mitigate these issues, expanding the retrieval corpus with additional text chunks can inject more knowledge into the compressor and learn domain-relevant entities, but this will drastically increase the cost of annotating the data and require enormous resources for retrieval to perform nearest-neighbor searches. Therefore, we consider extending these retrieval datastores an important

\begin{table}
\begin{tabular}{l c c c c c c} \hline  & **Lima-8h** & **Lima-70b** & **Mixtral** & **GPT-4e** & **MedApaca** & **Motifus** \\ \cline{2-7}  & \multicolumn{5}{c}{ModQuAD} \\ \hline
**Without compression** & & & & & \\ Top-1dcc & 2.96 & 3.11 & 2.63 & 4.27 & 1.65 & 3.14 \\ Top-5s codec & 3.22 & 3.36 & 3.07 & **4.73** & 1.85 & 3.53 \\
**With compressers** & & & & & & \\ HLC/Imgas & 2.29 & **3.80** & 3.51 & 3.92 & 1.48 & 3.68 \\ Linktopes & 2.41 & 2.87 & 2.53 & 3.68 & 1.61 & 3.21 \\ Four-tune & 3.28 & 3.46 & 3.27 & 4.15 & 1.44 & 3.68 \\ k-conv & **3.55** & 3.66 & **3.73** & 4.48 & **1.98** & **3.85** \\ \hline
**Whatout compressed** & & & & & & \\ Top-1dcc & 3.07 & 3.65 & 3.24 & 4.63 & 1.64 & 3.55 \\ Top-5 docs & 3.33 & 3.58 & 3.46 & **4.74** & 1.79 & 3.43 \\ \hline
**With compresser** & & & & & & \\ IRCO/imgas & 2.68 & 4.14 & **4.49** & 4.15 & 1.34 & 4.01 \\ ILML/Imgas & 2.97 & 3.18 & 3.77 & 3.99 & 1.88 & 3.47 \\ remote & 3.36 & 3.79 & 3.78 & 3.93 & 1.74 & 3.51 \\
**K-conv** & **4.23** & **4.62** & **4.06** & 4.52 & **1.99** & **4.53** \\ \hline
**Wishdown compressed** & & & & & & \\ Top-1doc & 3.44 & 3.75 & 3.64 & 4.33 & 2.09 & 3.76 \\ Top-5 docs & 3.58 & 3.66 & 3.68 & **4.53** & 1.32 & 3.64 \\ \hline
**While compressed** & & & & & & \\ RICQ/imgas & 2.64 & 3.92 & **4.19** & 3.58 & 1.18 & 4.00 \\ LMI+Imgas & 2.54 & 3.63 & 3.54 & 3.37 & 1.87 & 3.42 \\ PointTime & 3.63 & 4.05 & 3.59 & 4.01 & 1.48 & 3.88 \\ K-conv & **4.10** & **4.14** & 3.31 & 4.20 & **2.31** & **4.38** \\ \hline \end{tabular}
\end{table}
Table 5: Results evaluated using the G-Eval-4 metric Liu et al. (2023). The prompt used is presented in Table 13.

 

task in RAG, and this can be extended in future work.

Additionally, our study mainly focuses on English biomedical QA, which limits generalization to other languages and domains. Current studies in closed domains face challenges due to the scarcity of datasets, posing a considerable obstacle to the broader implementation of our methodology. We believe that, among closed domains, the medical QA has relatively more data, and we have proven our methodology in this domain. However, in other specific domains, not only QA data but also retrieval corpora are yet to be established. Furthermore, data availability in languages other than English is even more limited. Nevertheless, we recognize that our methodology has significant potential for extension to other languages and domains and that such expansion is necessary to demonstrate the generalizability of our training approach. Accordingly, we regard the application of retrieval-augmented QA in closed domains as a critical area of investigation, so we intend to extend our research to encompass additional domains in the future.

## Ethical Considerations

In our research, we employed publicly available datasets, including MedQuAD, MASHQA, BioASQ, and MEDIQA. When synthesizing ground-truth summaries, we ensure that no personally identifiable information is used and that all data are anonymized. Our methodology is still in its early stages and is not yet suitable for direct practical use in medical domains, where reliability and accuracy are paramount. In particular, hallucination can critically affect patient care and clinical decision-making. Therefore, our methodology is considered to mitigate hallucination by emphasizing domain knowledge in biomedical QA research rather than substituting professional medical judgment, thus posing no risk of harm.

## References

* A. Aghajanyan, B. Huang, C. Ross, V. Karpukhin, H. Xu, N. Goyal, D. Okhonko, M. Joshi, G. Ghosh, M. Lewis, and L. Zettlemoyer (2022)Cm3: a causal masked multimodal model of the internet. Preprint arXiv:2201.07520. Cited by: SS4.
* A. Ahmad, N. Constant, Y. Yang, and D. Cer (2019)ReQA: an evaluation for end-to-end answer retrieval models. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, Hong Kong, China, pp. 137-146. External Links: Link, Document Cited by: SS1, SS2.
* A. Asai, Z. Zhong, D. Chen, P. W. Koh, L. Zettlemoyer, H. Hajishirzi, and W. Tih (2024)Reliable, adaptable, and attributable language models with retrieval. PreprintarXiv:2403.03187. Cited by: SS1, SS2.
* M. Bavarian, H. Jun, N. Tezak, J. Schulman, C. McLeavey, J. Tworek, and M. Chen (2022)Efficient training of language models to fill in the middle. PreprintarXiv:2207.14255. Cited by: SS1, SS2.
* A. Ben Abacha and D. Demner-Fushman (2019)A question-entailment approach to question answering. BMC Bioinformatics20 (1), pp. 511:1-51:23. Cited by: SS1, SS2.
* A. Ben Abacha, C. Shivade, and D. Demner-Fushman (2019)Overview of the MEDIQA 2019 shared task on textual inference, question entailment and question answering. In Proceedings of the 18th BioNLP Workshop and Shared Task, Florence, Italy, pp. 370-379. Cited by: SS1, SS2.
* T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020)Language models are few-shot learners. In Advances in Neural Information Processing Systems, Vol. 33, pp. 1877-1901. Cited by: SS1, SS2.
* Z. Chen, A. Hernandez Cano, A. Romanou, A. Bonnet, K. Matoba, F. Salvi, M. Pagliardini, S. Fan, A. Kopf, A. Mohtashami, et al. (2023)Meditron-70b: scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079. Cited by: SS1, SS2.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 4171-4186. External Links: Link, Document Cited by: SS1, SS2.

 

Chris Donahue, Mina Lee, and Percy Liang. 2020. Enabling language models to fill in the blanks. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 2492-2501, Online. Association for Computational Linguistics.
* Du et al. (2022) Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. GLM: General language model pretraining with autoregressive blank infilling. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 320-335. Dublin, Ireland. Association for Computational Linguistics.
* Fried et al. (2023) Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. 2023. Incoroder: A generative model for code infilling and synthesis. In _The Eleventh International Conference on Learning Representations_.
* Frisoni et al. (2024) Giacomo Frisoni, Alessio Cocchieri, Alex Presepi, Gianluca Moro, and Zaiqiao Meng. 2024. To generate or to retrieve? on the effectiveness of artificial contexts for medical open-domain question answering. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 9878-9919, Bangkok, Thailand. Association for Computational Linguistics.
* Guo et al. (2021) Mandy Guo, Yinfei Yang, Daniel Cer, Qinlan Shen, and Noah Constant. 2021. MultiReQA: A cross-domain evaluation forRetrieval question answering models. In _Proceedings of the Second Workshop on Domain Adaptation for NLP_, pages 94-104, Kyiv, Ukraine. Association for Computational Linguistics.
* Han et al. (2023) Tianyu Han, Lisa C Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander Loser, Daniel Truhn, and Keno K Bressem. 2023. Medalpaca-an open-source collection of medical conversational ai models and training data. _arXiv preprint arXiv:2304.08247_.
* Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In _International Conference on Learning Representations_.
* Hu et al. (2024) Shengchao Hu, Li Shen, Ya Zhang, Yixin Chen, and Dacheng Tao. 2024. On transforming reinforcement learning with transformers: The development trajectory. _IEEE Transactions on Pattern Analysis and Machine Intelligence_.
* Hurst et al. (2024) Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welhiunda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. _arXiv preprint arXiv:2410.21276_.
* Izcard et al. (2022) Gautier Izcard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. _Preprint_, arXiv:2112.09118.
* Ji et al. (2023a) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023a. Survey of hallucination in natural language generation. _ACM Comput. Surv._, 55(12).
* Ji et al. (2023b) Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. 2023b. Towards mitigating LLM hallucination via self reflection. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 1827-1843, Singapore. Association for Computational Linguistics.
* Jiang et al. (2024) Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. _arXiv preprint arXiv:2401.04088_.
* Jiang et al. (2023a) Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023a. LLMLingua: Compressing prompts for accelerated inference of large language models. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 13358-13376, Singapore. Association for Computational Linguistics.
* Jiang et al. (2023b) Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023b. Active retrieval augmented generation. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 7969-7992, Singapore. Association for Computational Linguistics.
* Jin et al. (2021) Di Jin, Eileen Pan, Nassim Oufatole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. _Applied Sciences_, 11(14).
* Jin et al. (2024) Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Xiaojian Jiang, Jiexin Xu, Li Qiuxia, and Jun Zhao. 2024. Tug-of-war between knowledge: Exploring and resolving knowledge conflicts in retrieval-augmented language models. In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LRECOLING 2024)_, pages 16867-16878, Torino, Italia. ELRA and ICCL.
* Johnson et al. (2019) Jeff Johnson, Matthijs Douze, and Herve Jegou. 2019. Billion-scale similarity search with GPUs. _IEEE Transactions on Big Data_, 7(3):535-547.
* Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. _Transactions of the Association for Computational Linguistics_, 8:64-77 

Minki Kang, Seanie Lee, Jinheon Baek, Kenji Kawaguchi, and Sung Ju Hwang. 2024. Knowledge-augmented reasoning distillation for small language models in knowledge-intensive tasks. In _Proceedings of the 37th International Conference on Neural Information Processing Systems_, NIPS '23, Red Hook, NY, USA. Curran Associates Inc.
* Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6769-6781, Online. Association for Computational Linguistics.
* Kim et al. (2024) Jaehyung Kim, Jaehyun Nam, Sangwoo Mo, Jongjin Park, Sang-Woo Lee, Minjoon Seo, Jung-Woo Ha, and Jinwoo Shin. 2024. Sure: Summarizing retrievals using answer candidates for open-domain QA of LLMs. In _The Twelfth International Conference on Learning Representations_.
* Krithara et al. (2023) Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliquaras. 2023. Bioasqqa: A manually curated corpus for biomedical question answering. _Scientific Data_, 10(1):170.
* Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Linamin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In _Proceedings of the 29th Symposium on Operating Systems Principles_, SOSP '23, page 611-626, New York, NY, USA. Association for Computing Machinery.
* Lassance and Clinchant (2022) Carlos Lassance and Stephane Clinchant. 2022. An efficiency study for splade models. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR '22, page 2220-2226, New York, NY, USA. Association for Computing Machinery.
* Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7871-7880, Online. Association for Computational Linguistics.
* Li et al. (2023) Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. 2023. Compressing context to enhance inference efficiency of large language models. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 6342-6353, Singapore. Association for Computational Linguistics.
* Lin (2004) Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In _Text Summarization Branches Out_, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
* Lin et al. (2024a) Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024a. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. In _Proceedings of Machine Learning and Systems_, volume 6, pages 87-100.
* Lin et al. (2024b) Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Richard James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. 2024b. RA-DIT: Retrieval-augmented dual instruction tuning. In _The Twelfth International Conference on Learning Representations_.
* Liu et al. (2024) Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paramjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. _Transactions of the Association for Computational Linguistics_, 12:157-173.
* Liu et al. (2022a) Shuai Liu, Hyundong Cho, Marjorie Freedman, Xuezhe Ma, and Jonathan May. 2023a. RECAP: Retrieval-enhanced context-aware prefix encoder for personalized dialogue response generation. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 8404-8419, Toronto, Canada. Association for Computational Linguistics.
* Liu et al. (2023) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023b. G-eval: NLG evaluation using gpt-4 with better human alignment. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 2511-2522, Singapore. Association for Computational Linguistics.
* Long et al. (2023) Quanyu Long, Wenya Wang, and Sinno Pan. 2023. Adapt in contexts: Retrieval-augmented domain adaptation via in-context learning. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 6525-6542, Singapore. Association for Computational Linguistics.
* Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In _International Conference on Learning Representations_.
* Louis et al. (2024) Antoine Louis, Gijs van Di Jack, and Gerasimos Spanakis. 2024. Interpretable long-form legal question answering with retrieval-augmented large language models. _Proceedings of the AAAI Conference on Artificial Intelligence_, 38(20):22266-22275.
* Nan et al. (2021) Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Noguiera dos Santos, Henghui Zhu, Dejaiao Zhang, Kathleem McKeown, and Bing Xiang. 2021. Entity-level factual consistency of abstractive text summarization. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 2727-2733, Online. Association for Computational Linguistics 

Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019. ScispaCy: Fast and robust models for biomedical natural language processing. In _Proceedings of the 18th BioNLP Workshop and Shared Task_, pages 319-327, Florence, Italy. Association for Computational Linguistics.
* Nori et al. (2023) Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. Capabilities of gpt-4 on medical challenge problems. _Preprint_, arXiv:2303.13375.
* Nussbaum et al. (2024) Zach Nussbaum, John X. Morris, Brandon Duderstadt, and Andriy Mulayr. 2024. Nomic embeded: Training a reproducible long context text embedder. _Preprint_, arXiv:2402.01613.
* OpenAI (2024) OpenAI. 2024. Gpt-4o mini: advancing cost-efficient intelligence.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_, 21(140):1-67.
* Ram et al. (2023) Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. _Transactions of the Association for Computational Linguistics_, 11:1316-1331.
* Ren et al. (2024) Houxing Ren, Mingjie Zhan, Zhongyuan Wu, and Hongsheng Li. 2024. Empowering character-level text infilling by eliminating sub-tokens. _Preprint_, arXiv:2405.17103.
* Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. _Foundations and Trends@ in Information Retrieval_, 3(4):333-389.
* Ryu et al. (2023) Cheol Ryu, Seolhwa Lee, Subeen Pang, Chanycol Choi, Hojun Choi, Myeongge Min, and Jy-Yong Sohn. 2023. Retrieval-based evaluation for LLMs: A case study in Korean legal QA. In _Proceedings of the Natural Legal Language Processing Workshop 2023_, pages 132-137, Singapore. Association for Computational Linguistics.
* Ryu et al. (2024) Sangwon Ryu, Heejin Do, Yunsu Kim, Gary Geunbae Lee, and Jungseul Ok. 2024. Key-element-informed sllm tuning for document summarization. _Preprint_, arXiv:2406.04625.
* Sarthi et al. (2024) Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D Manning. 2024. RAPTOR: Recursive abstractive processing for tree-organized retrieval. In _The Twelfth International Conference on Learning Representations_.
* Shi et al. (2024) Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2024. REPLUG: Retrieval-augmented black-box language models. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 8371-8384, Mexico City, Mexico. Association for Computational Linguistics.
* Team et al. (2024) Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgan Riviere, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. _arXiv preprint arXiv:2403.08295_.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosalle, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Wang et al. (2024a) Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2024a. How far can camels go? exploring the state of instruction tuning on open resources. In _Proceedings of the 37th International Conference on Neural Information Processing Systems_, NIPS '23, Red Hook, NY, USA. Curran Associates Inc.
* Wang et al. (2024b) Yubo Wang, Xueguang Ma, and Wenhu Chen. 2024b. Augmenting black-box llms with medical textbooks for clinical question answering. _Preprint_, arXiv:2309.02233.
* Wang et al. (2024c) Zihao Wang, Anji Liu, Haowei Lin, Jiaqi Li, Xiaojian Ma, and Yitao Liang. 2024c. Rat: Retrieval augmented thoughts elicit context-aware reasoning in long-horizon generation. _Preprint_, arXiv:2403.05513.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delange, Anthony Moi, Pierric Cistac, Tim Rault, Remii Louf, Morgan Funtowicz, Joe Davison, Sam Sheifer, Patrick von Platen, Clara Ma, Yacne Jenrhe, Julien Pu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Xiao et al. (2024) Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024. C-pack: Packed resources for general chinese embeddings. In _Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR '24, page 641-649, New York, NY, USA. Association for Computing Machinery.
* Xiong et al. (2024) Guangzhou Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. 2024. Benchmarking retrieval-augmented generation for medicine. _arXiv preprint arXiv:2402.13178_ 

Fangyuan Xu, Weijia Shi, and Eunsol Choi. 2024a. RECOMP: Improving retrieval-augmented LMs with context compression and selective augmentation. In _The Twelfth International Conference on Learning Representations_.
* Xu et al. (2024) Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2024b. Retrieval meets long context large language models. _Preprint_, arXiv:2310.03025.
* Xu et al. (2023) Yan Xu, Mahdi Namazifar, Devamanyu Hazarika, Aishwarya Padmakumar, Yang Liu, and Dilek Hakkanitur. 2023. Kilm: Knowledge injection into encoder-decoder language models. _arXiv preprint_.
* Yagnik et al. (2024) Niraj Yagnik, Jay Jhaveri, Vivek Sharma, and Gabriel Pila. 2024. Medlm: Exploring language models for medical question answering systems. _Preprint_, arXiv:2401.11389.
* Yang et al. (2023) Haoyan Yang, Zhitao Li, Yong Zhang, Jianzong Wang, Ning Cheng, Ming Li, and Jing Xiao. 2023. PRCA: Fitting black-box large language models for retrieval question answering via pluggable reward-driven contextual adapter. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 5364-5375, Singapore. Association for Computational Linguistics.
* Yu et al. (2023a) Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023a. Generate rather than retrieve: Large language models are strong context generators. In _The Eleventh International Conference on Learning Representations_.
* Yu et al. (2023b) Wenhao Yu, Hongming Zhang, Xiaoman Pan, Kaixin Ma, Hongwei Wang, and Dong Yu. 2023b. Chain-of-note: Enhancing robustness in retrieval-augmented language models. _Preprint_, arXiv:2311.09210.
* Yu et al. (2023c) Zichu Yun, Chenyan Xiong, Shi Yu, and Zhiyuan Liu. 2023c. Augmentation-adapted retriever improves generalization of language models as generic plug-in. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2421-2436, Toronto, Canada. Association for Computational Linguistics.
* Zhang* et al. (2020) Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In _International Conference on Learning Representations_.
* Zhong et al. (2022) Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfeti Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards a unified multi-dimensional evaluator for text generation. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 2023-2038, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Zhu et al. (2020) Ming Zhu, Aman Ahuja, Da-Cheng Juan, Wei Wei, and Chandan K. Reddy. 2020. Question answering with long multiple-span answers. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 3840-3849, Online. Association for Computational Linguistics.

 

Experimental Details

### Model Details

To demonstrate that our contribution works universally regardless of reader models, we used a range of large language models (LLMs) [12, 30, 46], including a commercial model [57], with varying parameters and purposes. All open-source models were implemented based on HuggingFace's Transformers [89], and due to hardware constraints, we utilized AWQ [46] models for LLMs with a substantial number of parameters. The specific model details are as follows:

\begin{tabular}{l|l} \hline \hline
**Model** & **HuggingFace’s Repository** \\ \hline K-comp & google/gamma-2b \\ Retriever & nomic-ai/nomic-embed-text-v1.5 \\ Llama-3 8B & meta-llama/Meta-Llama-3-8B \\ Llama-3 70B & TechGens/Meta-Lambda-3-70B-AWQ \\ Mitxtal-8xTB & TheDikeRcv/mittal-8xTb-v0.1-AWQ \\ MedAlpaca 13B & medalpaca+medalapaca-13B \\ Meditron 70B & TheBloke/medium-70B-AWQ \\ Selective-Context & meta-llama/Llama-2-7b-hf \\ LLMLingua & meta-llama/Llama-2-7b-hf \\ SPLADE & naver/efficient-splade-V-large-query \\  & naver/efficient-splade-V-large-doc \\ \hline \hline \end{tabular}

Recognizing Question Entailment, and QA, but only the QA task was used.

### Implementation Details

We fine-tuned Gemma-2B [46] with our knowledge injection objective and also used the FineTune baseline (FineTune) with the same model. Table 16 shows the hyperparameter settings that were used for training and inference. K-comp and FineTune were trained with the same hyperparameters and selected their optimal checkpoints based on the performance of the development set. We also trained the model via the Transforming Reinforcement Learning [45]. When training on MASH-QA, we used 4 NVIDIA A100 GPUs with 80GB memory, otherwise we used 2 A100-80GB GPUs. For inference, we conducted experiments on a single A100-80GB GPU with vLLM [31] to accelerate and efficiently perform inference.

## Appendix B Analysis of Query-Agnostic

Figure 3 compares our approach to previous studies that compress prompts without relying on the query, as illustrated in the top-left panel. GPT-4omini is the result of summarizing only passages without questions. We referred to the prompts used when training RECOMP, which can be seen in Table 10. Selective-Context (SC) [12] is a method for the removal of contexts with low self-information at the token level. SC was compressed with Llama-2-7B [32] as that used for LLMLingua. Both methods compress prompts by considering token-level dependencies, so the resulting sentences are incomplete when decoding. As can be seen in Table 18, LLMLingua generates sequences in which words and symbols are merely listed. At the same time, SC produces fragments due to token-level pruning, which also results in incomplete sentences. Consequently, although embedding-based metrics such as BertScore [30] or lexical-based metrics like ROUGE [46] may exhibit robust performance, they are limited in qualitative evaluations. Moreover, the methodology of compressing passages without referencing the query is unsuitable for retrieval-augmented QA tasks. Thus, both GPT-4o-mini and K-comp outperform the aforementioned baselines. In addition, it is expected that K-comp, by referencing masked questions, provides a more relevant and specialized con



text compared to GPT-4o-mini, which does not reference the questions.

## Appendix C Additional Baseline

In Section 5, we compared various baselines for RAG methods that incorporate prompt compression. However, prior RAG approaches that do not utilize compression were not implemented. Although the primary focus of our study is not to exhaustively explore non-compression-based methods, we include additional state-of-the-art baselines to facilitate a more comprehensive evaluation of how K-comp performs relative to other approaches. Specifically, we adopt RAPTOR Sarthi et al. (2024) as a baseline and conduct experiments under several predefined conditions. Firstly, RAPTOR requires a summarizing model because it recursively summarizes to construct a tree structure. Therefore, we use FineTune model to perform context summarization, thereby eliminating the penalty for training. Second, due to computational resource limitations, we imposed a size constraint on the retrieval corpus to enable summarization. Given the massive scale of the MedCorp Xiong et al. (2024) corpus, we replace it with a smaller corpus consisting of the top 5 documents retrieved by the retriever across the entire dataset. We argue that these experimental conditions do not put RAPTOR at a disadvantage. Rather, we believe that restricting the corpus to the top-5 retrieved documents may offer potential advantages by improving the relevance and quality of the retrieved information.

As shown in Table 6, K-comp proves effective in QA tasks, whether using prompt compression or non-prompt compression approaches. Furthermore, as evidenced in Table 3, K-comp further underscores its cost efficiency, particularly compared to non-prompt compression methods. We also performed the GPT-4o evaluation described in Section 5, with the results presented in Table 7. The results indicate that the passages retrieved by RAPTOR approach are preferred more frequently in BioASQ than other datasets. As seen in Table 1, the FineTune model is particularly well-tuned for BioASQ, suggesting that RAPTOR effectively integrates both high-level concepts and detailed information during its tree construction process.

Despite RAPTOR's advantage of using a corpus limited to the top-5 documents, K-comp achieves higher accuracy than the state-of-the-art retrieval-augmented QA baseline (non-prompt compression). This validates our hypothesis that knowledge injection enhances QA performance. However, it is worth noting that RAPTOR was originally designed using closed models from the GPT family, which may not align as effectively with our work, particularly when using smaller models such as Gemma.

## Appendix D Licenses

MedQuAD, MASH-QA, and BioASQ are licensed under CC-BY-4.0, Apache-2.0, and CC-BY-2.5, respectively. The models Gemma-2b, Nomicembed-text-v1.5, Meta-Llama-3, Mixtral-8x7b-v0.1-AWQ, GPT-4o, MedAlpaca-13b, Mediterranean0B-AWQ, and Bge-reranker-large are licensed under Gemma, Apache-2.0, Llama 3 Community, Apache-2.0, OpenAI, Creative Commons, Llama 2 Community, and MIT, respectively.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{**LDA-3-BB**} & \multicolumn{2}{c}{**LDA-3-70B**} & \multicolumn{2}{c}{**Miktral-8x7B**} & \multicolumn{2}{c}{**GPT-4o**} & \multicolumn{2}{c}{**MedAlpaaca-13B**} & \multicolumn{2}{c}{**Meditron-70B**} \\  & **BertScore** & **UnifEval** & **BertScore** & **UnifEval** & **BertScore** & **UnifEval** & **BertScore** & **UnifEval** & **BertScore** & **UnifEval** & **BertScore** & **UnifEval** \\ \hline \multicolumn{9}{c}{MetaQuadAD} \\ RAPTOR & 74.96 & 60.84 & 77.66 & **71.79** & 66.4 & 61.49 & 84.20 & 57.78 & 83.59 & 48.42 & 74.67 & **60.30** \\ K-comp & **83.79** & **64.05** & **82.06** & 62.91 & **80.64** & **63.07** & **85.49** & **63.16** & **85.75** & **49.39** & **78.88** & 58.95 \\ \hline \multicolumn{9}{c}{MetaMiMi-QA} \\ RAPTOR & 79.61 & 60.62 & 84.64 & **66.20** & 73.37 & **62.95** & 83.31 & 58.60 & 80.19 & **50.27** & 80.49 & **64.83** \\ K-comp & **83.45** & **61.45** & **83.64** & 61.40 & **81.49** & 62.91 & **84.49** & **64.06** & **81.95** & 51.76 & **82.67** & 61.85 \\ \hline \multicolumn{9}{c}{BioASO} \\ RAPTOR & 82.63 & 60.62 & 85.57 & **70.32** & 83.39 & 66.10 & 87.16 & 56.64 & 83.50 & **46.40** & 85.44 & **65.48** \\ K-comp & **86.61** & **62.28** & **87.03** & 62.89 & **86.52** & **66.31** & **87.77** & **64.38** & **84.88** & 45.47 & **86.70** & 62.90 \\ \hline \hline \end{tabular}
\end{table}
Table 6: RAPTOR results. The implementation follows the same approach as that utilized for Table 1.

\begin{table}
\begin{tabular}{l l l l} \hline \hline  & **MedQuAD** & **MASH-QA** & **BioASQ** \\ \hline RAPTOR & 20.7\% & 30.1\% & 43.4\% \\ K-comp & 77.9\% & 69.6\% & 52.5\% \\ Tie & 1.4\% & 0.3\% & 4.1\% \\ \hline \hline \end{tabular}
\end{table}
Table 7: GPT-4o evaluation results with the RAPTOR approach.

 

Examples of Used Prompts

Please extract the content about the entity in fewer than four sentences.

### Passage Therapies in Aicardi-Goutieres syndrome.

Aicardi-Goutiere s syndrome (AGS) is a genetically determined disorder, affecting most particularly the brain and the skin, characterized by the inappropriate induction of a type I interferon-mediated immune response. In most, but not all, cases the condition is severe, with a high associated morbidity and mortality ...(skip)

Treatments in Aicardi-Goutieres syndrome.

Comprehensive reviews of the clinical characteristics and pathogenesis of Aicardi-Goutieres syndrome (AGS), particularly its contextualization within a putative type I interferonopathy framework, already exist. However, recent reports at attempts at treatment suggest that an assessment of the field from a therapeutic perspective is warranted at this time ...(skip)

Novel and emerging treatments for Aicardi-Goutieres syndrome.

<blntroduction</b: Aicardi-Goutieres syndrome (AGS) is the prototype of the type I interferonopathies, a new heterogeneous group of autoinflammatory disorders in which type I interferon plays a pivotal role. The disease usually manifests itself during infancy, primarily affecting the brain and the skin, and is characterized by cerebrospinal fluid chronic lymphocytosis and raised levels of interferon-alpha and by cardinal neuroradiological features: cerebral calcification, leukoencephalopathy and cerebral atrophy ...(skip)

Aicardi-Goutieres syndrome At the moment there are no therapies specifically targeting the underlying cause of AGS. Current treatments address the symptoms, which can be varied both in scope and severity. Many patients benefit from tube-feeding. Drugs can be administered to help with seizures / epilepsy ...(skip)

Aicardi-Goutieres syndrome Treatment

### Entity [research, clinical trial, Disorder]

\begin{table} \end{table}
Table 8: Prompt for synthesizing summaries used in K-comp.

 

Compress the information in the retrieved documents into a 2-sentence summary that could be used to answer the question: Question: what research (or clinical trials) is being done for Aicardi-Goutieres Syndrome Disorder ?

Retrieved documents: Therapies in Aicardi-Goutieres syndrome.

Aicardi-Goutieres syndrome (AGS) is a genetically determined disorder, affecting most particularly the brain and the skin, characterized by the inappropriate induction of a type I interferon-mediated immune response. In most, but not all, cases the condition is severe, with a high associated morbidity and mortality ...(skip)

Treatments in Aicardi-Goutieres syndrome.

Comprehensive reviews of the clinical characteristics and pathogenesis of Aicardi-Goutieres syndrome (AGS), particularly its contextualization within a putative type I interferonopathy framework, already exist. However, recent reports of attempts at treatment suggest that an assessment of the field from a therapeutic perspective is warranted at this time ...(skip)

Novel and emerging treatments for Aicardi-Goutieres syndrome.

<bIntroduction</b: Aicardi-Goutieres syndrome (AGS) is the prototype of the type I interferonopathies, a new heterogeneous group of autoinflammatory disorders in which type I interferon plays a pivotal role. The disease usually manifests itself during infancy, primarily affecting the brain and the skin, and is characterized by cerebrospinal fluid chronic lymphocytosis and raised levels of interferon-alpha and by cardinal neuroradiological features: cerebral calcification, leukoencephalopathy and cerebral atrophy ...(skip)

Aicardi-Goutieres syndrome

At the moment there are no therapies specifically targeting the underlying cause of AGS. Current treatments address the symptoms, which can be varied both in scope and severity. Many patients benefit from tube-feeding. Drugs can be administered to help with seizures / epilepsy ...(skip)

Aicardi-Goutieres syndrome

Treatment

Compressed documents:

\begin{table}
\begin{tabular}{l} \hline Compress the information in the retrieved documents into a 2-sentence summary \\ Retrieved documents: [(Top-5 retrieved passages)] \\ Compressed documents: \\ \hline \end{tabular}
\end{table}
Table 10: Prompt for synthesizing summaries used in query-agnostic GPT-4o-mini.

\begin{table}
\begin{tabular}{p{398.7pt} p{398.7pt}} \hline  & \multicolumn{1}{c}{} \\ \hline \multirow{2}{*}{Compress the information in the retrieved documents into a 2-sentence summary} \\ Retrieved documents: & \multicolumn{1}{c}{} \\ \hline \end{tabular}
\end{table}
Table 9: Prompt for synthesizing summaries used in RECOMP training and query-based GPT-4o-mini.

 



######Passage

Aicardi-Goutieres syndrome (AGS) is a genetically determined disorder affecting the brain and skin, characterized by inappropriate immune responses due to type I interferon. Current research focuses on understanding its pathogenesis and developing targeted therapies, with some recent attempts exploring treatments like Janus kinase inhibitors and anti-IFN-\(\alpha\) antibodies. Despite advancements, there are still challenges in assessing efficacy and addressing open questions related to treatment effectiveness. Ongoing clinical trials aim to evaluate the efficacy of new therapies and address the underlying causes of AGS. Overall, ongoing research and clinical trials are essential for developing effective treatments for AGS and type I interferonopathies.

### Entity

research: Systematic study undertaken to increase knowledge

clinical trial: Phase of clinical research in medicine

Aicardi-Goutieres Syndrome: Aicardi-Goutieres syndrome is a disorder that mainly affects the brain, the immune system, and the skin.Most newborns with Aicardi-Goutieres syndrome do not show any signs or symptoms of the disorder

######Questions

what research (or clinical trials) is being done for Aicardi-Goutieres Syndrome Disorder ?

######Passage

Therapies in Aicardi-Goutieres syndrome.

Aicardi-Goutieres syndrome (AGS) is a genetically determined disorder, affecting most particularly the brain and the skin, characterized by the inappropriate induction of a type I interferon-mediated immune response. In most, but not all, cases the condition is severe, with a high associated morbidity and mortality ...(skip)

Treatments in Aicardi-Goutieres syndrome.

Comprehensive reviews of the clinical characteristics and pathogenesis of Aicardi-Goutieres syndrome (AGS), particularly its contextualization within a putative type I interferonopathy framework, already exist. However, recent reports of attempts at treatment suggest that an assessment of the field from a therapeutic perspective is warranted at this time ...(skip)

Novel and emerging treatments for Aicardi-Goutieres syndrome.

<blIntroduction</b: Aicardi-Goutieres syndrome (AGS) is the prototype of the type I interferonopathies, a new heterogeneous group of autinflammatory disorders in which type I interferon plays a pivotal role. The disease usually manifests itself during infancy, primarily affecting the brain and the skin, and is characterized by cerebrospinal fluid chronic lymphocytes and raised levels of interferon-alpha and by cardinal neuroradiological features: cerebral calcification, leukoencephalopathy and cerebral atrophy ...(skip)

Aicardi-Goutieres syndrome

At the moment there are no therapies specifically targeting the underlying cause of AGS. Current treatments address the symptoms, which can be varied both in scope and severity. Many patients benefit from tube-feeding. Drugs can be administered to help with seizures / epilepsy ...(skip)

Aicardi-Goutieres syndrome

Treatment

######Questions

what research (or clinical trials) is being done for Aicardi-Goutieres Syndrome Disorde ?

Select the summary (Summary 1, Summary 2, Summary 3, or Summary 4) that is more relevant and informative as a rationale for the given question.

In particular, biomedical QA requires expertise to be credible, so choose a summary where expertise exists in the domain.

Choice: [Summary 1, Summary 2, Summary 3, Summary 4, Tie], Do not offer any opinions other than the choice.

### Summary 1 {{summary 1}}

## Summary 2 {{summary 2}}

## Summary 3 {{summary 3}}

## Summary 4 {{summary 4}}
 

You will be given a question, passage, and answer (both the provided answer and the correct answer).

Your task is to evaluate how coherent and relevant the provided answer is based on the question, passage, and correct answer. The passage is provided as a reference to determine if the provided answer appropriately aligns with the information required to answer the question.

Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.

Evaluation Criteria:

Coherence (1-5) - the degree to which the provided answer is logically consistent, well-structured, and directly relevant to the question. The answer should make logical sense and align with the correct answer, while addressing the question clearly and effectively.

Evaluation Steps:

1. Read the question carefully to identify the specific information it seeks.

2. Read the answer and compare it to the correct answer. Check if the provided answer addresses the question directly, aligns logically with the passage's content, and is presented in a clear, organized, and complete manner.

3. Assign a score for coherence on a scale of 1 to 5, where 1 is the lowest and 5 is the highest based on the Evaluation Criteria.

Example:

Passage:

{passage}

Question:

{question}

Correct Answer:

{correct answer}

Answer:

{answer}

Evaluation Form (scores ONLY):

Coherence:

\begin{table}
\begin{tabular}{l} \hline \hline  \\ \end{tabular}
\end{table}
Table 13: Prompt for G-Eval-4 evaluation metric [Liu et al., 2023b].

 

Extended Case Study

### Entry x-ray: Form of short-wavelength electromagnetic radiation rheumatoid arthritis: Type of autoimmune arthritis

Summary X-rays are the most common method for assessing the degree of joint destruction in rheumatoid arthritis (RA), revealing osteopenia and joint space narrowing as the disease progresses. They are particularly useful for identifying bony erosions and assessing treatment efficacy. While imaging techniques like MRI and ultrasound are also employed, X-rays remain the most common for evaluating joint damage in RA. Early diagnosis and treatment are crucial to modify the disease's course and reduce complications. Understanding the disease's pathophysiology and imaging findings is essential for effective diagnosis and management.

Llama-3-8B's Answer X-rays diagnose rheumatoid arthritis (RA) by revealing changes in the axial skeleton and joint space. As the disease progresses, early RA may not show significant changes on X-rays, but it can indicate osteogenic and joint space narrowing. Advances in imaging techniques like MRI and ultrasound enhance the ability to detect soft tissue changes and joint damage, which is crucial for accurate diagnosis and treatment management. Overall, X-rays serve as an important tool for evaluating the structural changes associated with RA.

Llama-3-70B's Answer X-rays can reveal joint space narrowing and osteopenia, which are common findings in RA. X-rays can show early signs of joint damage in RA, such as bone erosion and synovial inflammation. X-rays can detect soft tissue changes and joint damage in RA, which is important for diagnosis and treatment management. X-rays can assess the degree of joint destruction in RA, including changes in the axial skeleton and joint space.

## FineTune

Summary Rheumatoid arthritis (RA) is a chronic systemic disease characterized by inflammation of the synovial tissue, affecting 0.5%-1.0% of the global population, with a higher prevalence in females. Modern diagnostic techniques, including X-rays, MRI, and ultrasound, are crucial for assessing joint damage and evaluating treatment efficacy. Early diagnosis and treatment are essential to modify the disease's course and reduce severe complications. Advances in imaging techniques, particularly MRI, enhance the ability to detect and manage RA more effectively. Understanding the disease's pathophysiology and various imaging findings is essential for accurate diagnosis and tailored treatment strategies. Overall, imaging plays a significant role in the management and monitoring of RA.

Llama-3-8B's Answer X-rays are used to diagnose RA by detecting joint damage and erosion.

Llama-3-70B's Answer X-rays are a valuable tool in the diagnosis of RA, as they can detect joint damage and erosions caused by the disease. In the early stages of RA, X-rays may not show any abnormalities, but as the disease progresses, they can reveal joint space narrowing, erosions, and deformities. X-rays can also help monitor the progression of the disease and asses the effectiveness of treatment. However, X-rays have limitations, as they may not detect early changes in the joints or soft tissue inflammation. Therefore, other imaging modalities, such as MRI and ultrasound, may be necessary to provide a more comprehensive assessment of RA.

\begin{table} \end{table}
Table 14: Case study in section 4. (Above: K-comp, Below: FineTune) 



## Appendix G Dataset Statistics

Examples of Baselines

Question: Are there any adults who shouldn't get the meningococcal vaccine?

**Selective-Context**

Meningococcal disease Routine primary immunization against who planning Although conjugate vaccines in Since safety to Prevention Invasive disease The disease can rapidly or despite with required household Although cases often Sereogroup B accounts but contained Adolescents disease Because in contained they potentially recently containing This T-cell-dependent vaccine that should The Advisory Committee to entering The vaccine also may belong Meningococcal disease through caused While IMD incidence peaks both infants carriage rates often increasing Effective vaccines Because represent cases often have the highest carriage rate efforts in This review describes and characteristics which countries that will also Rates increasing. The burden due available older. Incorporation into offers affect The routine use will Awareness attitudes among among Meningococcal-under However MenB vaccination coverage We investigated MenB disease among An online survey parents among There were 606 total participants parents of MenB (<in</i = also Non-vaccinated cohorts reported ( though However all cohorts reported high interest MenB disease Vaccinated cohorts relied had SCDM at though those Barriers included PCP recommendation uncertainty There gaps MenB disease SCDM patients resulting discussing Targeted education on MenB may these opportunities

**LLMLingua**

Men ine most or are. Althoughss conjugo. Since established

disease In be ande.otic is for. population. B the.s Becauseoccalcups are the, are The This and The to who to certain the isidence in, are.., havination, regardingterm of in of efents. on with have ofs The limitations been populations In more TheA of and3old under population regardingold of (<ini adination, and SCDM among parents and patients in the US, resulting in missed opportunities for discussing and administering MenB vaccination. Targeted education on MenB and vaccination recommendations may increase these opportunities and improve MenB vaccination awareness and initiation.

**GPT-4o-mini (query-agnostic)**

Routine primary immunization against meningococcal disease is advised for adults in endemic areas or traveling there, with conjugate vaccines preferred for those under 55, while polysaccharide vaccines are used for older adults due to safety concerns. Adolscents and young adults are prioritized for vaccination due to their higher incidence and carriage rates of the disease, yet awareness and vaccination coverage for serogroup B remains low, necessitating targeted education to improve understanding and uptake.

**SPLade**

Meningococcal disease Routine primary immunization against meningococcal disease is recommended for most adults living in areas where meningococcal disease is endemic or who are planning to travel to such areas. Although conjugate vaccines are the preferred meningococcal vaccine in adults 55 years of age or younger, polysaccharide vaccines are an acceptable alternative for adults in this age group if the conjugated vaccine is unavailable. Since safety and efficacy of conjugate vaccines in adults older than 55 years of age have not been established to date, polysaccharide vaccines should be used for primary immunization in this group.

**RECOMP**

Adults who should not receive the meningococcal vaccine include those who are 55 years or older, those in high-risk groups, and those who are pregnant or breastfeeding. Additionally, polysaccharide vaccines are recommended for adults over 55, as conjugate vaccines are not recommended for this age group due to safety concerns.

**GPT-4o-mini (query-based)**

Most adults are recommended to receive the meningococcal vaccine, particularly those living in endemic areas or traveling to such regions, but adults over 55 years should use polysaccharides vaccines due to the unestablished safety and efficacy of conjugate vaccines for this age group. Overall, while there are no strict categories of adults who should not get the vaccine, individuals with specific health concerns or those who have experienced adverse reactions to prior vaccinations should consult wealthcare professionals for personalized guidance.

**K-comp**

_### Entity people: Plurality of persons considered as a whole meningococcal vaccine: Vaccine against meningococcal disease_

_### Summary The meningococcal vaccine is recommended for adults in endemic areas and for adolescents and young adults in high-risk groups. The meningococcal vaccine is safe and effective, but the polysaccharide vaccine is preferred for those 55 years and older due to safety concerns in older adults. Vaccination efforts are crucial, as awareness and uptake of the vaccine remain low among adolescents and young adults, particularly in the United States. Education and targeted efforts are needed to improve awareness and vaccination rates among this population. The vaccine is particularly important for preventing invasive meningococcal disease, which can have severe consequences. Overall, effective vaccination strategies are essential to reduce the incidence of meningococcal disease._

\begin{table} \end{table}
Table 18: Examples of summaries generated by all the baselines.