OSCAR: Online Soft Compression for RAG
Authors: Maxime Louis
Thibault Formal
Hervé Déjean
Stéphane Clinchant
Venue: ICLR 2026 Poster (2026)
Keywords: RAG
Compression
Embedding
Efficiency
Question Answering
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Oscar is the first online query-dependent compression method which enables x2-x5 speed-up of RAG pipelines with little to no accuracy loss.

Abstract: Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by integrating external knowledge, leading to improved accuracy and relevance. However, scaling RAG pipelines remains computationally expensive as context length grows. On one hand, hard compression methods have recently proposed to prune the retrieved text on-the-fly with a limited compression ration. On the other hand, soft compression method performs a costly offline compression thanks a dedicated LLM but with a higher compression rate. In this paper, we introduce OSCAR, a novel query-dependent online soft compression method for RAG. OSCAR bridges the gap between online hard and offline soft compression methods, bringing the best of both: OSCAR dynamically compresses retrieved information at inference time, eliminating storage overhead and enabling higher compression rates than existing methods. Our experiments demonstrate state-of-the-art performance with a 2-5x speed-up in inference and minimal, if any, accuracy loss, for LLMs ranging from 1B to 24B parameters.

Published as conference paper ICLR 2026
OSCAR: ONLINE SOFT COMPRESSION FOR RAG
Maxime Louis"
Thibault Formal
Naver Labs Europe
Naver Labs Europe
maxime. louis. x201 20gmail com
thibaul t. com
Hervé Déjean
Stéphane Clinchant
Naver Labs Europe herve. seree.ddeeanenhaerrabbbcco
Naver Labs Europe stephane.clinchhaee w.w.nchaneeevverraas..ccmm
ABSTRACT
Retrieval-Augmenned Generation (RAG) enhances large language models (LLMs) by integrating external knowledge, leading improved accuracy and relevance. However, scaling RAG pipelines remains computationally expensive as context length grows. To address this, hard compression methods prune the retrieved text on-the-fly, achieving only modest compression ratios, whereas soft compression methods rely on costly offline LLM-based compression to obtain higher rates. In this paper. we introduce OSCAR, novel query-dependent online soft compression method for RAG. OSCAR bridges the gap between online hard and offline soft compression methods, bringing the best of both: OSCAR dynamically compresses retrieved documents into a representation optimized for the query at hand, leading to efficient and accurate downstream answer generation. Our experiments demonstrate state-of-the-e performance with 2-5x speed-up in inference and minimal, any, accuracy loss, for LLMs ranging from 1B to 24B parameters.
INTRODUCTION
Serrival-AAugeeneee Generation (RAG) (Lewis et al.. 2020; Guu et al. 2020 Borgeaud et al.) 2022) has be- come pivotal for sol ving wide range of natural language processing challenges. RAG enhances Large Language Models (LLMs) by leveraging retrieved documents from curated datasets, enabling and up-to-date responses. However, one major issue when scaling up RAG pipelines is the high computational cost.
Average
0.778
Mistral-7B
0.765
Better
OSCAR-IIama
LLM eval
OSCAR-8 8-Layers
accurate, well-grounded,
OSCAR-10-Layers
0.757
PISCO
Recomp
Provence
To improve efficiency, a natural idea consists in replacing the retrieved documents with more compact represen- tation. A straightforward option is to perform hard com- pression on the text itself form summarized or pruned version as in Xu ct al. 2023): Kenton & Toutanova (2019); Wang et al. (023). These methods are L-gM-tgoottic and robust, but their compression rates are modest (z x2), limiting overall efficiency gains. Most hard compression
0.755
0.745
20
Total inference T-FLOPs
Figure 1: OSCAR models enable faster end-to- end inference with retrieval as well as improved accuracy compared hard compression methods.
"oorrsppndiing author.

---

Published as a conference paper at ICLR 2026
methods operate in an online, query-aware fashion, dynamically compressing the documents maximize utility for the task.
Another option is soft compression which maps retrieved texts to a continuous embedding space. Typically, texts are mapped to K/V cache (Qin et al., (024) or to an embedding which can be fed into the transformer
by bypassing its embedding layer (Chevalier et al. 2023 Ge et al. 2023 Hofstätter et al.) 2023 Louis et 2025 Rau et al. 2024b). These approaches achieve higher compression (= x16), but at the cost of substantial
performance degradation, and they fall short of empirical and theoretical efficiency bounds (Kuratov et al. 2025). In fact, we note that none of the existing soft compression methods use the query at compression time: all them rely on heavy, LLM-sized forward passes performed offline, as doing it online would not lead to overall efficiency improvements.
Both hard and soft compression thus have complementary strengths: hard methods are online and query-aware but limited in compression rate, while soft methods promise higher rates but suffer from quality loss are not usable online. Ideally, one would combine the advantages of both-high compression with query-dependent online operation. However, designing a fast enough compression operator remains open challenge: existing methods either sacrifice efficiency, accuracy or fail to scale to dynamic RAG scenarios. Developing an efficient online compression strategy would also facilitate dynamic RAG scenarios in which retrieved content originates from the open web or from large-scale corpora in a plug-and-play manner.
In this paper, we show how build an efficient compression model to obtain large efficiency improvements in RAG pipelines. The obtained OSCAR models- for Online Soft Compression for RAG- are novel soft-compression query-dependent methods for RAG. We obtain 2-5x faster end-to-end inference on a variety of L.MSs ranging from 1B to 24B parameter Crucially, the obtained models suffer from little no accuracy loss on a variety of in-domain and out-of-domain RAG benchmarks. Lastly, we notice, as discussed by Chirkova et al. (2025), that the compression operation can be exploited to simultancously re-rank the initial pool of retrieved documents. Since re-ranking is an integral part of efficient RAG pipelines (Rau et al. 2024a), this enables us obtain the compression representation of the documents for free.
2
RELATED WORKS
Long context optimizations for RAG RAG scaling problems relate to the long-contex (in)abilities of LLMs which is an active area of research. K/V caching techniques enable faster long context handling by diminishing the number of operations self-attention (Devoto et al.) 2024 Kwon et al. 2023 Li et al. 2024).
FINCH (Corallo& & Papotti, 2024) is more specifically designed for RAG: the retrieved content is chunked and only a small portion of the keys and values is kept in cache for each chunk for the subsequent attention computations but compression rates remain limited. TurboRAG and block-attention RAG (Sun et al.) 2024] Lu et al. 2024) propose to modify the attention causal mask to compute attention independently on each retrieved documents, while the query still attends to each previous token in the context. Overall, we note that these K V-cache compression methods are orthogonal to our work: combining both would be possible.
Hard compression methods aim shortening the retrieved documents by summarization pruning. Most of them have limited compression rates due to the nature of text but are agnostic to the LLM used for generation. Provence (Chirkova al. (025)) proposes fine-tune DeBERTa (He et al., 2021) model to prune retrieved contexts. It is fast, prunes the context in a query-dependent fashion and allows the simultaneous reranking of the retrieved documents-male
pruning essentially free in a standard RAG pipeline. Extractive RECOMP (Xu et al. 2023) prunes contexts based on sentences embeddings. Abstractive RECOMP summarizes input contexts using an autoregressive LLM: the efficiency improvement is less clear
We release open-source models on
naver/oscar well training code: github.commnaverrpiscoo
2

---

Published as a conference paper ICLR 2026
than Provence since generating the summary is an expensive operation. Other methods include FILCO (Wang et al. 2023 or COMPACT (Yoon et al. 2024), which also generate pruned contexts autoregressively.
Soft compression methods aim at compressing retrieved documents into vector representations, often be used as input embeddings or K/V cache to the LLM used for generation. These methods generally achieve higher compression rates but require a training specific to the I.I.M used for generation. xRAG (Cheng et al. 2024) proposes to use retrieval embeddings as precomputed compressed representations, and trains an adapter MLP to map these embeddings into inputs for the LLM - performances remain however limited. COCOM (Rau et al. 2024b), building on (Chevalier et al. 2023 Ge et al. 2023), proposes end-to-end training pipeline where both the compression LLM and the generation LLM are fine-tuned using a large QA dataset. PISCO (Louis et al.) (025) is extension of COCOM trained by sentence-level distillation from teacher LLM: it allows to compress contexts by a factor of 16x with very limited performance drops. All these approaches process documents independently from the query attempting compress all the information of the retrieved documents into the (compressed) vector representation. Fil--ighht (Hofstätter et al.) 2023) proposes a form of query-dependent soft compression by using an encoder-decoder LLM, where the encoder is fed in parallel with the input query and each retrieved document. FiD-light decoder then takes only the first 50 hidden states for each document and thus has very limited compression rate. Finally, DODO Qin eet al.] (2024) builds compressed representations of its past context dynamically. It is primarily intended as an efficient long-conted method but can be used for context compression. However, results when using DODO to compress multiple documents for RAG are weak (see Louis et al. (2025) Table 2.)
None of these methods can be used online and reach large compression rates. fact, soft compression merely succeeding with large compressors, and thus is really challenging with low-latency. OSCAR addresses this issue by using appropriate compressor backbone and training, as well as by computing query-dependent embeddings, which favor the task at hand.
3 METHOD
Figure 2 provides an overview of OSCAR. At inference, after retrieval, a compressor LLM maps each document-query pair a few embedding tokens and generator LLM generates an answer to the query based on the and RAG prompt. Provided the compression rate is high and the compression operation efficient, there can be efficiency gains compared to the no-compression RAG pipeline. We now give details about every component of OSCAR as well as the training procedure.
Compression The compression procedure is shown within Figure 2 (right). Contrary to Ge al. (2023); Rau et al. (2024b); Louis et al. (2025), the document compression operation is conditioned on the query. details, the query 4, the i-th retrieved document dj. set of learnable memory tokens [MEM j];=1... are fed forward to compressor LLM C. We collect the last layer hidden states corresponding to each of these tokens to form the query-dependent embedding representations (c).... 4) := c, = C(g. de) the document. [MEM j], tokens play a similar role as the [CLS] BERT token: it is task-specific token prompting the storage within the corresponding hidden states.
The embedding representations e of each document, as well as the query g, are fed within a RAG prompt (given in Figure 11 to generator LLM which generates the answer. Since each document is replaced by embeddings, generation is much faster compared to the original text.
Generation
Compressor architecture All prior work on compression for RAG Louis et al. 2025 Cheng et al. 2024 compressor architecture identical the generator LLM. In this setup, the hidden state representations the compressor are easily adapted to the generator hidden space, making the whole pipeline easier to learn and deploy. But running the compression inference time would negate any subsequent
Rau al. 2024b)
3

---

Published as a conference paper ICLR 2026
Inference
Training
Retrieved documents
Retheved documents
British fimmaker Ken Loach has wan the Palme d'Or at the Cannes Film Fostival
Brittah Nimakeer Ken Loach has won the prastigious Pains d'Dr at the Carres Film Festiral Iwice...
Dacument collection
OSCAR compressor
OSCAR compressor
Query
Compressed
Which fllms eamedd Ken Loach the Cannes Film Festival?
documents
Compressed documents
OSCAR generator
OSCAR generator
[MEM] Input toxons
LoRA adapters
Kon Loach wan the Paime d'Chr with The Wind That Shakes the Barley Daniel
Full fine-tuning
Teacher LLM
Distillation
Blake.
Figure 2: OSCAR overview.
generation time gains, making these methods inherently offline. OSCAR however intended to operate an online fashion with no possibility to pre-compute document compressions. Therefore, the compression needs be fast. To so, we propose two different architectures for the compressor backbone:
OSCAR-N-Layers:: construct headless transformers using the first N layers of the pretrained backbone (same architecture as the generator). As shown in $4.1 OSCAR-N -SCAR-N--Lyers models require pre-training to align hidden representations with the generator LLM. Efficiency is controlled by the choice of N. We typically set N to 1/4-1/3 the total number of layers.
OSCAR-llama: we use smaller LLM, primarily Ilama- 1 as our compressor. We apply two dense layers with ReLU non-linearity the compressor last layer hidden space to align with the generator embedding space. Learning this mapping, which a crucial contribution of OSCAR, requires some pretraining (see Appendix Table 8) on top of the QA fine-tuming Thus, following Rau al. (2024b). we pretrain the compressor/generator LLM on auto-encoding and text-continuation tasks. Pretraining details are provided in Appendix H
Training objective The end-to-end OSCAR RAG pipeline should produce results as close as possible its no-compression version. Therefore, we use a sequence-level distillation objective as in Louis et (2025): given training set of questions and collection of documents, we performm the retrieval stage and generate teacher labels from the standard no-compression RAG pipeline. These labels are then used as hupervised-fine-tuiing targets for the end-to-end OSCAR pipeline, as shown Figure 2 (right). Overall, denoting 4j,... ar the answer generated by the teacher LLM from the documents and query, then the training objective on the compressor C' and generator G is:
L(C, 9) = Σ log (a: q, Ci,..., Ck, aci), where C;
(4), i= == (1)
where denotes the total number of documents used for generation. The loss is kack-propagater both through the generator LLM and the compressor LLM at each step. Overall, OSCAR training does not require any ground truth labels. Initial experiments with the teacher choice and use of distillation objective gives identical
meta--llmal www.lllaaallaaaaa Iama-3.22 B-Instruut
4

---

Published as a conference paper ICLR 2026
conclusions to Louis et al. (2025): distillation is paramount and Mistral-7B labels offer good supervision. For simplicity, we use Mistral-77 7B as the teacher for all OSCAR models, whichever backbone they are based on. practice, we save the retrieval results as well teacher generations once on the training set so that OSCAR training is simple snpervseddfinn-tuniig between cresttons--ugmment with document embeddings within the RAG prompt- and teacher answers. The subsequent OSCAR model training is fast: between and 5 gpu-days for IB-24B generator backbones.
Simultaneous reranking Building on insights from Chirkova et al. (2025, ), query-dependent online context compression closely resembles document reranking. Rerankers, such as cross-encoders (Nogueira & Cho 2019), refine the ranking from the initial retrieval step. Unlike retrieval models, which encode queries and documents independently, rerankers contextualize documents with respect queries, yielding more informative representations. Since rerankers are already part of strong RAG pipelines Rau et al. (2024a). using a single forward-pass for both compression and reranking makes compression essentially free so long compression is no more expensive than typical rerankers.
We therefore add a reranking token [RR] to the compressor LLM prompt (Figure 2 right) and an additional dense layer which maps this token's 's hidden state to predicted relevance score T;- We train this added layer
with point-wiss distillation objective from reference reranker: add x Ei=1 (r: - rf)² to equation where À balances generation and reranking and r are scores from a reference reranker. While many training strategies exist (Hofstätter et al., 202T [Formal et al.. [022 Lin al. 2021 Schlatt al., 2024). simple point-wise distillation proved effective for OSCAR models.
4 EXPERIMENTS
Data Our training dataset comprises questions from Louis et al. 2025 along with 500k queries extracted from MS MARCO (Nguyen et al. (2016), resulting in total of 893/ queries" ! The document collection used for training is Wikipedia-KILT (Petroni et al., 2020), preprocessed into chunks of 128 tokens. Such chunking is typical in RAG pipelines (Rau al.) (024a) and not limitation as increasing the number of retrieved chunks still enables to extract long sequences of informative content. For each query. we retrieve the top-k chunks using SPLADE-v3 (Formal al. 2021 Lassance et al., 2024) and subsequently rerank them with a DeBERTa-v3 (He et al. (021)- bbased reranker (a robust RAG setting as shown by |Rau et al. (2024a)). We employ sentence-level distillation from Mistral-7B as recommended by Louis et (0025).
Training details During training, the number & of retrieved documents is set to 5. We empirically found that this value provides sufficient context for models to generalize to a larger number documents at inference time while keeping training costs low. Each document is then compressed into embedding vectors, where is fixed for each OSCAR model. Specifically, OSCAR models with a compression rate of 16 use 8 memory embeddings per document- given 128-sized input documents. All generators LLMs are trained with LoRA Hu et al. (2021) adapters. For SSCAR-N--Lyer models, we experiment with N = 5, 8, 10. OSCAR-Ilama relies on Llama-3.2-11 et al. 2024). All compressors are trained with full-fine tuning which was consistently more effective than LoRA adapters. For joint training ($.3) early experiments suggested that 0.05 usually offers the best compromise (in terms of compression quality and reranking effectiveness) on the validation set and we use this default value for all further corresponding experiments. Additional hyper-parameters are given in Appendix
Baselines and Backbones We compare OSCAR to Provence and Recomp models Chirkova et al. (025)) Xu et al. (2023) as they are the state-of thrt hard compression models for RAG. We also run evaluations of
'We will release the queries as well as the distillation labels upon publication
5

---

Published as a conference paper ICLR 2026
Accuracy
Cera-Floating point operations
Backhone
Compressor
ASQA
HotpotQA NQ
TriviaQA POPQA
BIOASQ Average Inference Compression
Total
RAG
0.51
0.34
0.46
11779
0.29
0.40
0.47
No compression
0.75
0.51
0.68
0.92
0.70
0.51
0.68
20.33
0.
20.33
RECOMP
0.73
0.49
0.67
0.92
0.67
0.53
0.67
7.29
0.84
8.13 (2.5x)
Mistral-7B
Provence
0.76
0.49
0.69
0.92
0.69
0.54
0.68
7,63
1.80
9,43 (2.2x)
PISCO
0.48
0.90
0.65
3.49
offline
(5.8x
OSCAR-Ilama
0.74
0.53
0.68
0.92
0.68
0.52
0.68
3.49
2.66
6.15 (3.3x)
OSCAR-S-Layee
0.73
0,50
0.66
0,91
0.66
0,50
0.66
3.49
3.04
6,53 (3.1x)
OSCAR-8- Layeers
0.53
0.67
0.92
0.68
0.68
3,49
4.87
8,36 (2.4x)
No compression
0.61
0.35
0.54
0.82
0.59
0.55
2.85
0.
2.85
Llama-18
OSCAR S-Cyers
0.64
0.43
0.59
0.86
0.59
0.46
0.60
0.50
0.88
1.38 (2.1x)
No compression
0.70
0,51
0.64
0,90
0.64
0.53
0.65
18.94
0.
18.94
Qwen-7B
OSCAR L-Lyyr
0,72
0.50
0.64
0.91
0.67
0.51
0.66
3.17
5.07
8.25 (2.3x)
OSCAR-Ilama
0.72
0.51
0.66
0.91
0.68
0.67
3.17
2.65
5.83 (3.2x)
No compression
0.74
0.54
0.68
0.92
0.70
0.53
0.68
64.29
0.
64.29
Mistral-24B
OSCAR-IIama
0.75
0.54
0.70
0.93
0.70
0.53
0.69
10.72
2.65
13.37 (4.8x)
Table 1: Accuracy and efficiency for OSCAR models and baselines based on various backhones. OSCAR models more effective and as accurate than their backbones with no compression. OSCAR models are also more efficient than the two hard compression baselines Provence and Recomp.
"PISCO is intended to be used offline and given for comparison. "We do not train an OSCAR-Ilama with 1lama-3.2-1B backbone as it would not increase global efficiency.
PISCO models, a state-of-the-art offline soft compression model. Finally we provide a no-retrieval baseline well the performances of the no-compression RAG pipelines. Unlike most hard compression methods, OSCAR models are backbone-specife and need to be retrained for every different generation LLM. To show Dwen2-7B-Instruus Mistral-24BB
how stable OSCAR training is, we produce models for Mistral-7B-Instruue and Llama-1 BB. We keep identical anramettr/ddatacoofiiuraaio for all backbones. Training times range between GPU-days from 1B to 24B backbones.
For most of the experiments, we train OSCAR models without reranking ability. In $4.1 we provide evaluation metrics for OSCAR when compared to competitive approaches. In $4.2 we run ablations to identify the critical components of OSCAR. In $4.3) we show results of OSCAR models with reranking ability.
Evaluation After training, we evaluate all models on multiple datasets: Natural Questions Kwiatkowski
et al. (2019), TriviaQA Joshi et al. (2017), HotpotQA Yang al. (2018). ASQA Stelmakk et al.) (2022), P
Mallen et al. (202)), ) and BIOASQ- Krithara et al. (2023 j. For each query, we retrieve documents from either KILT PUBMED a collection unseen during training. We measure three different evaluation
metrics to ascertain the quality of OSCAR models:
accuracy: for some question, accuracy if the (normalized) label is included the (normalized) generated answer, where normalization is described in Appendix
(ii) LLM evaluation: we prompt an LLM to determine whether the predicted answer corresponds to the ground truth answer. This evaluation metric is robust semantically-equivalle reformulation of the answer and better correlated to human judgements Kamalloo et al. (2023). Details in Appendix E
ImistralaiMMissral-SSllll.222 -nnttuuttttt--tttttttt----2-
6

---

Published as conference paper ICLR 2026
(iii) pairwise-comparison using gpt-4o: given generated answers from two models, we prompt gpt-4o determine which answer is best, or if they are equivalent. Pairwise evaluation is good complement to pointwise Zheng al. (2023). Details in Appendix
Altogether, these metrics enable thorough evaluation and comparisons. OSCAR models have 5 retrieved documents per query at training time, but we evaluate them and all other models in setting with 10 documents verify generalization to larger contexts.
Computational efficiency To evaluate computational efficiency, we sum the number of floating-poin operations required for compression and for answer generation. For consistency, perform this calculation on a standardized input query of 128 tokens, concatenated with 10 documents of 128 tokens each or their compressed embeddings. Measurements are obtained using torch. profiler. Further details, including computation times and peak memory usage, are provided in Appendix D
4.1
MAIN RESULTS
Table shows the accuracy results for all backbones, well as efficiency measures. First, the no-RAG baseline has very low performance, indicating that these datasets are appropriate for RAG evaluations: the models can- not rely on memorization. Second, OSCAR models are faster than hard compression baselines while preserv- ing the accuracy of the no-compression models. Using OSCAR models in-place of the underlying backbones en- ables a 2.2-4.8x inference speed-up. Among OSCAR vari- ants, OSCAR -Ilamm is generally the strongest and fastest, though it requires pretraining (.2).4.)). Most interestingly. aSCAR-Ilame model for Mistral-24B enables 5x de-
Mistral-7B
POPΩА
25.5%
55.3%
18.2%
ASQA
33.1%
37.1%
29.7%
HotpotQA
TriviQAA
23.3%
50.7%
18.0%
NQ
BloASQ
25.4%
43.155
31.5%
OSCAR wins
Provence wins
POPQA
51.19%
ASQA
40.0%
31 2%
28 0%
HotpotQA
30.5%
TriviOAA
35.8% 8%.8%
49.2%
15.0%
NQ
BloASQ
24 0%
43.5%
32.4%
OSCAR wins
Tie
PISCO wins
POPOA
13.0%
ASQA
38.2%
33.6%
28.25
crease in computational complexity while improving the overall results. In fact, OSCAR efficiency improve- ments are proportional to the backbone size, and hence particularly advantageous for larger language models.
HotpotQA
40.6%
35.8%
TriviaAA
25.7%
57.3%
17.0%
NQ
35.7%
39.4%
24.9%
BioASQ
29.9%
42.6%
27.6%
Figure
GPT-4 pairwise comparisons. OSCAR-IIama, while faster, is on par with no compression baseline, Provence and PISCO.
For OSCAR-N -Layerrs models, performance improves with more layers but the cost of efficiency. Beyond 10 layers, accuracy plateaus while efficiency worsens (details in Appendix
Figure shows LLM evaluation results for Mistral-7B models. These confirm the conclusions based on the accuracy metric. In fact, OSCAR models tend to favorably appreciated by this LLM-evaluation. We hypothesize that since the retrieval is embedding- based rather than text-based for OSCAR, then reformulation the answer into remantically-equieeee equivalent answer is more likely to occur and to comparatively penalize the accuracy measure. Detailed results for all backbones given on Table 3
Finally, Figure Beeure3pput provides the results of pairwise comparisons of EOCAARIlamas Mistral-7B, PISCO and Provence. These confirm that OSCAR, while faster, is on par with its uncompressed baseline and slightly better on average than Provence. Overall, OSCAR models offer an efficient alternative to regular RAG pipelines, with x2-5 speed-up but little to no loss in accuracy.
7

---

Published as conference paper ICLR 2026
ASQA
HotpotQA
NQ
TriviaQA
Faster
0.275
5.64
6,900
Better
LLM eval
5.63
0.902
0.915
6.3200
5.42
0.877
5.41
644150
0.755
0.425
2.50
0.782
6,80
20
POPQA
BloASQ
Average
Total inference T-FLOPs
0.85
Mistral-7B
LLM eval
OSCAR-IIama
0.62
9.82
OSCAR-5-Layen
0.733
OSCAR-8-Layen.
0.00
0.735
OSCAR-10- -ocyyess
9.79
PISCO
Recomp
23
10
15
Provence
Total inference T-FLOPs
Total inference T-FLOPs
Total inference T-FLOPs
Figure 4: LLM evaluation scores of each Mistcko--B-bacckbored models, in relation with the total number floating point operations required at inference. OSCAR models are faster and more effective on most datasets. OSCAR-Ilane R in particular offers the best alternative. For PISCO, we include in the FI.OPs the compression cost, as it was used online.
4.2 ABLATIONS
We ablations to understand the effect of the components of OSCAR, with results shown Table 2 All ablations use Mistral-7B backbone.
Query-dependeer and compression rate. First, Table shows that accuracy losses with 128 compression are limited, with only 2% decrease on average. Second, we show that not using the query at compression leads to strong performance degradation,
pronounced for large compression rate (-6%). Thus, OSCAR did succeed in using the query to optimize the compressed representation. Furthermore, in Appendix M we look into the content of the compressed embeddings, to assess that they do indeed depend the query. Figure 13 uses a sredle-in-a-hhasttee test gkamradt (2024) to show that cosine similarity between compressed embeddings and text tokens is highest near the needle, indicating strong query dependence. Second, Figure 14 examines OSCAR embeddings via logit attributions nostalgebraist (2020). revealing that they align closely vocabulary space with context relevant the query.
Freezing the generator For OSCAR, we train jointly the generator and compressor models. We tried keeping the generator frozen, allow preserve fully the pretrained model in its state, but not succeed obtaining satisfying performance, as shown on Table 2
Other compressor architectures Results shown in 44..reeice Llama- IB as the compressor LLM. To obtain further efficiency gains, we tested using smaller compressors: modern-bert modern-berte bert- (Warner et al. 2024) and DeBERTa-v3 (He et al. 2021). Table results after pretraining and fine-tuning with different compressors. Llama-1B performs the best. Modern-bert-lary may offer an interesting alternative for low- latency applications.
8

---

Published as conference paper ICLR 2026
Model
compr rate
ASQA
HotpotQA
NQ
TriviaQA
POPQA
Avg (A)
Compression rate 16
OSCAR ASCAR-Ilama llama
0.82
0.64
0.81
0.91
0.65
0.77
query- mnery-independent
x16
0.81
0.60
0.78
0.89
0.57
0.73 (-0.04)
compressor pretraining
x16
0.78
0.56
0.75
0.89
0.51
0.70 (-0.07)
frozen generator
x16
0.78
0.54
0.76
0.88
0.60
0.71 (-0.06)
Compression
OSCAR- -Illama
x128
0.81
0.61
0.79
0,90
0.63
0.75 (-0.02)
rate 128
query- guery-independent
x128
0.81
0.57
0.75
0.89
0.51
0.71 (-0.06)
Other compressor
DeBERTa-v3
x16
0.80
0.61
0.77
0,90
0.57
0.73 (-0.04)
Modeen-berttbase
x16
0.80
0.62
0.77
0.90
0.60
0.74 (-0.03)
architectures
Modern-berttlarge
x16
0.83
0.80
0.91
0.64
0.76 (-0.01)
BM25 retrieval No compression OSCAR-Ilama
0.57
0.56
0.57
0.81
0.37
0.58
pipeline
x16
0.57
0.52
0.56
0.80
0.37
0.56 (-0.02)
Table 2: Ablation study on compression rate, pretraining, compressor architectures and retrieval pipeline. The last column reports averages across the five QA tasks, and the difference compared to OSCAR-IIama (x16). We report point-wise LLM evaluation.
Robustness to retrieval changes. In all training and test experiments far, all documents using SPLADE-v3 and reranked with a DeBBRRTa-v3-based reranker - a robust RAG setup Rau et al. (2024a). Yet it still prompts the question of how OSCAR models perform when retrieval quality declines. In particulan laa,, the behavior of hard compression methods is clearly identifiable on noisy documents and shown to be correctly handled by Provence Chirkova al. (2025) or Recomp Xu al. (023)). It is more of an open question for soft compression models like OSCAR. To investigate this, we run evaluation experiments using Robertson et al. (1996) only (no reranking) and report results on Table Essentially, the performance drops of OSCAR models with respect to Mistral-7B are similar indicating that OSCAR models are able to handle noisy documents. Detailed results for all datasets are found in Appendix
retrieved
Long context abilities of OSCAR models. Since OS- CAR models are trained with 5 retrieved documents, we investigate whether they remain able to extract and use information from larger number of documents. Figure 5 shows the results when increasing the number of retrieved documents to up to 50 (which makes uncompressed con- texts around 7 tokens) on ASQA. Note that as the number of documents increase, because of the quadratic cost of the attention, the larger compression rate of OSCAR mod- els make them comparatively faster. With 50 documents, we measure 5x less FLOPs for OSCAR than Mistral-7B. Further analysis on the ability of OSCAR compress longer documents is in Appendix K
Mistral -
Mistrall
Qwen2
OSCAR
OSCAR
OSCAR
Number of etrieved documents
Figure 5: LLM evaluations with increasing num- ber of retrieved documents: OSCAR models are as robust as their no-compression basclines.
4.3 ADDING RERANKING CAPABILITY
Having demonstrated that OSCAR models function effec-
tively as standalone compressors, also train OSCAR models capable of both document compression and reranking. In a RAG pipeline incorporating reranking, the computational cost of compression becomes single forward pass produces both compressed representations and reranking scores.
virtually negligible,
The results in Table 4 in the appendix show the performance of such jointly trained models under two evaluation settings: standalone, which corresponds the previous setting (DeBERTa-y3 reranker), and
9

---

Published as conference paper ICLR 2026
<2e which corresponds compressing documents reranked by the OSCAR model itself. Essentially, we observe no drop in performance between standalone and e2e settings, indicating that OSCAR effectively learns rerank documents. This finding further supported by OSCAR's performance on the BEIR benchmark Thakur et al. (2021) where its reranking capabilities are nearly on par with the strong teacher model. Detailed BEIR results for individual datasets are provided Appendix (Table To match the teacher's performance on BEIR, OSCAR requires an increased model depth to 16 layers. However, this model is less efficient, and its actual e2c performance (evaluated via LLM-based metrics or accuracy) remains unchanged.
5 CONCLUSION
this paper, we introduce OSCAR, the first online soft compression methods for RAG. The key challenge is designing efficient compression technique for online setting, which we address with two variants: one using a small compressor model and another leveraging the generator's early layers. We compare OSCAR against hard compression methods (RECOMP, Provence) and soft ones (PISCO), showing that query- dependent compression is more effective than query-independent approaches. OSCAR also outperforms matches hard pruning methods while being more efficient, proving the potential of soft compression. Additionally, we extend OSCAR with reranking. thereby reducing compression costs by factorization in the RAG pipeline. Our ablations analyze different backbones, weak retriever performance, behavior with large number of retrieved documents and further validate the design and performance of OSCAR models.
REFERENCES
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katic Millican, George Bm Van Den Driessche, Jean- -can-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pp. 22400. PMLR, 2022.
Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, and Dongyan Zhao. xrag: Extreme context compression for retrieeal-augmeented generation with one token. arXiv preprint arXiv:2405.13792 2024.
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. arXiv preprint arXiv:2305. 14788, 2023.
Nadezhda Chirkova, Thibault Formal, Vassilina Nikoulina, and Stéphane Clinchant. Provence: efficient and robust context pruning for retrieval- augmennte generation. In The Thirteenth International Conference on Learning Representations, 2025. URL
Giulio Corallo and Paolo Papotti. Finch: Prompt-guided key-value cache compression for large language models. Transactions of the Association for Computational Linguistics, 12:1517-1532, 2024.
Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini. A simple and effective 2_2 norm- based strategy for kv cache compression. arXiv preprint arXiv:2406. 1430, 2024.
Grattafiori al. The llama herd of models, 2024. UKL//ttpss///rrii.or////////////
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. Splade: Sparse lexical and expansion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2288 2292, 2021.
10

---

Published as a conference paper ICLR 2026
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. From distillation hard negative sampling: Making sparse neural ir models more effective. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development Information Retrieval, SIGIR '22, pp. 2353-2359, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 978 9781430387323 doi: 10.1145/3477495.533 1857. URL httphhtss://oo.rrggggggr
1145/3477495.. 3531857
Tao Ge, Jing Hu, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in large language model. arXiv preprint arXiv:2307.06945, 2023.
gkamradt. Needle in haystack pressure testing llms, 2024. URL https://gittub.. wwwp:/ttuub..ooooggaaaaaaaaaaaaa
Pengeheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre- pee-ttriinnne training with LLMrest_NeedeeInAHaytttel
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language
model pre-training. In International conference on machine learning, pp. 3929-3938. PMLR, 2020.
embedding sharing, 2021.
Sebastian Hofstätter, Jiecao Chen, Karthik Raman, and Hamed Zamani. Fid-light: Efficient and effective cetrieal-augmmented text generation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1437-1447, 2023.
Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. Improving efficient neural ranking models with cross- architecture knowledge distillation, 2021. URL/ttpss///axx
Org/abs/2011 0010.00000000 0206666
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 10.iv::106.096888 2021.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.0333 2017.
Ehsan Kamalloo, Nouha Dziri, Charles Clarke, and Davood Rafici. Evaluating open- -onomiiin question
answering in the era of large language models. In Anna Rogers, Jordan Boyd-Grabes and Naoaki Okazaki
(eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I:
Long Papers), pp. 5591-5606, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. (e.11855/v1/2000 12153//1/202...o acc.oo.....-11107....
httpps://cclaatttloleggo//////// attp:/2222..cc11omooo acl-long. 307/
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naT,L- volume 1. Minneapolis, Minnesota, 2019.
Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras. Bioasq-qa: A manually curated corpus for biomedical question answering. Scientific Data, 10(1):170, 2023.
Yuri Kuratov, Mikhail Arkhipov, Aydar Bulatov, and Mikhail Burtsev. Cramming 1568 tokens into single vector and back again: Exploring the limits of embedding space capacity. arXiv preprint arXiv:2502. 2025.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-4666 2019.
11

---

Published as a conference paper ICLR 2026
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611 111-266, 2023.
Carlos Lassance, Hervé Déjean, Thibault Formal, and Stéphane Clinchant. Splade-v3: New baselines for splade. arXiv preprint arXiv:2403.06779, 2024.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. serrhval--augmeneer generation for knowledge- intensive nlp tasks. Advances Neural Information Processing Systems, 33:9459-9474, 2020.
Haoyang Li, Yiming Li, Anxin Tian, Tianhao Tang. Zhanchao Xu, Xuejia Chen, Nicole Hu, Wei Dong, Qing Li, and Lei Chen. survey on large language model acceleration based on kv cache management. arXiv preprint MMNV:2212..14422 2024.
Sheng-Chiel Lin, Jheng-Hong Yang, and Jimmy Lin. In-batch negatives for knowledge distillation with tightly-coupled teachers for dense retrieval. In Anna Rogers, Jacer Calixto, Ivan Vulić, Naomi Saphra, Nora Kassner, Oana-Maria Camburu, Trapit Bansal, and Vered Shwartz (eds.), Proceedings of the 6th Workshop on Representation Learning for NLP (RepLANLP-20211 pp. 163-173, Online, August 2021. Association for Computational Linguistics. doi: 10. 111.6653//1122221rpppppa.. 1.17.
2021 rep1.11p1111pp1111
Maxime Louis, Hervé Déjcan, and Stéphane Clinchant. Pisco: Pretty simple compression for retrieval- augmented generation. arXiv preprini arXiv:2501.1603 2025.
Songshuo Lu, Hua Wang, Yutian Rong, Zhi Chen, and Yaohua Tang. Turborag: Accelerating retrieval-
augmented generation with precomputed kv caches for chunked text. arXiv preprint arXiv:2410.07550 2024.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannanch Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non- con-parametric memories. arXiv preprint arXiv:2212.105/ 2022.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Deng. Ms marco: A human-generated machine reading comprehension dataset. 2016.
Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with bert. arXiv preprint 191.00099. 2019.
nostalgebraist. Interpreting GPT: The logit lens. LessWrong, jan 2020. URL httpp::/www..esswrrrrr hatp://sss.aBBBBssalllloe/eeeeet
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al. Kilt: benchmark for knowledge intensive language tasks. arXiv preprint arXiv:2009.02252. 2020.
G Qin, C Rosset, E Chau, N Rao, and B Van Durme. Dodo: Dynamic contextual compression for decoder- only Ims. Proceedings the 62nd Annual Meeting of the Association for Computational ..., 2024.
David Rau, Hervé Déjean, Nadezhda Chirkova, Thibault Formal, Shuai Wang, Vassilina Nikoulina, and Stéphane Clinchant. Bergen: A benchmarking library for aaugenel- generation. arXiv preprint arXiv:2407.01 102, 2024a.
12

---

Published as conference paper ICLR 2026
David Rau, Shuai Wang, Hervé Déjean, and Stéphane Clinchant. Context embeddings for efficient answer generation in rag. arXiv preprint arXiv:2407.09252 2024b.
Stephen E. Robertson, Steve Walker, MM Beaulieu, Mike Gatford, and Alison Payne. Okapi at trec-4. Nist Special Publication Sp. pp. 73-96, 1996.
Ferdinand Schlatt, Maik Fröbe, Harrisen Scells, Shengyao Zhuang, Bevan Koopman, Guido Zuccon, Benno Stein, Martin Potthast, and Matthias Hagen. systematic investigation of distilling large language models into cross-encoders for passage re-ranking, 2024. Ut..https://arxii.org//bb wotp.///rr/a.///5.....// 07920
Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming Wei Chang. Asqa: Factoid questions meet long- form answers. arXiv preprint rrXi:2204.00692, 2022.
East Sun, Yan Wang, and Lan Tian. Block-attention for efficient rag. arXiv preprint arXiv:2409. 2024.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A hetero- geneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https://openneviieeeneet/// (tw.orrummnndccCcT5555F555
Zhiruo Wang, Jun Araki, Zhengbao Jiang. Md Rizwan Parvez, and Graham Neubig. Learning filter context for retrieval-augmmentd generation. arXiv preprint arXiv:231 108377, 2023.
Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, et al. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. arXiv preprint arXiv:2412. 13663, 2024.
Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving retrieval-augmennte Ims with compression and selective augmentation. arXiv preprint arXiv:2310004408. 2023.
Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv: /00:::/8000000000 899009 2018.
Chanwoong Yoon, Taewhoo Lee, Hyeon Hwang, Minbyul Jeong, and Jaewoo Kang. Compact: Compressing retrieved documents actively for question answering. arXiv preprint arXhv:2407.09014 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances neural information processing systems, 36:46595 46623, 2023.
13

---

Published as conference paper ICLR 2026
ASQA
HotpotQA
NQ
TriviaQA
0.62
Faster
0.65
0312
0.907
0.80
Better
0.61
LLM evall
0.79
0
0.26
0.76
0.73
0.800
0.74
0.002
0.70
0.55
0.73
0.72
75
16.0
12.5
16.
17.5
75
100
325
16.
17.5
75
10.0
12.5
15.0
175
10.3 125
150 176
POPQA
BioASQ
Average
Total inference T-FLOPs
0.88
0.02
LLM eval
Qwen-7B OSCAR-IIama
OSCAR-5-Layers
0.50
0.78
OSCAR-B-Layers
0.50
6.70
OSCAR-10-Layers
0.52
Recomp
17. Total inference T-FLOPs
75
125
Provence
Total inference T-FLOPs
Total inference T-FLOPs
Figure 6: LLM evaluation of (0wen2-7B-backoone models, in relation with the total number of floating
point operations required inference. OSCAR-Ilama model is the fastest and best compression model.
A ADDITIONAL RESULTS
A.1 OSCAR WITH QWEN2-7B
We showed Figure 4 the efficiency/performance plots for Mistral-7B backbone, including comparison with Provence, Recomp and the uncompressed backbone. We provide in Figure the same results but for Qwen2-7B. OSCAR-Ilama, models remains the best compression model, both in terms efficiency and L.LM evaluation score. In particular, OSCAR-Ilama score is average 4 points above Provence and 6 points above RECOMP.
A.2 DETAILED LLM EVALUATION RESULTS
Section we provided LLM evaluation results for Mistral-7B models. On Table 3 shows all LLM- evaluation results. In Section 4.1 provided pareto plot efficiency/LLMM evaluation for Mistral-7B Backbone. We provide on Figure the corresponding efficiency/accuracy pareto plot. Conclusions are mostly identical the main results
A.3 FULL RESULTS ON THE BEIR DATASET
We report in Table the detailed BEIR results individual datasets.
14

---

Published as conference paper ICLR 2026
ASQA
HotpotQA
NQ
TriviaQA
Paster
0.53
0.00
0.52
Better
Accuracy
0.90
0.74
0.50
0.73
0.13
0.49
0.48
30
15
26
15
20
10
20
10
POPQA
BioASQ
Average
Total inference T-FLOPs
0.083
Mistral-7B
Accuracy
0.68
0,670
OSCAR-Ilama
0.52
OSCAR-5-Layers
0.67
OSCAR-B-Layers
0.00
0.51
5.603
OSCAR-10-Layers
PISCO
0.65
0.50
0.650
Recomp
15
20 Total inference T-FLOPs
Provence
Total inference T-FLOPs
Total inference T-FLOPs
Figure 7: Accuracy scores of each Beetrral-B-bacbboone models, in relation with the total number of floating point operations required at inference. OSCAR models faster and better most datasets.
B DETAILED EFFECTS OF BM25 RETRIEVAL
section we provided averaged effect across datasets the change of retrieval/reranking pipeline. We provide in Figure results for individual datasets. These results show that performance is preserved across all datasets, although it likely that retrieval for Bioasq is noisier.
C INFLUENCE OF NUMBER OF COMPRESSOR LAYERS
Section we proposed constructing a transformer by utilizing the initial layers of the backbone to develop an efficient compressor that operates without requiring pretraining. Since the inference cost scales with the number of retained layers, it is important to examine the impact reducing the number of layers used for compression. This analysis is presented in Figure 9 where the performance appears to plateau around 4-5 layers for Mistral-7B. Notably, increasing the number layers beyond does not seem to justify the additional computational cost.
MORE ABOUT EFFICIENCY
D.1 SETUP TO MEASURE EFFICIENCY
In Section 4.1
measured efficiency of models based on the total number of foating-poim operations as it is the primary indicator of the computational complexity. To generate these measures, we generate fake inputs of standardized size (a query/prompt of 128 tokens associated with 10 128-token documents)
15

---

Published as conference paper ICLR 2026
Backhone
Compressor
ASQA HotpotQA NQ
TriviaQA POPQA BIOASQ Average
No compression
0.82
0.62
0.80
0.90
0.61
0.82
0.76
RECOMP
0.83
0.61
0.78
0.89
0.58
0.83
0.75
Provence
0,82
0.60
0.80
0.89
0.58
0.85
0.76
Mistral-7B
PISCO
0.81
0.60
0.78
0.89
0.57
0.79
0.74
OSCAR-Ilama
0.82
0.64
0.81
0.91
0.65
0.81
0.77
L-yyesss
0.82
0.62
0.79
0.90
0.63
0.77
0.76
OSCeRs
0.82
0.64
0.80
0.91
0.64
0.79
0.77
No compression
0.69
0.48
0.66
0.81
0.52
0.76
0.65
Llama-1B
OSCAR-S-Layers" 0.71
0.53
0.70
0.85
0.55
0.72
0.68
No compression
0.80
0.67
0.78
0.91
0.65
0.84
0.78
Qwen-7B
OSCAR-8 OSCAR-&-Layers 0.81
0.61
0.78
0.90
0.64
0.80
0.76
OSCAR- 0.82
0.62
0.79
0.90
0.65
0.81
0.76
No compression 0.82
0.71
0.80
0.92
0.70
0.85
0.80
Mistral-24B
OSCAR-IIama 0.82
0.65
0.82
0.92
0.67
0.84
0.79
Table 3: LLM evaluation and efficiency for OSCAR models and baselines based on various backbones. effective and faster than their backbones with compression. OSCAR models
OSCAR models also more efficient than the two hard compression baselines Provence and Recomp.
"W do not train an OSCAR- with Ilama-32-1B backbone as it would not increase global efficiency.
LLM evaluation score
Model
Setting
BEIR
ASQA HotpotQA
NQ TriviaQA POPQA BIOASQ Average
standalone
0.83
0.64
0.80
0.91
0.66
0.80
0.77
OSCAR-Ilama
52.8
c2e
0.81
0.63
0.79
0.91
0.66
0.80
0.77
standalone
0.82
0.64
0.81
0.64
0.79
0.77
OSCAR-8-Layers
$2.5
<2e 0.81
0.63
0.79
0.90
0.64
0.78
0.76
standalone
0.82
0.64
0.81
0.91
0.64
0.80
0.77
10-LaerssLayers
54.3
c2e
0.81
0.65
0.82
0.91
0.66
0.78
0.77
Table 4: LLM evaluation and reranking performance the BEIR benchmark (mean nDCG@ @ 10 on the 13 BEIR datasets). We report results for three efficient OSCAR models on two RAG settings (with a Mistral-7B decoder). The reranking performance of the teacher (based on DeBERTa-v3) is 55.4. Note that the performance on the standalone setting might slightly differ from previous Tables as these models are trained with a different loss (joint training).
and do compression and the generation of a 32 token answe from an input of size computed from the compression rate of each method (e.g., for OSCAR with compression rates 16, the input the generator of size 128 + 10 128 16 ). To compute FLOP we set the batch size to and use torch. profiler. We provide additional measures regarding inference time and peak GPU memory in each case. We set the batch size at 256 (32 for the larger Mistral-24B) to compute the inference time (simulating busy service) and the
'The analysis for generated answers of 128 256 tokens leads similar conclusions
16

---

Published as conference paper ICLR 2026
Corpus
DeBERTa-v3
OSCAR- -llama
OSCAR- ---Leesss
OSCAR- L0-Layers
OSCAR- 16- L6-Laye
TREC-COVID
88.3
83.1
81.4
84.4
86.1
NFCorpus
34.2
34.5
36.5
36.9
NQ
66.7
63.3
61.3
64.1
67.2
HotpotQA
74.5
72.9
72.2
73.5
74.3
FiQA-2018
47.8
42.7
40.8
44.3
47.5
ArguAna
29.8
29.5
32.5
32.4
34.0
Touché-2020
33.5
29.3
31.6
31.9
31.3
Quora
84.8
86.0
86.0
87.5
87.9
DBPedia
48.9
47.5
46.5
48.2
49.2
SCIDOCS
19.2
17.2
17.6
18.6
19.3
FEVER
86.6
83.6
83.1
84.1
83.9
Clmatt-FEVER
27.4
25.9
24.2
25.3
26.3
SeiFact
75.8
71.2
71.2
75.2
75.5
average
55.4
52.8
52.5
54.3
55.3
Table 5: nDCG@1 the 13 open BEIR datasets. DeBERTa-v3 is the reranker teacher used to train OSCAR models.
ASQA
HotpotQA
LLM evaluation
0.79
retnever Deberta-/1
0.6
0.62
0.38
0.7-
0.52
0.52
0.6
0.57
0.57
0.5
0.5
retriever
0.3
NQ
TriviaQA
0.79
76
0.90
0.90
0.90
0.7-
Splad:
0.8
0.81
0.80
BN25
0.57
0.56
0.8
0.
0.4
Splade Debertsw3
BXQ5
0.2
0.2
BloASQ
retrever Splade Debertsv3
0.75
0.79
BN25
0.7
0.71
0.70
0.4
0.5
Splade Deberta
0.3
BM25
Mistrai-78
OSCARiEams
OSCAR-B-Layes
Mistrol-7
OSCAR tama
OoCyR-B-Layers
Figure 8: Effect of retrieval on OSCAR models, per dataset, compared to their uncompressed backbone.
peak GPU memory. In all cases we use hugging face implementation of the models. For memory usage and inference time, average the results over 10 runs.
17

---

Published as conference paper ICLR 2026
0.34
LLM-eval
0.72
Mistral - 778
OSCAR
No Compression
Qwen2
OSCAR
No Compression
16
14
Number of layers in compressor
Figure 9: Average accuracy general domain datasets for OSCAR models where the compressor has variable number of layers. Performances increase with the number of layers but plateau above 8-10 layers for both Qwen2-7B and Mistral-7B backbones.
Inference time (ms))
Backbone
Compressor
Architecture
Parameters
Inference
Compression
Total
Peak memory (Gb)+
No compression
141.6
0.
141.6
24.3
OSCAR-SL
1.2B
33.0
18.0
51.0 (2.3x)
16.2
Mistral
OSCAR-88
1.91B
33.0
28.8
61.8 (2.2x)
16.2
OSCAR-Ilame
1.1B
33.0
17.1
50.1 (2.8x)
16.2
No compression OSCAR-51.
30.2
0.
30,2 13.3 (2.3x)
8.6
Llama 3.2 1B
8.3
4.3
No compression OSCAR-SL
109
0.
109
30.2
Qwen-2-7B
1.7B
40.8 (2.7x)
23.3
OSCAR-Ilama
1.1B
25.6
17.1
42.7 (2.6x)
23.3
No compression
383.2
0.
383.2
69.2
Mistral-24B
OSCAR--Iama
1.1B
67.9
17.1
85.0 (4.5x)
51.9
Table 6: Inference time and memory for each model. Computed with 128-token queries and 128-token retrieved documents. computed with batch size 256 (32 for Mistral-245 -2BB) but brought down to individual query cost for a batch size 32.
Results are shown in Table 6 Gains observed in terms of floating-point operations mostly translate to computational time (as can be expected for sufficiently large batch sizes). OSCAR models enable to save about 50-75% of memory across the various backbones. In practice, this larger batch sizes to be used and hence further latency improvements.
LLM EVALUATION
Our primary evaluation metric follows the LLM-based assessment proposed in Rau et al. (2024a). This approach utilizes the SOLAR-107B mode prompted determine the correctness of predicted answer by comparing it against both the given question and reference answer. This metric can be viewed as an
18

---

Published as conference paper ICLR 2026
enhanced version of traditional accuracy, as it remains more robust surface-level variations that do alter the underlying semantic content. The prompt used is given in Figure I0.
Figure 10: LLM Evaluation Prompt
system: "You are evaluation tool. Answer with one of 1: Correct, 0.5: Partially correct, O: wrong. "Here is a question, a golden answer, and an Al-generated answer. Can you judge whether the Al-generated answer is correct according to the question and golden answer? Simply answer with one I: correct, 0.5: partially correct, O: wrong. Question: (question). Golden answer: (answer). Generated answer: (prediction)."
user:
PROMPTS
The prompt for generation is given on Figure II The prompt for GPT pairwise comparison is given on Figure 12
OSCAR TRAINING HYPERPARAMETERS
We provide in this section details enabling the replication of OSCAR training results. Note that all OSCAR models for all backbones (from Ilama-IB IB all the way to mistral-24B) were trained using this configuration. Our training code relies on HuggingFace trainer and an adaptation of the public Bergen library Rau et al. 22024a).
Note that OSCAR-N-layer models are directly trained by fine-tuning on the distillation data described in Section they do not need pretraining. This is similar effect as in Louis et al. (2025). On the contrary, OSCAR-Ilama models need pretraining described Appendix
Hyper-parameter search to build OSCAR models We took hyperparameters from Louis et al. (2025) and only conducted small grid search over 8 values to tune the learning rate required on the compressor, as we noticed performances were underwhelming with identical learning rates on compressor and generator. The total computation time to train an OSCAR model around Mistral-7B around hours on single high-end GPU.
H OSCAR-LLAMA PRETRAINING
OSCAR models using llama-15 1B as compressor models without any pretraining failed to reach satisfying performances (see Table 8). We attribute this effeet to the need of building map between the compressor
Figure 11: Main Prompt
system:
You are helpful assistant. Your task extract relevant information from provided documents and to answer questions as briefly as possible. user: Background:
{doc, }SEP(doc2) ... SEP(dock)
Question: {question)
19

---

Published as conference paper ICLR 2026
Figure 12: Gpt-4o Pairwise Comparison Prompt
system: "You are helpful assistant that ranks models by the quality of their answers. Please act impartial judge. Do not allow the length of the responses to influence your evaluation. Be as objective as possible."
user: "Here a question, ground truth answer, an Al-generated answer 1, and an Al-generated answer
2. Which answer the most correct one? Simply answer if the first better, 2 the second is better, and 3 if it's tic.
Question: {question].
Ground truth answer: {ref answer).
Answer I: {answer
Answer 2: (answer2)."
Hyperparameter
Value
Batch Size
128
LR generator
1 x 10 10-4
LR Ilama compressor
1
LR N-layers compressor
5 x 10
LR scheduler
linear
Optimizer
Adam W
Epochs
1
Max Tokens Teacher Generation
128
LoRA Layers (r)
all-linear
LoRA Rank (r)
16
LoRA Dropout
0.1
LoRA Alpha
32
Ilama compressor hidden dim
8096
Weight Decay
0.1
Warmup Ratio
0.05
Max Gradient Norm
1.0
Documents tokens
128
Table 7: Fine-tuning Hyper-parameters.
"Initial results with identical learning rates between the LoRA-trained decoder and fully fine-tuned N-layers compressor gave poor results: learning rates need differentiated between compressor decoder this case.
hidden space and the decoder hidden space. To achieve this, use the same pretraining proposed in Rau et al.] (2024b), with identical hyperparameters and a pretraining dataset consisting of chunks preprocessed from finewer Note that experiments show that as long as some form of extended pretraining is done which requires the decoder to use embeddings produced by the compressor, the ensuing OSCAR-Ilans models are crucial for replicating our work.
strong. Therefore, the exact recipe of the pretraining
20

---

Published as conference paper ICLR 2026
Model
ASQA HotpotQA NQ TriviaQA POPQA
OSCAR-
0.82
0.64
0.81
0.91
0.65
+ without pretraining 0.78
0.56
0.75
0.89
0.51
Table 8: Ablation on the pretraining for OSCAR-Ilama model.
I STRING NORMALIZATION FOR METRIC COMPUTATION
To measure accuracy, F1 score or recall between a ground truth label and a prediction, we check that the normalized label is included in the normalized prediction. When multiple labels possible, take maximum values across the available labels. Normalization consists in:
Converting the string to lowercase
Removing punctuation
Removing articles: "a", "an", "the"
Standardizing word splits by replacing multiple spaces and line returns with a single space
J RELATION BETWEEN EMBEDDINGS AND QUERY
While OSCAR offers greater computational efficiency and accuracy, it lacks the interpretability of hard compression methods. In this section, we offer glimpse into the content of the compressed embeddings, assess that they do indeed depend on the query. First, Figure 13 uses reedle-in-a-haystenr test gssamaadddd (2024) to show that cosine similarity between compressed embeddings and text tokens is highest near the needle, indicating strong query dependence. Second, Figure 14 examines OSCAR embeddings via logit attributions nostallebraist 2020), revealing that they align closely in vocabulary space with context relevant the query.
K COMPRESSING LONGER DOCUMENTS
So far systematically compressed 128-token documents. This offers valid RAG pipeline with excellent performances. However, to further understand the robustness OSCAR, we run additional experiments where expose the model to 256-token documents:
Doc:Merge-5x2: the top 10 retrieved documents concatenated pairwise, producing 5 documents whose lengths are doubled. This setting isolates the model's 's ability to exploit longer but noise-free context.
DoeMerge-Noisy-102 each of the top 10 retrieved documents is concatenated with randomly selected irrelevant document, yielding 10 new documents twice as long. This evaluates robustness to increased context length combined with noise.
Discussion. On the DocMerge-5x2 condition, OSCAR-despit not being trained on 256-token documents-exhibee only modest performance drop (approximately -1% on average) and still achieves competitive accuracy across benchmarks. In the DocMerrge-Nois--10 pled with injected noise, OSCAR performs par with its Mistral-7B backbone (within 0. 0.7%), highlighting its strong ability to process and utilize extended evidence even under noisier long-context regimes.
setting, where longer context is cou-
21

---

Published as conference paper ICLR 2026
Cosine similarity
Boltom document
Emtbedoing olocer the T3cT
Cats have excellent iight vision and can see at one sleth the light level required for human vision. This partly
eyes having tapetumn
Placed fact document depth
lucidum, which roflects light that passes through the retina back into the thereby increasing eye's sensttivity dim light. limbless reptiles of the suborder Serpentes. Like all other squamatess snaless ectothermic,
<01
elongated.
overlapping
What allows cats see light lnvels times dimmer than what humans require?
What type of body snakes have?
Query
Embedding
dim, <MEMU>, any cat, eye, back, reflect, sensibivity, thereby, reflected, any
el, lim, covered, Lim, Sub, of cats, Cover Se
The document
Context length tokens)
attributions
Figure Cosine similarity between document embeddings and document individual tokens, on a needle-in-a-hee -ha-t--hyyttck test. The document embed- dings are more similar to the area around the needle, indicating that compression focuses on query- related elements.
Figure 14: Logits attributions on OSCAR embed- dings. Attributed tokens predominantly correspond an area of the context relevant to the query.
Model
ASQA
HotpoiQA
NQ
TriviaQA
POPQA
BioASQI2B
Avg.
Mistral-7B
0.752
0.512
0.683
0.917
0.699
0.510
0.679
Mistral-7B 7B (DocMerge-54)) xx2)
0.732
0.484
0.650
0.911
0.649
0.495
0.654
Mistral-7B (DocMerge-Noixy-1022
0.674
0.513
0.682
0.918
0.696
0.502
0.664
OSCAR-LLaME
0.742
0.527
0.682
0.917
0.685
0,518
0.679
OSCAR-LLANE (xoMMere-5xx2)
0.703
0.494
0.648
0.904
0,615
0.495
0,643
OSCAR-LIAMA (DeeMerge-Noisy10022 10x2)
0.715
0.503
0.651
0.909
0.648
0.510
0.656
Table 9: Performance of OSCAR-LLaM and its Mistral-7B backbone under long-context robustness settings.
22


Reviews:
[{"review_id": "tsemWtNXKy", "reviewer": "Reviewer_4qhT", "summary": "The paper introduces OSCAR (Online Soft Compression for RAG), which tries to \"sit in the middle\" between (i) online, query-aware hard pruning methods such as Provence and RECOMP, and (ii) offline, high-ratio soft compression methods such as PISCO / earlier context-embedding work. The key idea is to run, at inference time, a query-conditioned compressor LLM that takes (q, dᵢ, MEM tokens) and emits a small set of embedding tokens per retrieved document, which are then fed to the generator LLM instead of the original text. Two realizations are proposed: (1) OSCAR-N-Layers (reuse the first N layers of the generator) and (2) OSCAR-llama (a small 1B Llama with a learned projection). The whole pipeline is trained end-to-end by sequence-level distillation from an uncompressed RAG teacher (Mistral-7B) and can optionally output a reranking score from the same forward pass so that compression is \"for free\" On six RAG QA benchmarks and on several backbones (Mistral-7B, Qwen2-7B, Mistral-24B, Llama-1B) the authors report 2–5× FLOP savings with little or no accuracy loss relative to uncompressed RAG and better speed/quality than online hard pruning (Provence, RECOMP).", "questions": "- Scope of \"online.\" In a production RAG service that already runs a DeBERTa-v3 reranker, what is the measured latency (ms) of OSCAR-llama / OSCAR-8-Layers for 10×128-token docs on an A100, and how does that compare to that reranker? Please report wall-clock, not only FLOPs.\n\n- Reusability across queries. Because OSCAR is query-conditioned, you can't pre-compress a corpus once. Did you try a two-stage variant (cheap query-agnostic soft compression at index time + light query adapter at run time)? If not, what prevents it?\n\n- Positioning vs. DODO / TurboRAG / KV-cache compression. What concrete capability do those methods lack that forces you to call OSCAR \"the first\" online soft compression? In particular, DODO can also run at inference and reduce hidden states. \n\n- Sensitivity to retrieval quality. You show BM25-only experiments, but only aggregated. Can you provide per-dataset breakdowns and failure cases (e.g., when none of the top-k docs is actually relevant), and how compression interacts with that?\n\n- Teacher choice. You always distill from Mistral-7B. If the teacher is smaller or of lower RAG quality, do you still beat Provence / RECOMP?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Well-framed gap and plausible contribution. The paper correctly observes that current online methods are mostly text-level and top out around 2× compression (Provence, RECOMP) while soft methods (PISCO, xRAG/COCOM-style context embeddings) reach 16× but are practically offline and often query-independent. Positioning OSCAR as \"query-dependent and online\" is a nice conceptual sweet spot. \n\n- Strong and unusually broad empirical section. The paper does not just show \"one backbone, one dataset\": it repeats the story on four backbones (including a 24B model), six RAG QA tasks, and also checks harder settings such as BM25-only retrieval, more retrieved docs (up to 50), different compressor depths, and different compressor families. That breadth makes the claim “this is not overfit to one setup” fairly convincing. \n\n- Factoring compression and reranking in one pass is practically useful. Existing strong RAG pipelines already pay for a cross-encoder reranker; using (almost) the same computation to also produce compressed representations is exactly the kind of engineering win production RAG wants, and it is consistent with how Provence also couples pruning and reranking.", "weaknesses": "- Novelty is a bit overstated. The paper calls OSCAR the \"first query-dependent online soft compression,\" but there is prior art that is close enough that it must be discussed carefully:\n\n    - DODO already performs dynamic contextual compression for decoder-only LMs and can be used as a context compressor at inference. It also reduces the number of hidden states and is explicitly meant to lower long-context cost. \n\n    - PISCO (2025) explicitly targets 16× compression for RAG and is trained by sequence-level distillation; the main difference is \"offline vs. online,\" but OSCAR’s evaluation makes PISCO pay an online cost, which weakens the fairness of the comparison. \n\n    - Provence / RECOMP already couple pruning with reranking or selective augmentation; OSCAR's \"free compression + rerank\" is in the same design space. \n\n- The paper needs to spell out why these do not qualify as \"online soft, query-dependent\" in the authors' definition.\n\n- Comparison protocol slightly favors OSCAR. Counting PISCO’s compression as if it had to be done online makes OSCAR look 3–5× faster, but the whole point of PISCO/xRAG/COCOM-like approaches is that you pre-compute or at least pre-compress. Likewise, Provence and RECOMP are mostly LLM-agnostic and don't require per-backbone re-training, while OSCAR has to be retrained for every generator (the authors themselves retrain for Mistral-7B, Qwen2-7B, Mistral-24B, Llama-1B). Deployment cost and model storage are not reported, so the advantages are not end-to-end in a systems sense. \n\n- Missing or underdeveloped baselines / metrics. The related-work section knows about FINCH, TurboRAG / block-attention RAG, and long-context KV-cache tricks but they are not in the experiments, even though some of them also run online and trade KV size for speed. We also never see actual wall-clock latency or QPS-under-load numbers, only FLOPs on synthetic inputs, so it’s hard to tell whether \"OSCAR compressor + generator\" < \"Provence + generator\" on a real A100/H100 with retrieval in front. And everything is single-domain QA with 128-token chunks; real RAG often has heterogeneous, noisy, and much longer pieces.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tt3IVUttiM", "reviewer": "Reviewer_6CpB", "summary": "This paper propose a compression method for RAG, called OSCAR. This method includes two designed models; the first one is called compressor, which is used to convert document chunks as well as query into fixed-length embeddings. The second model is called generator, which leverages the query and the embeddings obtained from the compressor to generate an answer. Both compressor and generator can be trained in an end-to-end fashion, using the RAG results from a teacher model without compression as training signals. Experiment results show that on a series of QA datasets, the proposed method can achieve similar performance compared to no compression and slighter better inference cost.", "questions": "Evidence that the method generalizes to longer context cases would be appreciated; does it maintain similar accuracy and end-to-end speed when k and document lengths increase?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well writen and easy to follow.\n2. The proposed method combines the advantages of both hard-compression and soft-compression, allowing online query-aware compression while ensures a better compression rate. It is a pretty straightforward and intuitively nice idea.\n3. The experiment results are conducted on a wide range of datasets, making the rseults look solid.", "weaknesses": "One possible concern is about the context length. This paper majorly conducts experiments on 128 token chunks, 5/10 chunks per query. This is one possible scenario, but other cases where longer context is applied is not tested in this paper. I would argue that results only under the settings from this paper is not comprehensive enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T364t3i7Do", "reviewer": "Reviewer_j7H9", "summary": "The paper proposes OSCAR, a query-dependent, online soft compression framework for RAG. Unlike prior hard or soft compression approaches, OSCAR compresses each (query, document) pair into a small set of embedding tokens on the fly and conditions generation on these embeddings. Across open-domain QA benchmarks, OSCAR targets 2–5× end-to-end inference FLOPs reduction with near-parity accuracy to uncompressed RAG, with stronger gains on larger backbones.", "questions": "- How does the LoRA-fine-tuned generator perform on standard non-RAG benchmarks ? Does fine-tuning lead to a degradation in general-purpose reasoning or language understanding?\n\n- The current cross-domain evidence is limited, focusing mainly on biomedical QA and retrieval-level BEIR, without end-to-end evaluation in other domains or non-QA RAG tasks. How does the proposed approach behave on cross-domain retrieval or non-QA RAG tasks?\n\n- Could the authors report results when the generator is kept frozen during training to isolate the effect of compression?\n\n- Providing a more comprehensive evaluation would offer a fuller assessment of the method’s contribution and generality.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method is elegantly designed, conceptually bridging the gap between traditional hard compression and offline soft compression in terms of both performance and usability. \n- Strong empirical results, achieving near-parity accuracy with up to 2–5× efficiency gains. \n- The integration with reranking allows shared computation, improving system efficiency without additional overhead.", "weaknesses": "1. My main concern is that fine-tuning the generator may impair generalization. Because the compressor is query-dependent, the resulting features can encode domain-specific priors; joint training encourages the generator to exploit these priors, increasing the chance of specialization and consequent degradation on cross-domain retrieval and non-RAG tasks. This potential loss of generality could limit the method’s practicality in real-world deployments.\n\n2. The experiments are mainly conducted on Mistral-7B and Qwen2-7B, while larger models such as Mistral-24B are only compared against an uncompressed version. Moreover, the evaluation focuses on accuracy and FLOPs, omitting critical system-level metrics such as end-to-end latency and throughput. Although Table 6 reports related values, it lacks direct comparisons to baseline methods.\n\n3. The paper claims that query-dependent embeddings preserve task-relevant semantics but provides limited theoretical analysis or formal justification to support this claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X02t62PNaC", "reviewer": "Reviewer_kf7i", "summary": "This paper proposes OSCAR, an online, query-dependent soft compression mechanism for RAG. A compressor LLM maps each (query, document) pair to a small number of learned “memory” embeddings, which a generator LLM then consumes to answer the query. Two compressor backbones are explored: (i) OSCAR-N-Layers, using the first N layers of the generator; and (ii) OSCAR-llama, a small LLM with a learned hidden-space mapping. Across Mistral-7B, Qwen-7B, Llama-1B and Mistral-24B backbones, OSCAR yields 2–5× end-to-end speedups with minimal accuracy loss.", "questions": "1. How does OSCAR compare to existing online soft compression approaches such as FiD-light mentioned in the prior work?\n\n2. In Figure 4 for BioASQ, why is the accuracy of OSCAR consistently worse than the baselines with the same compute T-FLOPs? \n\n3. In Figure 3 for the comparison of OSCAR with Mistral-7B, OSCAR also does not show clear wins as compared to Provence and PISCO. What are the possible reasons behind this?\n\n4. For the N value taken in OSCAR-N-Layers, what is the sensitivity of it when the backbone model changes?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Propose a novel online soft compression method that compresses the input in a query-dependent fashion.\n\n2. The training pipeline of OSCAR operates in a self-supervised way that does not rely on ground truth labels and generates them from no-compression RAG pipeline, providing flexibility over document choices.\n\n3. Comprehensive evaluation with different choices of datasets, backbones and evaluation metrics. The results show a better accuracy-efficiency tradeoff for OSCAR over existing methods.\n\n4. Detailed ablations on compression rate, query dependence, context length and compressor architecture.", "weaknesses": "1. OSCAR must be retrained per generator LLM to align hidden spaces; while the paper notes a few GPU-days for training, this could be operationally costly for trying out different backbone LLMs for different tasks.\n\n2. Under some dataset setups OSCAR has worse accuracy compared to existing compression methods or the original backbone models. The paper does not provide a good investigation into the reasons causing this.\n\n3. The baselines chosen by the paper are mainly hard compression and offline soft compression methods. There is a lack of comparison with existing online soft compression methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)