Estimating Optimal Context Length for Hybrid Retrieval-augmented Multi-document Summarization
Authors: Adithya Pratapa
Teruko Mitamura
Venue: COLM 2025 (2025)
Keywords: retrieval-augmented generation
long-context
multi-document summarization
TL;DR: We present a novel method to estimate optimal context length for retrieval-augmented generation. Our estimate is a function of the retriever, summarizer and the downstream task.

Abstract: Recent advances in long-context reasoning abilities of language models led to interesting applications in large-scale multi-document summarization. However, prior work has shown that these long-context models are not effective at their claimed context windows. To this end, retrieval-augmented systems provide an efficient and effective alternative. However, their performance can be highly sensitive to the choice of retrieval context length. In this work, we present a hybrid method that combines retrieval-augmented systems with long-context windows supported by recent language models. Our method first estimates the optimal retrieval length as a function of the retriever, summarizer, and dataset. On a randomly sampled subset of the dataset, we use a panel of LMs to generate a pool of silver references. We use these silver references to estimate the optimal context length for a given RAG system configuration. Our results on the multi-document summarization task showcase the effectiveness of our method across model classes and sizes. We compare against length estimates from strong long-context benchmarks such as RULER and HELMET. Our analysis also highlights the effectiveness of our estimation method for very long-context LMs and its generalization to new classes of LMs.

Published as a conference paper at COLM 2025
Estimating Optimal Context Length for Hybrid Retrieval- augmented Multi-document Summarization
Adithya Pratapa
Teruko Mitamura
Language Technologies Institute
Carnegie Mellon University
{vpratapa, teruko) cmu, edu
Abstract
Recent advances in long-context reasoning abilities of language models led to interesting applications in large-scale multi-document summarization. However, prior work has shown that these long-context models are not ef- fective their claimed context windows. To this end, retrieval-augmmeted systems provide an efficient and effective alternative. However, their per- formance can be highly sensitive the choice of retrieval context length. In this work, we present a hybrid method that combines retrieval-augmennt systems with long-context windows supported by recent language models. Our method first estimates the optimal retrieval length function of the retriever, summarizer, and dataset. On a randomly sampled subset of the dataset, we
panel of LMs to generate pool of silver references. We use these silver references to estimate the optimal context length for given RAG system configuration. Our results the multi-document summariza- tion task showcase the effectiveness of our method across model classes and sizes. We compare against length estimates from strong long-context benchmarks such as RULER and HELMET. Our analysis also highlights the effectiveness of our estimation method for very long-context LMs and its generalization new classes of LMs..
1 Introduction
Language models increasingly support longer context windows, leading to useful appli- cations in large-scale multi-document summarization. Recent work has shown that these models are not very effective at their claimed context windows (Hsieh et al., 2024; Yen et al., 2025). An alternative to the full context setting is retrieval-augmented generation (RAG), and previous work has illustrated its effectiveness for long input processing (Asai et al., 2024; Li et al., 2024). RAG systems facilitate better use of the LM context windows by passing only the most relevant information to the model. However, the choice of retrieval length that provides peak RAG performance is often unclear and sensitive to the choice of retriever, language model, and downstream task (Jin et al., 2025). In this work, we present a methodology for estimating this optimal retrieval length as function of the retriever, summarizer, and dataset. In addition to providing gains over the full context setting, our method also outperforms the lontext-length estimates identified by standard long-context evaluation benchmarks. Figure provides a schematic overview of our method.
Previous efforts to combine RAG and long-context LMs focused on query-based routing (Li et al., 2024), iterative RAG (Yue et al., 2025). While these methods are effective, they rely on the model's ability to accurately determine the scope of information need and self- evaluate its own output. This might not always be feasible option, especially for smaller LMs. In this work, we take a complementary approach to combine RAG and long-context and show its effectiveness for models ranging from 0.5B to 72B parameters. We evaluate on a challenging large-scale multi-document summarization dataset (Laban et al., 2024).
'Our code is publicly available at https: /ats:::///tthiie http://c///aiiaay///aaa
1

---

Published as a conference paper at COLM 2025
Baseline retriever length estimate f(summarizer) (RULER, HELMET)
Dataset
Retriever
Summarizer
Summary
Taec-fp estimate
(Proposed)
Silver
Llama
F1
Optimal context length fddataeet, retriever, summariter)
Silver reference
Sampled
Qwen
subset
summaries
Jamba
# retrieved tokens
LM panel
(used in a full-context setting)
RAG-based task performance as a function of retrieved tokens
Figure 1: A schematic overview of our proposed method. Unlike traditional benchmarks, estimate the optimal retrieval length as function of dataset, retriever and summarizer. Given a dataset, we first sample a fraction of examples. On this subset, we run panel of LMs in full-context setup to create silver candidates. We then identify the top silver candidates using Minimum Bayes Risk decoding. With the help of these silver candidates, we estimate the optimal retrieval length for the given experiment config.
In a recent work, Jin et al. (2025) compared the RAG performance of varying model sizes on the guestion-answering task and found that the optimal retrieval length varies considerably across model sizes. They also found that this length is sensitive to the choice of retriever. Similarly, Yu et al. (2024) noted the sensitivity of optimal retrieval length to the downstream task. Based on these observations from previous work, we hypothesize that the retrieval length that provides peak performance should be modeled components of the RAG pipeline: retriever, summarizer, and dataset. For our baselines, use two popular long-context evaluation benchmarks, RULER (Hsieh et al., 2024) and HELMET (Yen et al., 2025). They benchmark models on a suite of tasks with inputs of increasing lengths. RULER focuses on synthetic retrieval tasks, while HELMET includes NLP tasks such as LongQA and summarization. Although these provide effectine context length estimates for individual LMs, these estimates are often agnostic to the downstream dataset and the retrievers when used in the RAG setting.
function of the three main
Given a dataset, we first create subset of representative examples by random sampling. We then use panel of LMs to compile a candidate set of silver reference summaries. In our panel, we include LMs from the Qwen (Qwen et al., 2025), Llama (Grattafiori et al., 2024) and Jamba (Team et al., 2024) series. From the pool of candidate silver references, Minimum Bayes Risk decoding (Kumar Byrne, 2004) to identify the top silver reference summaries. For given combination of retriever and summarizer models, we perform search over context lengths on this silver subset to estimate the optimal retrieval length. Unlike baseline methods, our approach is customized the specific experiment configura- tion (dataset, retriever, and summarizer). Our method is based on two key observations. First, larger LMs show robust performance across a broad range of context lengths. This is mainly due to their enhanced ability to deal with noise in the retrieved input (Jin et al., 2025). Second, to identify a task-specific estimate, we can approximate the gold summaries with silver candidates sampled from strong long-context LMs.
use
We evaluated our method for the multi-document summarization task using the SummHay dataset (Laban et al., 2024). Our results show that all retrieval-basee methods (baselines and ours) significantly outperform full-context. Our method performs the best in most settings, followed by HELMET- and RULER-basse estimates. Although HELMETTbased estimates sometimes perform comparable to our method, neither the LongQA nor summarization task-based HELMET estimates consistently perform better. Notably, our method performs much better on very long-contess LMs such as Qwen 2.5 1M and ProLong 512k. Our analysis also shows that method generalizes well to model classes outside of our panel (e-g.,
2

---

Published as a conference paper at COLM 2025
Phi-3). We also perform ablation experiments our LM panel well as the size of our sampled subset.
2 Estimating Optimal Context Length for Retrieval
For the multi-document summarization task, given long input and a query, we have two possible systems. First, the entire input is fed directly into long-context summarizer that supports such lengths (full-context). Second, we use the query to rank the documents and only pass the top-k relevant documents to the summarizer (RAG). Previous work has shown that long-context models are not effective at their claimed context windows, and RAG can help improve task performance (Yu et al., 2024; Pratapa & Mitamura, 2025).
Benchmarks such as RULER and HELMET provide a comprehensive evaluation of long- context models across suite of NLP tasks, including QA and summarization. However, these benchmarks focus solely on the model and do not study the effects of unseen down- stream datasets and the retrievers used in RAG settings. Previous work has briefly studied this with Jin et al. (2025) noting significant variance in long-context RAG performance depending on the choice of LM and retriever. Yu et al. (2024) noted similar behavior for question-answeriig tasks. Giorgi et al. (2023) studied the effects of retriever and summarizer for short-context open-domain multi-document summarization. Therefore, we hypothesize that the optimal context length estimate for a RAG system should be function of the retriever, summarizer, and specific downstream task.
Our proposed method is centered on two key observations. First, large LMs show robust performance across a broad range of context lengths because of their enhanced ability to deal with noise in the retrieved input. Jin et al. (2025) observed this behavior for long input QA tasks. Second, gold references can be approximated by silver references sampled from strong long-context LMs. For a given dataset (D), retriever (R), and summarizer (S), our method involves the following steps. See Figure for an overview of our method.
1. We sample subset of the dataset (D). Each example in this subset constitutes a set of documents and a query. ($2.2)
2. We used a panel of LMs (52.1) to generate summaries for this subset. These sum- maries serve as our candidate silver references. (62.2)
3. We use Minimum Bayes Risk decoding identify the best silver references. ($2.2)
4. We perform search for retrieval lengths (8k to 80k) by comparing the system- generated summary (using & S) against the silver references. This search gives us the optimal retrieval length estimate for our RAG setup. (52.3)
5. Finally, on the full dataset, we retrieve the top-k documents that fit into this length estimate (using R) before generating summary (using S).
2.1 LM panel
In our LM panel, we include a diverse class of models. Panels of diverse LMs have previously been explored for evaluation and are considered strong alternative a single LM evaluator (Verga et al., 2024).
Large LMs: We choose Qwen-2.5 (Qwen et al., 2025), Llama-3.3 70B (Grattafiori al., 2024), and Jamba-1.5 Mini (Team et al., 2024). These are the largest models from each class that we could run locally.
Long-context LMs: We include two smaller LMs that are specifically trained for long-context tasks, Qwen-2.5-1M 14B (Yang et al., 2025) and ProLong 512K (Gao et al., 2024). ProLong is continually trained on long texts starting from the Llama-3 8B model.
In our pool, we focussed on including diverse models while being within our compute budget to run these models locally. Our panel can be easily modified with newer variants of these models as well as include API-based models.
We couldn't run Llama 405B and Jamba 1.5 Large (400B) locally on our setup.
3

---

Published as a conference paper at COLM 2025
2.2 Generating silver references
To begin, we select fraction of the examples from the dataset (default: 25%) using uniform sampling algorithm (without replacement). We run our silver reference generation on this data subset. We leave out the gold references and do not use them in our context-length estimation procedure.
Silver references: We run LM panel to create pool of candidate silver references. We used temperature sampling (T = 0.5) to generate three candidate summaries for each LM. We use LMs in full-context setup and do not assume any optimal context length.
We experiment with two ways to collect our final set of silver references. First, used single LM from the panel and select the three sampled candidates our silver references. Second, we collect many candidates by pooling outputs from all LMs in our panel. In this scenario, we use Minimum Bayes Risk (MBR) decoding identify the three highest scoring candidates. We follow previous work (Suzgun et al., 2023; Bertsch et al., 2023) to compute the similarity between each pair of candidates and obtain the alignment scores among the candidates. To be consistent with our downstream evaluation metric, we use the A3CU Fl score as our utility metric in MBR decoding.
Pooling:
Our use of MBR decoding here borrows ideas from previous summarization works, specifi- cally post-ensemble (Kobayashi, 2018) and crowd sampling (Suzgun et al., 2023). Similarly to Kobayashi (2018), use model ensemble in the post-processing stage. We follow Suzgun et al. (2023) to use temperature sampling and a neural utility metric. However, our utility metric differs from the BLEURT and BERIScore used in Suzgun et al. (2023).
2.3 Search for optimal retrieval length
To identify the optimal length for the retrieval step, we search wide spectrum of context lengths from 8K to 80K tokens in 8K intervals. For each context length C, we run the RAG pipeline on the silver dataset by retrieving up to C tokens (see $3.2) and passing them to the summarizer. We evaluate the system generated summaries against the silver references. We generate three predictions per example using temperature sampling (I 0.5) and take the average A3CU Fl score (see $3.1). For efficiency reasons, we choose the smallest context length that falls within standard deviation of the maximum score as our optimal context length. Previous works RULER and HELMET use coarser intervals for context lengths (multiples of 8K).
Yue et al. (2025) is closely related to our work. For the long input question answering task, they propose iterative RAG method that uses inference-time scaling. Unlike traditional RAG, their method iteratively generates subqueries and retrieves additional documents as needed before generating the final answer. They present computation allocation model that optimizes task performance based on three parameters: nunber of documents, number of demonstrations, and maximum number of iterations. Our setting differs considerably from this work. For nulti-document summarization task, we have a fixed set of documents, the prompt is often infeasible. We believe that our single- step retrieval solution can be combined with such iterative methods to further improve task performance. We leave this extension to future work.
and including demonstrations
Experimental Setup
In this section, we describe our dataset, the evaluation metric, baselines, and the systems used for the retrieval and summarization tasks.
3.1 Dataset & Metric
SummHay: Proposed by Laban et al. (2024), this is multi-documeett summarization curated using GPT-3.5 and GPT-4o, starting with summary insights followed by document generation. Each input typically consists of 100 documents (avg. length 884 words), and the
4

---

Published as a conference paper at COLM 2025
summary consists of an average of 185 words. This dataset includes 92 examples that cover the news and conversational domains.
Metric: For the summarization task, report the Fl score of the reference-based Atomic Content Unit (A3CU) metric (Liu et al., 2023b). This model-based metric is trained to predict score that measures the overlap of atomic content units (Liu et al., 2023a) between the gystee--generted and reference summaries. Previous work has found that this metric is strongly correlated with human evaluation for both single (Liu et al., 2023b) and multi- document summarization (Pratapa & Mitamura, 2025).
3.2 Retrieval systems
For our retrieval task, we use entire documents as retrieval units and obtain document embeddings using Qwen-2-based GTE models (Li et al., 2023). We then compute cosine similarity between document and query embeddings and pick the top-k documents that fit within the specified context length.
Jin et al. (2025) analyzed the effect of the retriever on optimal context lengths in RAG settings and found that the stronger retriever has shorter optimal lengths than the weaker retrievers. To see the impact of this in our setting, we experiment with two sizes of GTE embeddings, Qwen-2-1.5B and Qwen-2-7B.*
We acknowledge the impact of chunking strategies RAG performance (Chen et al., 2024), however, shorter chunks might need additional recontextualization. We leave the exploration of fine-grained chunking strategies to future work.
3.3 Summarization systems
For the summarization task, we use the instruction fine-tuned variants from Qwen-2.5, Llama-3, ProLong, and Phi-3 series of models.
Qwen-2.5: We experiment with multiple sizes from this series including 0.5B, 1.5B, 3B, 7B, 14B, 32B, and 72B (Qwen et al., 2025). The smaller models (S3B) only support a context length of 32K, while the larger models support up to 128K tokens. For the smaller models, we report RAG @ 32K as their full-context performance.
Qwen-2.5-1M: These are long-contex variants of the Qwen-2.5 7B and 14B models (Yang et al., 2025) supporting up to a context length of 1M tokens.
Llama-3: We include 1B, 3B, 8B and 70B models in our experiments. All models support context length of 128K tokens (Grattafiori et al., 2024).
ProLong:
Gao al. (2024) continually fine-tuned Llama-3-8B- -araa----B-Innsruu on long texts of up to 512K tokens. They are first trained on 20B training tokens of 64K data, followed by another 20B training tokens of 512K data. We experiment with the 64K and 512K variants.
Phi-3: We use three model sizes, Mini (3.8B), Small (7B) and Medium (14B). All of these models support context lengths of up to 128K tokens (Abdin et al., 2024).
We use vLLM (Kwon et al., 2023) for our inference runs, using up to four 48GB L40S GPUs in our experiments. For each set of input documents, we sample three summaries using temperature sampling (T 0.5). To provide a fair comparison of our systems, we limit all of our inputs to a maximum of 128K tokens. See Appendix $A.2 for additional details about our task prompts, tokenization, truncation strategies, and summary lengths.
3.4 Baselines
Full-context: In this setup, we utilize the full context window supported by the summa- rization model. Typically, larger models also tend perform well in long-context tasks. To
9nttps://huggingt http::/ginnaaaaa....oooaaaaaaaaa wtw.aa/w.ww....bbaaaaaaaattteeea
instruct http:://wgggggg..oooaaaaaa/a//// instructt

---

Published as a conference paper at COLM 2025
study this behavior, include models of varying sizes in experiments. Inputs longer than the supported context window are truncated starting with the longest documents.
For our RAG baselines, rely widely used long-context benchmarks RULER and HELMET that estimate efficient context windows for language models. For these baselines, limit the number of tokens retrieved to efficient context window of the corresponding summarization model.
RULER (Hsieh et al., 2024) benchmark consists of a collection of synthetic retrieval tasks at varying input lengths (8K, 16K, 32K, 64K and 128K). For a given LM, this benchmark evaluates its retrieval performance at these input lengths and determines an effective context window by using the performance of Llama-2-7B @ 4K as the threshold. We used the effective context windows reported in previous work our baseline estimates.
HELMET (Yen et al., 2025) benchmark covers a suite of NLP tasks, with multiple datasets included each task. The tasks are recall, RAG, citation, re-ranking, ICL, LongQA, and summarization. For each dataset, they evaluate system performance at varying input lengths (same set as RULER). They report task averages, as well as a HELMET average. As our baseline, we select the two most relevant subtasks, LongQA and summarization. For each task and L.M, we choose the context length with the highest task average as the effective context window for LM.
Note that both RULER and HELMET benchmarks evaluate model in a full-context setting but often find the optimal context window to be much lower than the claimed (or supported) context window by the LM. In our experiments, we used previously reported scores on the RULER and HELMET benchmarks. See Table 7 in Appendix $A.1 for full list of context length estimates from our baselines.
4 Results
In Table 1, we compare our method with the baselines the SummHay dataset. All RAG- based systems (baselines and ours) outperform full-context setup. Our method consistently shows strong performance across model classes, sizes, and retrievers. Although the RULER- or HELMET-based estimates do well in specific instances, neither is consistently better across all settings. Among our baselines, we find that the LonsQAd estimate from HELMET performs the best. In Table 8 in the Appendix, we report the context window estimates used in each experiment setting as well as the standard deviation across three random seeds.
Discussion & Analysis
We now analyze the effectiveness of our method in various settings. In $5.1, we look at very long context LMs (>500K). In $5.2, we evaluate the generalization of our estimation method to model class not included in our LM panel. In $5.3, we contrast our pooled estimate with those obtained using silver references from single large LM. We also evaluate the effect of the dataset sampling ratio on the quality of the estimated context length ($5.4). Finally, in $5.5, we discuss the performance and efficiency gains with our RAG setup.
5.1 Very long-context LMs
As LMs improve their long-context reasoning, there is often reduced need for RAG. Recent work (Yu et al., 2024) argues for the combination of long-context models and RAG, and our results in Table reinforce this argument. However, we want to test the effectiveness of our method on LMs carefully trained for long-contex: reasoning. For this analysis, we chose Qwen 2.5 1M models (Yang et al., 2025) (7B, 14B), and ProLong 512K (Gao et al., 2024). These models are continually trained on long texts and show almost perfect performance at 128K context length on HÉLMET. We report results in Table 2. Our method consistently outperforms the baselines. We leave the exploration of wlosed-weight API-based models such Gemini 1.5 Pro to future work.

---

Published as a conference paper at COLM 2025
Retriever
Summarizer
Full-context
RULER
HELMET
Ours
Summ
LongQA
Qwen-2.5 0.5B
16.7
20.6
Qwen-2.5 1.5B
26.3
26.3
28.7
27.4
Qwen-2.5
29.5
29.5
29.5
30.0
Qwen-2.57B
34.1
36.4
34.5
37.6
37.2
Qwen-2.5 14B 14B
35.7
35.6
37.4
Qwen-2.5 32B
33,9
35.1
36.6
GTE 1.5B
Qwen-2.5 72B
32.5
32.5
35.0
35.0
36.3
Llama-3.2 1B
17.7
24.6
24.6
25.8
Llama-3.2 3B
28.7
28.7
311
30.3
Llama-3.1 8B
33.3
34.9
34.9
34.0
34.5
Llama-3.3 70B
31.9
33.2
35.8
33.2
35.9
ProLong 64K
24.9
32.2
Qwen-2.5 0.5B
17.3
21.3
Qwen-2.5 1.5B
26.8
26.8
27.7
28.2
Qwen-2.5
30.2
30.2
30.2
32.7
Qwen-2.5
34.1
36.8
34.9
36.9
36.9
Qwen-2.5 14B
35.7
35.4
36.2
Qwen-2.5 32B
33,9
34.6
37.2
GTE 7B
Qwen-2.5 72B
32.5
32.5
35,9
35,9
35.3
Llama-3.2 1B
17.7
24.9
24.9
25,4
Llama-3.2 3B
28.7
28.7
29.7
31.4
Llama-3.1 8B
33.3
35.1
35.1
33.7
33.7
Llama-3.3 70B
31.9
34.4
35.8
34.4
33.3
ProLong 64K
25.9
32.3
Table 1: Comparison of our method against the baselines on the SummHay dataset. We report average A3CU Fl scores across three sampled summaries. For the baselines, we only report scores for models with context length estimates previously reported in prior work. See Table 8 in Appendix for context window estimate used in each experiment.
Retriever
Summarizer
Full-context
RULER
HELMET
Ours
Summ
LongQA
Owen-2.5-1NM 7B
32.1
33.3
32.1
32.1
33.6
GTE 1.5B
2 14B
35.6
35.6
35.6
35.6
37.4
ProLong 512K
31.0
31.0
31.0
32.3
Qwen-2.5-1M 7B
32.1
32.9
32.1
32.1
32.9
GTE 7B
Qwen-2.5-1M 14B
35.6
35.6
35.6
35.6
36.6
Prolong 512K
31.0
31.0
31.0
32.5
Table comparison of our method against the baselines for very long-context LMs. Except for RULER on Qwen-2.5-1M 7B, all baselines estimate full 128K context length.
Generalization to new models
In our LM panel, we included mixture of Qwen, Llama, and Jamba models ($2.1). test the generalization of our method to new model class, we report the performance for the Phi-3 series (Abdin et al., 2024). In Table 3, we compare our proposed method with the baseline using GTE 1.5B and 7B retrievers. We find that RULER estimates perform the best and our method is a close second. contrast to Table 1, the HELMET summarization estimate is better than its LongQA --bsedd estimate, but both underperform method.
7

---

Published as a conference paper at COLM 2025
Retriever
Summarizer
Full-context
RULER
HELMET
Ours
Summ
LongQA
Phi-3 Mini
11.0
30.6
30.4
30.4
30.6
GTE 1.5B
Phi-3 Small
27.8
31.1
30.3
31.9
Phi-3 Medium
29.4
30.7
29.9
29.4
30.7
Phi-3 Mini
11.0
29.9
28.3
28.3
29.9
GTB 7B
Phi-3 Small
27.8
32.4
30.6
31.5
Phi-3 Medium
29.4
30.7
30.5
29.4
30.3
Table 3: A comparison of our method against the baselines for the Phi-3 series.
Effectiveness system pooling
To test the effectiveness of pooling systems using MBR decoding ($2.2), we compared the pooled estimate of the system against two variants based on silver references from single LM. We experiment with Qwen-2.5 72B and Llama-3.3 70B, In Table 4, we compare the effectiveness of silver summaries. Notably, we find that the Qwen 72B-based estimate fares better than both the Llama 70B-based and pooled estimates. This could be because Qwen-2.5 provides slightly full-context performance compared to Llama-3.3 70B (see Table 1).
Summarizer
Silver Reference LM(s)
System Pooling
Qwen 72B
Llama 70B
Qwen-2.5 0.5B
21.3
21.3
21.3
Qwen-2.5 1.5B
28.2
27.7
28.2
Qwen-2.5 3B
32.7
32.7
32.7
Qwen-2.5 7B
36.9
36.9
36.9
Qwen-2.5-1M 7B
32.9
34.8
32.9
Qwen-2.5 14B
36.2
35.4
36.6
Qwen-2.5-1M 14B
36.6
36.6
36.6
Dwen-2.53322 32B
37.2
37.8
37.2
Qwen-2.5 72B
35.3
35.3
1lama-3.2 1B
25.4
26.3
25.4
Ilama-3.2 3B
31.8
31.8
31.8
Llama-3.1 8B
33.7
33.7
35.1
Llama-3.3 70B
33.3
34.7
Phi-3 Mini
29.9
29.9
29.9
Phi-3 Small
31.5
31.8
28.3
Phi-3 Medium
30.3
31.5
27.7
ProLong 64K
32.3
32.7
32.6
ProLong 512K
32.5
32.0
32.5
Table 4: A comparison of system pooling against Qwen and Llama-based silver references (GTE 7B retriever). We don't compute estimates for Qwen 2.5 72B based on Qwen 2.5 72B silver references (and similarly for Llama 3.3 70B).
Based on these results, we perform further analysis of our silver references in the pooling setup. In Table 5, we report the counts for how often each silver LM is chosen in the top-3 post-MBR decoding. The notable outliers here are Qwen-2.5 72B (picked least often) and Llama-3.3 70B (picked most often). This shows potential limitation of our pooling-based estimate. Although MBR decoding allows us to make better use of the target summary space, it is possible that low-quality summaries in the pool could adversely impact the overall performance, albeit only by a small margin. An interesting future work direction would be to explore Best-of-N sampling alternative to MBR decoding.
8

---

Published as a conference paper at COLM 2025
Silver Reference LM (full-context)
Count
Qwen-2.5 72B
33
Llama-3.3 70B
79
Jamba-1.5 Mini
54
Qwen-2.5-1M 14B
51
ProLong 512K
59
Total
276
Table 5: Counts of silver summaries from individual LMs post-MBR decoding. We pick
top-3 summaries per input, total of 276 summaries.
5.4 Effect of sample size
As we describe in $2.2, we sample subset of the dataset before generating silver references using our LM panel. To understand the effect of this sample size, we compare various sampling ratios in Table Our results show that even very small sample (10% examples) is sufficient for our estimation and shows superior performance to baselines.
Model
10%
25%
50%
75%
100%
Qwen-2.5 0.5B
21.3
21.3
21.3
21.3
21.3
Qwen-2.5 1.5B
27.7
28.2
27.7
27.7
27.7
Qwen-2.5 3B
32.7
32.7
32.7
32.7
32.7
Qwen-2.5 7B
36.9
36.9
36.9
36.9
36.8
Llama-3.2 1B
26.3
25.4
25.8
25.8
25.8
Llama-3.2 3B
31.8
31.4
31.4
31.4
31.4
Llama-3.1 34.6
33.7
34.6
35.1
35.1
Table 6: A comparison of our method at various dataset sampling ratios (GTE 7B retriever).
5.5 Performance Efficiency
Small LMs tend to have very limited effective context windows; therefore, optimal RAG necessary for improved task performance. For large LMs with longer effective context windows, optimal RAG can provide efficiency gains while maintaining improving task performance.
Our results from Table 1, Table 2, and Table 3 show the effectiveness of our method in models ranging from 0.5B to 72B parameters. For given downstream task, the can pick model size that is most suited to their computing budget. For example, Qwen-2.5 < 7B can run on single 48GB GPU, while larger models would require up to 4X48GB GPUs.
Performance:
Compared to the baselines, our method often provides a significantly shorter context length estimate (see Table in the Appendix SA.1). Therefore, the final summa- rization run on the full dataset is much more efficient with our method. However, we acknowledge that our method requires task-specific additional inference time compute to determine the optimal context length. Similar compute is also needed for benchmarks such as RULER and HELMET that compute task averages. In Table 6, we showed that our estimation requires very small sample of the dataset, so the marginal cost of our method would be lower as the size of the dataset increases.
Efficiency:
Conclusion & Future Work
In this work, we presented a methodology for estimating optimal context length for RAG- based summarization systems. Unlike traditional long-context benchmarks, method

---

Published as a conference paper at COLM 2025
is geared to specific downstream dataset and models the estimate function of the entire experimental configuration. We show the superior performance our method across model classes and sizes. We show a generalization of our method to new model classes, as well as its effectiveness on models with very long context windows (>500K). In future work, plan to apply our method to other tasks, such open-domain multi-document QA and long-document summarization (Zhou et al., 2023). Previous work has also shown that the relative performance of long context and retrieval varies between examples (Karpinska et al., 2024; Pratapa & Mitamura, 2025), so another future direction is to identify the optimal retrieval context length for each example. Using open-weight models allowed us analyze our method across various model sizes within reasonable compute budget. We expect future work expand our LM panel to include larger API-based models such as Gemini or GPT.
Another line of work studies input compression methods (Jiang et al., 2024; Xu et al., 2024) that fit long inputs to fixed context length. Although these are promising alternative to full-context setup, they may suffer irreversible information loss (Pratapa & Mitamura, 2025). In this paper, we focus the strengths of RAG while taking advantage of the long-context reasoning capabilities of recent LMs. We leave the exploration of input compression with long-context methods to future work.
Limitations
We discuss the limitations of our estimation method and the potential ways for future work to improve them. We use a silver panel consisting of medium-sized open-weight models ($2.1). This silver panel might not work as effectively to estimate the optimal context length for a much larger model such as Gemini-2.5 Pro. In such situations, we believe the silver panel should consist of models with similar capacity. Additionally, we use a full context setup to get silver summaries (62.2) and this might not work as well if the inputs are much longer than the context windows supported by the LMs our silver panel. An option is to perform RAG by retrieving tokens up to the LM's supported context window (similar to our approach, smaller Qwen models in $3.3). Our analysis in 55.3 also highlighted the limitations of system pooling, and future work could explore the use of Best-of-N sampling to improve the pooling mechanism. Finally, our method relies on the availability at least few examples from the downstream dataset and might not work well if this sample is not representative of the downstream task.
Acknowledgments
We thank Amanda Bertsch and Kimihiro Hasegawa for helpful discussions and feedback on this work. We also thank anonymous reviewers for their constructive feedback during the review process. Adithya Pratapa was supported by an LTI Ph.D. fellowship.
Ethics Statement
In this work, we limit our focus to the content selection evaluation of our summarization sys- tems. However, we acknowledge that the factual accuracy of summaries is of great practical importance and point the reader to related work on hallucination in text summarization. We believe that our RAG-based estimation procedure does not increase the chances of possible hallucination in text summarization systems.
References
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Mísha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao,
10

---

Published as a conference paper at COLM 2025
Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mah- moudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guan- hua Wang, Lijuan Wang, Shuohang Wang, Xin Wang, Yu Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Wei- jian Xu, Jilong Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang, Ziyi Yang, Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL httpp::/aarxiv.org//baao
14219.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth Interna- tional Conference on Learning Representations, 2024. URL http:: /ttp:::////eeevieeee/ net/forum? id=hSyW5go0v8.
Amanda Bertsch, Alex Xie, Graham Neubig, and Matthew Gormley. It's MBR all the way down: Modern generation techniques through the lens minimum Bayes risk. In Proceedings of the Big Picture Workshop. Association for Computational Linguistics, 2023.
URL http///cccannooooyoyoooooo orgg wogg/022/2 2023. bigpicture-1.9%.
Tong Chen, Hongwei Wang, Sihao Chen, Wenhao Yu, Kaixin Ma, Xinran Zhao, Hongming Zhang, and Dong Yu. Dense X retrieval: What retrieval granularity should we use? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. Association for Computational Linguistics, 2024. URL https://aclantholl emnlp-main, 8.45/.
Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. How to train long-context language models (effectively), 2024. URL https:://arae htrgg/bbb/111 021600260..000
John Giorgi, Luca Soldaini, Bo Wang, Gary Bader, Kyle Lo, Lucy Wang, and Arman Cohan. Open domain multi-document summarization: A comprehensive study of model brit- tleness under retrieval. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 8177-8199, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2000 2023. findings-emmlp.544 549/.
findings-emmll.554 URL https: / http::://aclanthollooo
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Ro- driguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bob- bie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu,
11

---

Published as a conference paper at COLM 2025
Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasuuuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Gird- har, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Ra- parthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Syd- ney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujiwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Chang- han Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpem, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanae Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, lean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan JePhie,JJnathaa Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Gro- shev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark,
Hakan Inan,
12

---

Published as a conference paper at COLM 2025
Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bon- trager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satter- field, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Vitoto, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024 URL https: //axxiv. http://axxxvvoooarb//////// org/ 21783.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What's the real context size of your long-context language models? In First Conference on Language Modeling, 2024. URL https:/ /otp::///peeeeee.... net/ forum?id=k10Bbc7655)
Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LongLLMLingua: Accelerating and enhancing LLMs in long context scenarios A
via prompt compression. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational o0g/2024.20.. long. 91/.
Linguistics, 2024. URL
Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan O Arik. Long-context LLMs meet RAG: Overcoming challenges for long inputs in RAG. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openrevient wwt.:ee///oooooe peet/forum?inn dorrumiddoUttpaRrr
Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit lyyer. One thousand and one pairs: A "novel" challenge for long-conter language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2024. URL htwpt: httt::::///caaahollllgooooo wtt.://......mmmmmmma..... 948/.
Hayato Kobayashi. Frustratingly easy model ensemble for abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018. URL https:://aalaatthllltii/ wwwpenthoooooooooDiit. wrgg/11--1111
Shankar Kumar and William Byrne. Minimum Bayes-risk decoding for statistical ma- chine translation. In Proceedings of the Human Language Techmology Conference of the North American Chapter of the Association for Compulational Linguistics: HLT-NAACI 2004, pp. 169-176, Boston, Massachusetts, USA, May 2 May 7 2004. Association for Computational Linguistics. URL https:h/accaanneeee http://aaaaaaattyy...oo org/Ne4-16222
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles. Association for Computing Machinery, 2023. URL https: //doi. www.:orggg.... 0145///801145/0800000000 3613165.
13

---

Published as a conference paper at COLM 2025
Philippe Laban, Alexander Fabbri, Caiming Xiong, and Chien-Sheng Wu. Summary of haystack: A challenge to long-context LLMs and RAG systems. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computa- tional Linguistics, 2024. URL https: hwtp://hlllaatoooooooyoooo org/2024, 1ttp//222.eme2m1mmmeeemmm emnlp-maing 552/.
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning, 2023. URL https://arxiv wttpaaaiiv...../go...
org. h////b///300 03281.
Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. Re- trieval augmented generation or long-context LLMs? comprehensive study and hybrid approach. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Pro- cessing: Industry Track, 2024. URL https: h/aps///hl/aaatooooo.y..o. org/2024/ emnlp- innustddutt.... 66/.
Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2023a. URL https: //aclanthology www.ology.rrgg/oooooooo org 2ttgg acl acl-long. 228/
Yixin Liu, Alexander Fabbri, Yilun Zhao, Pengfei Liu, S Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. Towards interpretable and efficient automatic reference- based summarization evaluation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2023b. URL https://aclanthollog. ptgg/222.3. emnlp cemnlp-minn... 1018/.
Adithya Pratapa and Teruko Mitamura. Scaling multi-document event summarization: Eval- uating compression vs. full-text approaches. In Proceedings of the 2025 Conference of the Na- fions of the Americas Chapter of the Association for Computational Linguisfics: Human Language Technologies (Volume 2: Short Papers), Albuquerque, New Mexico, April 2025, Association for Computational Linguistics. URL https: http//aalaantttt /achs///ollaa.ooogo...ooooooo... wwg.o///5522 maacl-shord short.44/ 44/.
Owen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan 240...//222.11111 15115.
Qiu. Qwen2.5 technical report, 2025. URL
Mirac Suzgun, Luke Melas-Kyriazi, and Dan Jurafsky. Follow the wisdom of the crowd: Effective text generation via minimum Bayes risk decoding. In Findings of the Association for Computational Linguistics: ACL 2023, Association for Computational Linguistics, 2023, org/ 2ww../2222.2222 findings-ann 222ss///c222.........
URL hattp::/aaclattiee
Jamba Team, Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, Daniel Gissin, Daniel Jannai, Dor Muhlgay, Dor Zimberg, Fdden M Gerber, Elad Dolev, Eran Krakovsky, Erez Safahi, Frez. Schwartz, Gal Cohen, Gal Shachaf, Haim Rozenblum, Hofit Bata, Ido Blass, Inbal Magar, Itay Dalmedigos, Jhonathan Osin, Julie Fadlon, Maria Rozman, Matan Danos, Michael Gokhman, Mor Zusman, Naama Gidron, Nir Ratner, Noam Gat, Noam Rozen, Oded Fried, Ohad Leshno, Omer Antverg, Omri Abend, Opher Lieber, Or Dagan, Orit Cohavi, Raz Alon, Ro'i Belson, Roi Cohen, Rom Gilad, Roman Glozman, Shahar Lev, Shaked Meirom, Tal Delbari, Tal Ness, Tomer Asida, Tom Ben Gal, Tom Braude, Uriya Pumerantz, Yehoshua Cohen, Yonatan Belinkov, Yuval Globerson, Yuval Peleg Levy, and Yoav Shoham. Jamba-1.5: Jambaa- Hybrid transformer-mambe models at scale, 2024. URL 12570.
https://arxiv.. hwtp://ogg////////////ooosss/s//
Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with
14

---

Published as a conference paper at COLM 2025
juries: Evaluating Ilm generations with panel of diverse models, 2024. URL https:
//arxiv.org///a/eaa///.a abs//bb/// 18796.
Fangyuan Xu, Weijia Shi, and Eunsol Choi. RECOMP: Improving retrieval-augmented LMs with context compression and selective augmentation. In The Twelfth International Conference on Learning Representations, 2024. URL https: httpss///ppeeeveeeeeee wwt. ffrdri m1JLVigNHp.
An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, Junyang Lin, Kai Dang, Kexin Yang, Le Yu, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He, Weijia Xu, Wenbiao Yin, Wenyuan Yu, Xiafei Qiu, Xingzhang Ren, Xinlong Yang, Yong Li, Zhiying Xu, and Zipeng Zhang. 1we technical report, 2025. URL wwt.hh:pt//tsttee
arggabs/2501. 15383.
Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. HELMET: How to evaluate long-context models effectively and thoroughly. The Thirteenth International Conference on Learning Representations, 2025.
URL httpss://ppereviiii hwt.:wee/....... www.teetooouoeee nett/forum1dd-22333bb0000
Tan Yu, Anbang Xu, and Rama Akkiraju. In defense of rag in the era of long-context wttp:/iv.org/bb////999. 01666.
language models, 2024. URL
Zhenrui Yue, Honglei Zhuang, Aijun Bai, Kai Hui, Rolf Jagerman, Hansi Zeng, Zhen Qin, Dong Wang, Xuanhui Wang, and Michael Bendersky. Inference scaling for long-conteds retrieval augmented generation. In The Thirteenth International Conference on Learning Representations, 2025. URL https::/openreeview http///pnrrrreeeeeeeeeorrrrrrree www.neet/oormmmidssssrrooooo
Yijie Zhou, Kejian Shi, Wencai Zhang, Yixin Liu, Yilun Zhao, and Arman Cohan. Odsum: New benchmarks for open domain multi-document summarization, 2023. URL https:
//arxiv. org/ http://bs/2399.99990 08960.
A
Appendix
A.1 Context length estimates
In Table 7, we list our models and the context length estimates from our baselines. In Table 8, we present our full set of results, including the standard deviation across the summaries of the three sampled systems summaries and the context length for each setting.
A.2 Experiment details
A.2.1 Dataset
We truncate the input documents to 128K tokens. We start by truncating the longest documents first. Due to slight differences in the tokenization methods between model classes, we calibrate the maximum number of summary tokens across models. We first get the 80th percentile summary length (in NLTK tokens) and use the model-specific word-to-token ratio to set the max summary tokens.
We use the following prompt for the summarization task,
[document)
Question: (question)
Answer the question based the provided document. Be concise and directly address only the specific question asked. Limit your response to maximum of (num_words) words.
15

---

Published as a conference paper at COLM 2025
Summarizer
Size
Supported
Estimated
RULER
HELMET
Summ
LongQA
Qwen-2.5 0.5B
0.5B
32,768
Qwen-2.5 1.5B
1.5B
32,768
32,768
16,384
Qwen-2.5 3B
3B
32,768
32,768
32,768
Qwen-2.5 7B
7B
131,072
32,768
65,536
16,384
Owen-2.5-1M 7B
7B
1,010,000
65,536
131,072
131,072
Qwen-2.5 14B
14B
131,072
65,536
Qwen-2.5-1M 14B
14B
1,010,000
131,072
131,072
131,072
Qwen-2.5 32B
32B
131,072
65,536
Qwen-2.5 72B
72B
131,072
131,072
32,768
32,768
Llama-3.2 1B
1B
131,072
32,768
32,768
Llama-3.2 3B
3B
131,072
131,072
65,536
Llama-3.1 8B
8B
131,072
32,768
32,768
65,536
Llama-3.3 70B
70B
131,072
65,536
32,768
65,536
ProLong 64K
8B 65,536
ProLong 512K
8B
524,288
131,072
131,072
Phi-3 Mini
3B
131,072
32,768
65,536
65,536
Phi-3 Small
7B
131,072
32,768
65,536
Phi-3 Medium
14B
131,072
32,768
65,536
131,072
Jamba-1.5 Mini
52B-A13B
262,144
131,072
131,072
Table A summary of LMs used our work. We report the model size and context lengths (supported and estimated). For RULER and HELMET, we use the results reported in prior works to identify the context length estimates. Since our proposed context length estimate is also dependent on the retriever and dataset, we do not include those numbers here (see Table 8).
A.2.2 Generation
For summary generation, we used temperature sampling (0.5) and generated three sum- maries for each input. All the results report are the average scores across three runs. For the retrieval task, we limit the length of each document to 1024 tokens.
A.2.3 Compute
We use a single L40S GPU for our retrieval runs. For our summarization task, we use up to four I.40S GPUs.
16

---

Published as a conference paper at COLM 2025
Summarizer
Full-context
RULER
HELMET
Ours
Summ
LongQA
Retriever: GTE 1.5B
Qwen-2.5 0.5B
16.7 (32K)
20.6 (8K)
Qwen-2.5 1.5B
26.3+0. (32K)
26.3 (32K)
28.7=12 (16K)
27.4 (8K)
Qwen-2.5 3B
29.5102 (32K)
29.5 (32K)
29.5 (32K)
30 (8K)
Qwen-2.5 7B
34.1+1.1 (128K)
36.441. (32K)
34.5 (64K)
37.6 (16K)
37.210.9 (24K)
Qwen-2.5-1M 7B
32.1+03 (128K)
33.3406 (64K)
32.1 (128K)
32,1 (128K)
33.640. (56K)
Qwen-2.5 14B
35.7=06 (128K)
35.6+07 (64K)
37.4+0. (24K)
Lwen-2.5-LM 14B
35.6=1.2 (128K)
35.6-1.2 (128K)
35.6 (128K)
35.641.2 (128K)
37.4 (24K)
Qwen-2.5 32B
33.9 (128K)
35.1 (64K)
36.610.6 (16K)
Qwen-2.5 72B
32.5+0.5 (128K)
32.5+ (128K)
35408 (32K)
35 (32K)
36.3 (24K)
Llama-3.2 1B
17.7 (128K)
24.61006 (32K)
24.6100
(32K)
25.8 11.9 (8K)
Llama-3.2 3B
28.7+1.4 (128K)
28.7 (128K)
31.1 10. (64K)
30.3 (56K)
Llama-3.1 8B
33.3 (128K)
34.9 (32K)
34.9+08 (32K)
34 (64K)
34.5 (40K)
Llama-3.3 70B
31.9 (128K)
33.2 (64K)
35.8 (32K)
33.2 (64K)
35.9 (40K)
ProLong 64K
24.940.6 (64K)
32.2+0. (16K)
ProLong 512K
31+08 (128K)
31 (128K)
31 (128K)
32.3 (48K)
Phi-3 Mini
11 (128K)
30.6 (32K)
30.4 (64K)
30.4-0.1 (64K)
30.6 (16K)
Phi-3 Small
27.8 (128K)
31.1 (32K)
30.3 (64K)
31.9
(48K)
Phi-3 Medium
29.4 1.3 (128K)
30.7+1.2 (32K)
29.9 (64K)
29.4 (128K)
30.7 (32K)
Retriever: GTE
Owen-2.5 0.5B
17.3+0.4 (32K)
21.3 (8K)
0wen-2.5 1.5B
26.8403 (32K)
26.8 (32K)
27.7 (16K)
28.210.6 (24K)
0wen-2.5 3B
30.2102 (32K)
30.2 (32K)
30.2 (32K)
32.7+1.1 (16K)
Qwen-2.5 7B
34.1+1.1 (128K)
36.8405 (32K)
34.9404 (64K)
36.9 (16K)
36.9+12 (16K)
Qwen-2.5-IM
32.1+03 (128K)
32.9+02 (64K)
32.1+0.3 (128K)
32.1 (128K)
32.9 (64K)
2.en-2.5 14B
35.7=06 (128K)
35.4+09 (64K)
36.2 10. (16K)
Qwen-2.5-LM 14B
35.6 (128K)
35.6 (128K)
35.6 1.2 (128K)
35.6 (128K)
36.6 (48K)
Qwen-2.5 32B
33.9 (128K)
34.64n2 (64K)
37.2+07 (32K)
Qwen-2.5 72B
32.5+0. (128K)
32.5+ (128K)
35.9 +0.4 (32K)
35.9 (32K)
35.3+ (24K)
Llama-3.2 1B
17.7 102 (128K)
24.9 (32K)
24.9
(32K)
25.4 10.7 (16K)
1lama-3.2 3B
28.7+1.4 (128K)
28.7 (128K)
29.7 (64K)
31.4 (32K)
Llama-3.1 8B
33.3 (128K)
35.1 (32K)
35.1 (32K)
33.7 (64K)
33.7+0. (56K)
Llama-3.3 70B
31.9 (128K)
34.4105 (64K)
35.8 (32K)
34.4 (64K)
33.3 (80K)
ProLong 64K
25.9 (64K)
32.3 (32K)
Prolong 512K
11s (128K)
10.8 (128K)
10.8 (128K)
32.5 (32K)
Phi-3 Mini
11 (128K)
29.9 (32K)
28.3 (64K)
28.3 (64K)
29.9
(32K)
Phi-3 Small
27. 1.3 (128K)
32.4 (32K)
30.6 (64K)
31.5106 (24K)
Phi-3 Medium
29.4 (128K)
30.7+ (32K)
30.5 (64K)
29.4 (128K)
30.3+ (80K)
Table 8: Full set of results the SummHay dataset. For each system, we report the average score and standard deviation across three runs. We also provide the (optimal) context length estimate used for each experiment configuration parantheses.
17


Reviews:
[{"review_id": "60I57dOABS", "reviewer": "Reviewer_jEKZ", "summary": "This paper describes a multi-document summarization proposal that searches for a good context length, uses it to decide how many relevant documents to retrieve, before generating a summary based on the retrieved docs.\n\nThe first step involves sampling a subset of the set of documents to be summarized. These are passed through a panel of different large-language models (LLMs) (e.g. Qwen, Llama, etc) to obtain a set of 'silver' reference summaries. Minimum Bayes Risk (MBR) decoding is then used to identify the top silver summaries. After this, a search is conducted over a fairly wide search space of varying context lengths by comparing outputs from a summarizer with the top set of silver summaries to identify what context length works best. With this estimated context length, top k documents which can fit into this length are retrieved, and used as input to the final summarizer step.\n\nEvaluations are done over the SummHay dataset, against baselines including 1/ one using the full context window of a summarizer, as well as previously reported runs for 2/ RULER, and 3/ HELMET on the dataset. Results generally show better A3CU F1 scores when smaller LLM summarizers are used. The effect is stronger when paired with a relatively larger embedding model. Further experiments conducted over LLMs. with very long context windows suggests that this proposal works well too.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Good set of experiments conducted, they build a good case for the efficacy of the approach - starting from baseline comparisons, to exploring different setups and baselines (e.g. against very long context LLMs, against new LLMs, etc)\n2. For a given RAG-based multi-doc summarization setup, the proposal has the potential to produce cost savings through the use of more optimal choice of context lengths. This can be significant over time.", "reasons_to_reject": "1. The paper lacks a clear motivation or value prop. My best guess right now is that the proposal could lead to cost savings through more optimal choices of context lengths. Yet the effect is more noticeable when a relatively larger embedding model is used for retrieval. It's not clear to me how the relative costs increase/decreases interact and whether this could be a nett gain. \n2. Re the generation of silver summaries used to estimate the right context width, a natural question that brings about is how these might compare against the full RAG-based pipeline that are used in the experiment.\n3. It took me several passes to get a good sense of what the proposal is (see suggestion #1). The paper could benefit from more editorial love.", "questions_to_authors": "Suggestion:\n\n1. It takes a few reads to properly piece together what the authors are proposing. There are attempts to explain what the proposed system is comprised of at various parts of the doc. With each attempt, there is progressively more context and details that are given. However, this means that each time, the reader has to pause, do a mental diff of the relatively more vague explanation they read earlier, and reconcile it with the more detailed explanation they have in front of them. After a while, it gets tedious. It would help to focus the explanation in a methodology section and give the reader a thorough read without having to build up a series of diffs over the course of several pages.", "extra_scores": {}}, {"review_id": "X8HazwBTIc", "reviewer": "Reviewer_iKj6", "summary": "The paper proposes an approach to estimate optimal input context lengths to be retrieved by a RAG for multi-document summarisation. First, a subset of the training dataset is sampled. A panel of LLMs (Llama, Qwen and Jamba) is used to generate a pool of silver-references for the sampled dataset. Then, the system is tested over the sample data (RAG+summarizer), with a varying window of context lengths from 8 to 80K in 8K intervals, generating three predictions per input, and evaluated against the silver reference pool. The smallest context length that falls within a standard deviation of the maximum score is selected as the optimal length. Finally, once the optimal lengths is selected for a particular dataset, RAG model and summarizer, the authors test the system on the test dataset. Overall, their test  results generally outperform the comparing baselines with full-context models and context-lengths optimised on the RULER and HELMET benchmarks, with generally shorter estimated context lengths, which make their approach more efficient at inference time. Additionally, the paper presents a sensitivity analysis on various parameters in their approach such as the percentage of samples used to estimate the context lengths, the use of LLMs with large context windows, and the use of LLMs from a different family compared to the ones used to create the pool of silver references.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well written and the main contribution clearly presented.\n- Interesting experimental results, demonstrating competitive performance compared to other strong baselines (outperforming the baselines in most settings) with generally shorter context lengths, which make their approach more efficient at inference time.\n- The approach seems to generalise well across various LLMs and RAG models.", "reasons_to_reject": "- The experimental analysis has been carried out over a single summarisation dataset. The authors claim that the optimal context-length depends on the dataset, RAG and summarizer. However, they don't explore if their context-length estimation works in other datasets. It would be interesting to understand if the approach works well with different tasks too, not only summarisation (although authors have already mentioned that they are planning to explore this in future work).\n- The RULER and HELMET baselines seem to be at a disadvantage, because their optimal  context-length is obtained over different datasets, not the sample obtained from the SummHay dataset. It would be interesting to compare the authors' approach with a simple baseline that uses the same sample dataset (potentially even with the respective ground-truth references) and the coarser varying input lengths proposed in the baselines.", "questions_to_authors": "- I suggest to include the estimated context lengths in table 1 for each baseline and the proposed approach. It is an important information to understand the contribution, it should be included in the main paper, not just the Appendix.", "extra_scores": {}}, {"review_id": "kFSPJdlgJB", "reviewer": "Reviewer_h4ue", "summary": "The paper proposes and validates a method to identify the context length as a function of a dataset, retriever, and summarizer combination in retrieval augmented generation. First a subset of pseudo examples is created from each dataset, then candidate silver summaries generated, then a single candidate summary selected, and finally a RAQ length chosen empirically by attempting to generate those silver summaries with different allowed context lengths.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(A1) The paper provides a novel,  empirical, reference-free method for choosing context/budget size in RAG systems. \n\n(A2) The method appears to perform well above a full-context baseline, and seems to generally perform similarly or better vs. ablations and other methods (Tables 1, 2), and adequate generalization performance to believe the result may be durable (Table 3)", "reasons_to_reject": "(R1) The fundamental premise is not theoretically sound -- both the question and amount of literature within the dataset addressing the question are ignored. For some topics where a thorough review is required, the methodology suggested here will continue to work, but for others (e.g. in medical literature reviewing), the recall burden is substantially higher. \n\n(R2) The actual description of the method is not particularly clear and should be revisited to be unambiguous: exactly what is sampled from the dataset (an \"example\" could be a set of documents with arbitrary relation, it could be a Q/A pair, or something else), how is it sampled (\"random\" is not a very clear description),", "questions_to_authors": "Line 53: I suggest the \"representative examples\" be clarified - examples of what? This underspecification is rather confusing.\n\nTable 1: it is not immediately obvious on first read that the correct comparison is actually the last column against the first column, and the others are ablations.\n\nSuggested related work: \n- \"Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrieval\n\" https://arxiv.org/abs/2212.10526 \n\nConsider this dataset: \nODSum: New Benchmarks for Open Domain Multi-Document Summarization https://arxiv.org/abs/2309.08960", "extra_scores": {}}, {"review_id": "31frnoZFF8", "reviewer": "Reviewer_3Jr7", "summary": "The paper addresses the task of Long Context Multi-Document summarization within a retrieval-augmented setting, where the retriever first selects a subset of the source documents which the LLM then uses to generate a query-focused summary. One of the main challenges in this setting is determining how much source content to retrieve to balance relevant content vs potential noise, given the available context window length of the summarizer LLM, and the paper introduces a novel approach for estimating an optimal retrieval size (in tokens), as a function of the retriever, dataset and summarizer. The idea is to use full-context LLMs to generate a pool of silver summaries for a sample of the dataset, and then to evaluate different RAG setups (retriever & summarizer) to determine the optimal context size. The experimental evaluation on the SummHay dataset covers an impressive range of Large and Long-Context LMs in various model sizes (0.5B - 72B), and the proposed approach shows strong performance across model classes, sizes, and retrievers.\n\n**Quality/Significance:** This paper has a very high technical quality, with sound experimental setup and reporting (multiple sample generations, averaged results and variance, detailed analyses of the method in various settings). It presents a well-executed approach to a timely, relevant problem in the area of long context generation.\n\n**Clarity:** The paper  well-structured and clearly written. The appendix provides the prompts used and more detailed experimental results (standard deviation, estimated context size)\n\n**Originality:** I found the idea interesting and novel, but I might have missed relevant literature.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Addresses an interesting and relevant problem in long context generative tasks\n- Well-executed and clearly documented approach\n- Impressive array of experiments / models analyzed, making for a convincing evaluation", "reasons_to_reject": "- The experimental evaluation uses only a single dataset (I assume because no others are available), and focuses on the single task of summarization, which might limit the generalizability of the approach.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper estimates the optimal input context length to be retrieved by a RAG system for multi-document summarisation. First, a subset of the training dataset is sampled. A panel of LLMs (Llama, Qwen and Jamba) is used to generate a pool of silver-standard references for the sampled dataset. Then, the system is tested over the sample data (RAG+summarizer), with a varying window of context lengths from 8 to 80K in 8K intervals, generating three predictions per input, and evaluated against the silver reference pool. The smallest context length that falls within a standard deviation of the maximum score is selected as the optimal length. Overall, their test results generally outperform comparison systems with full-context models and context-lengths optimised on the RULER and HELMET benchmarks, with generally shorter estimated context lengths, which make their approach more efficient at inference time. Additionally, the paper presents a sensitivity analysis on various parameters such as the percentage of samples used to estimate the context length, the use of LLMs with large context windows, and the use of LLMs from a different family compared to the ones used to create the pool of silver references. 

Reviewers were broadly positive, and the authors clarified questions and misunderstandings in the rebuttal period. It would be nice to see whether using the gold summary instead of the silver-standard ones leads to different length estimates and different/better summarization results.
Decision: Accept