Holistic Reasoning with Long-Context LMs: A Benchmark for Database Operations on Massive Textual Data
Authors: Seiji Maekawa
Hayate Iso
Nikita Bhutani
Venue: ICLR 2025 Poster (2025)
Keywords: long-context
reasoning
LLM
Primary area: datasets and benchmarks
TL;DR: We introduce HoloBench, a benchmark designed to evaluate and analyze the holistic reasoning capabilities of long-context language models on complex database operations over massive textual data.

Abstract: The rapid increase in textual information means we need more efficient methods to sift through, organize, and understand it all. While retrieval-augmented generation (RAG) models excel in accessing information from large document collections, they struggle with complex tasks that require aggregation and reasoning over information spanning across multiple documents--what we call  \textit{holistic reasoning}. Long-context language models (LCLMs) have great potential for managing large-scale documents, but their holistic reasoning capabilities remain unclear.  In this work, we introduce HoloBench, a novel framework that brings database reasoning operations into text-based contexts, making it easier to systematically evaluate how LCLMs handle holistic reasoning across large documents. Our approach adjusts key factors such as context length, information density, distribution of information, and query complexity to evaluate LCLMs comprehensively. 

Our experiments show that the amount of information in the context has a bigger influence on LCLM performance than the actual context length. Furthermore, the complexity of queries affects performance more than the amount of information, particularly for different types of queries. Interestingly, queries that involve finding maximum or minimum values are easier for LCLMs and are less affected by context length, even though they pose challenges for RAG systems. However, tasks requiring the aggregation of multiple pieces of information show a noticeable drop in accuracy as context length increases.  Additionally, we find that while grouping relevant information generally improves performance, the optimal positioning varies across models. Our findings surface both the advancements and the ongoing challenges in achieving a holistic understanding of long contexts. These can guide future developments in LCLMs and set the stage for creating more robust language models for real-world applications.

Published as a conference paper at ICLR 2025
HOLISTIC REASONING WITH LONG-CONTEXT LMS: A BENCHMARK FOR DATABASE OPERATIONS ON MASSIVE TEXTUAL DATA
Seiji Mackawa' Hayate Iso*
Nikita Bhutani
Megagon Labs
(seiji, hayate, nikita Jaaegaonem ai
ABSTRACT
The rapid increase in textual information means we need more efficient meth- ods to sift through, organize, and understand it all. While retrieval-augmentte generation (RAG) models excel in accessing information from large document collections, they struggle with complex tasks that require aggregation and rea- soning over information spanning across multiple documents-what we call holis- tic reasoning. Long-context language models (LCLMs) have great potential for managing large-scale documents, but their holistic reasoning capabilities remain unclear. this work, we introduce HoloBench, novel framework a novel frame- work that brings database reasoning operations into text-based contexts, making it easier to systematically evaluate how LCLMs handle holistic reasoning across large documents. Our approach adjusts key factors such as context length, in- formation density, distribution of information, and query complexity to evaluate LCLMs comprehensively.
Our experiments show that the amount of information in the context has big- ger influence on LCLM performance than the actual context length. Furthermore, the complexity of queries affects performance more than the amount of informa- tion, particularly for different types of queries. Interestingly, queries that involve finding maximum or minimum values are easier for LCL.Ms and are less affected by context length, even though they pose challenges for RAG systems. However, tasks requiring the aggregation of multiple pieces of information show a notice- able drop in accuracy context length increases. Additionally, we find that while grouping relevant information generally improves performance, the optimal posi- tioning varies across models. Our findings surface both the advancements and the ongoing challenges in achieving a holistic understanding of long contexts. These can guide future developments in LCLMs and set the stage for creating more ro- bust language models for real-world applications.
Code
https///gttuuub.cmommmmmmaaapaas
Benchmark
INTRODUCTION
The explosive growth of textual data has highlighted the need for efficient ways to process, organize, and comprehend large document collections. While retrieval- -augeeete generation (RAG) models have made accessing information from these vast sources easier (Lewis et al. 2020 Maekawa et al.) 2024). these models are limited by their reliance on localized context retrieval. This makes them less effective for complex tasks that require holistic reasoning the ability to synthesize, aggregate, and reason over information that spans across multiple documents. For instance, when asked, "Which company employed the most people?", traditional retrieval models might provide individual statis- ties for each company but fail to offer comprehensive comparison across all relevant entities. This reveals a fundamental limitation: RAG systems excel at retrieving specific information but struggle to aggregate and reason over broader cross-document contexts. This gap highlights the need for
"Equal contribution.

---

Published as a conference paper at ICLR 2025
Variable context
Variable
Variable info. position
Reasoning over info. pieces
Verifiable
Benchmark
info, amount
answers
Needle-in in a haystack (Kamradi 2023
X
Muli Needles-in-a haaa-haaatacc Kaanaa 2023
X
En. QA/Ee//n..aa MC (Zhang
anno.
BABILONG (Kiratov et 2024
X
NOCHA KKrppnska al. 2024
x
x human anno.
FLENQA (LaNQATLeyyoo (Levy al 2024
HOLOBENCH (Ours)
Table 1: answers of the benchmarks are created by human annotators.
Characteristics of Long-Context LM Benchmarks.
"human--nno." indicates that gold
more advanced models capable of holistic reasoning, particularly in tasks requiring multi-document synthesis, comparative analysis, and contextual integration across extensive datasets.
Long-Context Language Models (LCLMs) (OpenAl 2024a; Anthropic 2024 Reid et al. 2024
Dubey et al. 2024p. which can process hundreds of thousands tokens, present a promising solution to this challenge. With their ability to handle long text inputs, they can retrieve, aggregate, and reason over large amounts of relevant information, enabling more holistic approach to complex, multi-conditional queries. However, existing studies primarily evaluate LCLLMs as substitute for retrieval models that focus on localized contexts rather than evaluating their capability for holistic reasoning (Kamradt, 2023 Lee et al., 2024 Zhang al.) 2024 Hsich et al., 2024), summarized in Table I This gap persists partly due to the absence of suitable benchmarks that rigorously test models on tasks requiring reasoning across extensive document collections.
In this paper, we introduce the Holistic Reasoning Benchmark (HoloBench), a new framework specifically designed to evaluate LCLMs' ability to perform holistic reasoning over long contexts. HoloBench leverages database reasoning operations to systematically evaluate how well models can aggregate, compare, and draw conclusions from distributed information. By adapting existing text- to-SQL benchmarks (Yu et al. [2118] Batsogiannis-Meimaradd & Koutrika [023 Li et al. (2024a), HoloBench enables an automated and scalable evaluation process, eliminating the need for labor- intensive manual annotations.
A key innovation HoloBench its ability to control three critical factors that influence LCLM performance: (1) the length of the context and the amount of information contained within, (2) the position of relevant information within the context, and (3) the type and difficulty of queries. These factors allow for more comprehensive assessment of LCLMs holistic reasoning capabilities, Ex- isting benchmarks have primarily explored only one or two of these dimensions in isolation (Zhang) cet al. 2024 Hsich al. 2024: Levy al. 2024), making HoloBench the first framework to evaluate LCLMs across all three dimensions simultaneously.
Through detailed experiments, uncovered several important findings: 1) The amount of relevant information the context impacts model performance much more than the total length the context itself. 2) The complexity of the questions plays more important role in performance than the total amount of information being processed. For example, models easily handle questions that ask for maximum minimum values, but they struggle when the task requires aggregating several pieces of information. 3) How information is positioned in the text also matters. Some models work better when relevant information is grouped together, while others can handle more scattered information.
Our findings highlight both the progress and the remaining challenges in enabling models to perform holistic reasoning over long contexts. Our work addresses critical gap in evaluating long-context models and offers valuable insights into their capabilities and limitations when processing large- scale textual data.
2 RELATED WORK
Neural Databases Thorne al. (2021) have proposed a neural database architecture designed to process database- like queries over natural language text. While their approach demonstrates scalability for small medium-sized datasets, it is not sufficient for LCLMs, as the database size and complexity of queries are limited. Moreover, obtaining large-scale ground truth annotations for such tasks is prohibitively expensive and labor-intensive, posing challenge for comprehensive
2

---

Published as a conference paper at ICLR 2025
evaluation in this area. Trappolini al. (2023) extend neural databases to multimodal data, including images and text. However, their work focuses on multimodal retrieval, not on evaluating LCLMs for long-context tasks, leaving a gap in addressing the challenges of long-context understanding.
Long-context Language Model Benchmarks Many recent studies (Kamradi 2023 Lee ct al,
2024 Hsieh et al.] 2024 Kuratov et al.) 2024 Karpinska al.) 2024] Laban et al. 2024 Levy
et al. 2024| Zhang et al. 2024) have introduced benchmarks to evaluate LCLMs on tasks that
tasks, which involve locating specific "needles with magic numbers" within extensive texts. While these benchmarks effectively test models' retention and coherence across large texts, they predominantly focus on localized tasks or summarization. Specifically, most examples require aggregating up to approximately ten information pieces, which does not fully assess models' abilities to handle more complex reasoning tasks over vast amounts of information in lengthy documents.
require understanding
processing extended contexts, including "needle-in-a-haystae"
For instance, En.QA oeBENCH (Zhang et al., (024) includes only a few questions that require information aggregation over more than ten pieces. However, the gold answers for these questions are manually annotated by the authors, making the creation process prohibitively costly and prone to annotation errors. In contrast, we aim to investigate how LCLMs synthesize and integrate vast amounts of information distributed across extended contexts in an automated and scalable man- ner. By designing corpus generation framework that can systematically vary the complexity and distribution of information, we address the limitations of existing benchmarks and provide a more comprehensive evaluation of LCLMs' holistic reasoning capabilities.
Retrieval-Augmenter Generation Retrieval-Augmented Generation (RAG) (Lewis et al. 2020) combines the strengths of retrieval-based methods and generative language models enhance the generation of accurate and contextually relevant responses. Recent studies (Li et al. 2024b Yu al. 2024) have discussed the pros and cons of both LCLMs and RAG. However, their analysis relies on the abovementioned benchmarks that are designed for localized tasks. The insights are not applicable our task, which requires filtering information based on given complex constraints and aggregating a larger amount of information.
THE HOLISTIC REASONING BENCHMARK
The Holistic Reasoning Benchmark (HoloBench) is an evaluation framework designed to assess ability of LCLMs to perform holistic reasoning over massive textual data. HoloBench leverages database operations to create complex reasoning tasks that require models to aggregate and synthe- size information distributed across extensive contexts.
3.1 DESIGN PRINCIPL.ES
The design of HoloBench is motivated by the need to evaluate LCLMs holistic reasoning tasks that require comprehensive understanding and reasoning over large-scale document collections. To achieve this, HoloBench is built upon the following key principles:
The benchmark includes tasks that require LCLMS to perform oper- ations such aggregation, comparison, and inference massive textual data.
Complex Reasoning Tasks:
The benchmark systematically varies critical factors influencing LCLM performance, including context length, information density, information distribution, and query complexity.
Controlled Evaluation Factors:
Automated and Scalable Evaluation: The benchmark dynamically generates gold answers based on evaluation parameters, eliminating the need for labor-intensive manual annotations.
To adhere to these principles, HoloBench utilizes text-to-SQL benchmarks construct holistic rea- soning data. This approach ensures that the benchmark provides verifiable answers by executing queries databases while dynamically controlling context size and information distribution.
3

---

Published as a conference paper at ICLR 2025
Stop Iddendly Relevant BQL query
Stop Verbalize each row reaching longth
Stop Executo query subtables
P5GL
VBQL EHO
IKNER C0IN
Gold snower
HAnD Gzape
ccape
12
prcopess
hggel Sond
Step Marge Relevant Irrelavant context
Contest
Step Reaconing
PNIL
D2-
uppel Linn
Irolenant contast:
Figure I: Overview of the HoloBench instance generation process, illustrating the partitioning of databases into relevant and non-relevant subsets, verbalization of table rows into textual contexts, and construction of the final inference context based on information positioning parameters.
3.2 BENCHMARK CREATION
HoloBench is synthesized from text-to-SQL benchmarks by adapting queries and their correspond- ing databases into natural language tasks. Each question INL. in HoloBench is paired with SQL query ISol and database D, which allows automatic generation of dynamic gold answers y as eval- uation parameters are varied. The benchmark creation process, illustrated in Figure [ involves the following steps:
1. Identifying Relevant Tables: Based on the SQL query QSQL, tables relevant to the query in the database D = (d1...., dn are identified and isolated into subset D,, while the remainder are designated D -.. This process includes writing a pair of natural language questions INL. and its corresponding SQL query that can be executed on the database.
Each row from both the relevant tables d E D, and the non-relevant tables d E D... is iteratively sampled and transformed into its natural language text 4, using predefined templates. This verbalization continues until the lengths lcgl and |C-gl match the predetermined context lengths, ensuring balanced representation of relevant and irrelevant data line with the total input length |c] and predefined information density.
Verbalizing Data Rows:
3. Merging Contexts:
The relevant context Cg and the irrelevant context C-u are then merged based on an information positioning parameter that dictates the structure and placement of relevant and non-relevant information within the overall context This ensures that context c representative of both necessary and supplementary data, optimizing the distribution for evaluation.
4. Executing SQL. Queries: An SQL executor runs @SQL. on the subtables derived from D' to extract the gold answer This answer used to assess the accuracy of the LCLM's response ğ when provided with the context and the natural language question (NL- The evaluation measures how effectively the model can utilize the structured context to generate correct responses.
This process enables the creation of dynamic benchmark for holistic reasoning with desirable properties while requiring minimal to no human effort.
3.3 IMPLEMENTATION DETAILS
HoloBench is implemented using a variety of domains from the Spider dataset (Yu et al.) 2018), i.e. wine_1, college coll_e, 2, flight_4,, soccer and store_1. 1.
We categorize query types into five groups based on SQL. components:
1. Aggregation (e.g., SUM, AVG, and COUNT) which requires extracting relevant information and cor- rectly aggregating it.
2. Max/Min (e.g., MAX, MIN), which involves finding extreme values within the data.
3. Join (e.g.. JOIN and INNER JOIN), which requires two or more documents to retrieve the infor- mation requested in the query.
Comparison (c.g., >, S, =, and BETWEEN), which requires numeric comparisons to filter data
5. Ranking (ORDER BY and LIMIT) which requires identifying top-k results and sorting them.
4

---

Published as a conference paper at ICLR 2025
Query Type
Natural language / SQL queries
Aggregation
What the total number unique instructors?
SELECT
count (DISTINCT instructor_name
FROM advisor;
Max/Min
What is the maximum capacity of any classroom?
SELECT
MAX(capacity)
FROM classroom;
Join
List course titles and the corresponding buildings. SELECT
course. title, INNER JOIN section ON course. title section. course, _itle;
section. on. building
FROM
course
Comparison
Find the buildings which have rooms with capacity more than 50.
SELECT DISTINCT building FROM classroom WHERE capacity > 50;
Ranking
What are the titles of the top 5 posts with the highest popularity?
SELECT Title FROM posts ORDER ViewCount
5;
Table 2: Categorization of query types with example natural language and SQL queries.
Difficulty Level
Natural Language SQL Queries
Easy
What the of the wine with the highest price? SELECT wine_name
WHERE price (SELECT MAX(price) FROM wine);
Medium
List the names of flights that depart from 'JFK' and arrive at 'LAX'. SELECT flight_name FROM flights WHERE departure_airport
'JKK' AND
arrival_airport "LAK";
Hard
What the names of colleges that have more than 10,000 students and are located in Californi??
SELECT college_nane FROM college WHERE nun_students 18000 AND state 'California';
Table 3: Categorization of Query Difficulty Levels with Example Natural Language and Corre- sponding SQL Queries.
Uniform
Beginning
Middle
End
Bimodal
Figure 2: Illustration of how relevant context Gg (highlighted in green) is placed within the entire context for different information positioning strategies.
Table 2 shows example queries for this categorization. Note that a single query can be associated with multiple types. These query types are automatically decided based on the type of database operator(s) in the SQL query. We also define three levels query difficulty: Easy, Medium, and Hard. These are automatically decided based on the number of SQL components. Table shows example queries by difficulty level. We provide more detailed definitions in Appendix A.3.1 and
A.3.2 To observe the impact of information positioning, we consider three clustered positioning strategies -Beginning, Middle and End- and two distributed positioning strategies -Uniform and Bimodal (concentrated both ends). These illustrated in Figure 2
For each database, the authors write six questions per query difficulty to cover all query types, result- ing 90 questions for HOLOBENCH since queries in the Spider dataset do not cover various query types. To ensure quality and variety, more than two authors reviewed the questions. We provide several examples Appendix A.3.3 and the complete query set available in the supplementary material. To convert each table row into textual representation, we need ensure that all necessary table information is included in the output while any extraneous details that could conflict with the ground truth answer y is excluded. To achieve this, develop template- based text generation system. We manually analyze the queries the seed dataset for each database to identify the rele- vant table columns needed for verbalization. Based this analysis, create custom templates for
5

---

Published as a conference paper at ICLR 2025
each table that incorporate all required columns. Additionally, to ensure diversity in the generated text, we design five distinct templates for each table. Example templates are given in Table
Table/Database
Template
customers/store_1
(city), (country), (full_ name)) can be reached at (phone) for assistance from suuport_rop..mmpooeee..oo
airports/flight_4
The {nane} located (city), {country) elevation of {elevation) feet.
Player/soccer_1
With height of (height) cm, (player_name) was born (birthday) and weighs (weight) 1b.
Table Templates for text generation. (placeholder) indicate column names that can be replaced with actual values from the tables.
4 EXPERIMENTS
4.1 SETUP
Models: We use nine LCLMs that support 128k or longer context length: Llama 3.1 84/70B/405B-instruct (Dubey al., 2024). GPT-40-mini/GPT-44 (OpenAl] (OpenAA|2024a,, [2024a], o1-mini (Ope- nAT 2024b), Claude 3.5 Sonnet (Anthropic 2024), Gemini 1.5 Flash/Pro (Reid et al.) (2024) and five more recent LCLMs: 03-mini (OpenAl, 2025), Gemini 2.0 Flash/Flash-Thinn ThinshhhThiing... and DeepSeek- V3/RI (DeepSeek-AT
2024
2025).
We used the same prompt across all the models shown in
Appendix|A.. A.4
We evaluate LCLM performance across varying context lengths and information amounts. The context lengths |c] tested 4k, 8k, 16k, 32k, and 64k tokens. To better understand how the amount and density of relevant information affect performance, we employ two setups: a) constant information amount: The total relevant information fixed 2k tokens, regardless of context length, and b) constant information density: The relevant information is distributed at fixed density of 50% throughout the context.
Configurations:
Evaluation Metrics: We evaluated the holistic reasoning performance of LCLMs by determining
whether each element in gold answer y is included in the output ý (Kamradt) 2023 Fu al. 2024). Since the model produces the answer ý in natural language, while the gold answer is a structured result set, we evaluate the performance on per-row basis. We use the gpt-4o-min each row as Exact Match, Partial Match, or No Match, assigning scores of 100%, 50%, and 0%, respectively. This recall-oriented accuracy for each instance is calculated the average score across all rows in the result set. We manually tested this evaluation metric for 20 questions, achieving 93.8% agreement between the model evaluations and human judgments]
to classify
We report three types of performance measures: the unweighted average (Avg.), and two weighted averages based on context length. (1) wAvg. (inc), where the weight increases linearly with context length, simulating scenarios where longer sequences dominate usage, and (2) wAvg. (dec), where the weight decreases linearly with context length. (Hsich al. 2024).
4.2
MAIN RESULTS
4.2.1
EFFECT OF CONTEXT LENGTH AND INFORMATION AMOUNT
Table summarizes the performance of various LCLMs as the average score across all questions in HoloBench. Specifically, we present performances for different context lengths under both con- stant information amount and constant information density settings. Note that we use a uniform positioning strategy for the experiments.
In preliminary study, experimented with basic 5-point scoring system LLu et al. 2024), but it demonstrated low alignment with human judgments.
2023 Padlewski

---

Published as a conference paper at ICLR 2025
Constant Information Amount (2k)
Constant Information Density (80%)
LCLMs
5k
16k
32k
64k
Avg.
w:vg
(inc)
WAvg- (dec)
4k
16k
32k
641k
Avg-
wArgg
(inc)
wArg
(dec)
Llama-311-0056
64.0
61.0
67.5
65.1
44.8
56.3
53.5
09.4
59-4
56.1
41.4
30.2
50.00
44.3
56.7
GPT-4o
62.5
52.2
53.6
59.4
52.0
55.9
55.0
56.8
62.5
57.5
15.3
13.3
37.5
19.2
45.0
58.5
ol-mini
63.5
55.8
48.8
47.6
40.3
51.2
47.5
54.8
61.5
51.2
45.8
34.4
29.2
44.8
39.1
50.5
Claude-3 S-Soneet C.oone-3.--SSonet
49.3
02.2
46.4
42.3
41.9
46-4
44.8
48.1
49.3
50.7
47-2
42.1
34.7
44.8
42.3
47.3
Llama-3,1 1-100
60.5
51.2
11.1
12.2
33.6
15.7
11.5
19.9
60.5
58.7
15.9
39.0
28.6
15.5
10.3
50.8
GPT-4o-mins
51.5
50.1
45.1
45.2
38.7
45.3
43.2
47.5
51.5
45.6
37.8
35.0
26.3
39.2
35.2
43.3
Gemini-1.5 Pro
45.8
433.4
322.8
39:22
42.12
40.6
39.3
41.4
45.6
35.1
31.9
30.0
33.5
34.8
32-6
37.0
Gemini-1 Flush
36.8
35..0
31.3
27.2
26.8
31.4 4
299 8
355.1
36.8
30.7
25.8
27.7
20.5
25.3
25.9
30.7
L.-
39.1
27.4
28.7
9.8
10.0
23.0
18.0
28.1
30.1
34.7
21.0
6.6
1.1
20.5
13.5
27.4
Gemini-2.n O Flash
66.2
02.3
00.1
57.3
53300
19.6
67.5
61.7
66.2
60.1
57.0
52.5
48.2
M6.8
13.9
19.7
Gomini-2.0 Flash Thinking
69.5
54.2
43.0
67.2
00.3
51.9
50.1
o3 -3-mmni
68.6
62.6
51.6
46.8
42.7
54.5
50.0
59.0
68.6
58.0
47.2
33.6
22.4
46.0
38.2
53.8
D3
59.5
51.7
5408
47.3
53.6
02.6
51.5
53.7
58.5
56.9
52.5
45.8
36.0
50.1
46.2
54.0
DeepSook-R1
61.1
41.8
36.4
34.4
40.7
31.9
35,9
43.6
Table 5: Model performances for different context lengths under both constant information amount and constant information density settings. Arg. refers to the average across context sizes, and wArg. denotes the weighted average, with weights either increasing (inc) or pydecreasing (dec) linearly with context length. The best score of each column is underlined. Generally, performance declines as context length increases. The drop is wider for information density than information amount.
Information amount impacts more than context length The experimental results indicate that the amount of information has a more significant impact on LCLM performance than context length. For instance, 1 1-05BBss accuracy drops from 64.0% 30.2% information density in- creases, compared to a smaller drop to 44.8% when only the context length increases. Similarly, GPT-4o achieves 52.0% accuracy with 64k context and 2k of relevant information, but perfor- mance decreases to 45.3% when processing 8k of information in a shorter 16k context (with 50% information density). These results suggest that increased information density negatively impacts accuracy more than simply extending the context length.
Llama-3.1-405b is best for short contexts, while GPT-4o dominates in long contexts Overall, Alama-3.1-4033 achieves the highest average and weighted average (dec) scores, indicating strong performance with shorter contexts. We observe this trend both information amount and informa- tion density settings. However, GPT-4o achieves notably higher average (inc) score than 1lama-3.1- 405b (55.0 vs 53.5), indicating GPT-4o is more consistent and robust in handling longer contexts. This is also evident in the significantly higher scores achieved by GPT-4o for 32k and 64k context lengths.
Model size is key for holistic reasoning in long contexts The smaller models used in this study (GPT-4o-mini, Llama-3.1 1--8,, Gemini-1.5 Flash) claimed high accuracy in retrieval tasks such as the Needle-in-a-haysttde test. However, for tasks that require holistic reasoning. model size signif- icantly affects performance, especially as context length increases. For instance, in the Llama-3.1 series, the 8b model achieved only 1% accuracy under the 64k context with 50% information den- sity, whereas the 405B model reached around 30% accuracy under the same conditions. This trend consistently observed across other model series well.
Reasoning- LCLMs struggle with extended contexts Reasoning-focusee models-ol mini, o3-mini, Gemini-2.0 Flash Thinking, and DeepSeek-R1- R1 strong performance on shorter contexts but experience significant performance degradation as context length increases. This sug- gests that current training approaches for reasoning-foote focusee models may not adequately address the complexities of processing longer contexts. While general-purpose models like Gemini-2.0 Flash maintain consistent performance across context lengths, the challenges faced by reasoning-foee models highlight important area for future research: developing training strategies that can main- tain robust reasoning capabilities even with extended contexts.
4.2.2 EFFECT OF INFORMATION POSITION
For brevity, we focus on the best-performing models, Llama-3.1-400 and GPT-4o, to examine the effect of information position. We use constant information density the baseline setting unless
7

---

Published as a conference paper at ICLR 2025
Uniform
64.0
59.4
55.1
41.4
30.2
50.0
Uniform
62.5
57.5
45.3
43.3
37.5
49.2
Beginning
66
60.1
49.0
40.3
32.5
49.6
Beginning
57.2
49.7
43.3
38.6
50.8
Middle
64.0
55.0
46.0
38.9
27.2
46.2
Middle
54.7
48.8
43.1
39.0
50.5
End
63
61.7
49.2
41.7
28.8
43.0
End
6.3
55.5
55.4
46.7
46.1
53.5
Bimodal
70.8
61.0
46.4
44.4
32.3
51.0
Bimodal
69.3
59.0
56.1
50.4
43.9
55.7
4k
16k
32k
64k
Avg.
8k
32k
Avg.
Context Length
Context Length
Alamma-3.1-4055.
(b) GPT-4o
Figure 3: Variations model performances with position of relevant information. Concentrated information benefits most models, although optimal position varies for each model.
otherwise specified. uniform, beginning, middle, end, and bimodal.
the model performances across five information positions:
Allama-3.140055 is robust for information position, while GPT-4o is more sensitive Llama-3.1 405b maintains stable performance across various information positions, showing resilience whether clustered. In contrast, GPT-4o performs better when relevant information placed at the end or bimodal, but its accuracy declines when the information grouped at the beginning and middle, suggesting that it tends favor details presented the end.
the information is distributed uniformly
4.2.3 EFFECT OF QUERY TYPE AND DIFFICULTY
Aggregation
Max/Min
Join
Comparison
Ranking
1lama-3.1-4055
GPT-4o
75
Accuracy
25
4k
8k
16k
32k
4k
16k
32k
64k
Context Length
Context Length
Figure 4: LCLMs' performance various query types. Max/Min queries easier task across context lengths than other types. In contrast, We observe a performance drop for Aggregation queries, particularly for longer contexts.
Next, we evaluate the performance of Llama-3, and GPT-4o using constant information den- sity and uniform information placement, considering query type and difficulty.
Query type affeets performance: Max/Min is easier while aggregation is harder. Figure summarizes the performance of models across different query types. As shown, model performance varies significantly depending on the query type. Max/Min queries consistently easier than other types of queries. This is likely because extreme value retrieval tasks are relatively easier and do not scale in complexity with larger context lengths, unlike other tasks.
Interestingly, GPT-4o performs on par with other query types for Aggregation queries at 4k context length. However, as the context length increases, there is notable drop performance, indicating the model struggles to aggregate relevant information effectively in longer contexts. To investigate this behavior, we select random aggregation query: "Find the destination airport and its country that receives flights from the highest number of distinct source airports, and include that number the result." " and inspect the model predictions. We find that GPT-4o correctly lists relevant information and produces the correct answer when the context length is 4k. At 8k context length, can find relevant information but begins to struggle with correctly retrieving and aggregating the information. Specifically, it returns the correct airports but miscounts the number of airports. At 64k context length, skips the detailed analysis of the content and instead jumps straight to answering the question, leading to incorrect responses. We provide model outputs in Figures 14] 15 and 16 in Appendix. This highlights the limitation of LCLMs' step-by-step reasoning abilities as the context size increases.
8

---

Published as a conference paper at ICLR 2025
Easy
Medium
Hard
Llama-3,1-405b
GPT-4o
80
Accuracy
60
40
Context Length
Context Length
Figure 5: LCLMs' performance on various query difficulties. Model performance declines as query difficulty increases. Performance drops further context length increases.
Query complexity often affects performance than the total volume of information. Fig- ure the performance of 1lama33.1-405h and GPT-4o across different query difficulties. We observe that the relative order difficulty-ease easy. medium, and hard-remains consistent across different context lengths, highlighting the alignment between the inherent complexity of the queries and the models' capacity for holistic reasoning over extended contexts. However, hard queries con- sistently show the lowest performance across all context lengths, with a steep decline context length extends, indicating the increasing challenge of reasoning over longer contexts with complex queries. Upon close inspection of failure cases, we observe that even though models can think step- by-step, they struggle retrieve information for each condition for multi-conditional hard queries. Additionally, we find that when the volume of relevant information is too large. LCLMs complain about the imput format being inappropriate and suggest using programmatic parsing. This points to an important research gap in the reasoning capability of LCLMs that demands further research. We observe similar trends for other models and provide further details in the Appendix A.5
4.3 DISCUSSION
Does the RAG help for holistic reasoning? Holistic reasoning requires analyzing entire long contexts to answer questions comprehensively, making the RAG approach suboptimal for this task. RAG relies retrieval models that focus on localized context retrieval, and they cannot effectively determine how much information should be retrieved. However, some studies suggest that reducing noise in the context may improve downstream tasks (Wu al.) 2024 Xu et al. 2024). With this in mind, we tested whether a retrieval model can filter irrelevant information and enhance holistic reasoning.
To test this hypothesis, we used constant information density and uniform information placement across models. We compared two large LCLMs, Llama-3. 1-40bbbbbbb and GPT-4o, and smaller LCLM, Jlama-3.1-8h As the document retriever, we used BAAT/bge-large-n-1 55, a strong embedding- based retrieval model, retrieving 2k tokens with the highest similarity score the query. In a shorter context (e.g., 4k tokens), the retrieval model might capture all relevant information, potentially enabling to solve the holistic reasoning task without interference from extraneous data. However, in longer contexts, it becomes impossible to retrieve all relevant information, which tests the limits of this approach.
Figure 6 compares the performance of vanilla LCLMs and RAG-based models. For short contexts (4k tokens), RAG slightly outperforms vanilla LCLMs with GPT-4o and Llama-3. This may be because, in this specific scenario, the retrieval model retrieves exactly the amount of relevant information, providing a cleaner context for holistic reasoning than the vanilla LCLM setting.
However, this configuration is not realistic for most practical scenarios, where cannot know the exact amount of relevant information in advance. As the context length increases beyond 4k tokens, larger vanilla LCLMs consistently outperform RAG-based models. This suggests that LCLMs are more effective at handling longer contexts where RAG models may fail to retrieve sufficient relevant information.
Interestingly, with smaller models like 1aama-3.18bb RAG shows better performance when the context length exceeds 16k tokens. This aligns with existing studies (Maekawa al. 2024). which

---

Published as a conference paper at ICLR 2025
LCLM
RAG
Llama-3.1-408
GPT-4o
Llama-3.1-80
75
Accuracy
50
25
16k
16k Context Length
16k Context Length
Context Length
Figure 6: Performance comparison of L.LM and RAG. For long contexts, large models outperform RAG but retriever helps a small model due to its limited ability to handle long contexts.
suggest that retrieval models assist weaker models by mitigating their limitations of reasoning capability, even accounting for retrieval errors.
Further, detailed comparisons hetween RAG and LCLMS provided in Appendix A.5.5 A promising future direction would be to develop dynamic mechanism for determining the optimal amount of information retrieve based on the query and context length, particularly when working with weaker models for holistic reasoning.
Does CoT prompting improve performance? We evaluated how chain-of-thouge (CoT) prompting, which is our default configuration, impacts performance across different context lengths and query types (Wei et al., 2022, Kojima et al. 2022). We instructed LCLMs to provide answers di- rectly without reasoning in a non-CoT setting. Figure compares model performance with and with- out CoT prompting. The results indicate that CoT significantly and consistently enhances LCLM performance across all context lengths and query types. Specifically, aggregation queries are rarely solved without COT, whereas max/min queries are relatively easier and less influenced by context length. These findings highlight the importance of CoT prompting in enabling LCLMs handle complex holistic reasoning tasks more effectively, particularly in scenarios requiring deeper infor- mation synthesis.
with CoT
without CoT Join
Aggregation
Max/Min
Comparison
Ranking
Accuracy
50
25
4k
8k
16k
32k
64k
8k
16k
32k
64k
4k
8k
16k
32k Length
64k
16k
32k
64k
4k
8k
16k
32k
64k
Context Length
Context
Length
Context
Context Length
Context Length
Figure 7: Performance comparison of GPT-4o with CoT and without CoT. Generally, CoT improves
performance, particularly for tasks that require reasoning multiple information sources.
5 CONCLUSION
In this paper, we introduced HoloBench, a novel benchmark designed to evaluate the holistic rea- soning capabilities of LCLMs when handling complex, multi-document queries. Our framework systematically controls key factors such as context length, information density, information posi- tioning, and query complexity, enabling a comprehensive assessment of LCLMs. Through extensive experiments, we demonstrated that the amount of information in the context has larger impact on model performance than the actual context length. Additionally, certain query types, such as ag- gregation tasks, present unique challenges for LCLMs, especially context length increases. This study fills a critical gap in evaluating LCLMs' holistic reasoning capabilities and provides valuable insights into their strengths and limitations in processing large-scale textual data. Future work could extend this framework to a broader range of reasoning tasks and explore optimization techniques to enhance LCLM performance in complex information aggregation scenarios.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We have made several efforts to ensure the reproducibility of our work. First, the design principles, methodology, and evaluation setup for HoloBench are detailed in Sections 4 and The SQL queries, natural language counterparts, and data preprocessing steps are provided in the Appendix (Sections A.2 and A.3.3). Moreover, the benchmark creation process, which includes database construction and verbalization steps, is clearly outlined in Section 4.2, and additional dataset details are available in the supplementary material.
To further enhance reproducibility,
will release all code and benchmark datasets, including queries, natural language questions, and corpora, upon paper acceptance. In all experiments, we set the temperature of LCLMs to 0.0, ensuring deterministic outputs for consistent model perfor- mance across multiple runs. All models used in the experiments are well-documented, with the prompt templates described in Appendix AA.4 Upon paper acceptance, we will release the bench- mark, codebase, and detailed documentation for both the experimental setup and data generation to promote ease of replication.
REFERENCES
Anthropic. Introducing the next generation of claude. Technical report, 2024. URL https://www. anttroppi..com//ee///ccccc com/news/claude f-fmmily
DeepSeek-AL. Deepseek-v? technical report, 2024. URL httttss//aartvv.rr//////12111111
DeepSeek-AL. Deepseek-rl: Incentivizing reasoning capability in ilms via reinforcement learning, 12948
2025. BRLLittts///axxxvoorg/bbb/
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Leuman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The Ilama 3 herd of models. 21783
arXiv preprini 1rXiv:2407.21783, 2024. http:/tpss///xxi.oor////aa//////
Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128K context. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 14125-14134, PMLR, 21-27 Jul 2024. URL https://proceedingg mlr. wwwppp/essss/. wtt.perr//vv3555244255 html
Cheng-Ping Hsich, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What's the real context size of your long- long-context language models? In First Conference on Language Modeling, 2024. URL httt::://ppenrevieeweeeffrroroo kloBbc76Sy
Gregory Kamradt. Needle in a haystack pressure testing 1ls.. Github repository, 2023. URL hcttps://github.comm/ee hww.aaarrtttttteeetteeeeeeeeeaer
Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit lyyer. One thousand and one pairs: novel" challenge for long-context language models. arXiv preprint arXiv:2406. 16264, 2024. URL
2406. 16264
George Katsogiannis-Miimaaaad and Georgia Koutrika. survey on deep learning approaches for text-to-sql. The VLDB Journal, 32(4):9559936, 2023. HRL/https:///in https//////n.n....iggrr...r com/ article/10. 000.0/00000//00070022--000000
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwa- sawa. Large language models are zero-shot reasoners. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.). Advances in Neu- ral Information Processing Systems, volume 35, pp. 22199-22213. Curran Associates, Inc., 2022. URL https:://roceediii reriiinnssnneruii wttppppcppp//////./ee 8bb0d291 0bbbad99acchaa000e11119916666630
cc//pppp.ffilee/ppprr//////1// .dff

---

Published as a conference paper at ICLR 2025
Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of Iims with long context reasoning iaaaiiin--------ttteeeee a-hasstack. arXiv preprint arXiv:2406.11111 2024.
10149
Philippe Laban, Alexander R Fabbri, Caiming Xiong, and Chien-Sheng Wu. Summary of haystack: A challenge to long-context ilms and rag systems, arXiv preprint arXiv:24007001370, 2024. URL
fttps://aaxxivoor/abae
01370
Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Sébastien MR Arnold, Vincent Perot, Siddharth Dalmia, et al. Can long-context lan- guage models subsume retrieval, rag, sql. and more? arXiv preprint arXiv:2406. 13121, 2024. URL
http://rwxi....rr/a////66666 23122
Mosh Levy. Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact of input length on the reasoming performance of large language models. arXiv preprint arXiv:2402.1488. 14848, 2024.
URL wttphhttts//xiiiv www. wwtgg//bs/2200 ab/p///....22..... 14848
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 9459-9474. Curran As- sociates, Inc., 2020. URL htttss://proceedings.oo neurips File//b493230226f788888 1000000047444444454555555555 750.://222828.000002200088644810 pat.e///.dd/7e11eeeae...r..d .dff
fiaaper_ileesppppeeee
Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang. Binhua Li, Bowen Li, Bailin Wang. Bowen Qin, Ruiying Geng, Nan Huo, et al. Can Ilm already serve as database interface? big bench for large-scale database grounded text-to-slls. Advances in Neural Information Processing Sys- tems, 36. 2024a. URL /tL/ newrips.rpppp.... cttp:ccccpppppeeeec filles/paaprrr2223//asttt 83fc8fab17103338 SSffaaa67770330000000ccccc00006-
hat.raaassssttananneennnnrnnrss html
Zhuowan Li, Cheng Li, Mingyang Zhang. Qiaozhu Mei, and Michael Bendersky. Retrieval aug- mented generation or long-context Ilms? comprehensive study and hybrid approach. arXiv preprint arXiv:2407.16833, 2024b. /tt/stttss///rxxi /tattpps///axii.orr/abbbsss// htrgg/bbb/400/ 16833
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NI.G evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 2511-2522, Singapore, December 2023. Association for Computational 153. URL httppss://aaaaanhollog.. wrgp://0oooyogg/22....
Linguisties. doi: 11286533/112023.mmllppp emmlp-main. 153
Seiji Maekawa, Hayate Iso, Sairam Gurajada, and Nikita Bhutani. Retrieval helps or hurts? deeper dive into the efficacy of retrieval augmentation to language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1; Long Papers), pp. 5506-5521, Mexico City, Mexico, June 2024. Association for Computational URL
Linguistics. doi: 10. 36653/v1/2024.aaalllaaa haac1-1ong. 308
2024.
OpenAl. Model release blog: Gpt-4o. Technical report, 2024a. UR..ittpp://qpaaa.ccm//eeo dello-gpt-4ol
OpenAl. Model release blog: Openai o1-mini. Technical report, 2024b. URL https://openaa. com/
OpenAl. Model release blog: Openai 03-mini. Technical report, 2025. URL https://openai.
com Comm/index/openai-----iii
Piotr Padlewski, Max Bain, Matthew Henderson, Zhongkai Zhu, Nishant Relan, Hai Pham, Dono- van Ong. Kaloyan Aleksiev, Aitor Ormazabal, Samuel Phua, Ethan Yeo, Eugenie Lamprecht, Liu, Yuqi Wang, Eric Chen, Deyu Fu, Lei Li, Che Zheng, Cyprien de Masson d' Autume, Dani Yogatama, Mikel Artetxe, and Yi Tay. Vibe-eval: A hard evaluation suite for measuring progress of multimodal language models, 2024. http:::ts://aaat www.aaawwwvvooora/a//aaaaaa 02287
12

---

Published as a conference paper at ICLR 2025
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. 2024.
05530
James Thore, Majid Yazdani, Marzieh Saeidi, Fabrizio Silvestri, Sebastian Riedel, and Alon Halevy. Database reasoning over text. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the Ith International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pp. 3091-31044 Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v 12203//11/221..2 lonl-llon.221,
org/202 2221 acl-long. 241
Giovanni Trappolini, Andrea Santilli, Emanuele Rodolà, Alon Halevy, and Fabrizio Silvestri. Multimodal neural databases. In Proceedings of the 46th International ACM SIGIR Confer- ence on Research and Development in Information Retrieval, SIGIR '23, pp. 2619-2628, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 978 9781490344086. doi: N0.1145/3339618.3599999
URL httLhhtps:://// htttt:::/ddoi.rrgg/// 1101111351/31111 3591930
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language mod- els. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Ad- vances in Neural Information Processing Systems, volume 35, pp. 24824-24837. Curran As- sociates, Inc., 2022. URL
neurips cc/ wwwpapcrppppeeeeecceeeeeeeeeee www.lllspppe/e//ee 2022/ pdf]
Yunshu Wu, Hayate Iso, Pouya Pezeshkpour, Nikita Bhutani, and Estevam Hruschka. Less is more for long document summary evaluation by LLMs. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computa- tional Linguistics (Volume Short Papers). pp. 330-343, St. Julian's s, Malta, March 2024. Asso- ciation for Computational Linguistics. URL htttps://aclanthooloo wtwpeaanttooooooooo///2222
shorts.
29
Fangyuan Xu, Weijia Shi, and Eunsol Choi. RECOMP: Improving retrieval-augmenter LMs with context compression and selective augmentation. In The Twelfth International Conference Learning Representations, 2024. URL / wttp://wwwweeoooooooooco/ocll//e
Tan Yu, Anbang Xu, and Rama Akkiraju. In defense of rag in the era of long-context language models. arXiv preprint wrXiv:2409.016666 2024. URL httts:://////ii/ org. abs/2409. 01666
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Ellen Riloff, David Chiang, Julia Hockenmaier, and inhi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3911-3921, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/v1/DIJ 1/21
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. o:Bnnch: Extending long context evaluation beyond 100K tokens. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Pupers), pp. 15262-15277, Bangkok, Thailand, August 2024. Association for Computational Lin- / htt.://pps...c..aatoollgooo/////
guisties. URL
1oo.e..oooo111 814
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 MODEL DETAILS
We show the model details in Table 6
Model
Size
Context
HuggingFace API
GPT-4o (OpenAl
2024a
128k
gpt-4o-2824-08-65
GPT-4o-min ppp 2024a
128k
gpt-40-mini-2224-9777
03-mini (OpenAl 2025
200k
01-mini [OpenAT] 2024b
128k
01-min1-2024-0911
Claude-3.3-Somgg (Anthropic 2024
200k
LLuud=-2-5-sommete22244-00-
Gemini-1.5-Pru RReid el al. 2024
2M
gemini-1.5-pro-20
Gemini-1.5-Flash [Rerd ct a...
2024
1M
gemini-1.5-flaste
Gemini-2.0-Flash [Rerd ct al.
2024
1M
Gemini-2.0-Flast 2.a.--0---aaah--hannnnn
Tuunking Rerd cl
al. 2024
1M
Llama-3.1-4005 (Dubey et
2024
405B
128k
Llama-3.11700 Dubey ct ai.
2024
70B
128k
Llama-3. 1-8b
(Dubey ct al.
2024
8B
128k
meta-1Laam/LLaaa333--8-8-----
DeepSeek- 33 DeepSeck-AA
2024
671B
128k
DeepSeek-RI (DeepSeck-A] 2025
671B
128k
Table 6: Models used in experiments. Model sizes are not publicly disclosed (-).
A.2 DATASET DETAILS
A.2.1 DATABASE PREPROCESS
To ensure that each verbalized document is comprehensive, we add entity name information for each table. This preprocessing involves mapping various identifiers (such course IDs, instructor IDs, student IDs, etc.) to their corresponding human-readable names within the database. By integrat- ing this descriptive information into the database tables, the data becomes more interpretable and contextualized, facilitating subsequent natural language processing tasks. Additionally, only entries with complete information are retained to ensure the consistency and reliability of the data, thereby improving the quality of the analysis. With this preprocessing, tables containing only identifiers and entity names are excluded from the verbalization process.
A.2.2 DATASET SELECTION
To create HoloBench for evaluating LCLMs, we required large tables capable of generating long contexts. The following criteria were applied ensure diversity in both domains and answers:
1. Databases must have more than two tables and at least 500 rows total.
2. Each selected database must belong to a distinct domain.
Among the databases in the SPIDER dataset, only 15 meet the row count requirement:
[' soccer_1', 'wta_l' 'baseball_1', 'sakila_1', formula_1", 'college_2' 2', 'bike_1', 'store_1', 'chinook_1', 'orld_1', "csu_1' 'flight_2', 'wine_1', 'car_1']
'flight_4'
From these, we selected five databases spanning different domains while also diversifying the num- ber of tables and rows. The selected databases summarized in Table
This selection balances domain variety, table count, and row diversity to create a robust evaluation benchmark.
14

---

Published as a conference paper at ICLR 2025
Base DB
Tables
Total Rows
Wine
3
555
Store 1
4
2,548
College_2
9
3,554
Flight_
3
20,989
Soccer_
5
185,608
Table 7: Summary of Selected Databases for HoloBench
A.2.3 EXAMPLES OF DUMMY DOCUMENTS
In cases where query pertains to all tables, resulting in an empty non-relevant table, we write ten dummy documents that not affect any query result for each database. Example dummy documents the following:
soccer_1: Injuries are a common concern soccer, affecting players' careers and team perfor- mance. Proper training, conditioning, and recovery protocols are essential to minimize risks. Teams invest in medical staff and technology to ensure players remain fit and healthy.
wine_1: Wine festivals celebrate the culture and craftsmanship of winemaking. These events often feature tastings, workshops, and opportunities to meet producers, allowing attendees to immerse themselves in the world of wine.
A.3 QUERY
A.3.1 QUERY TYPE
The detailed explanation of each query type is described below:
1. Aggregation: Queries that perform data summarization using aggregate functions such as SUM(). AVG(,, COUNT(), etc. These queries often involve combining multiple rows of data into single summary value, usually with the use of GROUP BY.
Example: SELECT SUM(sales) FROM transactions GROUP BY region
2. Max/Min: Queries that retrieve the maximum or minimum value from dataset. These queries use the MAX() or MIN() functions to find the highest or lowest value in a specified column.
Example: SELECT (AX(salary) FROM employees
3. Join:
Queries that combine rows from two or more tables based on a related column between them. Joins such INNER JOIN, LEFT JOIN, RIGHT JOIN, and FULL JOIN are used to retrieve data from multiple tables in a relational database.
Example: SELECT employees. name, departments name FROM employees INNER JOIN departments ON employees department_id = departments id
4. Comparison: Queries that compare numerical values against condition or threshold using comparison operators like >, <, =, BETWEEN, or IN. These queries filter records that meet specific criteria.
Example: SELECT
FROM products WHERE price BETWEEN 100 AND 200
5. Ranking: Queries that involve ordering results based certain metric, often using ORDER in conjunction with LIMIT to rank data and return only the top or bottom results.
Example: SELECT player_name FROM players ORDER score DESC LIMIT 10
A.3.2 QUERY DIFFICULTY
The definition of our query difficulty is described below:
Components Definition:
SQL Components 1: WHERE, GROUP BY, ORDER BY, JOIN, OR, LIKE, HAVING, BETWEEN
15

---

Published as a conference paper at ICLR 2025
SQL Components 2: EXCEPT, UNION, INTERSECT, NESTED subqueries (including EXISTS, IN, ANY, ALL with subqueries)
Other Complexity Factors:
Number of Aggregates: More than aggregate function (e.g., SUM, COUNT, AVG)
Number of SELECT Columns: More than 2 columns the SELECT clause
Number of WHERE Conditions: More than condition in the WHERE clause
Number of JOIN Clauses: More than JOIN
Number of GROUP BY Columns: More than column in the GROUP BY clause
Difficulty Levels:
1. Easy:
The query uses zero or exactly one keyword from SQL Components
No keywords from SQL Components 2.
Does not satisfy any conditions under Other Complexity Factors.
2. Medium:
The query uses up to two keywords from SQL Components 1.
No keywords from SQL Components 2.
Satisfies more than one condition from Other Complexity Factors.
3. Hard:
The query uses more than two keywords from SQL Components 1.
OR
The query uses one more keywords from SQL Components 2.
OR
Satisfies more than one condition from Other Complexity Factors.
Since our questions are manually derived from SQL queries, more complex SQL queries with ad- ditional conditions result in longer and more intricate questions. To quantify this, we calculated the average word count per question across different levels of SQL query difficulty. The summarized results presented in Table
SQL Difficulty
Average Question Length (words)
Easy
10.8
Medium
13.8
Hard
17.0
Table 8: Average Question Length by SQL Query Difficulty
A.3.3 QUERY EXAMPLES
We give several example queries that are used our experiments.
wine_1:
Give me the average prices of wines that are produced by appellations in Sonoma County.
Question:
Query: SELECT (G((T.Prric) FROM appellations AS T1 JOIN wine AS T2 ON
T1 Appelllat = Appellato WHERE T1 County 'Sonoma'
Type: ['Aggregation', 'Join']
Difficulty: Medium
store_1:
16

---

Published as a conference paper at ICLR 2025
Question: How many tracks are longer than 12 minutes?
Query: SELECT COUNT(DISTINCT name) FROM tracks WHERE seconds 720000
Type: ['Aggregation', 'Comparison']
Difficulty: Easy
college_2:
Question:
Which course is laught in the building with the largest capacity classroom and what is the name of the building?
Query: SELECT course title, section. section.building FROM course JOIN section ON course. title = section. course_title JOIN classroom ON section. section.building busssroombbuildie WHERE classroom. caasrrom.capacity ty (SELECT MAX(capacity) FROM classroom)
Type:
["Max/Min', 'JoinI
Difficulty: Hard
flight_4:
Question:
Which are the top airlines with the most active routes by counting the number routes without codeshare for each airline?
Query: SELECT airline Sarme COUNT (+) AS active_routes FROM routes WHERE
codeshare IS NULL GROUP BY airline_name ORDER BY active_routes DESC LIMIT
3
Type: ['Aggregation', 'Ranking']
Difficulty: Hard
soccer_1:
Question: Which players are shorter than 170 cm?
Query: SELECT player_name FROM Player WHERE height < 170
Type: ("Comparison")
Difficulty: Easy
A.4 PROMPT TEMPLATES
First, we provide a prompt used for our main experiments in Figure g Since ol-mini is reasoning model, we remove the reasoning output part from the prompt for this model. Next, we show prompt used for the "without CoT" setting in Figure(9] Also, Figure 10 shows an evaluation prompt.
A.5 ADDITIONAL EXPERIMENTAL RESULTS
A.5.1 INFORMATION POSITION
To generalize our insights described in Section 4.2.2 we additional conducted experiments for smaller models, GPT-40-mini, Gemini-1.5 Flash, and Llama-3.1-8b, with all information positions and report results in Figure [T] We observe that smaller models also have their preferences of in- formation positions. For example, GPT-4o-mini achieves better performance on "beginning" and "bimodal" positions than other positions. For Gemini-1.5 Flash, "end" and "bimodal" positions better than other positions. Interestingly, Llama-3. 1-bb achieves better performance not only "end" and "bimodal" but also "uniform" positions. Since this preference "uniform" is shared with its larger model Llama-3. 3.15-0 their training process enhance models' information retrieval ability even if information pieces distributed over documents.
17

---

Published as a conference paper at ICLR 2025
You'll be given set of sentences to read through carcfully. Once you' 'ee reviewed them, I'll ask you question related the information those sentences. Your job is think critically about the details, analyze the sentences in relation to the question, and then provide your answer. the information clearly supports a partial answer, provide that. However, if the evidence is unclear or insufficient, it is okay to respond with "No answer."
**Input:"*
**Sentences:**
[context)
***uestton::**
[question]
""Response=**
**Reasoning::*3
[Describe how you thought through the sentences and how they helped you reach your conclusion. the evidence is unclear or insufficient to provide reliable answer, explain why. Your reasoning should not exceed 10,000 words.]
**Answer:** [Provide an answer only if it is clearly supported by the information in the sentences. the evidence is unclear insufficient, respond with "No answer."]
Figure Prompt template.
You'll be given set of sentences to review carefully. Once you't reviewed them, I'll ask you question related to the information in those sentences. Your task to provide a direct, clear answer hased only on what is clearly supported by the information. the evidence does not fully support the answer, provide partial answer. If the information is unclear insufficient, respond with "No answer." Do provide explanations reasoning.
*IIput::*
***entences:**
[context]
***uestion:**
{question]
*Response:**
** * [Provide only the direct final If the information supports only part of the answer, provide partial answer. the evidence unclear or insufficient, respond with "No answer." Do not include reasoning.]
Figure 9: Prompt template for the without CoT setting.
A.5.2 QUERY TYPE
We demonstrate the results of ol-mini, Claude-3.5-Sonnee Llama-3,1-70b. GPT-4o-mini, Gemini- 1.5 Pro, Gemini-1.5 Flash, and Llama-3.1-88 in Figure 12 The experimental settings except models the same those in Figure 4 i.e. information placement. We observe the same trend as Figure [2] i.c., all models achieve better performance Max/Min queries than other queries and Aggregation queries harder task.
the proportional information amount and uniform
18

---

Published as a conference paper at ICLR 2025
A.5.3 QUERY DIFFICULTY LEVEL
We demonstrate the results of ol-mini, Claude-3.--Sonnet, Llama-3,1-70b, GPT-40-mini, Gemini- 1.5 Pro, Gemini-1.5 Flash, and Llama-3,1-8 from the lens of query difficulty levels in Figure [3] We observe the same trend as Figure [] the relative order of difficulty levels remains consistent over context lengths across various models.
A.5.4 QUALITATIVE ANALYSIS
We observe that GPT-4o's behavior varies based on the context length provided, especially for ag- gregation queries, e.g., "Find the destination airport and its country that receives flights from the highest number of distinct source airports, and include that number in the result.". At 4k context length (Figure l4p. GPT-4o delivers an exact match to the gold answer, correctly identifying "Kazan International Airport, Russia" with 7 distinct source airports. The concise context enables the model to effectively parse and enumerate the relevant information without error, providing an accurate re- sponse. As the context length increases to 8k (Figure 15 GPT-4o begins to struggle with correctly retrieving and aggregating the information. It returns of airports, representing partial match. This is due to the limitation of models' retrieval ability over long context. This highlights a potential issue in the model's ability to maintain consistency in rea- soning context size increases, impacting its accuracy. When the context reaches 64k (Figure [6), the performance further deteriorates. GPT-4o produces substantial error by incorrectly stating that Kazan International Airport has 10 distinct source airports, which fails to match the gold answer. Interestingly, unlike on 4k and 8k, it does not show the contents of its analysis over the sentence after describing a solution to the question.
correct airport but miscounts the number
A.5.5 RAG COMPARISON
We conducted additional experiments to explore how different retrieval settings affect model perfor- mance.
First, we varied retrieval sizes and analyzed their impact accuracy. We evaluated scenarios where half of a 64k context (32k tokens) contained query-relevant information and tested retrieval sizes of 2k, 4k, 8k, 16k, and 32k using the Llama-3.1 3.1-405b model. As shown in Table performance peaked when the retrieval size (32k) matched the amount of relevant information, highlighting the benefit of minimizing irrelevant context enhance reasoning.
# Retrieved Tokens
2k
4k
8k
16k
32k
64k (LCLMs with full context)
RAG
21.70
25.28
27.02
34.91
37.34
30.18
Table 9: Accuracy with Information Density = 0.5
In real-world scenarios, the density of relevant information is often unknown. To study this, we conducted experiments where all 64k tokens in the context were query-relevant. Note that absolute performance values are not directly comparable between scenarios with information density 0.5 and 1.0 due to differing answer distributions. As shown in Table TO using all 64k tokens yielded the highest performance, but the improvement over a 32k retrieval was marginal. This indicates that current LLMs struggle with ultra-long contexts, revealing significant opportunities to enhance long-context reasoning.
# Retrieved Tokens
2k
4k
8k
16k
32k
64k (LCLMs with full context)
RAG
14.85
19.18
21.79
29.36
38.69
39.06
Table 10: Accuracy with Information Density = 1.0
We examined whether RAG performs optimally when the distribution of relevant information is known. In scenario with 32k relevant tokens within 64k context, using the bge-large-en-vi retrieval system yielded the following metrics: Precision: 83.92%, Recall: 80.89%. These findings demonstrate that significant retrieval errors persist even under ideal conditions.
19

---

Published as a conference paper at ICLR 2025
To summarize, RAG outperforms LCLMs when the retrieval size matches the relevant information. However, this alignment rarely achievable in practical scenarios. Even with advanced retrieval models, retrieval emrors remain significant highlighting the difficulty of accurate information lection. Addressing long-context reasoning requires developing adaptive retrieval mechanisms that adjust dynamically varying information densities enhancing I.CLMs' ability process and reason over ultra-long contexts effectively. Simply increasing retrieval size or context length is insufficient and substantial improvements both retrieval and model reasoning capabilities are sential for progress.
20

---

Published as a conference paper at ICLR 2025
You will be given a question along with response generated by an assistant and the corresponding ground truth data. Your task is to assess the response based on its accuracy and completeness in comparison the ground truth. For each entry in the ground truth, determine whether the information provided by the assistant is an "Exact Match," "Partial Match," a "No Match."
#i# =*Ealuatiio Criteria:**
**Exact Match**; The assistant's response precisely matches the ground truth in both content and
detail.
**arttill Match*s; The assistant's response includes some correct information but either incomplete, incorrectly ordered, or contains inaccuracies.
No Match+*;; The assistant's response does not accurately reflect the ground truth is missing entirely.
##### "*Special Cases:
**Ground Truth is None**
If the ground truth is 'None' (represented empty list '[I"):
** Exxact Match*#; the assistant's 's response indicates that there is information content.
* *N Match**; If the assistant's response provides any information when the ground is 'None..
##### #*Uutput #ormat:
output should a of objects where each object contains:
An 'iid" that matches the 'id' the corresponding ground truth entry.
A '"label" indicating whether the assistant's 's response is an ""Exact Match" '"Partial Match"
"No Match"
The number oulput objects should match the number entries in ground truth.
**Exmppes:***
[Several Examples)
Your task starts here
*Question:
{question)
AAssistant's Response:**
[pred)
*Groundd Truth:*
(gold)
...
O*utput ss
foutput_forman
Figure 10: Evaluation prompt.
21

---

Published as a conference paper at ICLR 2025
Unitorm
51.5
15.6
37.8
35.0
26.3
39.2
Unitorm
36.8
30.7
25.8
27.7
20.5
28.3
Beginning
50.7
49.7
42.0
37.1
27.8
43.1
Beginning
39.5
36.5
28.1
22.4
21.7
29.6
End
End
Bimodal
58.8
19.5
40.0
35.9
29.4
42.7
Bimodal
40.1
37.1
32.3
26.4
24.0
32.0
8k
16k
32k
64k
Avg.
8k
16k
32k
64k
Avg.
Context Length
Context Length
(a) GPT-4o-mini
(b) Gemini- 1.5 Flash
Uniform
Beginning
29.9
10.9
1.1
14.6
Middle
33.0
8.3
1.1
14.5
End
34.2
32.5
18.6
7.7
1.1
18.8
Bimodal
40.4
27.9
17.8
6.3
1.9
18.8
16k
32k
Avg.
Context Length
(c) 1.amm--3.1-88
Figure 11: Results of smaller models on various information positions.
22

---

Published as a conference paper at ICLR 2025
el-mini
Claude-3.5 3.5
100
oo Maxthh
Agg opatinn
Ma
80
80
Comparkon
Compar
Accuracy
60
Rencing
Accuracy
60
Rankinng
40
40
20
20
4Kc
8K
16k
32k
64k
4K
8k
16k
32k
64k
Context Length
Context Length
(a) ol -mmii
(b) Claude-3.5-Sonee
-l2-a-3,1-70b
GPT-4o-mini
100
regation Max/hh
Aggregation Ma
80
80
Compar
Accuracy
Rancing
Accuracy
60
60
40
40
20
20
4k
8k
16k
32k
64k
4k
8k
16k
32k
64k
Context Length
Context Length
(c) Llama- -.1---3.1-0bb
(d) GPT-4o-mini
Gemini-1.5 Pro
Gemini-1.5 Flash
100
regation MaxWii
Appregation MaxNin
80
80
Comparson Rancing
Comparann
Accuracy
Accuracy
Rank
60
60
40
40
20
20
4k
8k
16k
32k
64k
4k
8k
16k
32k
64k
Context Length
Context Length
(e) Gemini- 1.5 Pro
(l) Gemini- 1.5 Flash
Llama-311-8b
100
Apgregstion
MawMin
Jan
Accuracy
Rankinng
60
40
20
4k
8k
16k
32k
64k
Context Length
(g) Llama-3.1-8h
Figure 12: Performance of various models with proportional information amount across different query types.
23

---

Published as a conference paper at ICLR 2025
80
Easy
Modium Hard
Easy
Medium
Hard
Accuracy
Accuracy
60
40
40
4k
8k
16k
32k
64k
4k
8k
16k
32k
64k
Context Length
Context Length
(a) ol-mini
(b) Claude- 3.5- Sonnett
Easy
Easy
Accuracy
60
Accuracy
40
20
4k
8k
16k
32k
64k
4k
16k
32k
64k
Context Length
Context Length
(c) Llama3.1-700
(d) GPT-4o-mini
Easy
Medium
Hard
60
Accuracy
Accuracy
40
20
10
4k
8k
32k
64k
4k
16k Context Length
32k
64k
Context Length
(c) Gemini-1.5 Pro
(f) Gemini-1.5 1.5 Flash
Medium
50
Accuracy
40
30
20
10
Context Length
(g) Llama-3.1
Figure 13: Performance of various models with proportional information amount across different difficulty levels.
24

---

Published as a conference paper at ICLR 2025
Question:
Find the destination airport and its country that receives flights from the highest number of distinct airports, and include that number in the resuli.
"Reasoning:** ... I need to analyze the provided sentences and identify all the flights mentioned, **Kazan International Airport, Russia:* Source Airports: Sochi International Airport
Koltsovo Airport- Number of distinct source airports: 7
... 10.. Yakutskk Airport, Russia:**... Number of distinct source airports:
After analyzing the data, Kazan International Airport in Russia receives flights from the highest num- of distinct source airports, with
of distinct source airports.
Answer:**
Kazan International Airport, Russia; 7 distinct source airports. [Exact Match
Gold Answer: "Kazan International Airport, "Russia", 7 distinct source airports
GPT-4o's 's output with 4k context. An evaluation score is shown after the model output and a gold answer is also provided below the dashed line. GPT-4o accurately list all relevant information and then provide correct answer.
Figure 14:
""Reasoning:** ... reviewed the sentences for flight routes mentioned ... The destination airport that appeared most frequently with different source airports was Kazan International Airport in Russia. It received flights from the following distinct source airports: Sochi International Airport Astrakhan Airport- Belgorod International Airport- Chelyabinsk Balandino Airport Domodedova International Airport Pulkovo Airport...
*nswer: Kazan International Airport, Russia; 6 distinct source airports. [Partial Match]
Gold Answer: "Kazan International Airport, "Russia", 7 distinct source airports
GPT-4o's output with 8k context. It retrieves most relevant information but misses one of relevant airports, resulting Partial Match.
Figure 15:
**Reasoning:**: +* ... Ined analyze the provided sentences and identify the destination airports along with their corresponding source airports. ...
***Identify Destination Airports:** will go through the sentences to extract the destination
2. **Track Source Airports:** For each destination airport, will keep track of the source airports
6. ** s*Provide the Result:** The final result will include the destination airport,
** *nswer::** After analyzing the sentences, ... ** **Kazan International Airpor in **Russia** with flights arriving from 10 distinct source airports. [No Match]
Gold Answer:
"Chengdu Shuangliu International Airport, "China", distinct source airports
Figure 16: GPT-4o's 's output with 64k context. Instead of analyzing the context as does with 4k and 8k contexts, it attempts answer the question directly, resulting in incorrect answer.
25


Reviews:
[{"review_id": "IUk141vszU", "reviewer": "Reviewer_KK9H", "summary": "This paper presents HoloBench, a benchmark designed to evaluate the holistic reasoning capabilities of Long-Context Language Models (LCLMs). While Retrieval-Augmented Generators (RAGs) effectively access information from large document collections, they struggle with complex tasks that require aggregation and reasoning across multiple documents. The authors refer to this as holistic reasoning, for which long-context language models have been developed. HoloBench aims to evaluate LCLMs against RAGs under various conditions.\n\nAlthough similar datasets have been created in the past, this dataset is more comprehensive in terms of context length, information density, information distribution, and query complexity, and it includes automated evaluation methods.", "questions": "How do you differentiate between long but simple questions and genuinely complex ones?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The code has been released, and the data is reproducible. I believe the authors are providing a valuable service to the community.", "weaknesses": "From a technical standpoint, this method is relatively straightforward.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I82YmGhVof", "reviewer": "Reviewer_53mu", "summary": "The authors propose a new comprehensive benchmark to test the reasoning capabilities of LCLM across multiple dimensions. The authors unveil a range of novel insights from this benchmark, by comparing the performance of recent LCLM on diverse scenario (position of the information, density of information, complexity of the query).", "questions": "/", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The authors :\n- Proposed a new, original, comprehensive and timely benchmark dataset and framework based on text-to-sql for consistent long context LM benchmarking\n- Important benchmark dataset and frameworks, showing novelty into aspects of LCLM seemingly not thoroughly explored yet\n- Identified a range of novel and important insights about LCLM reasoning (lost in the middle comparison, model size, RAG comparison, query complexity effect, CoT importance)\n- Tested recent LCLM (Llama 3.1, GPT4o, Claude 3.5, Gemini 1.5)\n- Used LLM-based evaluation metrics, but which showed 93.8% agreement with human judgements\n- Good paper layout to quickly identify important insights derived from the benchmark\n- Good discussion points on RAG contribution", "weaknesses": "- Minor: \"Information density\" concept could be further detailed", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uLLwMPApeO", "reviewer": "Reviewer_p5kH", "summary": "The paper introduces HoloBench, a benchmark aimed at evaluating the holistic reasoning capabilities of Long-Context Language Models (LCLMs) when handling multi-document contexts. The framework aims to systematically assess LCLMs by controlling factors like context length, information density, distribution, and query complexity. The study uses a text-to-SQL dataset and corresponding databases to generate dataset for evaluating long-text language models. Based on proposed benchmark, the study shows that the quantity of information within the context more significantly impacts model performance than the pure length of the context. The findings provide insights into the strengths and limitations of LCLMs in processing large-scale text.", "questions": "Please answer the concerns in the above weaknesses section and the questions below:\n\n1. Selection of databases:\nAccording to reference [1], Spider consists of 200 different databases, yet as stated in line 214 of the paper, HoloBench only employs five databases from it. What was the reason behind selecting these specific five databases? Would the findings apply consistently across other databases and other scenarios as well?\n[1] Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task\n\n\n2. Potential information leakage:\nThe benchmark is constructed based on a publicly released text-to-SQL dataset from 2018, which might have already been included in the pre-training corpus of existing large language models. This could lead to potential information leakage, where LLMs might arrive at correct answers simply due to prior exposure to the data. Could the authors address this concern?\n\n3. Comparative analysis using long-context models:\nIn Section 4.3, the authors utilize a document retriever that handles only 2k tokens while experimenting with data ranging from 4k to 64k tokens. This undermines the reliability of the performance comparison between LCLM and RAG. The authors should employ a RAG model that supports long-context processing to conduct this experiment accurately.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.The paper offers a reasonable definition of the holistic reasoning capability of LLMs. The motivation of employing a text-to-SQL dataset to construct an evaluation framework for the holistic reasoning capabilities of Long-Context Language Models is reasonable.\n2.The methodologies of benchmark construction are reasonable, for example, constructing data with varying difficulty levels and types based on the inherent difficulty classifications of the text-to-SQL dataset and the types of queries.\n3.Given that the extractable information from databases can be freely controlled, the proposed Automated and Scalable Evaluation is practical.\n4.The paper is written in a concise and coherent manner, offering illustrative figures that facilitate comprehension. The experiment results are clearly presented and easy to read.", "weaknesses": "1. Limited benchmark size:\nAccording to Section 3.3, HoloBench consists of only 90 questions, which is significantly smaller in scale compared to other benchmarks like BABI Long and NOCHA. The small size of the benchmark makes it susceptible to randomness, compromising the reliability of the evaluation results.\n\n2. Lack of diversity and practicality:\nSince each database table stores data of a similar type, the data constructed by HoloBench tends to be highly homogeneous and confined to a single scenario dictated by the original database. Compared to other benchmarks that use richly varied books as a base corpus for long-context tasks, HoloBench, built from individual databases within a single text-to-SQL dataset, lacks diversity and richness, offering limited scenarios.\n\n3. Limited challenge:\nIn HoloBench, rows from relevant and irrelevant tables are merged as relevant and irrelevant contexts, respectively. However, data stored in different tables of a database typically exhibit noticeable differences, allowing LLMs to potentially distinguish between relevant and irrelevant contexts with ease. This suggests that the tasks provided by HoloBench may not be sufficiently challenging.\n\n4. Evaluation fairness and reliability:\nHoloBench uses GPT-4-mini as its evaluation metric, which leads to potential uncontrollability and unfairness in evaluation. The human judgm", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LqvQdPXkzt", "reviewer": "Reviewer_Epz7", "summary": "This paper argues that existing RAG systems struggle with complex tasks that require aggregation and reasoning over information spanning across multiple documents, while Long-Context language models may have the potential to this task but there is no study clearly show that. This paper proposed a new framework called HoloBench, which used database operations to generate context to test whether LCLMs can answer the target type of questions. \nThe experiments indicate that the amount of information in the context has a bigger influence on LCLM performance than the actual context length as well as the complexity of queries.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. It is great to see a set of design principles in a paper that is to propose a benchmarking framework.\n\n2. The paper did a comprehensive study of experiments and found interesting observations about factors, such as information amount, context length, information positions and query types and complexity to query performance.", "weaknesses": "1. The reality and the quality of the corpus/context is unclear. The author decided to generate the context by verbalizing data rows, which may lead to a set of unrealistic documents/passages with a simple description of attribute values in each row. Furthermore, there is no study in this paper that can show that the verbalization process is faithful to the original tables/rows.\n\n2. The discussion about RAG and LCLMs are flawed. It does not make sense at all that limited RAG only retrieving 2k tokens and compared it with a language model that can be fed with 4k to 16k context.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)