NeedleInATable: Exploring Long-Context Capability of Large Language Models towards Long-Structured Tables
Authors: Lanrui Wang
Mingyu Zheng
Hongyin Tang
Zheng Lin
Yanan Cao
Jingang Wang
Xunliang Cai
Weiping Wang
Venue: NeurIPS 2025 poster ()
Keywords: Long-context
table understanding
structured data
Primary area: applications

Abstract: Processing structured tabular data, particularly large and lengthy tables, constitutes a fundamental yet challenging task for large language models (LLMs). However, existing long-context benchmarks like Needle-in-a-Haystack primarily focus on unstructured text, neglecting the challenge of diverse structured tables. Meanwhile, previous tabular benchmarks mainly consider downstream tasks that require high-level reasoning abilities, and overlook models' underlying fine-grained perception of individual table cells, which is crucial for practical and robust LLM-based table applications. To address this gap, we introduce \textsc{NeedleInATable} (NIAT), a new long-context tabular benchmark that treats each table cell as a ``needle'' and requires models to extract the target cell based on cell locations or lookup questions. Our comprehensive evaluation of various LLMs and multimodal LLMs reveals a substantial performance gap between popular downstream tabular tasks and the simpler NIAT task, suggesting that they may rely on dataset-specific correlations or shortcuts to obtain better benchmark results but lack truly robust long-context understanding towards structured tables. Furthermore, we demonstrate that using synthesized NIAT training data can effectively improve performance on both NIAT task and downstream tabular tasks, which validates the importance of NIAT capability for LLMs' genuine table understanding ability. Our data, code and models will be released to facilitate future research.

NEEDLEINATABLE: Exploring Long-Context Capability of Large Language Models towards Long-Structured Tables
Lanrui Wang"*i Mingyu Zheng!.2 Hongyin Tang Zheng Lin Yanan Cao 1,2, Jingang Wang") Xunliang Cai²; Weiping Wang'
IIstitttee of Information Engineering, Chinese Academy Sciences School of Cyber Security, University of Chinese Academy of Sciences Meituan
oliveerwangtencent com
(zhengmingyu, 1in.hengg@eee.eeeec.... tanghongyinneelttuun.
Abstract
Processing structured tabular data, particularly large and lengthy tables, constitutes a fundamental yet challenging task for large language models (LLMs). However, existing long-context benchmarks like Needle-in-a-Haystack primarily focus on unstructured text, neglecting the challenge of diverse structured tables. Meanwhile, previous tabular benchmarks mainly consider downstream tasks that require high- level reasoning abilities, and overlook models' underlying fine- ane-grained perception of individual table cells, which is crucial for practical and robust L-MM-based table applications. To address this gap, we introduce NEEDL FINATABLE BLEE (NIAT), a new long-contex, tabular benchmark that treats each table cell as "needle" and requires models to extract the target cell based on cell locations or lookup questions. Our comprehensive evaluation of various LLMs and multimodal LLMs reveals a substantial performance gap between popular downstream tabular tasks and the simpler NIAT task, suggesting that they may rely on dataset-specific correlations shortcuts obtain better benchmark results but lack truly robust long-com long-context understanding towards structured tables. Furthermore, we demonstrate that using synthesized NIAT training data effectively improve performance on both NIAT task and downstream tabular tasks, which validates the importance NIAT capability for LLMs' genuine table understanding ability.
1
Introduction
The long-contex: modeling ability has laid the foundation for a wide range of LLM-based applications, such as lifelong conversational chatbots, document analysis tools, and advanced agent-based systems Correspondingly. lots of effort has been dedicated to scaling up the context window of L.Ms, improving their proficiency in handling extremely long input texts 5.9 parallel, various benchmarks like beedle-n---HHasttad, (NIAH) [10] have been developed to evaluate the long-context capabilities LLMs from diverse perspectives, such as extracting key information from long texts, answering questions about extended passages or multiple documents 11
14
Equal contribution. Work was done when Lanrui Wang was graduate student of IIE, CAS.
Joint work with Meituan
Zheng Lin is the corresponding author.
4 'Our code and are available at:
In InArable
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Inpui C Codtect
Imput Carbeo
Input Contect
Rell oul theeche modil traly undervianal wach cill table like
Oase
woten rersed
there lived
Bate
Oppnrel
Adka
Reat
20
All dawe
hnman?
The thing do
ked bmsgght moch hugprren ielo bas lif..
francisco to sandwrich
City
10
Cell L Locating
38 Ak
Libeda
What the ool value la the 3Paw snd. 1P colerez?
Teputt Queediin
Inputt O arslive
Inpurt Questtion
What the last thing San Prenokecs?
little identt sub that best
Caption: wallking
Whst tie lotil nembe of ataeelonos (uating this sween?
Cell Lookep
What's Augurt 15907
The di 20
LUM
molches provided cxption
What at doncee nemberr Baethda 20 agaa 1960?
Orggett Respinee
MLLMM
Onippt Response
Looks
Acconting to the proeoied information the bei thing todo Sas Frencico ls sad a sandwrich...om
We roed to add the streciine
The asest crutcchigg sull sub-irnage the 8th sub-iisage becasse
Oulput Respatee
atbendares - 43,7% 20,011 21,07 #89,554
rumber fine aach meichh Total
voler Priee Puokaa "4-001"
(MLLUM
Therseat of 19931e 1-4
(a) Needle in Heystack
(hi multi Nordle in Haystack
(c) Traditional Tabular Benchmadks
(6) Neadle Table (0 urs)
Figure 1: Comparison of previous long-context benchmarks, tabular benchmarks and the proposed NIAT benchmark. Existing long-context benchmarks overlook the structured tabular data, while traditional tabular benchmarks mainly focus on high-level complex reasoning ability. Both of them ignore the model's basic fine-grained comprehension of individual cells the table context.
Beyond lengthy unstructured textual data, structured tables also play critical role in real-world long-context scenarios. Featured with unique two-dimensional structures and diverse formats, tables are widely used in different domains to organize and present information, ranging from financial documents, scientific research to medical records [16] Moreover, LLM- based chatbot users often feed serialized tables in different formats like Markdown into LLMs context windows to perform various table understanding tasks such as table question answering (TQA) and table fact verification (TFV) 19 20 Consequently, the lengths of these table-related task requests often exceed that of common instructions with a few sentences, presenting a distinct long-text challenge for LLMs.
However, existing long-context benchmarks primarily focus on evaluating the ability of LLMs to comprehend long unstructured text, overlooking in-depth exploration of long-context scenarios of structured tables. On the other hand, previous tabular LLM studies mainly assess LLMs' proficiency with traditional benchmarks of downstream tabular tasks that demand high-level reasoning abilities based on partial table information, but neglect the importance of pressure testing LLMs' underlying fine-grained understanding of each individual table cell. This is crucial for practical -baed table applications as real-world users could input questions about any table cells. Besides, it can help determine whether we have developed a truly robust table understanding ability or if we are barking up the wrong tree by overly pursuing the model performance on specific benchmarks.
To bridge this gap. we introduce a new task and construct benchmark based on publicly available tables, termed NTEDLEINATABLE (NIAT), As illustrated in Figure I unlike previous long-context benchmarks and tabular benchmarks, the proposed NIAT task treats each table cell as "needle" and requires the model to extract the target cell according to two types of questions, respectively. The cell-locating questions ask the model to locate the specific cell at the given position, and the cell-lookup questions require the model to retrieve the answer cell based on the information lookup demand. Our benchmark contains 750 tables and up to 287K questions total, covering three table structures (flat, horizontal and hierarchical tables), three table formats (Markdown, HTML. and image) and diverse table sizes (fixed and arbitrary).
With the help of NIAT benchmark, we evaluate a wide spectrum of LLMs and MLLMs including mainstream open-source models and close-sourced GPT-4o. These models demonstrate varying performance on the NIAT task and they obtain much better results on cell-lookup questions than cell- locating questions. This shows that current LLMs can retrieve answer cells from input tables using the attention mechanism, but they struggle in interpreting the basic two-dimensional table structures in a human-like perspective. More importantly, our thorough evaluation reveals a substantial performance gap between more complex downstream tabular tasks (like TQA and TFV) and the NIAT task, suggesting that they may rely on dataset-specific correlations or shortcuts to obtain better benchmark results but lack truly robust long-context understanding towards structured tabular data.

---

Furthermore, we explore whether enhancing NIAT capabilities can fundamentally improve the overall table understanding ability, To this end, we propose a strong2weak data synthesis method to create NIAT fine-tuning data. Specifically, we first utilize the in-context learning with GPT-4o to generate various NIAT task queries based on tables from training splits, and then synthesize detailed chain-of-thought (CoT) reasoning processes as target responses, which instruct weak LLMss to progressively understand structured tabular data. Experimental results with Llama3. Instruct and Qwen2.5-7B-Ine
demonstrate that our synthesized 12K NIAT training data can not only improve models' long-context performance on the NIAT task, but also achieve a significant performance gain (14.55%f in average accuracy) on four downstream benchmarks (WTQ, TabFact, HiTab and TABMWP), outperforming strong baselines such as long-conten context LLMs and specialist tabular LLMs, which further underscores the importance of NIAT capabilities. We hope that this work could foster advancements in both long-context L.M and tabular L.M communities, facilitating the development of L.MMs with robust understanding of complex and lengthy structured tables.
We conclude our contributions as follows:
Targeted at the limitations of existing benchmarks for long-conted and table understanding, we introduce a new task named NIAT and construct the first benchmark for evaluating LLMs' long-context table understanding ability.
On this basis, we conduct a thorough evaluation of various types of LLMs and MLLMs, revealing their shortcomings in long-context understanding of lengthy tables and providing valuable insights.
We propose simple yet effective data synthesis method to enhance LLMS'' long-table com- prehension capabilities, which also significantly improves their performance on downstream table-related tasks, outperforming recent state-of-the-an baselines.
2
Related Work
Long Context LLMs and Benchmarks. Recently, the long-context capabilities of LLMs have gamered significant attention in the research community. Numerous efforts have focused on enhancing positional embeddings to develop more powerful long-context LLMs (21H24), Extensive studies have also explored and evaluated LLMs' ability understand long-context data across various modalities and generate long-form content [27 28 To evaluate the long-context capabilities of large language models (LLMs) in the text modality, benchmarks such as RULER [5], Infini-Bench [Л]. and LongBench [T] have introduced diverse long-con context understanding tasks across varying context lengths, providing a comprehensive evaluation framework for developing robust long-context LLMs. However, these benchmarks primarily focus on unstructured text, largely overlooking structured table data. In this paper, we address this gap by proposing the NIAT tasks.
25 26 3
Notably, a concurrent long-context benchmark, LongBench-v2 [12, evaluates the high-level reason- ing ability of LLMs over long structured tables using 18 question-answeriinn samples. In contrast, our work focuses on assessing LLMs' comprehension of every individual cell within the global context of long tables, making it complementary LongBench-v2.
Table Collection
Query Construction
Diverse Table Fermats
Necdle-in-
Structures
Naee
CellLecating
Task
Predefined Teapline
Quarias
Taol
Quary
Ausiver
Mackdann
sdectal celb
WTO mitab AIT-QA
Flat
Harienetal
GITT-to
selidad salt
HTMMI
Querio
Ilatsechical
Lookep
Gencrehol Auswes
Table sampling
Imsgee
Ansoard
Format Conversion
Queey Generatian
Scffsomistency Amwer Vating
Figure 2: The construction pipeline of Needle-in-a- T-ble Benchmark.
Tabular L.MSs and Benchmarks. Recent years have witnessed paradigm shift to developing Tabular LLMs for table understanding, where different strategies have been applied to enhance LLMs'
3

---

ability to fulfill tabular tasks, such as prompt engineering [29 30 and instruction tuning [1 34]. Moreover, a series of benchmarks have been constructed to evaluate LLMs' proficiency table understanding tasks within diverse scenarios 33 Nonetheless, these benchmarks primarily focus on assessing L.LMs' high-level table-based reasoning ability with complex tasks as testbeds, such as table question answering and aable-basdd fact verification, and fail to evaluate the fine-graines understanding of each input table cell from the long-context perspective, which is the foundation for humans to comprehend tables and perform more complex tasks. To this end, we propose the NIAT benchmark to address this issue and provide a comprehensive landscape of LLMs' table understanding capacity over the whole input table.
3
NEEDLEINATABLE Benchmark
3.1
Benchmark Construction
3.1.1 Table Collection
To fill the gap in existing benchmarks and provide the community with the first long-context bench- mark towards structured tables, it is most cost-effective to build such a benchmark based on tables from publicly aviable datasets. Moreover, real-world tables possess various structures, ranging from simple flat tables to hicrarchical tables with multi-level headers and merged cells [16 38 39] With
these considerations in mind, we collect tables of three different structures from existing tabular benchmarks. (1) Flat tables: we randomly select 250 tables from WTQ benchmark [9] which contains flat tables from Wikipedia. (2) Hierarchical tables: we randomly select 220 tables and 30
tables from HiTab [4) and AIT-QA [41] benchmarks respectively. which contain diverse hierarchical tables from Wikipedia, statistic reports, and annual reports of airline companies. (3) Horizontal tables: we write Python scripts to transpose flat tables in WTQ so that the first column serves as row headers, resulting in 250 horizontal tables.
Another unique feature of structured tables lies in the diversity of table representation formats 30 42 Chatbot users may input tables in different formats into LLMs' context windows, such as Markdown, HTML, csv, Latex, and even image format. Therefore, we carefully design Python scripts to transform the collected tables into three commonly used formats including Markdown, HTML and image formats. Table sequences in Markdown and HTML formats are used to evaluate textual LLMs. For image format, we render tabular data into images following the previous work (34), and the resulting table images serve as inputs for multimodal LLMs to evalute their NIAT capabilities.
3.1.2 Query Construction
Despite the remarkable performance of LLMs complex tabular benchmarks, even surpassing human levels, does this really represent that LLMs possess the same robust long-context table understanding abilitics as humans? To answer this question, take step back to the fundamental or atomic ability of humuan-like table comprehension, focusing on fine- fine-grained perception of individual table cells. Concretely, "needle", and design two types of NIAT queries that are simple and trivial for humans but constitute the foundation of more complex based tasks.
treat the input table as the context, the single table cell the target
Cell Locating queries require the model to extract the cell content according to the given cell location (i.e., row index and column index), which aims at evaluating the understanding of basic wwo-dimensional table structures. An example query could be "What is the cell value in the 3rd row and 4th column".. If the model has correct understanding of the basic table structure, it should be easy to locate the target cell by enumerating along the row and column directions.
Cell Lookup queries simple lookup questions where answers specific table cells and do not need any further aggregate operations (such as min/max/sum/coont) or multi-step reasoning. Using the table in Figure aas an example, multiple lookup questions can be raised based on different table cells, e.g.. "What is the result on 20 August 1960". This task requires the model to use the keywords in the questions cues and retrieve the answer cell from the table by intersecting the and the column with overlapped keywords.
To save the cost of model inference, we randomly select 60% table cells from each table as target cells to create NIAT queries. Since the cell locating task only takes the table and the target cell location as input, we fill in the pre-defined prompt template with selected row indices and column

---

Table 1: Performance of LLMs NIAT benchmark. 'Flat', 'Hori' and 'Hie.' denote flat, horizontal and hierarchical tables, respectively. 'MD.' and 'HT.' indicates tables in Markdown and HTML formats. The best results in each category are highlighted in bold.
Cell-Locating
Cell-Lookup
Over
Model
Flat
Hor.
Hie.
Flat
Hor.
Hic.
Ave
Ave
MD.
HT.
MD.
HT.
MD.
HT.
MD.
HT.
MD.
HT.
MD.
HT.
Open-source LLMs
Mistral-7BB-nsstuu1--..33
7.75
6.79
4.35
6.33
2.11
4.60
33.72
38.79
27.99 27.55
25.62
33.79
31.98
18.29
Deepseck-Iim-7B
2.88
0.16
2.31
2.03
1.13
0.70
1.54
47.46
19.37
32.21
27.03
38.29
26.05
31.74 16.64
MiniCPM-3-4B
1.48
2.01
0.87
1.57
0.73
0.40
1.18
64.60
66.51
51.54 49.65
59.12
57.38
58.13 29.66
IntemLMM2.57B-chht
1.36
1.36
1.05
1.34
0.54
0.09
0.96
36.31
37.34
27.85
27.03
43.15
39.89
35.26 18.11
Yi-1.5-9B-chat-Be
7.29
5.36
4.31
7.45
1.77
0.26
4.41
45.26 52.81
43.14
44,26
(4.31
54,52
49.05 26.73
GLM--49B-chat
9.76
5.76
7.32
2.45
6.74
55.09
48.48
42.40
36.89
61.20
50.98
49.17 27.96
Qwen2.5-7B-Instne
16.24 13.64
7.63
15.85
2.90
0.49
9.46
44.80
48.20
38.92
42.39
56.40
54,90
47.60 28.53
Oween2,5-14--Instrure
Inssruutttt
19.18
18.28
11.41
26.27
4.69
0.39
13.37
42.73
36.80
38.29
36.46
56.16
51.84
43.71 28.54
Llama3.1-88 1-8B--Isstruct
10.38
8.75
7.18
7.70
2.54
0.38
6.16
67.30
65.10
57.10
66.25
72.50
66.20
65.74 35,95
Qwen3-14B
17.6 18.73
18.41
23.81
4.33
0.81
13.95
74.41
70.76
64,14
62.2
72.79
79.59 70.65 42,30
Qwen3-32B
22.41
22.6
21.32
31.05
5.77
17.39
76.6
71.24
69.23
63.8
80.6
45.48
Qwen3-30B-A3B
23.17
20.41
24.12
26.2
3.65
1.39
16.49
78.87
78.83
72.2
69.8
85.41
86.59
78.62 47.55
Tabular LLMs
StruciLLM
2.00
2.10
1.30
4.20
1,20
0.10
1.82
58.65
46.70
25.88 85
29.65
51.85
43.55
42.72 22.27
TableGPT2
12.67
12.83
6.32
17.52
3.14
0.52
8.84
77.02
71.31
61.03
71.64
80.73
81.50
73.87 41.36
Reasoning LLMy and GPT-4o
GPT-4o
38.50
29.70
36.50
39.40
10.40
1.50
26.00
61.50
63.80
60.30
59.50
83.60
10
68.30 47.15
(wen-QQQ-32B
63.76
54.40
36.36
52.29
3.90
1.80
35.42
48.78
45.21
45.14
45.84
56.55
56.90
49.74
42.58
DeepSeek- RI
76.20 79.50 75.46
82.30
72.39
9.60
65.91
81.00
77.82
79.19 77.50
85.60
84,80
80.99 73.45
indices create final input queries. For cell lookup task, we provide GPT-4o with the table and the selected target cell, and utilize in- mn-conexxt learning with GPT-4o to generate corresponding lookup questions. We add JSON output requirements prompts of two tasks minimize errors during answer parsing. To guarantee the validity of synthesized cell lookup questions, we use GPT-4o with self-consistency 43) answer these questions and filter out questions whose answers do not align with the target cells. Finally, we obtain 142K cell-lookup queries and 145K cell-locating queries.
3.2 Benchmark Statistics
We have developed NEEDLEINATARE comprising 750 table and 287K test cases, with an average of 382.67 test cases per table. 80% tables have a row size ranging from I to 40, and a column size spaning from to 23. 80% test data have prompt length within 4.2K, with the maximum length up to 120K. For tables in image format, we randomly select 120 tables from each structure and their corresponding queries as test data for multimodal LLMs. The key statistics about collected tables and constructed queries are shown Tables 5 and such as average row number and column number, average input length, and on. Traditional tabular benchmarks mainly evaluate models' complex tabular reasoning with limited test cases, which can only span a small number of table cells. By contrast, the NIAT benchmark evaluates the underlying fine-grained perception of individual table cells with much more test cases, covering various table structures, formats and sizes. Detailed information about NIAT benchmark shown in Appendix
3.3 Evaluation and Analysis
Baselines. We evaluate models across four categories: (1) Generalist LLMs and MLLMs, such as
Mistral-7B- Instrruttttt- [44], Llama3. 1--.1118B--nstie [ and Intern VL-2.558-225-88 45). (2) Tabular
LLMs and MILMM that are fine-tuned with table instruction-tuninn data, including StructLLM (46).
TableGPT2 33 and Table-LLaa 34 (3) Recent Reasoning L.LMs including DeepSeek-RI R1 47
QwQ-32B 48 with enhanced reasoning ability derived from reinforcement learning. (4) Close-
sourced GPT-4o. Considering the high cost of GPT-4o and DeepSeck-Ri R1 API, for tables in Markdown,
5

---

HTML and image formats, we sample 3K test samples of two types of NIAT tasks, respectively. For all baselines, we adopt the zero-shot setting during evaluation.
Performance of LLMs. From the results in Table 1 we have the following findings. (1) Different open-source L.Mss demonstrate varying performance NIAT tasks, with Qwen2.5 family performing best on cell-locating task and Llama3. 1--B--Insttee on cell-lookup task. The competitive performance of Qwen2.5 series models on both NIAT tasks could be stemmed from the specially collected table- related post-training data for enhancing table understanding capability [4]] MiniCPM3-4P also obtains strong performance on cell- cell-lookup task and even greatly surpasses some LLMs with larger sizes, showcasing the potential for developing efficient LLMs with great table understanding ability,
(2) Existing LLMs and tabular LLMs demonstrate much better performance cell- cell-lookup task compared to cell-locating task. This suggests that current L.LSs can utilize semantic co-occurrence between the question and the table to identify question-related rows and columns, and subsequently extract the answer cells via row-column intersections. However, they struggle in accurately compre- hending the basic table structures, indicating the fundamental gap between the ways of LLMs and humans in understanding tabular data. To shed more light on the LLMs' perspective of structured tables, we further analyze the attention weights distribution from representative LLMs.
As illustrated in Fig. 3 we have observed certain attention patterns. This figure visualizes the attention weights, where the input table is serialized into sequence in a left-to-right horizontal and vertical coordinates (m, n) correspond to the cell in the m-th row and n-th column, and the color intensity represents the attention Bright--righher colors indicate larger weights. Based on this, we categorized the observed patterns into the Multi-Slash and Local-Triangee patterns. In the Multi-Slash pattern, the attention weights concentrated on the cell tokens in the same column, thereby exhibiting multiple slash lines at fixed intervals (i.e., the number of cells in one row). In the Aocal-Triangle pattern, the attention weights are concentrated on the cell tokens in the same row within local windows, especially the row header. With these attention patterns, LLMs could retrieve the target cell located in the row and the column that contain keywords of cell-lookup questions. Nevertheless, such table understanding approach may still fail to guarantee the robust interpretation of table structures, leading to failed edge cases that restrict the reliability of LLM- applications.
-o--tt-bottom order. The
(3) Results in Table T on tables of arbitrary sizes show that seemingly powerful LLMS actu-
(1.1
ally lack robust long-context table understand-
(1.2
ing ability, especially table structure compre- hension. To more intuitively demonstrate the
(1.3
(2.1)
influence of different table sizes and increased context length, we crop flat Markdown tables into fixed sizes (15 tables for each size), rang- ing from 8x8. 12x12, to 32x32, and evaluate the ability of representative LLMs to extract cells from each table position with newly constructed 48K cell-locating queries.
[2.2]
Trian-Triangle
[2.3]
[3.1]
[3.2]
[3,3]
[4.1]
(4.2)
From the per-cell accuracy heatmap illustrated
(4,3)
in Figure 4 we can find that GPT-4o demon-
[5.1]
strate much better table structure understand-
[5.2]
ing capacity than open-source LLMs like Qwen-
and Llama3. 1-BB ----B--Intuutt... with more green color in the heatmap. More impor- tantly, we observe that there exists lost-in-the- middle-table phenomenon for LLMS including GPT-4o in understanding lengthy tables [50]. i... LLMs possess better perception of table cells in the first row and the last row, but struggle with cells in the middle part of tables. Besides,
(5.3)
(1. 222 112. 11/3,333 (3.320(3.3)))) 244433443
Figure 3: The illustration of LLMs' attention pat- terns for structured tables. The input tables are in Markdown format and "(m.n)" indicates cell tokens the m-th row and n-th column.
as the table size increases, LLMs suffer significant performance decline, highlighting the unique challenge of lengthy tabular data to LLMs' long-context abilities. Unlike traditional 'Needle-in-al haystack' where current LLMs have achieved near-perfect performance, there remains significant room for improvement in the NIAT task.

---

8X8
12X 12
16X 16
20 20
24 X 24
28 X 28
32 X 32
Qwen2.5-78
Qwen2.5-7B+ours
Llama3.1-8B
1amaa.1-8B+ours
Qwen2.5-14B
GPT-4o
Figure 4: The per-cell accuracy heat map of cell-locating task on tables of fixed sizes, with redder indicating lower accuracy and greener indicating higher accuracy. The x-axis represents tables of different sizes (e.g.. 8 8 denotes tables with 8 rows and 8 columns), while the y-axis lists evaluated LLMs in instruct version. + ours denotes models fine-tuned with our synthesized NIAT training data.
(4) The performance of hierarchical tables relatively worse than the other table structures. demonstrating the increased difficulty due to multi-level headers and merged cells. Compared with Markdown format, most models obtain worse results with HTML format, showcasing the unrobustness of current LLMs in handling various table formats and structures, which is consistent with findings from prior works [42 30.
(5) Compared to generalist LLMs, recent tabular LLM such as TableGPT2 achieve better results, validating the effectiveness of its 2.36M in-house table instruction tuning data. Among various base- lines, DeepSeek-R achieves the state-of-the-arr performance across different settings, demonstrating that test-time scaling techniques could effectively enhance LLMSs ability process long-structured tabular data. By analyzing model outputs, we find that its in-depth reasoning capability enables more accurate understanding table structure and extraction of target cells, thereby facilitating more complex downstream tasks.
Performance of MLLMs. MLLMs directly take table images input, which more closely mimic the way humans understand tabular data. From the results in Table we have the following observa- tions. (1) Recent state-of-the- of-the-art MLLMs like Skywork-Re. V3-38B achieve superior performance in NIAT tasks compared to earlier models, demonstrating the advantages of their optimizations in understanding larger table images, such supporting dynamic high image resolution and introducing extra post-training data for text-intensive images.
(2) Similar to the performance trend observed in LLMs, multimodal LLMs also perform significantly worse on cell-locating tasks compared to cell-lookup tasks, which reveals their defects in perceiving basic table structures. We speculate that the way the visual encoder of MLLMs tokenizes input images could disrupt the integrity of table structures, e.g., traditional ViT models segment input images into fxed-size grid patch tokens, thereby resulting in fragmented table images. Developing advanced semantic equivalent visual tokenizers that can maintain the completeness and the structure of input table images may alleviate this problem [51).
(3) Combining with the experimental analysis of L.LMs, we can infer an important fact: although seemingly powerful large models including table-oriented models have achieved good per-

---

Table 2: Performance of representative MLLMs NIAT benchmark. 'Flat', 'Hori.' and 'Hie.' denote flat, horizontal and hierarchical tables, respectively. The best and suboptimal results are highlighted in bold and underlined.
Cell-Locating
Cell-Lookup
Model
Flat
Hori.
Hic.
Ave.
Flat
Hori.
Hie.
Ave.
GPT-4o
28.46
15.34
11.4
18.40
66.9
61.43
76.65
68.33
DeepScek-VLL2-tny--55
5.89
4.08
5.47
5.15
55.29
25.44
54.68
45.14
GL.M-4--BB-hht
4.73
3.72
2.68
3.71
52.63
24.41
50.00
42.35
IntemWI-2.5-8B-Chal
15.53
10.76
12.44
12.91
37.89
33.37
24.39
31.88
Llava-1.5-7B
0.69
0.56
0.93
0.73
2.71
4.63
1.50
2,95
Table-Llava-7B
7.65
4.90
11.62
8.06
9.58
5.59
2.68
5.95
1.laa-1.6-7B
2.57
1.61
3.50
2.56
16.88
5.39
7.80
10.02
MiniCPM-V-2.6-8B
23.99
16.21
20.12
20.11
59.50
36.10
37.37
44.32
1.....-Viiion-nntrueee
2.28
2.61
4.13
3.01
41.09
22.17
36.48
33.25
Phi-4-multimodda-instnuuu
4.06
4.49
6.47
5.01
57.86
31.22
26.87
38.65
Qwen2-VL-7B. Be-n2--V--B-IIstruuu
5.53
4.43
5.07
5.01
63.04
21.83
53.24
46.04
Qwen2.5- 1------B-Innttte
8.01
5.45
7.77
7.08
59.71
38.02
65.96
54.56
Qwen2.55vLL-7B-Insttu
10.53
7.92
9.39
9.28
70.24
51.83
72.32
64.80
Llama-3.2- 1omna-..2-1BB-Vsiioo Iistonn-nnsttue
4.46
4.25
5.99
4.90
37.35
36.46
51.74
41.85
Qwen2.5-VL-32. 1---------B---ttuuue
23.30
21.41
14.60
19.77
74.82
44.01
84.77
67.87
Bive2.55-VL-722--mmss
25.80
25.13
18.60
23.18
72.72
48.94
87.68
69.78
Skywork-RIV3-388
36.07
26.11
11.33
24.50
56.87
65.86
69.04
63.92
GLM--.IV-
27.77
31.54
18.24
25.85
76.93
64.15
84.28
75.12
formance many tabular benchmarks involving complex tasks, their success may not be grounded on a robust and reliable long-context table understanding ability, i.e., they indeed can solve certain complex problems in existing benchmarks, but they still can not reliably perceive and understand the information of each cell lengthy input tables. This also shows that existing models may rely on dataset-specific correlations or shortcuts to obtain better benchmark results or there may be serious data leakage problems for some models.
4 Experiments on Downstream Tasks
4.1 Data Synthesis Method
Based on above findings, we further explore whether enhancing NIAT capabilities can fundamen- tally improve models' overall table understanding ability. If the long-context table understanding truly lays the foundation of more advanced high-level table-related capacities, its improvements should also benefit the performance on downstream tasks. To this end, we propose a strong2weak data synthesis method to create NIAT fine-tuning data. Specifically, we collect Markdown tables from the training split, and generate various NIAT queries and corresponding responses using GPT-4o as instruction-tunde benchmark and automatically construct queries based on randomly selected table cells. For cell- lookup task, to avoid overlapping with lookup questions in the test data of downstream benchmarks, we synthesize queiries
data. For cell-locating task, we adopt the same prompt templates in the NIAT
more difficult lookup tasks to improve data diversity, which are listed in the Appendix C. These tasks present greater challenges than the simplest lookup questions, requiring LLMs to develop a more thorough understanding of table structures and header information. For instance, cell-retrieval task requires LLMs to perform lookup operations within the whole input table and identify all locations given cell content.
Besides, previous tabular LLMs often directly transform existing tabular datasets into fine-tuning data with short answers as target responses [1 52), which can not provide enough supervised information for LLMs and may also lead to overfitting towards dataset-specific shortcuts. Therefore, we meticulously craft demonstrations and utilize GPT-4o with in-context learning to synthesize (CoT) reasoning processes as target responses, which aims at instructing
detailed chain-of-thouge weak I.LMs to progressively understand tabular data and complete NIAT tasks. In the end, 6K instruction-tumie tuning data were constructed for cell- cell-locating and cell-lookup tasks, respectively.

---

Table 3: Performance results downstream tasks. The best results in different model categories highlighted in bold, and NIAT denotes models further fine- tne-tuned on our synthetic data.
NIAT Task
Downstream Task
Method
Cell- Locating
Cell- Lookup
Ave. Acc
WTQ
TabFact
HiTlbb
TABMWP
Ave. Acc
Open- LLMs
GLM4-9B-Chal
6,74
49.17
27.96
45.60
43,50
25,90
47,51
40.63
MiniCPMB-4B
1.18
58.13
29.66
40.58
62.88
24.43
69.39
49.32
In1emL..M..577BBCHM
0.96
35.26
18.11
34.76
33.00
18.78
54.83
35.34
Yi-1-5-9BChhat
4.41
49.05
26.73
34.00
45.40
32.70
34.93
36.76
Tabular L.Ms
StruciLLM
1.82
42.72
22.27
31.08
29.45
15.74
39.37
28.91
TableLLM
35.86
31.47
17.96
28.91
28.55
TableGPT2
8.84
73.87
41.36
60.01
61.17
36.04
56.19
53.35
Ours
Owen2.5-7B-Insttree
9.46
28.53
52.90
70.00
30.50
54.42
$1.96
+NIAT
10.84
58.41
34.63
60.28
61.28
62.28
72.39
64.06
1-ama..--8B-Inaeee
6.16
65.74
35.95
49.90
62.80
26.10
54.78
48.40
+NIAT
8.38
66.46
37.42
67.43
78.57
49.41
66.15
65.39
Reasoning LLMs and GPT-4o
GPT-4o
26.02
68.30
47.16
83.50
39.10
84.38
68.19
QwQ-32B
35.42
49.74
42.58
78.26
75.23
61.35
53.76
67.15
DeepSeek-RI
65.91
80.99
73.45
84.21
66.44
66.86
64.74
70.56
4.2 Experimental Setup
Baselines. Apart from open-source LLMs, reasoning LLMs like DeepSeck-RI. and close-sourced GPT-4o, we mainly compare our method with tabular LLMs that are fine-tuned with table instruction tuning data, including TableLLM [53) with 80K synthesized training data of downsteam tabular tasks, T2bleGPT2 [33] with 86B tokens for continuous pre-training (CPT) and 2.36M in-house data for table instruction tuning, and StructLLM fine-tuned with 1.1M samples of structured knowledge grounding tasks. We use the released model weights for inference, and fine-tune Llama3. 1-8B-Instruct and Qwennt.5-7B-Instreet with 12K synthesized data our methods.
Benchmarks. We evaluate model performance on 4 downstream benchmarks. WTQ 19 and HiTab [40] are TQA benchmarks based on flat tables and hierarchical tables, which contain lookup questions and reasoning questions that require additional aggregation operations. TabFact [20] is TFV benchmark that requires models to verify whether textual hypothesis holds based the given evidence in a table. TABMWP [54] is table-based math word problem benchmark that evaluates mathematical reasoning ability format for WTQ and FabFact, and HTML format for HiTab and TABMWP to maintain hierarchical table structures. We exact match accuracy the evaluation metric and report model performance under the zero-shot setting. Complete implementation details are provided in Appendix C.
structured tabular data. Tables represented in the Markdown
4.3 Results and Analysis
Table 3 compares model performance on NIAT tasks and downstream tabular tasks. We can find that there is a substantial performance gap between NIAT tasks and more complex downstream tasks, demonstrating their weaknesses in the robust long-context table understanding ability. The proposed NIAT benchmark helps mitigate the data leakage issue among rapidly evolving LLMs, where training data and even test data in downstream tabular benchmarks could be excessively utilized for fine-tuning to achieve better benchmark results. We believe that a model with truly robust table understanding ability should not only perform well traditional downstream tasks, but also maintain strong capability in simple but foundamental NIAT tasks, which is the foundation for reliable LLM--bseed table applications, especially in scenarios involving lengthy tables.
Compared to vanilla models, fine-tuning with our synthesized 12K NIAT data brings substantial per- formance enhancements for Qwen2.5-7B. 7---5---Bnnnnuu and Llama3. 1-8--nnsruuc.. improving the average

---

accuracy in both NIAT tasks and downstream tasks by 3.78% and 14.55%. Notably, our synthesized data does not contain training data of downstream benchmarks and solely focuses on enhancing the basic long-context understanding ability towards table structures and content. However, our fine-tuned models achieve better overall performance than baseline I.L.s of similar scale, even surpassing tabular L.Mss like TableGPT2 that use much more training data. This validates the effectiveness and efficiency of our synthesized data, and verifies our initial assumption that NIAT capabilities constitute the very foundation of more advanced table-related tasks. Figure 4 also intuitively demonstrates the improvement of our synthesized data on every cell position within cropped tables. Due to space limitation, more experimental results such as ablation experiment provided in Appendix D.
4.4 Ablation Study
Table 4: Ablation results of different tasks further fine-tuned on Llama3. 1-atee.--8B-Insttuee The best results are highlighted in bold. WTQ, TabFact, and HiTab the three downstream datasets selected for sine-tunigg with chain chain-of-thought (CoT) reasoning processes generated by GPT-4o on foundation LLMs, listed in the table.
NIAT
Model
WTQ
TabFact
HrTab
TABMWP
12
16
20
24
28
32
Llama3.-88--nstriee
16.88
13.56
12.94
8,98
6.34
7.25
3,83
49.90
62.80
34.78
Cell Locating & Cell-Lookup
20.10
12.75
10.21
5.51
4.51
4.88
2.49
67.43
78.57
49.41
66.15
Cell-Locating
19.76
13.49
7.86
5.31
6.22
3.07
67.33
33.44
Cell-Lookup
18.85
13.13
12.15
8.36
6.22
5,83
3.21
59.00
53,50
35.00
69.44
+4 downstream datasets
16.28
11.12
11.41
6.11
5.92
5.40
2.17
64.78
78.13
48.22
81.79
We conduct ablation experiments to investigate the impact of different training tasks systematically.
The experimental results are presented in Table 4
We compare the foundation model fully trained on
our proposed mixed synthetic data variants trained on individual sub-tasks, well to models fine-tuned on the training splits of WTQ. TabFact, and HiTab, which serve as oracle experiments representing the performance ceiling.
Fine-tuning on our synthesized training data achieves comparable performance to variants fine- tuned on three downstream datasets. Surprisingly, when fine-tuned our mixed synthetic data, Llama3. 1-88--8888--- 1-8--nntruct achieves comparable accuracy on the TabFact dataset and even outperforms other models on the remaining downstream tasks. For variants trained solely Cell-Locating and Cell-Lookup tasks (+ Cell-Locating and Cell-Lookup), removing NIAT training data significantly drops performance across all four downstream tasks.
5
Conclusion
This paper makes the first systematic exploration of long-coniext table understanding problem that has been overlooked by prior work, together with a new benchmark NEEDLEINATABLE. which covers tables of diverse structures, formats and sizes, and serve as valuable testbed for evaluating models' underlying fine-grained understanding and perception of individual cells within tabular context. On this basis, conduct extensive evaluation of existing LLMS and MILMss revealing that current large models lack truly robust long-context table understanding ability, which could influence the reliability of LLM-based table applications. Furthermore, propose a data synthesis method for enhancing L.LMs' basic NIAT capabilities, and demonstrate its improvements in models' overall table understanding abilities, outperforming strong baselines including recent tabular LLMs.
Acknowldagment
This work was supported by the National Natural Science Foundation of China (Nos. 62472419, 62472420)
References
[1] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. In Lun-We Wei Ku, Andre Martins, and Vivek
10

---

Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 31 19-3137. Association for Computational Linguistics. 2024. doi: 10. 10.18653/V1/2024,
ACI-LLONG.122. Utt.https:: /oaetpss//dof.ooo www.:/o..r...... 18650/...11111111/
acc ong 172
[2] Xiao Liu, Hao Yu, Hanchen Zhang. Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang. Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating ilms agents. In The Twelfth International Conference Learning Representattins, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. net forum id zAdUBOaCTQ
URL httphht:t://ppeeeeee
[3] Jun Zhao, Can Zu, Hao Xu, Yi Lu, Wei He, Yiwen Ding, Tao Gui, Qi Zhang, and Xuanjing Huang. Longagent: Scaling language models to 128k context through multi- collaboration. CoRR, abs//402. 11550, 2024. doi: 10.4855/ARRXIV2402..11
URL /ttts:///or
erg/10.48550/
11550
[4] Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longwriter: Unleashing 10,000+ word generation from long context Ilms. CoRR, abs/2408.07003 2024. doi: 10.48550/ARRXV.24088..8
URL httpp:///// orgg/00
48550/arXiv 2408 07055
[5] OpenAI. Gpt-4 and gpt-4 turbo documentation. 2024. URL https://plas form ahenai.com/doctt modelss httpeeessspppttppeeeeeeeee -pt--p------ -ad--ann-gp- -nd--gt---u-rrb Accessed on January 23,
2024.
[6] Anthropic. familyy 2024. URL https///ww.atthrooic.coo///
claude- 3- familamlll Accessed on March 4, 2024.
[7] GeminiTeam. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. 2024. URL /tctps:/sstorage: goo.leepp.oggg.p......oo...... com deepm ind-iniddmmaaaa gemini/gemin v1 5 report pdff Accessed on March 17, 2024.
[8] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and et The Ilama 3 herd of models, 2024. URL http:: wwwp:/iii.orraaggooeeee abs///aa//aa 220.//227777273 21783
[9] An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, Junyang Lin, Kai Dang. Kexin Yang, Le Yu, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He. Weijia Xu, Wenbiao Yin, Wenyuan Yu, Xiafei Qiu, Xingzhang Ren, Xinlong Yang, Yong Li, Zhiying Xu, and Zipeng Zhang. Qwen2.5-Im technical report, 2025.
//axxiv.. org/ abs/ 2501 15383
[10] Greg Kamradt. Needle in haystack pressure testing llms.
https: httt:s:/ggituu..oom
2023. URL https: com/
[11] Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. inftybench: Extending long context evaluation beyond 100k tokens. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 15262-15277. Association for Computational Linguistics, 2024. doi: (0.18653/V1/0022 ACL-LONG.814, URL https:
www.orggg//. 18653/ 2024 ac - long .814
[12] Yushi Bai, Shangqing Tu, Jiajie Zhang. Hao Peng. Xiaozi Wang. Xin Lv, Shulin Cao, Ji- azheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks, 2024. https
15204 15204
[13] Mo Li, Songyang Zhang, Yunxin Liu, and Kai Chen. Needlebench: Can Ilms do retrieval and reasoning in million context window? CoRR, abs/2407. 2024. doi: 10.48550/ARXIV/ 2407.11963, URL https:: /ww.////ii////////////// org 10. 4w550/axxiv.2407..mm. 11963
11

---

[14] Julian Schnitzler, Xanh Ho, Jiahao Huang, Florian Boudin, Saku Sugawara, and Akiko Aizawa. Morehopqa: More than multi-hop reasoning. CoRR, 4bss/4400..3337, 2024. doi: 10.48550/ ARXIV.2406.1333 13397.
doi wrg/10. 48550/arxix 2406. 13397
[15] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: what's the real context size of your long-context language models? CoRR, abs/2404.06654, 2024. doi: 10448550/ARXIV.244..666
URL
https::/doi. org/ wtl.g/1..4855500 48550/555/a/0 arxiv 2404. 06654
[16] Yucheng Ruan, Xiang Lan, Jingying Ma, Yizhi Dong, Kai He, and Mengling Feng. Language modeling tabular data: A survey of foundations, techniques and evolution, 2024. URL
petps://azxis> orq/ abs/ 1
[17] Xi Fang. Weijie Xu, Fiona Anting Tan, Jiani Zhang. Ziqing Hu, Yanjun Qi, Scott Nickleach, Diego Socolinsky, Srinivasan Sengamedu, and Christos Faloutsos. Large language models(llms) on tabular data: Prediction, generation, and understanding a survey, 2024. URL https?
7/arxiv. org abs /ttp///
[18) Weizheng Lu, Jing Zhang, Ju Fan, Zihao Fu, Yueguo Chen, and Xiaoyong Du. Large language model for table processing: a survey. Frontiers Comput, Sci., 19(2):123300 2025. doi: 1007/ $11704-024-40763-6 URL /ttpttts////....//10000100000000
024 424-44433---44-3---- 6
[19] Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers, pages 1470-1480, The Association for Computer Linguistics, 2015. /LL/E//t..///oo...r.///0/33111/1
doi:
[20] Wenhu Chen, Hongmin Wang. Jianshu Chen, Yunkai Zhang. Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-based fact verification. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020, OpenReview.net, 2020. URL httpp:://pperreiieee net/ forum? id=rkeJRhNYDDF
[21] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. URL
abs/ 2104.214..0 09864
[22] Himashi Peiris, Munawar Hayat, Zhaolin Chen, Gary Egan, and Mchrtash Harandi. Hybrid window attention based transformer architecture for brain tumor segmentation, 2022. URL
https: /ttpss///axile org. abs 2209. 2200007704000
[23] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yar:: Efficient context window extension of large language models, 2023. URL.
2309. 20071
[24] Xiaoran Liu, Hang Yan, Shuo Zhang, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling laws of rope-based extrapolation, 2024. URL
[25] Zijia Zhao, Haoyu Lu, Yuqi Huo, Yifan Du, Tongtian Yue, Longteng Guo, Bingning Wang, Weipeng Chen, and Jing Liu. Needle in A video haystack: scalable synthetic framework for benchmarking video milms. CoRR, abs/2406.09938 2024, doi: 10.45550/ARXII22440.0036 /ooi org/10. 18550/arXiv 2406. 09367
URL https::
[26] Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang, Akshay Nambi, Tanuja Ganu, and Hao Wang. Multimodal needle in a haystack: Benchmarking long- context capability of multimodal large language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3221 -2241, Albuquerque, New Mexico, April 2025. Association for Computational Linguisties. ISBN 979-8-89176-188 189-66 URL httpp:: / / antho l logy
org / 2025 naacl- -oong.n. 166/
12

---

[27] Chau Pham, Simeng Sun, and Mohit lyyer. Suri: Multi-constraint instruction following in long-form text generation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguisties: EMNLP 2024, Miami, Florida, USA, November 12- 16, 2024, pages 1722-1753, Association for Computational Linguisties, 2024. htt.http:: acc ac anthaa ogy. 202/ 2024. f Findings-emenn 94
[28] Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang. Ge Zhang, Junran Peng. Zhaoxiang Zhang. Songyang Zhang, and Kai Chen. Hellobench: Evaluating long text generation capabilities of large language models. CoRR, abs/2409. 2024. doi: 10.48550/ARXIV.24091100 https://dos http://oo...//1.......//////// 2wt.:/V........1191
URL
[29]
Wenhu Chen. Large language models are few(1 >-shot table reasoners. In Andreas Vlachos and Isabelle Augenstein, editors, Findings of the Association for Computational Linguisties: EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, pages 1090-1100. Association for Computational Linguisties, 2023. doi: 10. FINDIIN 10. 18653/ wt//////222323 wig.inngnnsssl....
URL http::
[30] Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. Table meets llm: Can large language models understand structured table data? benchmark and empirical study, 2024. URL
org/ ab// 2305. 13062
[31] Tianshu Zhang, Xiang Yue, Yifei Li, and Huan Sun. Tablellama: Towards open large gen- eralist models for tables. In Kevin Duh, Helena Gómez-Adorno and Steven Bethard, ed- itors, Proceedings of the 2024 Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies (Volume I: Long Pa- pers), NAACI. 2024, Mexico City, Mexico, June 16-21, 2024, pages 6024-6044, Association URL
for Computational Linguistics, 2024. doi: /TT.:////111222AAAAAALLLALA http:::///oo..rr/1000 01..1...65611/1 /2024 naacl - long. 335
[32] Qian Liu, Fan Zhou, Zhengbao Jiang, Longxu Dou, and Min Lin. From zero to hero: Examining the power of symbolic tasks in instruction tuning. CoRR, abs/2304.07995,, 2023. doi: 10.48550/ ARXIV.2304.07995.
[33] Aofeng Su, Aowen Wang, Chao Ye, Chen Zhou, Ga Zhang, Gang Chen, Guangcheng Zhu, Haobo Wang, Haokai Xu, Hao Chen, Haoze Li, Haoxuan Lan, Jiaming Tian, Jing Yuan, Junbo Zhao, Junlin Zhou, Kaizhe Shou, Liangyu Zha, Lin Long. Liyao Li, Pengzuo Wu, Qi Zhang, Qingyi Huang. Saisai Yang. Tao Zhang. Wentao Yc, Wufang Zhu, Xiaomeng Hu, Xijun Gu, Xinjie Sun, Xiang Li, Yuhang Yang. and Zhiqing Xiao. Tablegpt2: A large multimodal model with tabular data integration, 2024. URL htttss://aarette erg abs/ 24112 02059
[34] Mingyu Zheng, Xinwei Feng. Qingyi Si, Qiaoqiao She, Zheng Lin, Wenbin Jiang, and Weip- ing Wang. Multimodal table understanding. Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume Long Pupers), pages 1102-9124, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18653/v 1/228653/1/ 493. URL 493/
https://aclantho.le wwtp:yy...ooo2222..ccaa.
[35] Xianjie Wu, Jian Yang, Linzheng Chai, Ge Zhang, Jiaheni Liu, Xinrun Du, Di Liang, Daixin Shu, Xianfu Cheng, Tianzhen Sun, Guanglin Niu, Tongliang Li, and Zhoujun Li. Tablebench: comprehensive and complex benchmark for table question answering, 2024. URL https
drexiv.org/aae
2408.4.0 09174
[36] Wei Zhou, Mohsen Mesgar, Heike Adel, and Annemarie Friedrich. FREB-TQN- A fine-grained robustness evaluation benchmark for table question answering. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pupers), pages 2499, Mexico City, Mexico, June 2024. Association for Computational Linguisties. doi: 10. 10.18633/v/2004.00 http://2222nnaaloooooo
naacl-long. 137. URL https:// 137..137.
aclanthology. 2024 naacl 111.020....11111111131333.3.
13

---

[37] Justin Payan, Swaroop Mishra, Mukul Singh, Carina Negreanu, Christian Poelitz, Chitta Baral, Subhro Roy, Rasika Chakravarthy, Benjamin Van Durme, and Elnaz Nouri. InstructExcel: A benchmark for natural language instruction in excel. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4026-4043 Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v windings-iiiiiiinneeeein findings-emml.2655. Findings emn emnip. 26.5
URL berttss:/aclanthoolee org 2023
[38]
Wei Zhou, Mohsen Mesgar, Heike Adel, and Annemarie Friedrich. FREB-TQA: A fine-grained robustness evaluation benchmark for table question answering. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pupers), pages 2479-2497, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 10.18653//1/2024.0o naacl-long..13.. URL https://
aclanthology.oo org/ 2024 www.aaalllaaaa 1aaeclloong.1111
[39] Mingyu Zheng, Yang Hao, Wenbin Jiang, Zheng Lin, Yajuan Lyu, QiaoQiao She, and Weiping Wang. IM-TQA: A Chinese table question answering dataset with implicit and multi-type table structures. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, ed- itors, Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guisties (Volume 1: Long Papers), pages 5074-5094 Toronto, Canada, July 2023. Associ- ation for Computational Linguistics. doi: 10.18653/v 1/2333////2..cll acl-long..2255 URL https logy.ogy....oooooooo....o ooy...o.ooo..................... org/ http://222..allllooo
2787
[40] Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian- Haan-Gungg Lou, and Dongmei Zhang. Hitab: A hierarchical table dataset for question answering and natural language generation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 1194-1110. Association for Computational Linguistics, 2022. doi: 10. URL httt::::/doi.ooo/// htt:://iii.oog/////// 1ttp://....1153/1112222
acl- long. 78
[41] Yannis Katsis, Saneem Chemmengath, Vishwajeet Kumar, Samarth Bharadwaj, Mustafa Canim, Michael Glass, Alfio Gliozzo, Feifei Pan, Jaydeep Sen, Karthik Sankaranarayanan. and Soumen Chakrabarti. AIT-QA:
Question answering dataset over complex tables in the airline indus- try. In Anastassia Loukina, Rashmi Gangadharaiah, and Bonan Min, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies: Industry Track, pages 305-314, Hy- brid: Seattle, Washington + Online, July 2022. Association for Computational Linguistics. URL https://aclantholl/ wtw.llggyy.gg///2/2
doi: 10. 10.188553/11/2222..aall neacl-industry. 34/]
[42] Ananya Singha, José Cambronero, Sumit Gulwani, Vu Le, and Chris Parnin. Tabular represen- tation, noisy operators, and impacts on table structure understanding tasks in llms, 2023. URL
https://aaxii: org abs 22100. 2230.11035
[43] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023. URL http:ttpps.//axi..ooror//////eee wbtp://wwwwwr/p/////////// 222.2.11..711
[44] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL ptttps://arxiv.oor www.:v...rg//bbooooeeeooo 0...1...........
[45]
Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge, Kai Chen, Kaipeng Zhang, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Expanding
14

---

performance boundaries of ooen- multimodal models with model, data, and test-time scaling, 2025. URL URL.attps///xxx..ccrr. wttp://////////222/////// 05271
[46] Alex Zhuang, Ge Zhang, Tianyu Zheng, Xinrun Du, Junjie Wang, Weiming Ren, Stephen W. Huang, Jie Fu, Xiang Yue, and Wenhu Chen. Structim: Towards building generalist models for structured knowledge grounding, 2024.
URL http:ttpp:////m /ttp:::///xivi.aa//a www.wwwwww.aaaaaaaaaaa abs/ 2202/44222222 16671
[47] DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, and et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. 2025. URL htttps://arxii........ /
2501 12948
[48]
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https: qw/elm.eemmmmmmmmmmmm..mmmmm.m.. github. wt/bloq/////oooo hotploog/waa-22202222 qwa-
[49] Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin. Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https: / arw.aiivvv. org/ abs/ /t/b/////// 2412 15115
[50] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the A Association for Computational Linguistics, 12:157-173, 2024. doi: 10.1162/tacl 00638.
URL htt.lhtttss:///om ac antho loooooooyy..
2tw.............. tac1-1.9/
[51] Shengqiong Wu, Hao Fei, Xiangtai Li, Jiayi Ji, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Towards semantic equivalence of tokenization multimodal 1lm. ICLR, 2025.
[52] Naihao Deng and Rada Mihalcea. Rethinking table instruction tuning, 2025. URL https
7/arxiv.org/ abs 11.00..444 200.b//65911.3
[53] Xiaokang Zhang. Jing Zhang, Zeyao Ma, Yang Li, Bohan Zhang, Guanlin Li, Zijun Yao, Kangli Xu, Jinchang Zhou, Daniel Zhang-Li, et al. Tablellm: Enabling tabular data manipulation by Ilms in real office usage scenarios. arXiv preprint arXiv:2403. 19318, 2024.
[54] Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi- structured mathematical reasoning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://
net/ forum? Dir.umm?@id-HHHHBBBBBUUU
[55] Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid-training incentivizes reinforcement learning scaling. CoRR, abs/2506. 20512, 2025. doi: 10.48550/ARXII.25666 20512. URL
wtt.o///.......0//r......... wtt.aarwwi............. 2506 2551.
[56] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. CoRR, abs/2503.20783, 2025. doi: 10448500ARXIV.25002207 URL https: wrw.ooioorgg/000 48550/885555aa0
2503.20783
NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
15

---

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence), justificatin right after your answer (even for NA).
integral part of your paper submission. They visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The checklist answers
The reviewers of your paper will be asked use the checklist one of the factors in their evaluation. While "[Yes] is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for dataset we used"). "[No] " or "[NA] is not grounds for rejection. While the questions are phrased in binary way, acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract provides a concise summary of the key findings and experiment results. The introduction in Sec. 1 outlines the research questions and objectives.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses the limitations of the work performed by the authors in detail in Appendix. A.2
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
16

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms
and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [No]
Justification: This paper did not contain theory assumptions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper provides a detailed description of the experimental data setup and Justification: hyperparameter settings in Section and Appendix B and C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
17

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The datasets, baseline methods, and models used in the paper are fully open- source and available Hugging Face. The complete code and data is provided in the supplementory materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
pub public//n guides / SubeSubii ss ssion
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips //tppsccc/ppbbb///ggidcd//////// Sewediddee//eoeSeeeeeeeeeeeeeeee Ctt//CooeSSuuaa Submissuuiissiee onn cwwssiionooooyoyyoooooo olicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
18

---

Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes)
Justification: The paper provides a detailed description of the experimental data setup and in Sec. 4.2, and more information is provided in Appendxi C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars reported because it would be computationally expensive, but we calculate the average performance of the LLMs and shown in Table 3.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Shown in Appendxi and C, we report the GPUs we used, the memory, and detailed training and inference information.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
19

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpt:://tttrrrrt http://www.....oooooooooccccccea
Answer: [Yes]
Justification: Appendix B and
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The discussion of the ethics and impact can be consulted in Appendix. A.2. We are open and transparent throughout the study and do design for human subjects. privacy data bias, or other issues.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: The safeguard of our paper is dissussed in Appendix A.
Guidelines:
The answer NA means that the paper poses such risks.
20

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: This article uses assets reasonably in compliance with the license, and the assets used are cited in the article.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL.
The name of the license (c.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package ppppesswithseeeeo snneerswittee swithcode com/ dat aseas has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
should be provided. For popular datasets,
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The documentation is provided in the supplementory files.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
21

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: LLMs are only used for editing grammar.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (ht htw.earrrppsccc/ooooeeeneee////e
LLM) for what should or should not be described.
22

---

A Broader Discussion
A.1 Limitations
Although this work presents a comprehensive exploration of the long-context table understanding problem, certain limitations could be alleviated by future research. First, the proposed benchmark focuses on a single table in English. The long-context understanding in multi-table scenarios with broader language coverage deserves future explorations. Second, real-world applications such as document understanding could also require processing the hybrid content of tabular, textual, and image data. Therefore, valuable to explore the long-context understanding ability towards data consisting of multiple modalities. Third, we build the NIAT benchmark based on tables from public datasets to save cost, which primarily contain medium-sized tables. Future work could collect larger tables that span diverse structures to provide longer context for long-context table understanding, especially tables with more than thousands of rows.
A.2 Ethical Considerations
The proposed NIAT benchmark is constructed based on the academic datasets like WTQ and TabFact, which are free and open datasets for research use with licenses like MIT Licens or CC-BY-SA4.0 License The resulting synthetic training data also a free and open resource for the community to study long-stmn structurd table understanding. Considering that LLMs may generate harmful content, we used Llama3.1 --nttintttttt to conduct LLM-as-a-judge to check the content of synthetic data and the generated data by APl-available models, and we also randomly sampled part of the data for manual checking and did not observe unsafe data in the synthetic samples. Thus, the authors foresee no ethical concerns with the research in this paper.
A.3 Different fine-tuning benefits between Qwen2.5 and Llama3.1
One notable finding from our experiments is the significant disparity in performance gains on the cell-locating task after NIAT fine-tuning (Figure 4 Specifically, Qwen2.5-7B demonstrated substantially greater improvement than Llama3. 1-8B. We hypothesize this phenomenon is attributable the models' differing pre-existing capabilities, stemming from two potential factors. First, the pre-training regimen of the models plays a crucial role. As detailed in its technical report [49], the Qwen2.5 series underwent specialized post-training on structured tabular data. This inherent familiarity with table structures may provide more effective foundation for our Co'T-based fine- tuning. Consequently, the fine-tuning process could more readily "activate" " and enhance this latent ability, leading to more pronounced boost in table structure comprehension. Second, existing research reinforcement learning [55] (66] has indicated that the Qwen2.5 series possesses strong reasoning capabilities, even without instruction tuning. A superior innate reasoning capacity could make the model more adept at acquiring complex, multi-step procedural tasks, such as the cell- locating process of interpreting table row-by-ron and then pinpointing cell by its column index. In contrast, the Llama3. series might require additional mid mid-training to achieve a comparable level of performance such tasks.
A.4
B
Details of NIAT Benchmark
B.1
Data Statistics
To construct a benchmark for long- long-context understanding of tabular data at low cost, we leveraged existing open-source datasets to sample tabular data. Table 5 represents the statisties results of sampled tables, we calculate the average length of tokenized tabular data. Figure shows the distribution of input length of the NIAT benchmark.
http:////pennonneee.ooo//lcceece "ntpss//ceaatvvcoommnn.ooo http:///wwweeotooooooo//////eena
23

---

Distribution of Input Lengths
60th Percentile: 4690
11516
23032
34548
46064
57580
69096
80612
92128
103645
Length
Figure 5: The distribution input lengths for our proposed NIAT benchmark. The tokenizer of Llama3. 1-BBB 1-8B-Instruct is adopted to calculate the token length.
Although the largest tables reach length of 100k tokens, the average length of tables sampled from public dataset is not long enough. We anticipate that future researchers will build upon this work to develop more comprehensive benchmarks for long-table understanding tasks with longer tabular text.
Table Types
Format
Table Num.
Avg. Row Num.
Avg. Num.
Avg. Length
Max Lengh
HTML
6.43
26.78
1831.27
29791
Horizontal
Mardown
25.77
981.92
16012
HTML
362
26.32
6.51
5226.04
103430
Flat
Markdown
26.24
843.05
13203
HTML
19.81
8.87
878.39
5339
Hierarchical
Markdown
1086
19.81
8.87
1481
9420
Total
11.46
1594.32
103430
Table Detailed table statistics of sampled tables of NIAT benchmark. The Avg. Length and Max Length represent the average and maximum token lengths of encoded tabular text for specific table categories. We use the tokenizer of 1-8B-Instruct to calculate the token length.
B.2 Prompt Templates of Evaluating LLMss
For text-LLMs and MILMs, we adopt unified prompt templates to evaluate the performance of them. For Cell-Locating the prompt templates shown in Figure for Cell-Lookup task, the adopted template are provided Figure
C Implementation Details
C.1 Implementation Details
To investigate the relationship between the fundamental table structure comprehension capabilitics of LLMs and their performance in downstream table understanding tasks, selected four popular datasets spanning various table understanding tasks. These datasets include Table Question An- swering (HiTab, WTQ), ), Tabular Numerical Reasoning (TABMWP), and Table Fact Verification (TabFact). Each language model was instructed to generate a mhain-of-thoughh reasoning process before providing its final answer. To evaluate performance, calculated accuracy based on the last 30 characters the generated content for open-source LLMs, which corresponds to the model's 's final answer.
provide an example Cell-locating question and answer pair, along with the chain-of-thoure (CoT) reasoning process, in Figure II

---

C.2 Training Details
To efficiently fine-tune 1ann2.5----nnsruuee and Llama3. (-amma.1--B--.. Instruct on our synthetic data, we employ the NVIDIA Megatron framework. To avoid out-of-memory errors caused by long-context input sequences, we configure tensor parallelism (tp) 8 and model pipeline (pp) 4. Each model is fully fine-tuned on the mixed data for 2 epochs.
C.3 Evaluation Details
For the Data Mixing Strategy, we randomly sample 8,400 NIAT training instances and 6,497 table- lookup question-answer pairs, combining them into the training dataset. The longest sample in our training data reaches 92k tokens. For evaluating all foundation models and the further fine-tuned models. we adopt the original generation configurations of the models and utilize the vLLM framework for deployment on a machine equipped with 8 NVIDIA A100-80G GPUs, as the input sequences can be extremely long.
C.4
Details of Cell-Lookup Synthesis Tasks
Locate single cell Given header and a column header, LLMs are tasked with identifying the cell value at the intersection of these two headers. The uniqueness of the ground truth answer is strictly enforced.
Synthetic Data Generation Prompt Template: I'In the table above, what is the element located in the cell the intersection of the row header <row headers and the column header ecolumn header>? ?"
Retrieve the Nth cell based on row header Counting ability is crucial for LLMs when interpreting tabular data. In this task. given row header (a specific cell value in the first column), the LLM is required to return the value of the Nth cell in the corresponding row. Synthetic Data Generation Prompt Template: W'What is the Nth cell value with header row <row headers?"
Retrieve the Nth cell based column header Similarly, LLMs are expected locate the Nth cell value within a column specified by given column header. Both tasks require LLMs to retrieve target cell contents simultaneously based natural language queries and row/column IDs. Synthetic Data Generation Prompt Template: f"What is the Nth cell value with header col <col header=?"
Retrieve all location IDs of certain key In some cases, cell values may be duplicated in large table. This task requires identifying the row and column IDs (location IDs) of all cells that contain the target value. Synthetic Data Generation Prompt Template: f'In the table above, how many cells contain the value acell to ask*?"
Cell value counting In this task, LLMs are asked to count the number of cells that contain specific value, further testing their ability to process and summarize tabular data. Synthetic Data Generation Prompt Template: HoHow many cell contains the value <random element>? Please provide all row IDs and column IDs of all cells contain "<random element>"."
Table navigation Understanding the relative positions of different cells in table is chal- lenging task for L.LMs. Given a specific base cell position and row/column offsets, LLMS are required to navigate the two-dimensional table by calculating the target position IDs and retrieving the corresponding cell values.
Synthetic Data Generation Prompt Template:" are provided with a two-dimensional table and need to locate the content of a specific cell. The following information is given: Base Position: Row index: wbase row> Column index: æbase cols Relative Position: Row offset: ærelative row> Column offset: wrelative cols Search Instructions: If orelative row> > O, move downwards from the base row index. If ærelative row> < O, move upwards from the base row index. <relative cols > 0, move rightwards from the base column index. If <relative cols < O, move leftwards from the base column index. "Calculate the new target position (<new row*, <new cols) using these offsets from the base position, and return the content of the cell located at this new position"
25

---

Format
Avg. Queries Num.
Horizontal
HTMI.
317.56
Markdown
376.33
HTML
290.57
Flat
Markdown
372.67
HTML
909
Hierarchical
Markdown
1041
Table 6: The average number of queries of question-based NIAT. We prompt GPT-4o to generate simple questions to extract cells of given table and check the correctness of
Prompt Template of Synthesized Cell-Lookup tasks Cell-Lookup training data synthesized by GPT-4o are shown in Figure [2 RReriiev the Nth cell based on column header) and 13 (TTble navigation).
We randomly sampled examples of
D Additional Experiments
D.1 Results on Cropped Tables
Table presents the detailed performance metrics of various MLLMs. In Table we compare LLMs fine-tuned on our proposed NIAT synthetic data against mainstream LLMs and T.b on Cell-Locating tasks with cropped tables. This task demands deep structural understanding of two-dimensional tabular data, we evaluated model performance across varying context window sizes. Consistent with expectations, longer context windows improve accuracy Cell-L Gell-Locating tasks.
The Qwen2.5 seriesdominates performance, likely due to its specialized training on tabular data (as noted in its technical report). Notably, Qwen2.5-7B-Instrueet-le long-context variant, significantly outperforms its standard counterpart (Owen2.5-7BBIInstrue suggesting benefits from structured pre-training and post-traine post-training data.
Among Thinker LLMs, DeepSeek-R achieves state-of-the-an results, leveraging its Long-Cor CoT capability to accurately retrieve target cells in large 32x32 tables using row/column identifiers. This underscores the potential of test-time scaling techniques for processing long-structured tabular data. Surprisingly, --ee...--B- Instruct (with only 7B parameters) excels on NIAT tasks, likely due to: 1) Structural alignment between tabular data and code (both rely on separators for semantic organization). 2) Extensive pre-training on markdown, which may have included markdown tables, priming it for Cell-Locating NIAT tasks.
Multi-Modal LLMs (MLLMs), which process table images directly, more closely mirroring human tabular understanding, demonstrate superior performance in Cell-Locating tasks. Notably, recent state- of-the-art models like Qwen2.5. --ntuu-utuuut and 6-2.6M---2..6.8 significantly outperform earlier approaches (e.g.. Llava-1.5-7B) while they still underperform GPT-4o with pure text input.
As illustrated in Figure 6 both mainstream MILMss and text-ILMMs exhibit the lest-i--the-middle- tables phenomenon- their performance degrades monotonically as table size increases. For MI.LMs, this limitation arises because visual encoders struggle to accurately parse table structures when rows and columns expand beyond 32x32, leading to unreliable target cell retrieval. Nevertheless, recent MLLMs (e.g.. MiniCPM-V-2.6-888 show marked improvements in Cell-Locating tasks, suggesting advances in structural understanding.
D.2
Case Study
We provide two cases of the model further fine-tuned on our synthesized data on Cell-Locating and Cell-Lookup, respectively in Figure aadd
26

---

Table 7: Performance results on Cropped Tables mainstream text-LLMs, Tab MLLMs, and foundation models fine-tur tune on our synthetic data. The best results are highlighted in bold, and + ours denotes models further fine-tuned on our proposed synthetic data.
Cell-Locating on Cropped Tables
Model
20
24
Open Source LLMs
GLM4-9b
9.9
10.88
10.26
8.43
4.91
8.65
4.29
GI.4--9-chat--1M
5.63
6.39
5.94
5.43
3.84
3.99
2.57
MiniCPMB-4B
1.98
4.35
5.73
5.75
4.28
9.51
5.02
IntemL.M2.5-7B
4.9
6.25
4.84
5.23
3.24
3.85
2.25
Mistral-7B-v0.3
8.54
10.65
9.17
6.2
3.47
5.49
5.81
Yi-1.5-9B-16K
12.81
8.98
5.55
5.67
3.32
4.53
2.12
@wen2.5-Instruc
Owen2.5-7B-Insttee
26.35
18.98
15.91
12.83
7.55
9.72
2.9
Bren2.5-7B-nnstruee
32.08
24.44
20.6
13.8
9.99
12.13
-- 54B-Inrrtttt
28.96
19.81
19.69
15.93
14.38
17.54
Qwen2.5-1444 11102.5-11B-Iii I4B-Instruct-IM
39.69
33.84
33.83
26.7
20.98
25.54
16.54
Qwen2.5-775 Instruct ours
37.60
30.83
25.76
17.67
18.66
18.24
10.85
-lama3..-IInsruue.
1ama--31-8B-Instruee
16.88
13.56
12.94
8.98
6.34
7.25
3.83
Llama-3.1-8B-Insiiee ours
20.1
12.75
10.21
5.51
4.51
2.49
TableLLM
TableLLM
StructLLM
5.62
5.23
4.29
4.02
2.07
3.26
0.92
TableGPT2
21.15
15.46
12.99
10.12
8.34
9.57
3.4
Thinker LLM and Coder LLM
GPT-4o
50.21
33.36
30.89
24.15
27.54
26.01
17.32
QwQ-32B
51.15
45.23
43.91
14.57
6.01
6.32
6.56
DeepSeek-RI
73.33
64.63
64.60
41.87
38.07
54.06
51.17
Owen2.5-7BB-Coder
65.42
54.86
44.90
28.42
17.79
29.88
28.67
MLLMMs
GPT-4o(Text)
41.67
18.34
19.94
9.75
8.84
18.46
3.92
GLM-4V-9B
2.80
2.61
4.30
1.39
1.43
0.89
0.09
Vnt55VL-22.-8BB
28.52
16.64
12.60
10.24
10.96
13.18
5.09
Llava-1.5-7B
2.68
4.31
1.19
0.20
1.26
0.39
0.30
Table-Llava-7B
1.67
2.37
2.65
2.12
2.97
1.82
Llava-1.6-70
6.60
4.03
4.24
5.57
4.11
4.33
0.26
MiniCPM-V-226-8B1
43.51
30.57
15.57
11.74
10.01
12.88
3.94
Phi-3.5-Vision Insriitt-.
4.25
2.84
5.92
2.68
4.68
1.56
0.82
mhii4-muliinodal-nnsttee
6.04
8.86
8.84
5.77
6.12
8.69
3.68
Bwve2--VL-7B-Instruue
7.27
8.72
8.62
9.64
8.19
8.07
2.30
Qwen2.5.
15.66
10.95
10.53
9.70
Qwen2.5-V
19.80
14.64
11.98
10.67
5.29
9.59
3.02
Llama- 2-112111 Vision-Instruce
10.63
8.96
9.89
6.98
4.34
5.30
3.03
Table 8: Performance results of mainstream MILMS cropped tables.
The best results
highlighted in bold.
Cell- Cell-Locating NIAT task cropped tables
Model
8*8
12*12
16*16
20*20
24
28
32#32
Ave.
Multi-Modal LLMs
DeepSeck-VL2-ting
2.8
4.08
2,46
4.27
3,47
2.49
1
2.94
GLM-4V-9B
2.8
2.61
4.3
1.39
1.43
0.89
0.09
1.93
Llava-1.5-7B
2.68
4.31
1.19
0.2
1.26
0.39
0.3
1.48
Table-Ilava-7B
1.67
2.37
2,65
2.12
2.97
1.82
2.01
2.23
1lava-1.6-7B
4.03
4.24
5.57
4.11
4.33
0.26
Phi-3.5--esioo--nnsruc-----
4.25
2.84
5.92
2.68
4.68
1.56
0.82
3.25
hol-4--ultinoddaiinnssruttt
6.04
8.86
8.84
5.77
6.12
8.69
3.68
6.86
Qwenn-VV--7B-nnstee
7.27
8.72
8.62
9.64
8.19
8.07
2.3
7.54
15.66
13.6
10.95
10.53
9.7
8.11
2.67
10.17
Qwen2.5- S-----L--B-----ee-
19.8
14.64
11.98
10.67
5.29
9.59
3.02
10.71
Llama-3, 3.B- LeLL-B-Viiii--niiie Instruct
10.63
8.96
9,89
6.98
4.34
5.3
3.03
7.02
MimiCPM-V-2.66-88
43.51
30.57
15.57
11.74
10.01
12.88
3.94
18.32
IntermVL-2.5-88B
28.52
16.64
12.6
10.24
10.96
13.18
5.09
13.89
GPT-4o
41.67
18.34
19.94
9.75
8.84
18.46
3.92
17.27
27

---

8X8
12X
16X
20
24
28
32
Qwen2.5-78
Qwen225-78+ours
Llama3.1-8B+ours
Qwen2.5-14B
GPT-4o
50
8X8
12X 12
16X 16
20 X 20
24 X 24
28 X 28
32 X 32
Laaa-332-118 2-118- Vision-Instruct
Qwen2255-V-33. Instruct
Qwen2.5-VL-7B Instruct
Qwen2.5-7B.
Instruct
MiniCPM-V-2.6
Figure 6: The figure presents a NEEDLE/NATABIES evaluation performance comparison cropped tables between MLLMs and text-LLMs.
28

---

An example of finetuned Llama3. 1e-.11-----eee 1-0ma3.1-8-Instueee output:
To determine the value at the specified cell, we will follow these steps:
1. Identify the Row:
We are looking for the row with row ID of 8. In the provided table the rows are numbered from 1 to 8, corresponding to each province listea.
The with row TD of corresponds the province "Seville
2. Identify the Column:
We are looking for the column with a column ID of 1. The columns are labeled as Follows:
Column 1: Province
Column 2: Capital
Column 3: Population
Column 4: Density
Column Municipalities
Column Legal districts
3. Locate the Cell:
With the identified row (Seville, row ID 8) and colurn (Province, column ID 1), we locate the cell at the intersection of these two. The cell at row ID 8, column ID contains the value "Seville".
4. Output the Result:
The cell value located at the specified position is "Seville",
Therefore, the final answer is:
I"row ID": 8, "column ID": 1, "cell_value": "Seville")
This matches the provided ground truth answer, confirming that the value in the specified cell "Seville".
Figure A case response from Llama3. 1-mma3.--B-IInstrree I-struutt further fine-tuned on synthesized data, demonstrating its performance on the Cell-Locating task.
29

---

An example of finetuned 1lama3..-8-Instruct output:
To determinc the opponent in Week 16, we need to carefully examine the table provided. let s go through a step-by stepp reasoning process:
1. Identify Week 16:
We need locate Week 16 in the table. The weeks listed sequentially from Week to Week 17. we look for the entry that correspends to Week 16.
2. Locate the Date for Week 16: In the table, the date for Week 16 is December 23, 2007.
3. Find the Opponent for Week 16:
Once we have identified the cerrect week, we Look at the "Opponent" column for that row. For Week 16, the opponent listed is "philade phia Eagles."
4. Verify the Details:
To ensure accuracy, we check the other details in the same row to confirm consistency. The "Game Site" is "Louisiana Superdome, the "TV channel is "Fox, and the "Result/Score" is "L 38-23," which matches the pattern of losses (l.) followed by a score in the format of "score Score."
5. Conclusion:
Based the information extracted from the table, the opponent in Week 16 is confirmed to be the Philadelphia Eagles.
Pinal Answer:
jssson
{
"answer"; "Philadelphia Eagles"
Figure A case response from Llama3.1 1-8B..118---I--- further fine-tuned on synthesized data, demonstrating its performance on the Cell-Lookup task.
Read the table image carefully and retrieve the value from specified cell, based on given row ID and column ID. Let's first think step by step and then output final answer in the following JSON format:
-jjsson 11
"row ID"; "row_ID": "m"
"column ID"; "n" "cell_value"; "<Corresponding Cell Value>"
!!.
Please note:
Row and column IDs start from 1.
The header row is considered as the first row, with a row_ID of i, so ensure you include it when locating the target row.
the target cell empty, the cell_value should be an empty string ("").
Now, retrieve the cell value the position where the ID is () and the column ID is ().
# Your Response:
Figure 9: Prompt template for evaluating Cell-Locating Task.
30

---

Read the table image carefully and answer the following question.
Let's first think step by step and then output final answer in the JSON
format:
json
I|
"answer"; "<the final answer to the question
!!.
# Question:
()
Your Response:
Figure 10: Prompt template for evaluating Cell-Locating Task.
31

---

Question: Retrieve the cell value at the position where the row_ID is and the column 26.
Answer:
To find the target cell value in the given table, where the row_ID is 5 and the column_ID is 26, we should follow these detailed step-by-stop instructios::
1. Identify the Header Row:
The header row is the first rew the table and provides the nanes for each column, In the provided table, this the first row of data:
Name (8) Council | Core County Other Counties | 2006 Tot 2011 Tot
change Tot Tot area Tot dens 2006 Mun 2011 Mun change Mun Mun Mun dens 2011 Council Core County Mun area | Name area I change
Mun dens 2009 elect 2006 Sub 2011 Sub
2006 Mun Other Counties 2011 Sub not
2009 elect
change Tot 2006 Sub 2011 Mun
2. Identify Row 5 (not skipping the header row)
Based the given instructions, the header should be counted row 1. Thus, row_ID 5 corresponds to the fifth row in the table, including the header row.
Looking at the provided table, the fifth row is:
Waterford City Waterford Kilkenny 49,213 | 51,
4.7
44 1 1,111 45, 748 46, 732 2.2 3521 1,219 26,
| 3,46
4,787 1, 219 51,519 | 45, 748 Kilkenny | 4,787
| 44
1 2.2 City
1 Waterford | 38.35 | Waterford
26, 598
4.7 3, 465
46,
3. Identify Column 26:
Each vertical group of data is identified as column. Starting from the first column, count until reach the 26th column as specificd.
According to the header, the columns are ordered as follows:
1. Name
2. Council
3. Core County
4. Other Counties
5. 2006 Tot
6. 2011 Tot
7. & change Tot
8. Tot area
9. Tot dens
10. 2006 Mun
11. 2011 Mun
12. & change Mun
Mun area
14. Mun dens
15. 2009 elect
16. 2006 Sub
17. 2011 Sub
18. Mun dens
19. 2011 Tot
20. 2006 Mun
21. Other countios
22. 2011 Sub
Tot area
24. $ change Mun
25. Council
26. Core County <- Target Cell
4. Cell Location and Determination:
In the fifth row, column 26 corresponds to the "Core County" column.
The cell at the intersection of row and column 26 in this row contains the value: "Core County."
Thus, following this step-py-step process, the value located at row_ID and column_ln 26 is Core County.
32
An example of the chain-of of-tthughtt (CoT) reasoning process for retrieving a target cell using row ID and column ID, step by step.
Figure 11:

---

Carefully review the table, questicn, and provided ground truth answer.
Table: stable_str>
Question:
In the table above, what is the element located in the cell at the intersection of the row header 12 and the column header game?
Ground truth Answer:
Generate detailed, step-by-step reasoning process accurately answer the questicn. Conclude with the final answer.
GPT-4o Answer:
To determine the element located in the specified cell, follow these steps:
1. Understand the Table Structure:
The table organized with columns labeled as: "gane", "date", team", score', 'high points', "high rebounds", 'high assists", location attendance", and "record".
Each row corresponds to particular game's data, with the rows numbered from 1 to 13.
2. Identify the Cell to Examine:
The question asks for the element at the intersection of the row header and the column header "game". The rew header refers to the game's s sequence or identifier within this particular dataset; the row number is essentially the order of games played.
3. Locate Row 12:
Go to the row labeled with the number 12 in the table. This correspends to all the data associated with the 12th game played.
4. Find Column Header "game"
The "game" column is the first column in the table. This column indicates the game number or identifier for that particular data.
5. Extract the Information:
At the intersection of row 12 and the column "game", locate the number, which indicates the sequence position of that game.
6. Interpret the Table Data:
If we review the table, we can see that row under the column game the cell contains the number "12", This means that the 12th zow corresponds to the 12th game played.
Finally, using this step-by-step examination, the element located in the cell at the intersection of the row header 12 and the colurn header "game is indeed:
Final Answer: 12
Figure 12: An example of the chain-of-thought (CoT) reasoning process for synthesized Cell-lookup tasks.
33

---

Carefully zeview the table, questicn, and provided ground truth answer.
Table: <table_str>
Question:
You are provided with txo-dimensicnal table and need locate the content of specific cell. The following information is given:
Basc Positicn:
Row incex: 32 Column index:
Relative Position:
Row offset: -17
Column offset: Search Instructions:
If relative_ row > 0, move downwards from the base row index. row
If relative_ron row < 0, move upwards from the base row index.
If relative_col > 0, move rightwards from the base column index. relative_col < 0, move leftwards from the base column index. Calculate the new target positicn (new row, new new_col) using these offsets from the base position, and return the content of the cell located this pesition
Ground truth Answer: One of the criginal 19 parishes. Generate detailed, step-by-step reasoning process to accurately answer the questicn. Conclude with the final answer.
GPT-4o Answer:
To determine the content of a specific cell using the given information, we neea to follow the search instructions carefully. Let's go through each step:
1. Identify the Base Position:
The base index is 32. The base column index is 4.
2. Understand the Relative Position:
The row offset is -17, meaning we need to move 17 rows upwards From the base row.
The column offset is 0, meaning there is no movement in the colunns, and we stay in the same column as the base column.
3. Apply the Row Offset:
Starting from the base row index of 32, move 17 rows upwards. Calculate the new row index: (332 - 17 = 151)
4. Apply the Column Offset:
The base column index is and the column offset is 0, so the new column index remains 4.
5. Locate the New Position in the Table:
The The new column index is 4.
index is 15.
6. Find the Content of the Cell at the New Position:
Check the table at row index 15 and column index 4
According to the table, this corresponds to the "Origin" column for the 16th parish in the list, which is East Baton Rouge Parish since the table index starts from zerc
7. Verify and Extract the Content:
The content at row index 15 and column index 4 is "One of the original 19 parishes."
8. Conclusion:
The content of the cell located at the new position is "One of the original 19 parishes."
Thus, the final answer is:
"One of the original 19 parishes."
34
Figure 13: An example of the chain-of-thought (CoT) reasoning process for synthesized Cell-lookup tasks.


Reviews:
[{"review_id": "4OgNhDjZ66", "reviewer": "Reviewer_5fM7", "summary": "This paper introduces NeedleInATable (NIAT), a novel benchmark for evaluating the long-context understanding capabilities of large language models (LLMs) in structured tabular data. The authors argue that prior long-context benchmarks have focused almost exclusively on unstructured text, overlooking structured tables’ unique challenges—particularly the need for precise structural reasoning and cell-level localization.\n\nNIAT is built upon public table datasets and defines two task types: (1) cell-locating queries that require retrieving the content at a specified row and column, and (2) cell-lookup queries based on simple natural language questions. The authors evaluate a range of LLMs and multimodal LLMs (MLLMs) and reveal large performance gaps between the two query types, highlighting weaknesses in structural understanding. They further propose a data synthesis method using GPT-4o with Chain-of-Thought (CoT) reasoning to fine-tune models, showing improvements on both NIAT and downstream table QA tasks.", "questions": "See weakness.", "limitations": "The authors acknowledge that their benchmark uses medium-sized tables and only supports English. However, they do not explicitly discuss the risk of table content overlap with LLM training corpora, or the format-structure mismatch issues. These should be addressed in the limitations section.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### **Strengths**\n\n* **Clear motivation**: The work tackles an under-explored but critical problem—the structural localization of information in long tables—complementing existing long-context benchmarks that focus solely on plain text.\n* **Well-constructed benchmark**: NIAT covers different table formats (Markdown, HTML, image) and structures (flat, hierarchical, horizontal), and provides detailed statistics and task definitions.\n* **Empirical insight**: The finding that LLMs perform poorly on cell-locating but decently on lookup-style tasks provides valuable diagnostic information for the community.\n* **Effective synthetic data strategy**: The CoT-style data generation and fine-tuning approach leads to tangible gains across NIAT and traditional table QA datasets.\n\n### **Weaknesses**\n\n1. **Lack of contamination analysis**: Since the tables are sampled from existing datasets (e.g., WTQ, HiTab), many of them may appear in LLM pretraining corpora, especially for large proprietary models like GPT-4o. While the NIAT questions are novel, the table content may not be. This raises concerns about whether the observed performance (especially improvements on downstream benchmarks) could stem from memorization. A data contamination analysis is essential.\n\n2. **Limited discussion of format appropriateness**: The paper compares Markdown and HTML formats but doesn’t examine their suitability for different table types. For instance, Markdown cannot express hierarchical table structures (e.g., multi-row headers, merged cells), which HTML can. It would strengthen the paper to analyze when one format is preferable and explore alternative formats like JSON or XML that are common in table structure recognition tasks from computer vision.", "was_revised": true, "final_justification": "The rebuttal has addressed my concerns and I would like to keep my positive rating.", "rebuttal": "We sincerely thank you for your thorough review and for recognizing our contributions in this work! The questions and problems will be properly addressed as we reply, and your valuable advice will be fully incorporated when we refine our paper.\n\n**Q1: Lack of contamination analysis about used table data.**\n\n**A1**: Thank you for this valuable suggestion. The NIAT queries are constructed based on tables from previous academic datasets, which could indeed have been used in the pre-training stage of some proprietary models like GPT-4o and even some open-source models (as they did not fully open-source their training data). Nevertheless, none of these models achieved a near-perfect performance on the NIAT benchmark especially on the cell-locating task, which indicates that **only memorizing table content during pre-training does not guarantee a robust fine-grained table understanding ability**. In the future, we will utilize data contamination detection methods like MM-Detect [1,2] to more thoroughly investigate the influence of table content contamination on the NIAT benchmark performance. Moreover, constructing NIAT queries based on synthetic tables or tables from more recent table datasets can also alleviate this problem.\n\nAs for the performance improvements of our synthetic NIAT training data, our experimental setup was **specifically** designed to investigate \"whether enhancing NIAT capabilities can improve models' overall table understanding ability\". As a result, **we carefully synthesize fine-tuning samples of NIAT tasks that are different from the test samples in the downstream benchmarks in order to prevent test data memorization.** The experiment results demonstrate that the learned table structural and content interpretation ability with our synthesized NIAT samples can transfer to downstream tasks and thus can improve model performance. **We will use more recent tabular benchmarks with less data leakage risks like RealHiTBench [3] to evaluate model performance and further validate our findings.** In our revised manuscript, we will add a dedicated subsection titled \"Discussion on Data Contamination and Memorization\" to incorporate your advice together with related discussions and experiments.\n\n**Q2: Discussion about suitable format for different table structures and more alternative table formats.**\n\n**A2**: To save the cost of model inference, we consider three common table formats (Markdown, HTML and image) to control inference sample number. Our experimental results show that models obtain better average performance on Markdown formats for flat and hierarchical tables, but HTML leads to better performance for horizontal tables. We also find that different models can excel at different table formats. Besides, we notice that existing studies have not yet reached an agreement on this issue. For instance, [3] and [4] find that Latex and HTML formats results in the best performance in their experiments, respectively. Thus, we will dedicate more efforts to investigate this problem and evaluate model performance with other alternative table formats like XML and JSON. Related discussions and experiments will be incorporated in the limitations section.\n\n[1] Both Text and Images Leaked! A Systematic Analysis of Multimodal LLM Data Contamination.\n\n[2] Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination.\n\n[3] RealHiTBench: A Comprehensive Realistic Hierarchical Table Benchmark for Evaluating LLM-Based Table Analysis.\n\n[4] Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study"}, {"review_id": "sNjGUggVgq", "reviewer": "Reviewer_E6j3", "summary": "The paper explores the problem of dealing with long-context table question answering. The authors propose a testing benchmark NEEDLEINATABLE, which covers tables of diverse structures, formats and sizes suited to evaluate models’ underlying fine-grained understanding and perception of individual cells within tabular context.", "questions": "- The literature on tabular QA (https://ojs.aaai.org/index.php/AAAI/article/view/34739) and LLM for table understanding (https://link.springer.com/article/10.1007/s11704-024-40763-6) is quite extensive. The related work section is quite concise and not sufficiently detailed. What are the key differences with SOTA?\n- A more detailed description of the experiments, parameter settings, and ablation studies is necessary to ensure reproducibility of the research. \n- What are the execution times and memory requirements?", "limitations": "- The cost of using proprietary LLMs \n- The risks of LLMs' hallucination/bias\n- The lack of human-in-the-loop (no relevance feedback)", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "+ Clear problem formulation and motivations\n+ Interesting research direction\n+ Promising results\n\n- Incomplete/too high-level related works overview\n- Lack of details on reproducibility  (code/experiments/settings)", "was_revised": true, "final_justification": "The paper contribution is solid and well presented. The empirical evaluations are sufficient to support the initial claims. Therefore, I recommend paper acceptance.", "rebuttal": "We sincerely thank you for your valuable comments and positive feedback. Your questions will be properly addressed as we reply and your constructive suggestions will be fully incorporated in the next version of our paper.\n\n**Q1: Add more detailed discussion of related work and difference with previous SOTA methods.**\n\n**A1:** Due to the strict page limits of the initial submission, we provide a relatively concise discussion of related works and focus on distinguishing our work from the most recent and relevant lines of research. For instance, we clarify how NIAT is complementary to related long-context benchmarks and tabular benchmarks. Moreover, we totally agree that providing a more detailed context can better highlight our work's unique contributions. A more extensive discussion of related literatures like [1] and [2] will be included in the future version.\n\nWhen it comes to previous SOTA models like TableGPT2 and QwQ-32B, existing studies primarily evaluate their high-level table understanding ability with complex tasks as testbeds, such as tabular QA and fact verification. By contrast, our constructed NIAT benchmark aims to evaluate the fine-grained understanding of each input table cell from the long-context perspective, making it complementary to previous benchmarks. In addition, our systematic evaluation provides a comprehensive performance landscape of different models including previous SOTA methods, revealing their potential drawbacks towards robust long-context table understanding capacities.\n\n**Q2: Add detailed experimental settings and ablation studies.**\n\n**A2:**  Due to space limitation, we provide important experimental settings in the Section 3.3 and Section 4.2. More detailed experimental settings such as parameter settings and training details were provided in the Appendix C. Ablation studies were also conducted and results were shown in Appendix D.2. Preliminary benchmark data were also included in the supplementary materials. We will revise the main body of the paper to include more key details and add more explicit pointers to the relevant sections in the appendix. Besides, related data, model and code will be open-sourced to further improve reproducibility and facilitate future research in this area.\n\n**Q3: What is the the execution time and memory requirements.**\n\n**A3:** As mentioned in the Appendix C.3, we use 8 NVIDIA A100-80G GPUs for all experiments. For model training, we utilized the Megatron-LM framework for distributed training of Llama3.1-8B and Qwen2.5-7B on 12K synthetic data samples, which took approximately 3 hours on 32 A100 GPUs. For inference, the vLLM framework was employed to accelerate the process for our 287K test samples. The inference time is dependent on model size; for instance, Llama3.1-8B-Instruct achieved a throughput of approximately 700 samples per hour using tensor parallelism on 4 A100 GPUs.\n\nThank you once again for helping us improve our paper. Above information together with more discussions like the cost of proprietary LLM-APIs and the risk of LLMs' hallucination will be included in the future version.\n\n[1] TableBench: A Comprehensive and Complex Benchmark for Table Question Answering\n\n[2] Large language model for table processing: a survey"}, {"review_id": "TFPv08JfiO", "reviewer": "Reviewer_UMxE", "summary": "This paper introduces a new long-context tabular benchmark dataset, named Needle-in-a-table (NIAT), to benchmark language models' performance on extracting the target cell based on location or lookup queries in long tables. It performs a thorough analysis to evaluate various LLMs and MLLMs performance on different table structures and formats. The authors also propose a simple data synthesis method, using strong models to generate reseasoning processes to create the fine-tuning data on the NIAT tasks, demonstrating significant improvement on four downstream tasks, including table question answering and table fact verifications.", "questions": "* Maybe I didn't understand it correctly. Could the authors clarify the difference between Multi-Slash and Local-Triangle patterns in Figure 3? It seems to me that in the Local-Triangle pattern, the attention weights are also concentrated on the cell tokens in the same column. Also, it would be clearer to include the legend in Figure 3.\n* In Figure 4, for the Llama3.1-8B models trained with the NIAT data, it seems to me there is no significant improvement on per-cell accuracy, unlike Qwen2.5-7B models. There seems to be even a degradation in training using 32x32 cropped data on Llama3.1-8B models. Could the authors comment on the potential reasons? Also in Table 5 in Appendix, the table shows improvement on training Llama3.1-8 B-Instruct models with cell-locating and cell-lookup data. Why is there a difference?\n* In Table 3, there is also a degradation on TabFact downstream task on Qwen2.5-7B-Instruct trained using NIAT, could the authors comment on the potential reasons?\n* In Table 5, there is a typo in the second row, second column: 17.33 should be in bold for NIAT with 12 table size. Also, it would be more consistent to use 8x8, 12x12 to denote the table size. Additionally, in this table, there seems to be a degradation on the TABMWP dataset with including cell-locating and cell-lookup data. Why is it inconsistent with the results in Table 3, where we see consistent improvement on TABMWP dataset with Llama3.1-8B-Instruct models?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n* Proposes an important benchmark dataset to measure LLM and MLLM's long-context tabular understanding capabilities. \n* The proposed NIAT dataset creation is quite scalable and reliable, as it only involves cell locating and cell lookup queries.\n* The proposed benchmark dataset covers different table structures, formats and varying sizes.\n* Achieves significant improvement gains on downstream tasks using the proposed data synthesis on NIAT tasks.\n\n**Weakness**\n* The maximum length in the NIAT dataset is up to 120K, still not close to the token limit in some long-context LLMs. For example, Gemini supports the context window up to 1M tokens.\n* The NIAT tasks only involve cell locating and cell lookup, and could involve more complex tasks.", "was_revised": true, "final_justification": "The authors have clarified my confusion on Figure 3 and also corrected the ablation study results during the rebuttal phase.\n\nIn general, I believe this is an insightful work to investigate LLM's capabilities in understanding long structured tables. Currently, I haven't seen a lot of work in this area. The proposed benchmark creation is technically sound and quite intuitive. Therefore, I have given the final rating of 5.", "rebuttal": "We sincerely thank you for the precious effort and time spent in carefully reviewing our paper. The questions will be properly addressed as we reply, and your valuable advice will be fully incorporated when we refine our paper.\n\n**Q1: The NIAT benchmark does not involve more complex tasks, and the prompt length is less than the token limit of some long-context LLMs like 1M tokens.**\n\n**A1:** To provide the community with the first benchmark on the long-context ability of individual cells within structured tables while aiming to save the cost of data collection, we build the NIAT benchmark based on public table datasets and design two table understanding tasks (cell-locating and cell-lookup), which are simple for humans but can provide a basic testbed for LLMs' understanding of table structures and individual table cells. However, as pointed out in the limitation section, we also totally agree that more complex tabular tasks together with larger tables and more hybrid content (like table-text scenario) should be considered to provide a more thorough evaluation of long-context table understanding. For instance, as mentioned in the related work, a concurrent benchmark LongBench-v2 evaluates the high-level reasoning ability of LLMs over long structured tables using 18 question-answering samples of more complex tasks, which we think is really a good start along this direction.\n\n**Q2: The difference between Multi-Slash and Local-Triangle patterns in Figure 3.**\n\n**A2:** We apologize if our descriptions lead to some misunderstanding. The horizontal and vertical coordinates (i.e., (m, n) ) in Figure 3 represents the table cell in the m-th row and n-th column. Note that the input table is serialized into Markdown sequence in a left-to-right and top-to-bottom order. The cell color represents different attention weights, where brighter colors (yellow and green) indicate larger attention weights and darker colors (dark purple) indicate smaller attention weights.\n\nIn the Multi-Slash pattern, the attention weights of one cell are concentrated on the preceding cells in the same column, e.g., the cell at (3,3) (3-rd row and 3-rd column) attend to cells at (2,3) and (1,3). In the Local-Triangle pattern, the attention weights of one cells are concentrated on the preceding cells in the same row, e.g., the cell at (3,3) attend to cells at (3,1) and (3,2). We will add these examples and legend to Figure 3 to make it more intuitive.\n\n**Q3: The performance of Llama3.1-8B on cell-locating (on cropped markdown tables) and TABMWP task differs between Figure 4, Table 3 (main body) and Table 5 (ablation study in appendix).**\n\n**A3:** Thank you for this valuable question which helps us find an important reporting error in Table 5 (ablation study) in the appendix, where we mistakenly confused the ablation experiment results on a subset of test samples (current wrong version) with results on all test samples (the intended version). We sincerely apologize for this oversight, and provide the correct ablation results below, which aligns with the trend in Figure 4 and Table 3 in the main body. The Table 5 will be corrected accordingly in the revised manuscript. Thanks again for helping us catch this error.\n\n| **Model** | **8 x 8** | **12 x 12** | **16 x 16** | **20 x 20** | **24 x 24** | **28 x 28** | **32 x 32** | **WTQ** | **TabFact** | **HiTab** | **TABMWP** |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| *Llama3.1-8B-Instruct* | 16.88 | **13.56** | **12.94** | **8.98** | **6.34** | **7.25** | **3.83** | 49.90 | 62.80 | 26.10 | 54.78 |\n| + Cell-Locating & Cell-Lookup | **20.10** | 12.75 | 10.21 | 5.51 | 4.51 | 4.88 | 2.49 | **67.43** | **78.57** | **49.41** | 66.15 |\n| + Cell-Locating | 19.76 | 13.49 | 12.06 | 7.86 | 5.31 | 6.22 | 3.07 | 67.33 | 67.45 | 33.44 | 70.50 |\n| + Cell-Lookup | 18.85 | 13.13 | 12.15 | 8.36 | 6.22 | 5.83 | 3.21 | 59.00 | 53.50 | 35.00 | 69.44 |\n| + 4 downstream datasets | 16.28 | 11.12 | 11.41 | 6.11 | 5.92 | 5.40 | 2.17 | 64.78 | 78.13 | 48.22 | **81.79** |\n\n**Q4: In the cell-locating task on cropped markdown tables (Figure 4), NIAT fine-tuning benefits Qwen2.5-7B far more than Llama3.1-8B.**\n\n**A4:** This is indeed a curious phenomenon and we present our results honestly. We speculate the potential reasons are two-fold and are related with models' pre-existing capabilities. On the one hand, as noted in the Qwen2.5 technical report [1], Qwen2.5-series underwent specialized post-training on structured tabular data, which may lead to better table structure understanding ability that our CoT-based fine-tuning data could more effectively \"activate\" and build upon, leading to a significant boost in table structure comprehension. On the other hand, exisiting RL studies [2, 3] have shown that Qwen2.5-series even the base version possesses better reasoning ability, which makes it more easier to incentivize advanced capabilities for complex multi-step tasks, e.g., interpreting the table structure row by row and then pinpointing the answer cell by column index. By contrast, Llama3.1-series needs extra mid-training to achieve the comparable performance.\n\n**Q5: In Table 3 (main body), the performance of Qwen2.5-7B decreases with synthetic NIAT fine-tuning data.**\n\n**A5:**  We speculate a potential reason is that the model may be confused between the GPT-4o generated NIAT fine-tuning data and the learned table fact verification ability during Qwen2.5's post-training, leading to some performance fluctuation, as Qwen2.5 could have used these datasets within post-training stage. Another potential reason is that the NIAT fine-tuned Qwen2.5 does not strictly follow the output format requirements on some test samples, where the evaluation script may wrongly judged them as incorrect samples (i.e., false negative). We will further look into the model output and improve the answer parsing and evaluation scripts if necessary.\n\n**Q6: Typo and other presentation suggestions**\n\n**A6:**  The paper presentation will be revised based on your valuable suggestions. Thanks again for your careful reviewing.\n\n[1] Qwen2.5 Technical Report\n\n[2] OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling\n\n[3] Understanding R1-Zero-Like Training: A Critical Perspective"}, {"review_id": "ubuZf1Dp5g", "reviewer": "Reviewer_QGzx", "summary": "The paper is concerned with evaluating LLMs and MLLMs on structured data.\nA benchmark called NIAT (Needle in a Table) is proposed that tries to evaluate the fine-grained, cell-based understanding of these models.\nThis is achieved using two tasks cell lookup and cell locate, trying to find a needle (the cell) in a table.\nMultiple LLMs and  MLLMs fail at this seemingly simple task while being able to do more difficult reasoning tasks on the tables.\nA dataset (similar to the benchmark task) is constructed  and fine-tuning on this dataset improves performance.\nFinally, the paper conducts analysis and ablation using the benchmark and dataset. Notably, it reports similar findings to lost in the middle but for tabular data.", "questions": "- The models under-perform in locate task compared to  look-up task. It seems like more of a \"geometry\" problem than table understanding problem (The model might understand the table but have difficulty finding 5th column 7th row). There are various other encodings (some of which would not even support cell based referencing). This raises two questions 1) Why is locating a cell important? 2) Would other encodings (like JSON, Data Matrix, DFLoader [1]) mitigate or solve the problem?\n- In section 4.1 for the locate task the train/test split is used while for lookup, some other datasets are used. It is stated the reason is \"to avoid overlapping with lookup questions in the test data of downstream benchmarks\". What explains the difference between locate and lookup?\n- Can more details be shared regarding Figure 3? What layer, model, size is used? Is it one example or average of many? Does the pattern occur across various models and layers and encodings? \n\n[1] Fang, Xi, et al. \"Large Language Models (LLMs) on Tabular Data: Prediction, Generation, and Understanding--A Survey.\" arXiv preprint arXiv:2402.17944 (2024).", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n- The proposed method is general and applicable to LLMs, MLLMs and (some) models for tabular data.\n- Identifies a simple failure mode (cell value by location).\n\n# Weaknesses\n- The paper tries to address many subjects while not going deep into any. (New benchmark creation that is applicable to wide range of models, Analysis on size and location, long context behavior, creating a new dataset and fine-tuning. details below)\n- Long context behavior is not sufficiently explored. 80\\% of the tables are less than 4k tokens. The largest table in lost in the middle analysis is 32 x 32. Assuming 10 token per cell, this results in ~10k tokens which is not long context. \n- The part regarding attention pattern analysis (Figure 3) seems under-explored. It is not clear which model on which task and which layer is used. Is the pattern repeated in many examples?\n- The models used for evaluation (both LLM and MLLM) seem limited in scope. Many of them have newer variant of the same size and larger variants with better performance. (The same 3k sub-sample method can be used to mitigate cost.) \n- The claims in lines 257 \\~ 259 and in 251 \\~ 255 are not completely justified due to the above points. Various models like PaliGemma can perform segmentation even with image patches. To claim \"their success may not be grounded on a robust and reliable long-context table understanding ability\" requires investigating various table encodings, model sizes and more long context analysis. \n- There are mismatches between main text and appendix. Main text reports 120k to be longest context while Table 1 in B.1 says otherwise. (different tokenizers?)\n- The results of fine-tuning in Table 5 (ablation in appendix) are not conclusive and while there are some improvement in some tasks, some times there is decreased performance for NIAT itself (especially for smaller tables.)\n- The paper claims success in some tasks might be due to leakage (line 263) yet it uses existing tables to create the benchmark.", "was_revised": true, "final_justification": "After discussion some of my concerns were addressed and I decided to increase the rating from 3 to 4.", "rebuttal": "We sincerely thank you for the precious effort spent in carefully reviewing our paper. The questions will be properly addressed as we reply and your valuable advice will be fully incorporated.\n\n**Q1: The paper tries to address many subjects while not going deep into any.**\n\n**A1:** We totally understand the concern about the research depth in this work. However, **facing the gap in existing long-context and tabular benchmarks regarding long-context table understanding problem, we think that taking a solid first step is equally or even more important.** In this work, we not only introduce the new NIAT task and construct a large-scale benchmark for future study, but also conduct heavy and massive experiments to evaluate many LLMs and MLLMs with 300K+ samples, revealing their pros and cons in this new problem and providing valuable insights. Moreover, we also attempt to improve models' performance with a simple yet effective data synthesis method, serving as a fundamental baseline for following methods. **Therefore, we strongly believe that we make solid contributions towards this new problem with necessary in-depth investigations. Nevertheless, we also think this new problem deserves future efforts more than just one paper and we would like to further going deep into this problem**.\n\n**Q2: The NIAT table is not long enough to reach extremely large context-window.**\n\n**A2:** As mentioned in the Limitation section, to provide the community with the first benchmark on the long-context ability of structured tables with limited resources, we build NIAT benchmark based on tables from public datasets. The experimental results have shown that some powerful models struggle within these medium-sized tables even small tables, thus we anticipate that models would suffer more significant performance decline on longer and larger tables. We agree that more complex tasks together with larger tables and more hybrid content (like table-text scenario) should be considered to provide a more thorough evaluation of long-context table understanding, which we leave to future work.\n\n**Q3: The experimental settings about attention pattern analysis (Figure 3). Which model on which task and which layer is used? Is the pattern repeated in many examples?**\n\n**A3:** The input samples are from the WTQ benchmark (TQA task) and tables are serialized into Markdown format. We observe attention patterns from different attention heads from different layers of Llama3.1-series models, where we found that the 'Multi-Slash' attention pattern prevalently exist in the upper layers (20+ layers) while the 'Local-Triangle' attention pattern could exist in each layer. The illustrated attention patterns are not anomalies and are observed in many examples (100+). We will add these experimental details in the next version together with the corresponding attention distribution images.\n\n**Q4: Evaluating recent models of new version and larger variants.**\n\n**A4:** **Based on the sampled 6K test data, we evaluate some recent models especially those with R1-style reasoning abilities such as GLM-4.1V-Thinking and Qwen3 (in no-thinking mode).** From the results provided below, we can find that recent models achieve better performance on NIAT benchmark with enhanced reasoning abilities, but there is still room for improvements.\n\n| Models | Cell-Locating |  |  |  | Cell-Lookup |  |  |  |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|  | Flat | Hie. | Hori. | Ave. | Flat | Hie. | Hori. | Ave. |\n| Qwen2.5-VL-32B-Instruct | 23.30 | 14.60 | 21.41 | 19.77 | 74.82 | 84.77 | 44.01 | 67.87 |\n| Qwen2.5-VL-72B-Instruct | 25.80 | **18.60** | 25.13 | 23.18 | 72.72 | **87.68** | 48.94 | 69.78 |\n| Skywork-R1V3-38B | **36.07** | 11.33 | 26.11 | 24.50 | 56.87 | 69.04 | **65.86** | 63.92 |\n| GLM-4.1V-Thinking-9B | 27.77 | 18.24 | **31.54** | **25.85** | **76.93** | 84.28 | 64.15 | **75.12** |\n|  |  |  |  |  |  |  |  |  |\n| Qwen3-14B | 18.22 | 2.61 | 22.00 | 13.99 | 72.60 | 76.20 | 63.00 | 70.60 |\n| Qwen3-32B | **22.51** | **3.52** | 27.67 | **17.34** | 73.90 | 80.31 | 66.50 | 73.57 |\n| Qwen3-30B-A3B | 21.82 | 2.51 | 25.47 | 16.48 | **78.90** | **85.39** | **71.02** | **78.44** |\n\n**Q5: The calculated benchmark prompt length differs between main body and appendix.**\n\n**A5:** In section 3.2 Benchmark Statistics in the main body, we reported an rounded word number of input prompt by whilespace-splited tokenizer (line 165). In appendix, the accurate token length of input prompt by Llama3.1-8B-Instruct tokenizer is reported, leading to a mismatch which will be unified in the next version.\n\n**Q6: In Table 5 (ablation study), fine-tuning with NIAT data brings improvements on some tasks, but not bring consistent improvements in cell-locating tasks on cropped tables of all sizes.**\n\n**A6:**  This is indeed a curious phenomenon and we present our results honestly. As shown in Figure 4, we can find that NIAT fine-tuning brings significant performance increase for Qwen2.5-7B but less gains and even slightly degeneration for Llama3.1-8B in cell-locating tasks on some cropped tables. For one thing, one potential reason is that the used cropped tables (15 tables for each size) for evaluation are still limited in amount, thus leading to normal performance fluctuation. From Table 3 in main body, we observe that fine-tuning with synthesized data effectively improves Llama3.1-8B performance in two NIAT tasks when evaluated with full 287K test data.\n\nFor another, this could be attributed to models' pre-existing capabilities. On the one hand, Qwen2.5-series underwent specialized post-training on structured tabular data [1], which may lead to better table structure understanding ability that our CoT-based fine-tuning data could more effectively \"activate\" and build upon, leading to a significant boost in table structure comprehension. On the other hand, exisiting RL studies [2, 3] have shown that Qwen2.5-series possesses better reasoning ability, which makes it more easier to incentivize advanced capabilities for complex multi-step tasks, e.g., interpreting the table structure row by row and then pinpointing the answer cell by column index. By contrast, Llama3.1-series needs extra mid-training to achieve the comparable performance. We will perform ablation studies with Qwen2.5-series to further investigate this issue.\n\n**Q7: Why cell-locating is important? Can using other table formats like DataFrame mitigate or solve the problem?**\n\n**A7:** We fully understand the concern that cell-locating seems to be a trivial task compared to more complex tabular tasks like data analysis or table manipulation. However, **we think it serve as a fundamental probe task to evaluate models' basic understanding of diverse table structures, just like many efforts that have been spent to identified simple yet important failure modes of seemingly perfect (M)LLMs**. For instance, an MLLM is asked to analyze a table in a document image, how can we fully trust its results if it even does not know what is the content at the 1st row and 1st column.\n\n**Using other table formats (like DataFrame) with code-based solutions (like Text2SQL or Python) can indeed mitigate this problem in certain scenarios.** (1) For scenarios which only consider flat tables or database-like tables, we also think that semantic parsing methods like Text2SQL are preferable [4] as they can manipulate tables with executable code. (2) However, real-world application scenarios often involve processing large tables with complex structures, such as hierarchical headers[5], merged cells, nested tables and even multiple tables within a single document[6], whose table formats could not support standard processing of DataFrame-like semantic parsing approaches. Such practical use cases make it valuable and necessary to study the LLM-based understanding ability of long and structured tables of diverse formats within its context window.\n\n**Q8: Why synthesize cell-lookup questions of new types for NIAT fine-tuning?**\n\n**A8:**  Our synthesized NIAT fine-tuning data are based training tables from public datasets. As the downstream benchmarks like WTQ could also contain lookup questions, we generate cell-lookup questions of new types for NIAT fine-tuning to maximumly avoid the overlap between training data and test data in benchmarks.\n\n**Q9: Potential data contamination of used tables from previous datasets.**\n\n**A9:** The NIAT queries are constructed based on tables from previous academic datasets, which could indeed have been used in the pre-training stage of some proprietary models like GPT-4o and even some open-source models (as they did not fully open-source their training data). Nevertheless, none of these models achieved a near-perfect performance on the NIAT benchmark especially on the cell-locating task, which indicates that only memorizing table content during pre-training does not guarantee a robust fine-grained table understanding ability. In the future, **we will utilize data contamination detection methods like MM-Detect [7] to more thoroughly investigate the influence of table content contamination on the NIAT benchmark performance. Moreover, constructing NIAT queries based on synthetic tables or tables from more recent table datasets can also alleviate this problem.** A dedicated subsection will be added to incorporate your advice together with related discussions and experiments.\n\n[1] Qwen2.5 Technical Report\n\n[2] OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling\n\n[3] Understanding R1-Zero-Like Training: A Critical Perspective\n\n[4] DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction\n\n[5] HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generation\n\n[6] MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data\n\n[7] Both Text and Images Leaked! A Systematic Analysis of Multimodal LLM Data Contamination."}]

Author rebuttal:

Meta-review: The paper introduces NIAT (NeedleInATable), a new benchmark to evaluate long-context understanding in LLMs and MLLMs on structured tabular data. The authors highlight that existing benchmarks overlook the unique challenges of tables, such as precise structural and cell-level reasoning. NIAT includes two tasks: cell-locating queries (finding a cell by its row/column) and cell-lookup queries (finding a cell based on natural language).

Consensus & Decision
- All reviewers rated the paper positively, finding its motivation and contributions to be significant and novel. They praised the well-constructed benchmark and the empirical finding that LLMs struggle with cell-locating tasks despite performing well on more complex lookup tasks. The proposed data synthesis method for fine-tuning models was also considered effective.
- The primary concerns raised by reviewers—data contamination, limited discussion of table formats, and reproducibility—were all satisfactorily addressed by the authors in their rebuttal. They committed to conducting further analysis, expanding discussions, and open-sourcing their code.
- Based on the paper's clear strengths and the authors' effective responses to all concerns, the AC recommend acceptance. The work provides a critical tool for diagnosing and improving LLMs' ability to handle structured data, a key area of future research.

Suggestions:
- The authors' approach to synthetic data generation should be contextualized within recent related work like "TableRAG: Million-Token Table Understanding with Language Models" (NeurIPS'24) and "Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding" (ICLR'24).
Decision: Accept (poster)