Title: NeedleInATable: Exploring Long-Context Capability of Large Language Models towards Long-Structured Tables
Venue: NeurIPS 2025
Accepted: yes
Keywords: Long-context

table understanding

structured data
Abstract: Processing structured tabular data, particularly large and lengthy tables, constitutes a fundamental yet challenging task for large language models (LLMs). However, existing long-context benchmarks like Needle-in-a-Haystack primarily focus on unstructured text, neglecting the challenge of diverse structured tables. Meanwhile, previous tabular benchmarks mainly consider downstream tasks that require high-level reasoning abilities, and overlook models' underlying fine-grained perception of individual table cells, which is crucial for practical and robust LLM-based table applications. To address this gap, we introduce \textsc{NeedleInATable} (NIAT), a new long-context tabular benchmark that treats each table cell as a ``needle'' and requires models to extract the target cell based on cell locations or lookup questions. Our comprehensive evaluation of various LLMs and multimodal LLMs reveals a substantial performance gap between popular downstream tabular tasks and the simpler NIAT task, suggesting that they may rely on dataset-specific correlations or shortcuts to obtain better benchmark results but lack truly robust long-context understanding towards structured tables. Furthermore, we demonstrate that using synthesized NIAT training data can effectively improve performance on both NIAT task and downstream tabular tasks, which validates the importance of NIAT capability for LLMs' genuine table understanding ability. Our data, code and models will be released to facilitate future research.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is concerned with evaluating LLMs and MLLMs on structured data.\nA benchmark called NIAT (Needle in a Table) is proposed that tries to evaluate the fine-grained, cell-based understanding of these models.\nThis is achieved using two tasks cell lookup and cell locate, trying to find a needle (the cell) in a table.\nMultiple LLMs and  MLLMs fail at this seemingly simple task while being able to do more difficult reasoning tasks on the tables.\nA dataset (similar to the benchmark task) is constructed  and fine-tuning on this dataset improves performance.\nFinally, the paper conducts analysis and ablation using the benchmark and dataset. Notably, it reports similar findings to lost in the middle but for tabular data.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- The proposed method is general and applicable to LLMs, MLLMs and (some) models for tabular data.\n- Identifies a simple failure mode (cell value by location).\n\n# Weaknesses\n- The paper tries to address many subjects while not going deep into any. (New benchmark creation that is applicable to wide range of models, Analysis on size and location, long context behavior, creating a new dataset and fine-tuning. details below)\n- Long context behavior is not sufficiently explored. 80\\% of the tables are less than 4k tokens. The largest table in lost in the middle analysis is 32 x 32. Assuming 10 token per cell, this results in ~10k tokens which is not long context. \n- The part regarding attention pattern analysis (Figure 3) seems under-explored. It is not clear which model on which task and which layer is used. Is the pattern repeated in many examples?\n- The models used for evaluation (both LLM and MLLM) seem limited in scope. Many of them have newer variant of the same size and larger variants with better performance. (The same 3k sub-sample method can be used to mitigate cost.) \n- The claims in lines 257 \\~ 259 and in 251 \\~ 255 are not completely justified due to the above points. Various models like PaliGemma can perform segmentation even with image patches. To claim \"their success may not be grounded on a robust and reliable long-context table understanding ability\" requires investigating various table encodings, model sizes and more long context analysis. \n- There are mismatches between main text and appendix. Main text reports 120k to be longest context while Table 1 in B.1 says otherwise. (different tokenizers?)\n- The results of fine-tuning in Table 5 (ablation in appendix) are not conclusive and while there are some improvement in some tasks, some times there is decreased performance for NIAT itself (especially for smaller tables.)\n- The paper claims success in some tasks might be due to leakage (line 263) yet it uses existing tables to create the benchmark.\n\nQUESTIONS:\n- The models under-perform in locate task compared to  look-up task. It seems like more of a \"geometry\" problem than table understanding problem (The model might understand the table but have difficulty finding 5th column 7th row). There are various other encodings (some of which would not even support cell based referencing). This raises two questions 1) Why is locating a cell important? 2) Would other encodings (like JSON, Data Matrix, DFLoader [1]) mitigate or solve the problem?\n- In section 4.1 for the locate task the train/test split is used while for lookup, some other datasets are used. It is stated the reason is \"to avoid overlapping with lookup questions in the test data of downstream benchmarks\". What explains the difference between locate and lookup?\n- Can more details be shared regarding Figure 3? What layer, model, size is used? Is it one example or average of many? Does the pattern occur across various models and layers and encodings? \n\n[1] Fang, Xi, et al. \"Large Language Models (LLMs) on Tabular Data: Prediction, Generation, and Understanding--A Survey.\" arXiv preprint arXiv:2402.17944 (2024).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new long-context tabular benchmark dataset, named Needle-in-a-table (NIAT), to benchmark language models' performance on extracting the target cell based on location or lookup queries in long tables. It performs a thorough analysis to evaluate various LLMs and MLLMs performance on different table structures and formats. The authors also propose a simple data synthesis method, using strong models to generate reseasoning processes to create the fine-tuning data on the NIAT tasks, demonstrating significant improvement on four downstream tasks, including table question answering and table fact verifications.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n* Proposes an important benchmark dataset to measure LLM and MLLM's long-context tabular understanding capabilities. \n* The proposed NIAT dataset creation is quite scalable and reliable, as it only involves cell locating and cell lookup queries.\n* The proposed benchmark dataset covers different table structures, formats and varying sizes.\n* Achieves significant improvement gains on downstream tasks using the proposed data synthesis on NIAT tasks.\n\n**Weakness**\n* The maximum length in the NIAT dataset is up to 120K, still not close to the token limit in some long-context LLMs. For example, Gemini supports the context window up to 1M tokens.\n* The NIAT tasks only involve cell locating and cell lookup, and could involve more complex tasks.\n\nQUESTIONS:\n* Maybe I didn't understand it correctly. Could the authors clarify the difference between Multi-Slash and Local-Triangle patterns in Figure 3? It seems to me that in the Local-Triangle pattern, the attention weights are also concentrated on the cell tokens in the same column. Also, it would be clearer to include the legend in Figure 3.\n* In Figure 4, for the Llama3.1-8B models trained with the NIAT data, it seems to me there is no significant improvement on per-cell accuracy, unlike Qwen2.5-7B models. There seems to be even a degradation in training using 32x32 cropped data on Llama3.1-8B models. Could the authors comment on the potential reasons? Also in Table 5 in Appendix, the table shows improvement on training Llama3.1-8 B-Instruct models with cell-locating and cell-lookup data. Why is there a difference?\n* In Table 3, there is also a degradation on TabFact downstream task on Qwen2.5-7B-Instruct trained using NIAT, could the authors comment on the potential reasons?\n* In Table 5, there is a typo in the second row, second column: 17.33 should be in bold for NIAT with 12 table size. Also, it would be more consistent to use 8x8, 12x12 to denote the table size. Additionally, in this table, there seems to be a degradation on the TABMWP dataset with including cell-locating and cell-lookup data. Why is it inconsistent with the results in Table 3, where we see consistent improvement on TABMWP dataset with Llama3.1-8B-Instruct models?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

comments: SUMMARY:
The paper explores the problem of dealing with long-context table question answering. The authors propose a testing benchmark NEEDLEINATABLE, which covers tables of diverse structures, formats and sizes suited to evaluate models’ underlying fine-grained understanding and perception of individual cells within tabular context.

STRENGTHS AND WEAKNESSES:
+ Clear problem formulation and motivations
+ Interesting research direction
+ Promising results

- Incomplete/too high-level related works overview
- Lack of details on reproducibility  (code/experiments/settings)

QUESTIONS:
- The literature on tabular QA (https://ojs.aaai.org/index.php/AAAI/article/view/34739) and LLM for table understanding (https://link.springer.com/article/10.1007/s11704-024-40763-6) is quite extensive. The related work section is quite concise and not sufficiently detailed. What are the key differences with SOTA?
- A more detailed description of the experiments, parameter settings, and ablation studies is necessary to ensure reproducibility of the research. 
- What are the execution times and memory requirements?

LIMITATIONS:
- The cost of using proprietary LLMs 
- The risks of LLMs' hallucination/bias
- The lack of human-in-the-loop (no relevance feedback)
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces NeedleInATable (NIAT), a novel benchmark for evaluating the long-context understanding capabilities of large language models (LLMs) in structured tabular data. The authors argue that prior long-context benchmarks have focused almost exclusively on unstructured text, overlooking structured tables’ unique challenges—particularly the need for precise structural reasoning and cell-level localization.\n\nNIAT is built upon public table datasets and defines two task types: (1) cell-locating queries that require retrieving the content at a specified row and column, and (2) cell-lookup queries based on simple natural language questions. The authors evaluate a range of LLMs and multimodal LLMs (MLLMs) and reveal large performance gaps between the two query types, highlighting weaknesses in structural understanding. They further propose a data synthesis method using GPT-4o with Chain-of-Thought (CoT) reasoning to fine-tune models, showing improvements on both NIAT and downstream table QA tasks.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n* **Clear motivation**: The work tackles an under-explored but critical problem—the structural localization of information in long tables—complementing existing long-context benchmarks that focus solely on plain text.\n* **Well-constructed benchmark**: NIAT covers different table formats (Markdown, HTML, image) and structures (flat, hierarchical, horizontal), and provides detailed statistics and task definitions.\n* **Empirical insight**: The finding that LLMs perform poorly on cell-locating but decently on lookup-style tasks provides valuable diagnostic information for the community.\n* **Effective synthetic data strategy**: The CoT-style data generation and fine-tuning approach leads to tangible gains across NIAT and traditional table QA datasets.\n\n### **Weaknesses**\n\n1. **Lack of contamination analysis**: Since the tables are sampled from existing datasets (e.g., WTQ, HiTab), many of them may appear in LLM pretraining corpora, especially for large proprietary models like GPT-4o. While the NIAT questions are novel, the table content may not be. This raises concerns about whether the observed performance (especially improvements on downstream benchmarks) could stem from memorization. A data contamination analysis is essential.\n\n2. **Limited discussion of format appropriateness**: The paper compares Markdown and HTML formats but doesn’t examine their suitability for different table types. For instance, Markdown cannot express hierarchical table structures (e.g., multi-row headers, merged cells), which HTML can. It would strengthen the paper to analyze when one format is preferable and explore alternative formats like JSON or XML that are common in table structure recognition tasks from computer vision.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nThe authors acknowledge that their benchmark uses medium-sized tables and only supports English. However, they do not explicitly discuss the risk of table content overlap with LLM training corpora, or the format-structure mismatch issues. These should be addressed in the limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}