Title: Table as a Modality for Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large language model

Table reasoning
Abstract: To migrate the remarkable successes of Large Language Models (LLMs), the community has made numerous efforts to generalize them to the table reasoning tasks for the widely deployed tabular data. Despite that, in this work, by showing a probing experiment on our proposed StructQA benchmark, we postulate that even the most advanced LLMs (such as GPTs) may still fall short of coping with tabular data. More specifically, the current scheme often simply relies on serializing the tabular data, together with the meta information, then inputting them through the LLMs. We argue that the loss of structural information is the root of this shortcoming. In this work, we further propose TAMO, which bears an ideology to treat the tables as an independent modality integrated with the text tokens. The resulting model in TAMO is a multimodal framework consisting of a hypergraph neural network as the global table encoder seamlessly integrated with the mainstream LLM. Empirical results on various benchmarking datasets, including HiTab, WikiTQ, WikiSQL, FeTaQA, and StructQA, have demonstrated significant improvements on generalization with an average relative gain of **42.65%**.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper reports an approach of treating tables as a separate modality - so tables are encoded using a hyper graph transformer with multi set semantics to handle permutation invariance,  and then 'fused' with the text representation of the table using standard multi modal architectures to perform better table QA.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.  The use of hyper graphs as table encoders is interesting, as is the use of pretraining autoregressive tasks that teach the model about the structure of tables.\n2.  The results seem interesting although I do have some questions about the baselines.\n\nWeaknesses:\n1.  Use of the inference only baseline to make all comparisons seemed a bit weak to me - was this a few shot prompt case?  Were some structural examples provided?\n2.  What exactly are the specialist SOTA methods - why do they not constitute a baseline and why is the fact that their performance better in those papers ignored?  Also, why aren't they evaluated on StructQA?  If the point is that these specialize SOTA methods (maybe refer to them by the citation in the table?) were specialized for the task - we do not know that do we we?  As in, those methods have not been evaluated on all the datasets mentioned here?\n3.  If the point also is that this approach is more 'general' somehow - I'd question that as well because the table does need to be encoded by a specialized network.\n\nQUESTIONS:\n1.  Use of the inference only baseline to make all comparisons seemed a bit weak to me - was this a few shot prompt case?  Were some structural examples provided?\n2.  What exactly are the specialist SOTA methods - why do they not constitute a baseline and why is the fact that their performance better in those papers ignored?  Also, why aren't they evaluated on StructQA?  If the point is that these specialize SOTA methods (maybe refer to them by the citation in the table?) were specialized for the task - we do not know that do we we?  As in, those methods have not been evaluated on all the datasets mentioned here?\n\n\nThank you for addressing my questions - I believe you have addressed them satisfactorily.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces TAMO which emphasizes on treating tables as an independent modality for tabular Question-Answering (QA) tasks. The paper presents a novel instruction tuning paradigm by combining structural information in tables and semantic information in text towards augmenting the capabilities of a pretrained LLM towards robust QA. TAMO further introduces the StructQA benchmark which evaluates various impending challenges like permutation invariance, structural variations etc. important for downstream tabular learning tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths -**\n\n1. TAMO introduces a novel instruction tuning architecture which combines text and hypergraph representations of tables towards Tabular QA tasks.\n\n2. The introduction of StructQA benchmark to evaluate permutation and structural invariance - a necessary evil in tabular QA tasks is a key contribution.\n\n**Weaknesses -**\n\n1. To the best of my knowledge, the introduction of Multi-Modal graph and text based tabular learning is not novel to TAMO. An example would be TabGLM (Majee et al., 2025) which combines graph and textual representations from tables for auxiliary tasks. However, the introduction of a multimodal instruction tuning model for Tabular QA would be a more appropriate positioning for TAMO.\n\n2. The claim in line 182 that TAMO is cost-effective may not be true or not clearly supported. The model in Figure 4 uses two encoders and an LLM instead of single encoders in unimodal pipelines like TAPAS (Herzig et al., 2020), TAPEX (Liu et al., 2022) ALTER (Zhang et al., 2024) and TaBERT (Yin et al., 2020) which might be more cost-effective than TAMO. \n\n3. The experiments seem to be insufficient - TAMO (a specialized multimodal model) is contrasted against general purpose LLMs like GPT-3.5, 4 etc. without contrasting against several existing tabular LLMs like TabLLM, TAPAS, TAPEX, TaBERT etc. which have already been established for Tabular QA and argue that representing a table as a natural language can largely mitigate permutation invariance (a major pitfall analyzed in this paper).\n\nQUESTIONS:\n1. What is the importance of using only WikiTQ over other tabular datasets in StructQA ?\n\n2. Does the experimental results of TAMO depend on the serialization template used ? It would be good to know the variation in performance when using direct serialization as in TAPAS, SQL based serialization in TAPEX etc. \n\n3. It is not clear whether a graph-only pipeline would be sufficient for the improvements discussed in TAMO. Following Figure 6 where it is clear that text-only pipelines are insufficient the authors must include experiments comparing graph-only pipelines like GCondNet (Margeloiu et al., 2022), IGNNet (Alkhatib et al., 2024) etc. against TAMO. This is critical to establish the necessity of a multimodal model in TabularQA tasks.\n\n4. TAMO utilizes the serialized tables as additional input in their pipeline. Does this indicate that during the hypergraph transformation the learnt representation loses semantic understanding of the table ?\n\n**Minor Suggestions -**\n1. The section on Hyper-Trans (Chen et al., 2021) (lines 133-152) is not unique to TAMO and should definitely be moved to the appendix. The authors should use this space to highlight the core differences between TAMO and other existing methods in Tabular QA.\n\n2. The use of non-standardized mathematical notations in equations 1 through 3 makes the paper very hard to follow. I would request the authors to use meaningful notations and elucidate them in the text wherever possible. Eg. usage of $rFF$ in line 148 seems to be $r$ times $FF$, $X$ and $H$ in equation 3 was never defined etc.\n\nLIMITATIONS:\nYes, Section 5 of the paper does mention several limitations of the paper which are valid and encourages future research in this space.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

comments: SUMMARY:
This paper addresses the topic of LLM-based question answering regarding tables.
It presents a new framework for this, TaMo, and shows that it outperforms previous methods, including using GPT4, GPT 3.5 and Llama 2 7B, and it is robust to permutations. The framework is based on a new method for representing complex tables as hypergraphs and encoding them. It then uses a fine-tuned Llama 2-7B model to answer questions based on the embeddings of both the structure and content of the table. The paper presents comparisons based on 5 question answering datasets, one of them, StructQA, was developed as a part of this work and includes 7500 examples based on 500 tables. The reported results are 42.65% average improvement in these comparisons (ranging from 4.5% to 86%), as well as outperforming the OpenAI models in 4 out of the 5 datasets. Further analysis in done on the changes in attention and the good generalization of the proposed method.

STRENGTHS AND WEAKNESSES:
The paper addresses an important problem and suggests a new approach for handling it. It includes significant experimentation, and it also provides a new dataset. It is clearly written and original.
The main quality weakness in my view is that it compares with old models (from over a year ago), which also raises concerns about its current significance. To be considered for NeurIPS, it would be important to show comparisons using the current state of the art models, including GPT4.1, O4-mini, Llama 3.3 and DeepSeek-R1, to clarify whether this method is still useful after the many improvements in LLMs that took place in the recent year.

QUESTIONS:
Please perform a comparison with recent models, such as GPT4.1, O4-mini, Llama 3.3 and DeepSeek-R1.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a multimodal large language model called TAMO: by converting table contents into hypergraphs, then further transforming them into embeddings, which are fed into the LLM to enhance its overall understanding capabilities. Results show that this method can achieve significant improvements on multiple benchmarks. Besides, a novel tabular-structure-understanding benchmark is released for the community.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The approach is innovative: For table understanding scenarios, this paper proposes a special encoding scheme for table reasoning tasks, which introduces a graph-based approach into the LLM to enhance model performance.\n\n2. The logic is clear: The paper first highlights the importance of table representation, as shown in Fig. 2, then designs a specific table encoder to address issues in table understanding scenarios (Sec. 2), and demonstrates the benefits of this approach through experimental validation (Tab. 2).\n\n\nWeaknesses:\n1. As described in Sec. 2 & Figure 4, TAMO adds an additional multimodal processing component to represent the content of tables, while the original table content serialized as text is still preserved as input to the LLM. Based on experience, this approach essentially provides the LLM with detailed inputs from multiple perspectives, which naturally enables the model to achieve better performance. However, does this approach (keep the serialized table text) introduce redundancy in information representation? In other words, why does the parsed table text still need to be input into the LLM in text modality? Additionally, it is recommended to further clarify the statistics in Table 2 (Main results).\n\n2. The experimental conclusions are not sufficient enough: Since the addition of new modalities on LLM is not a completely innovative solution, but rather a situation where ongoing work is continuously innovating solutions on this topic. As a new modality bridging solution, has it been better compared with other multimodal solutions, such as DocLLM (Layout-Aware) and QWEN2-VL / TextMonkey (Image-Aware)? Can further analysis of this part be provided in Section 3.3?\n\nQUESTIONS:\nPlease refer to the Strengths and Weaknesses.\n\nLIMITATIONS:\nPlease refer to the Strengths and Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces TaMo, a framework that treats tables as a distinct modality to improve LLM performance on table reasoning tasks. Instead of flattening tables into text, TaMo uses a hypergraph encoder to capture structural relationships in the table. TaMo feeds these structure embeddings into the language model as additional input, improving its ability to reason over tables. Experiments across five benchmarks show that TaMo improves accuracy compared to standard LLM approaches. \n\nThe authors also introduce StructQA, a new benchmark to evaluate if permutation invariance is respected\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- Proposes treating tables as a separate modality, enhancing structural reasoning in LLMs.\n- Leverages hypergraph-based encoders to model hierarchy and permutation invariance — key properties of tabular data\n- The authors show large performance gains overall\n- Introduces StructQA, a new benchmark for testing if permutation invariance in tables is respected\n- Plug-and-play integration with LLMs\n- The paper is well-written and easy to follow\n\n**Weaknesses**\n- While the combination of parts is novel, many components (e.g., hypergraphs, structural encodings) are close extensions of existing work\n- While generally clear, some aspects (e.g., multiset functions and alignment details) could be simplified or clarified further\n- The evaluation does not seem thorough, the authors do not compare against other models that also incorporate table structure (e,g, TAPAS, TAPEX and others) and are baselines for table QA\n- Specialist SoTA scores are included but not clearly explained, making it hard to assess the fairness of the comparison\n- TaMo underperforms GPT-4 on WikiTQ, but this is not adequately discussed or analyzed\n- Lacks ablation studies to isolate the contribution of key architectural components\n\nQUESTIONS:\nMy main concern is the evaluation. \n- The term “Specialist SoTA” is used in Table 2, but the paper does not discuss which specific models were used for each dataset. Please provide a breakdown of which models are used for with brief descriptions of how they were trained or evaluated. \n- TaMo underperforms GPT-4 on WikiTQ, but this is not adequately discussed or analyzed - please include some minor comment on this\n-  The paper does not include comparisons to well-established structure-aware baselines such as TAPAS/TAPEX. TAPAS and TAPEX are more relevant baselines than GPT-3.5 for table QA tasks\n- The paper lacks ablations that show the contributions of TaMo’s components. Consider adding or discussing ablation experiments, this would clarify which parts of TaMo are important\n- While generally clear, some aspects (e.g., multiset functions and alignment details) could be simplified or clarified further - please do that\n- I would also like the authors to make more clear their contribution compared to prior works because it looks like the paper borrows a lot from existing approaches (e.g. hytrel)\n\nLIMITATIONS:\nThe authors have addressed several limitations and I have no concerns regarding negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-07"}