Table as a Modality for Large Language Models
Authors: Liyao Li
Chao Ye
Wentao Ye
Yifei Sun
Zhe Jiang
Haobo Wang
Jiaming Tian
Yiming Zhang
NINGTAO WANG
Xing Fu
Gang Chen
Junbo Zhao
Venue: NeurIPS 2025 poster ()
Keywords: Large language model
Table reasoning
Primary area: deep_learning
TL;DR: A pioneering multimodal LLM framework for table modality.

Abstract: To migrate the remarkable successes of Large Language Models (LLMs), the community has made numerous efforts to generalize them to the table reasoning tasks for the widely deployed tabular data. Despite that, in this work, by showing a probing experiment on our proposed StructQA benchmark, we postulate that even the most advanced LLMs (such as GPTs) may still fall short of coping with tabular data. More specifically, the current scheme often simply relies on serializing the tabular data, together with the meta information, then inputting them through the LLMs. We argue that the loss of structural information is the root of this shortcoming. In this work, we further propose TAMO, which bears an ideology to treat the tables as an independent modality integrated with the text tokens. The resulting model in TAMO is a multimodal framework consisting of a hypergraph neural network as the global table encoder seamlessly integrated with the mainstream LLM. Empirical results on various benchmarking datasets, including HiTab, WikiTQ, WikiSQL, FeTaQA, and StructQA, have demonstrated significant improvements on generalization with an average relative gain of **42.65%**.

Table as a Modality for Large Language Models
Liyao L Chao Ye Wentao Ye', Yifei Sun', Zhe Jiang', Haobo Wang' Jiaming Tian', Yiming Zhang', Ningtao Wang', Xing Fu?, Gang Chen', Junbo Zhao !Zhejiang University Ant Group University of Michigan (liliyao, ye.chao, j J.zhao) edu.cn cn
Abstract
To migrate the remarkable successes of Large Language Models (LLMs). the community has made numerous efforts to generalize them to the table reasoning tasks for the widely deployed tabular data. Despite that, in this work, by showing probing experiment on our proposed StructQA benchmark, we postulate that even the most advanced LLMs (such as GPTs) may still fall short of coping with tabular data. More specifically, the current scheme often simply relies on serializing the tabular data, together with the meta information, then inputting them through the LLMs. We argue that the loss of structural information is the root of this shortcom- ing. In this work, we further propose TAMo, which bears ideology treat the tables as an independent modality integrated with the text tokens. The resulting model in TAMO work as the global table encoder seamlessly integrated with the mainstream LLM. Empirical results on various benchmarking datasets, including HiTab, WikiTQ, WikiSQL, FeTTaQA, and StructQA, have demonstrated significant improvements on generalization with an average relative gain of 42.65 %.
multimodal framework consisting of a hypergraph neural net-
Introduction
Table reasoning, the process of generating task-specific responses based one more pre-structured tables text, has emerged as a key research area. This encompasses various tasks such as table question
answering [Pasupat and Liang] 2013]. table fact verification [Chen al.) 2019]. text-to-SQL Yu
ett al.
2018). and predictive tasks (Ye et 2024a et al. 2022] Van Breugel and Van Der Schaar
2024
Classical methods employ baselines such as BART [Lewis al. 2020 or T5 [Raffel et al.
2020 to generate answers, often augmented with external retrieval frameworks Patnaik et al. 2024]
With the advent of large language models (LLMss, like GPT-4 [OpenAl 2023 and L.L.MA [Touvron
ct al. 2023]. this field undergoes substantial transformations. These LLMs generally adopt identical strategies, which involve serializing tables into text formats, often using markdown-like markup
languages to represent tables, occasionally accompanied by a few examples Herzig et al. 2020). as
shown in Figure] !
However, through principled and empirical observation, we find that existing serialization strategies may cause LLMs to lose their understanding of structural semantics of tables. This observation is conducted on specially designed diagnostic benchmark, named. StructQA. This benchmark considers structural semantic property unique tabular data: permutation invariance, which means that most tables should retain consistent semantics regardless of row and column reordering. Ideally, LLMs should maintain approximately the same downstream task performance for structurally equivalent tables. However, as illustrated in Figure our experiments reveal: leading LLMs. including Llama2- 7B Touvron al. 2023], GPT-3.5 [OpenA] 2022), GPT-4, and even TableLlama [Zhang al.
"Equal contribution. *Correspondigg author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

2023b
trained specifically for table dasksdeemonstraee significant performance degradation when presented with permuted versions of the same table. While resisting these perturbations trivial for humans, these models excluding GPT-4, show answer robustness below 40%. 'his phenomenon indicates the significant limitations in LLMs' generalization capability for table comprehension through serialization.
This limitation stems from fundamental mismatch: tables are inherently structured data with permu- tation invariance. while text serialization cannot naturally preserve this property. Although this concept has been recognized in previous research [Herzig et al., 20200] Yang et al.) 2022], it has been largely overlooked contemporary L.IM develop- ment. When faced with structural perturbations, LLMs exhibit ballucinations [Huang et al., [023] and unstable reasoning patterns, indicating their limited generalization of tabular structure.
Notes
Inceptisn
Don Cabb
Festere
2015
Quay
Himmett
Shart liles
Previous Works
TAMo (Ours)
Table Input
Table Input
Question
Sequence:
Table Structure Emb..
What is the
[TAB] col: Year Title
value the
Role Notes [SEP]]
column Role
2010 Inceptionn
Text Sequence: [TAR] cnl: Year Title
sample
Feature fim
row
To tackle this issue, we propose a novel perspective: encode tables as an independent modality inte- grate their complex relational structures. Just like images and audio which contain rich semantic in- formation, tables possess inherent structural nuances that textual serialization fails represent alone. The multimodal large language models (MLLM) learn the semantics of specialized modalities through separate encoding architectures and align different modalities in a unified and more expressive embedding space. By employing similar approach, we can bridge the gap in LLMs' comprehension and achieve a holistic understanding of tables' structure comparable to human cognition through learnable table features.
Large Language Models
Answer
Answer
Short film
X
The Protagonist
Figure 1: Current tabular LLMs oversimpli- fies tables into text sequences, ignoring struc- tured information and hindering basic table cell localization tasks. This work is the first to direct table structure integration into L.Mss
In this paper, we propose TA T pio- neering tabular language model framework to reimagine Table representation as an indepen- dent Modality. TAMO leverages theoretically permutation-invarian hypergraph structures to independently capture the intricate relationships and global structures within tabular data. Further, we integrate this hypergraph- based encoding into LLMs through learnable features, achieving dy- namic and efficient injection of structural infor- mation without tuning the L.M''s M fixed parame- ters. We exhibit extensive empirical validation on five table reasoning datasets ToMo demon- strates substantial performance improvements against previous baselines-up up to a 42.65 % in- crease in average performance. Meanwhile, our methodology validates superior efficacy and broad applicability when integrating hypergraph- encoded tables with diverse LLMs.
TableLiama
Llama2-78
GPT-3.5
GPT-4
Liama2-78 w/ Prompt Tuning
TAMO Llama2-78 (ours)
0.8
Accurcy
0.4-
0.0
StructQA
Permulation
Robustness
4
Figure 2: We evaluated LLMs' understanding of table structures through our StruciQA dataset (Sec- tion 3.1). The evaluation focused on permutation invariance by assessing answer robustness across randomly permuted rows and columns. Results show that TAMo achieves state-of-thee of-theeart perfor- mance, comparable with GPT-4
Contributions. Benchmark: We introduce
StructQA, the first open-source benchmark on the robust tabular structure understanding. Our findings reveal that current LLMs struggle with this human-friendly task. Position: Our research represents a pioneering step in integrating tables as an independent modality into LLMs. Method- ology: We explore the semantic alignment of tabular structures in LLMs" embedding space via
1 Code and datasets are on pattps://gittee http:ttthb...mmlly/t//e wttp://wwo.c/c//ccoooooooooooott TAMO
'Hitab Cheng et al. 2022 WikiTQ Paasupat and Liang 2015), WikiSQL Zhong et al.
2017 and FeTaQA
Nan et al. 2022 and proposed StructQA benchmark (Section 3.1

---

hypergraph architectures, effectively modeling complex relational patterns across diverse table tasks. Feasibility: Experiments show that TAMo enhances LLMs' generalization on table reasoning by encoding irructureinvvriaant table representations. We demonstrate the generality of our framework in two primary aspects: (1) Robust Generalization across structural variations, such as the permuta- tions tested in our proposed StructQA benchmark, and (2) Architectural Generality, functioning as a "plug-and-play" module that integrates with diverse decoder-only LLMs without intrusive architectural modifications.
2 Methodology
2.1 Problem Definition
Following Wang et al. [2024b], table reasoning can be defined as a unified task that acts on samples for- matted as triplets (T, Q A). Here, T represents a pre-structured table containing information clearly organized in rows and columns, with cell types en- compassing numerical values, text entries, and dates.
Sabject
Mark
Plysks
Stadent
Greder
Piesirs
Mark
Jack
Male
7
Jack
15
11
Fikes
96
Q = {q1, 92, .... 4m }o denotes the question or state- ment related to the table T, typically in a natural lan- guage sequence with m tokens. Meanwhile, A is the expected answer or output of Q, usually simplified into an n-tokens sequence {a1, 02 .... ). J. Briefly, given the table T and the question Q. the objective of table reasoning is to predict the corresponding answer A, i.e., ().
Eduas
Mals
90
66
Sqiie
90
17
(a) Simple Flat Table
(b) Complex Hierarchical
Ky
Seadint
Subject
Plysiet
Erken
99
2.2 Hypergraph-enhanced Table Encoder
A table encoder is essential for our multimodal tab- ular LLMs paradigm. To develop the table encoder capable of learning structural information, we first address fundamental question: "How define the structural properties in tabular data?" As illustrated in Figure we provide the answer based on prior human observations: (i)-most real-world tabular data possess a hierarchical structure, with ordinary flat tables being a special case of this hierarchy; (ii)-cells within each hierarchy and hierarchies at the same level exhibit permutation invariance. For example, arbitrarily swapping rows or columns in table does not distort its original meaning. This implies that learning the relationships between table cells should not be pairwise but rather set- based. Building on the inherent hierarchical structure of tables, we introduce the hypergraph (Yadati et al., [019] architecture to model tabular data. This approach incorporates both high-order hierarchical structure and permutation invariance as inductive biases, enabling the precise modeling of complex structural properties in tabular data. For the first time, allows us to successfully model all types of tables, from simple flat tables to complex hierarchical forms [Cheng et al. 2022].
(e) Uniffed Hypergraph Madellng
Figure An example of converting arbitrary simple or complex tables into hypergraphs. A simple flat table is a special case of the com- plex hierarchical table. A hyperedge (e.g., ta- ble headers) in the hypergraph a set of reg- ular nodes. We construct the corresponding hypergraph format according the hierarchi- cal relationships of the table.
We re-construct the structure of tabular data via hypergraph. Specifically, a hypergraph G == (V,E) consists of a set of nodes V and hyperedges E. Each hyperedge e E & is subset of V. i.e., e C V. For table T, we represent each leaf cell, defined as a cell that does not contain any other cells within the hierarchy, as a node v E V and each branch cell, defined as a cell that contains other cells within the hierarchy, as hyperedge hierarchical level. For example, in simple flat table, each table cell is node, and each column or row is hyperedge encompassing all nodes within that column or row. Under this modeling, altering rows or columns maintains a consistent graph structure (both nodes and edges), effectively reflecting the permutation invariance of tables.
E. Each hyperedge e consists of nodes that belong to its
3

---

LoRA (Optional)
LLMs (Llama Mistral ...)
Answer
Table Structure Emb.
Textual Tokens Embeddings
Alignment Projector
[TAB] ... col rowl:
Question
Hypeeggapp--ehanned
Serialize Table into Formatted Text Sequence
Tabular Encoder
Year
Title
Role
Notes
Unified Hypergraph Modeling
2010
Inception
Dem Cobb
Feature film
2015
Quay
Himself
Shart film
Pretagunist
Figure 4: The proposed framework for tabular LLMs, TAMo. Given table input, the hypergraph- enhanced table encoder (Section (.2) is used to capture the unique structure properties of the table modality. Simultaneously, input both the table structure and textual embeddings into LLMs, generating answers using the next token prediction paradigm. LoRA is optional.
senalize the original table into formatted text sequence. Finally,
Furthermore, model information propagation within the hypergraph structure, employ Hyper- Trans [Chen et al., (2024), bypergraph-structur--avvae transformer, as the backbone architecture of the eypergraph-ennaneed table encoder. The encoder architecture integrates two distinct mul- tiset functions [Chien et al. 2021] to effectively capture higher-order hierarchical structures in hypergraphs. These multiset functions, characterized by their permulation invariance property, combined sequentially as depicted in Eqa annd Eqa Each layer of the table encoder comprises two primary components, with the initial component implementing a multiset function that aggregates node-level information to update hyperedge representations:
Fusion (x',, Multiseti (fx!, e eee))),
(I)
where refers to the current layer number; x, the embedding of the node ;; x, is the embedding of the hyperedge e; the fusion layer is employed integrate hyperedge information from the last layers, typically utilizing a multilayer perceptron (MLP) network.
The second part is another multiset function that aggregates hyperedge information update node representations:
Multisetz((xx"" (+1 et).
(2)
Finally, we use the Set Transformer Lee et al. 2019 to parameterize these multiset functions for learning. Each set attention block is defined as:
M Multisct(X)
++F()(H)
Layer + MultiHeau((SXX..
where S is a trainable parameter vector; rFF is the row-wise feedforward layer; Layer Norm is layer normalization [Ba et al. [016]; MultiHead the multi-head attention mechanism [Vaswani et al. 2017]. By facilitating the mutual propagation of information between nodes and hyperedges, the model effectively learns complex hierarchical relationships among table cells thus outputting learnable table features. This approach conceptually reframes the table from serialized string into distinct modality, which we position as a complete multimodal reasoning framework distinct from modality-specific encoders. A detailed discussion this positioning is provided in Appendix D.]

---

2.3 A Modality Interface for Integrating Table Structure Representations with LLMs
Most LLMs (Meta] 2024 Jiang et al. 2023a OpenAl 2022 2023 are pre-trained on large-scale unlabeled corpora in an autoregressive manner. thereby learning rich linguistic structures and patters,, To maximize the utilization of LLMS' powerful text understanding and reasoning capabilities for table reasoning tasks, we design a fully autoregressive interface to integrate structure representations from the tabular modality with LLMs for table reasoning tasks. The overall framework of our proposed TAMo shown Figure 4 We inject the structure representations learned by the hypergraph- enhanced table encoder in Section 2.2 into the LLMs in manner similar to the soft prompt (Lester et al. 202T]. This allows the LLMs to globally perceive the structural information of the tabular data before reading the textual information, thereby enhancing their understanding and reasoning abilities regarding tabular tasks.
Aligning Table Structure Representations to LLM Semantic Space. Assuming the node repre- sentations obtained through the table encoder are Xy {x,,|v E v} E | and the hyperedge representations are X {&. le E} E Rl dg is the hidden dimension of the table encoder. We use a multilayer perceptron (MIP) network to learn the transformation table structure represen- tations X into the semantic space, where adopt single-layer MLP as the alignment projector for simplicity and efficiency:
X.t (ALP(Pooling(Xv.XX Xe)) k
(4)
where pooling an information aggregation function for nodes and hyperedges, set up as mean pooling in our experiment; d; is the hidden dimension of LLMs.
Generating Answers based on both Tabular and Textual Modality Information. Following previous works (Zhang et al., [223b Wang et al., 2024b] Herzig et al.) 2020). we serialize tabular data into formatted text sequences and obtain the text embeddings of tabular data Xxe R xd, through the LLMs embedding layer. Lg is the length of text sequences. For questions in natural language form, we obtain the corresponding question tokens Xgt E RL through the embedding layer similarly. I.g is the length of question sequences. The final answer is generated following the next token prediction paradigm:
P(AIT, Q) = II p(a: X Xue, Xuts aj<s),
(5)
where is the number of answer tokens A {a1, 02, On During training on downstream table reasoning datasets, we can choose to freeze the parameters of the LLMs and only learn the table encoder and alignment layers. We retain both input streams they serve complementary, non-redundant roles. The serialized text X.: provides the fine-grained semantic content (the "what"), which LLMs are exceptionally skilled at processing. The hypergraph-based structure embedding X.r provides the global relational context (the "where"), such as row-column relationships and hierarchies, which is inherently lost during serialization. The necessity of both modalities is empirically validated in our ablation study (Appendix/Ca C.4), which shows that the graph-only approach fails on generative tasks and the full TAMo model significantly outperforms the text-only counterpart, confirming the synergistic benefit. This method allows us to capture structure representations in the tabular modality while integrating them with LLMS in a cost-effective and scalable manner.
Experiments
In this section, we will demonstrate the advantages of treating tables an independent modality. Section introduces inntoouuess our novel benchmark. StructQA, designed evaluate LLMs' understanding of table structures and their robustness. Sections 3.3
presents the performance gains of our approach across mainstream datasets and fine-tuning methods. Section B.4feexor.eexp the interpretability of our method through attention visualization. Section 3.5
showcases the robust performance of our method discusses the ability of the hypergraph-enhance table encoder to extract table structure information.
under different fine-tuning techniques. Section 3.6
5

---

3.1 StructQA: Table Structure Understanding
We present StructQA, a novel benchmark focusing on table structure understanding, containing 7500 QA pairs from 500 tables across 5 structural reasoning tasks as described in [ Unlike prior table QA datasets [Pasupat and Liang 2015 Cheng al., 2022]. StruciQA evaluates models' capabilities through three dimensions: direct performance, permutation accuracy under row/column shuffling, and answer robustness before and after permutation. This benchmark also addresses potential data contamination concerns Yee et al. 2024b] present in existing datasets.
Figure 2 indicates the performance mainstream LLMs on StruciQA. We find that current text-based method do not perform well on this dataset, with overall answer robustness below 40 %. This phenomenon may suggest that current approaches face significant limitations in understanding tabular data.
3.2 Experimental Setup
To enhance the model's comprehension of structured data, we propose TAMO (refer to Section [.2] for de- tails). This section primarily outlines the experimental setup for the experiments of TAMo.
(1) Cell location: identify cell value by row number and column name.
(2) Colamn lookup: identify the column based on row numiler and cell value.
Datasets & Metrics. To conduct a more comprehensive assessment, we choose five datasets including Struc- IQA, HiTab, WikiTTableQuesionn (WikiTQ), WikiSQL, FeTaQA, for our experiments. To establish the perfor- mance upper bounds of TAMo for different tasks inde- pendently, we trained separate instances from scratch on each task-specife training set and evaluated them on the corresponding test s
Row lookup: identify name and cell value.
based the column
(4) Column comprehension: summarize all distinct val- column based the column name.
comprehension: summarize all distinct values in a row besed on the row number.
Table I: Five different types of structure reasoning tasks in the StruciQA dataset. More details are in Appendix A
Competing Methods. To demonstrate that incorporat- ing tabular modality into LLMs, referred as tabular language models, can enhance performance in table rea-
soning tasks, we compare TAMO against using only pure text modality in four different settings: (i) Inference Only (Zero-shot): directly reasoning on serialized table sequences and questions: serves as diagnostic baseline (primary claims rely on stronger tuned comparisons). (ii)-Frozen I.I.M: com- paring with prompt tuning (Lester et al., [02T]. which adds some parameterized and trained tokens in front of serialized table sequences.
LLM (LoRA): using LoRA (Hu et al., (02T] to finetune the parameters of LLMs. We add optional LoRA in our method TAMO Lo (iv)-Tunee) LLM (SFT): supervised finetuning of all parameters of LLMs. TAMO SFT SFT means supervised training of TAMO and LLMs jointly.
A key baseline in our comparison is TableLlama [Zhang et al. 2023b which represents the current state-of-the-art (SOTA) across multiple table reasoning tasks through supervised fine-tuning (SFT) on extensive tabular datasets. For fair comparison with TableLlama, we implement TAMo using the same base model, Llama2-7B. We provide additional performance analysis across more advanced LLMs in Appendix C.8 We further evaluate our approach against powerful peneral-purpose LLMs HPPT-4-turbo-222--00
like GPT-3.5-turbo-0125,
GPT-4. I and DeepSeek-R [Guo et al. 2025).
Regarding the "Specialist SOTA" results presented in Table 2 these refer to state-of-the-art methods specifically tailored for each dataset: TableL [Zhang et al. [2023b] for HiTab, CABINET [Patnaik et al. 2024) for WikiTQ and FetaQA, and SeaD [Xu et al. 2022 for WikiSQL. We included these to highlight different methodological directions, contrasting with TAMO's focus on enhancing the
LLM's end-to-end understanding via modality integration. Unlike TAMO's approach, these specialist methods often rely on external modules (like scorers or specific pre-processing steps) or task-specific objectives, which limit their direct applicability across the full range of table reasoning tasks without adaptation.
*Details of these datasets can found in Appendix B "Appendix demonsttassss that TAMO exhibits notable cross-dataset generalization capabilities.

---

Dataset
StruciQA Structural QA Accuracy
HiTabb Hierarchicul QA
WikiTQ Table QA
WikiSQL Table QA Accuracy
TetaQA
Setting
Type Evaluation Metric
Free-form QA BLEU
Accuracy
Accuraey
Inference Only
Zero-shot
8.60
7.77
14.50
21.44
20.08
Prompt tuning
37.80
26.26
29.86
61.24
29.94
Frozen LLM
TAMO
59.07
48.86
37.06
76.45
36.52
Promgt Exiving
56.27%
86.06%
124.11%
21.84%
21.98%
LoRA
43.67
50.76
37.13
57.10
35.80
Tuned LLM
(LoRA)
TAMo
70.80
59.22
43.53
84.43
37.43
LoRA
33.03%
16.67%
17.244%
47.86%
4.33%
Tama 2023b
6.47
63.76
31.22
46.26
38.12
Tuned LLM
62.73
54.80
43.28
79.86
37.37
(SFT)
TAMo
45.81
39.01
ASFT
14.14%
16.50%
5.85%
7.56%
4.39%
GPT-3.5
41.93
43.62'
53.13'
41.91"
26.49'
GPT-4
51.40
48 48.40"
68 40"
47.60"
21.70"
Others
GPT-4.1
60.33
60,54
68.14
71.21
36.75
DeepSeek-Ri
57.47
75.76
13.10
Specialist SOTA
64.71
2023b
69.10
2024
92.07
2022
40.50
2024
Table 2: Results on our table structure understanding dataset StruciQA and four table reasoning benchmarks. TAMO adds additional table modality information compared the pure text baseline. Specialist SOTA refers to methods that design models and training tasks specifically for each dataset. indicates data sourced from Zhang et al. [2023b]. The first best result for each task is highlighted in bold and the second best result is highlighted with an underline.
3.3 Main Results
Main results are exhibited in Table we conclude that:(i> Explicitly inpulting the table modality significantly enhances LLM's performance in various table reasoning lasks; (ii--aMM significantly outperforms the SFT models that rely solely on the text modality; (iii)-TMM is competitive with specialist SOTA methods, highlighting the utility of using hypergraphs to model complex table structure relationships.
Across all datasets, TAMO achieves substantial improvements in both frozen and tuned LLM settings. For example, TAMO shows an average improvement of +42.65 over inputting pure text modality on the frozen LLM setting, with maximum improvement of +86.06 % the HiTab dataset. In the tuned LLM setting, both TAMO and TAMO F show substantial improvements, outperforming the pure text modality by an average of +28.27% and +9.71%, respectively. Meanwhile, TAMO SFT achieves SOTA performance across all tasks under our settings and TAMO LORA secures a close second on 3 out of 5 datasets. This reveals that TAMO provides LLMs with more comprehensive table embeddings that remains unattainable through text sequences regardless of training setting.
The Llama2-7B based TAMo achieves closed SOTA performance HiTab, FetaQA, and WikisQL, where HiTab is a complex hicrarchical table dataset. This indicates that hypergraph- enhanced table encoder can effectively learn complex hierarchical relationships within tables, thus further improving the model's accuracy in table reasoning tasks. Although slightly behind the specialist SOTA methods the other datasets, it's worth noting that they all utilized dataset-specific model architectures, training methods, or other enhancement tricks. Additionally, TAMO LORA and TAMo consistently surpass GPT-3.5 and GPT-4 on 4 out of 5 datasets. For example, t achieves an average improvement of over +19.83 score compared GPT-3.5. When compared against the more recent powerful foundation models. GPT-4.1 and DeepSeck-RL. our 7B 7B-parameter TAMO FF demonstrates strong competitiveness, particularly on structure-centric benchmarks. As shown Table 2 TAMo significantly outperforms both GPT-4.1 (71.60 vs 60.33) and DeepSeek- R1 (71.60 vs 57.47) on StructQA, and notably surpasses them on WikiSQL (85.90 vs 71.21 and 71.91, respectively). While these larger models show stronger performance on knowledge-intensive tasks like WikiTQ (e.g., DeepSeek-RI achieves 75.76). the competitive results achieved by our much smaller model underscore the effectiveness of the TAMO architecture in enhancing structural reasoning capabilities.
7

---

Input [[T AB col 3 Pick Player| Position National ity NIIL teum
[table siructure token] ... Input : [T AB J col Pick Player
[College jj un ior
team SEP ||
War
Position National IHHL
Collage /j un los I club
Defence Canada| Florida Panthers| Susk atoen Dlades (W HL )
[SEP 27 Rh ett War re net Defence Canada Florida Panihers
[SEP] I... 335 Joset Mar ha Center Czech Republic QQuchoc
Sask atoon Blades HL ) SEP ... 35 Josef Mar ha Center
Nord iques D uk la ib lava C zech Republic ]| SEP 36 Ryan
Czech Republic Queebec Nord iques |D uk la J lava zech
Johnson Centre| | Canada| Florida Panthers| Thunder Bay Flyers
Republic I SEP ] 36 Ryan Johnson Centre Canada Florida
(US HL ... Question What are the national
player
Panthers Thunder Bay Flyers (
... Question
the
picked from Thunder Bay flyers ( ush I
national itees of the player picked from Thunder Bay Flyens msh ))
Inference only
TAMo (Ours)
Figure 5: A real visualization case in the WikiSQL dataset results of attention weights from other input tokens to the label answer cell "Canada". Intuitively, the darker the color, the more closely the token is associated with "Canada". We observe that with the of TAMo, the LLM better focuses on information relevant to the correct answer, as indicated by the darker background colors associated with those tokens.
token]"
3.4 Case Study
To further investigate the table encoder's comprehension of tabular data, we conduct visual analysis of specific cases. Specifically, we visualize the attention importance of all input tokens for the correct answer as perceived by the LLMs. During this process, we adopt the visualization method from the PromptBench (Zhu et al., 2023b], which uses the gradients of the input embeddings to estimate token importance. We randomly select a sample from the WikiSQL test sets for visualization analysis, where the base method (inference only) is incorrect but TAMO is correct.
As shown in Figure 5 we can find: (i)-TAMo thinks "Canada" (correct answer) and 'US HL" (relevant contextual information) tokens are more important for the final answer, while the base method largely ignores these crucial tokens. (ii)-TAMo shows a certain level of attention to " ttt_sssuuuuu_eeteeeeee
and adding srrabl_ssructurre "ooken] affects the impor- tance distribution of other input tokens, prompting LLMs to focus more on tokens relevant to the correct answer. We observed some error cases with the LoRA setting that resemble those shown above. For example, when the correct answer is far from the question in the serialized input, TAMo can utilize the overall table structure to locate the correct answer, compared to LoRA in text-only mode, which primarily focuses on the content immediately before and after the question. This case study indicates that the structural information learned from TAMO enhances LLMs' reasoning capabilities by optimizing attention relationships for key table information, thereby indicating potential for mitigating hallucination phenomena.
3.5 Generalization to Structural Variations
Compared to image/text data, permutation invari- ance-any any permutation of the rows and columns does not change the original interpretation of the ta- ble-is a unique structural property of tabular data. To further explore whether TAMO can effectively perceive table structure information, we construct experiments assess its robustness regarding per- mutation invariance. Specifically, we use the permu- tation version test set by randomly shuffling the rows and columns of tables in the StructQA test set (the training set is unchanged). In the frozen LLM setting, we compare the performance of TAMO with pure text modality methods (inference only & prompt tun- ing) on the new test set and check the consistency of answers after permutation. Results are shown in Figure @ and Figure [6] we find that for both frozen LLMs and tuned LLMs (LoRA and SFT), TAMO consistently outperforms pure text modality methods. Additionally, TAMO demonstrates the best robust- ness in maintaining consistent results after permuta-
Text Only
TAMO (ours)
0.75
Accurnay
0.45
0.30
0.30
0.15
0.15
0.00
0.00
Strusio
Permsstation
Robustress
S
Pommstation
LoRA Tuning
Supervised Finetuning
Figure 6: Evaluate the robustness of TAMO to permutation invariance on the StructQA dataset. Permutation: randomly permuting rows and columns in the StructQA test set. Robustness: the proportion of samples that remain consistent after random permutation.
8

---

tion. These indicate that TAMO can enable superior generalization of LLMs across diverse table structural variations.
3.6 Generalizable Table Encoding Analysis
To further validate the table encoder's effectiveness in learning table structure, we conduct series of additional experiments. Specifically, we propose a binary classification task to predict whether cells belong to specific rows or columns. For this experiment, we use the WikiTQ (Pasupat and Liang 2015] as base dataset, train all models for 50 epochs with a learning rate of 3e-4. Finally, we evaluate them with Fl metric.
The results are shown in Table 3 Without table representation learned from the encoder, baseline MLP classifier only achieve 5.39% Fl score, while a randomly initialized table encoder with MLP head reach 49.73%. Most notably, when using our pre- trained table encoders from various datasets (StructQA, HiTab, WikiTQ, WikiSQL, and FetaQA) with MLP classifier, all mod- els exceed 60% Fl score, with StructQA achieving the highest at 71.32%. The superior performance on StructQA can be attributed to its focused structural learning with minimal reasoning complex- ity. These results, evaluated consistently on the WikiTQ test set, demonstrate that TAMO's 's hypergraph-basee table embeddings effectively encode and generalize structural relationships across different datasets.
Settings
Fl Score
MLP head
5.39
+ randomly enc.
49.73
+ pretrained enc.
StructQA
71.32
HiTab
66.39
WikiTQ
62.63
WikiSQL
68.00
FetaQA
64.99
Table 3: Evaluation of table
Combined with the interpretability analysis in Section 3.4 these findings suggest that table structure representations can enhance LLMs' understanding of table and answer localization capabili- ties during reasoning, aligning with observations from previous work (Yang et al. 2022]. Additional ablation studies are provided in Appendix
structure representations learned from encoders pretrained on dif- ferent datasets, measured on the WikiTQ test set.
4 Related Work
LLM-based Table Reasoning. Recent advances in Large Language Models (LLMs) have led to their widespread adoption in tabular reasoning tasks [Zhang et al. 2024], establishing Tabular Large Language Models as the dominant approach. These methods primarily follow two strategies: fine- tuning on tabular data and prompt engineering. The fine-tuning approach enhances LLMs' structured
data capabilities through supervised training on tables (Zhang et al. 2023b Zhuang et al. 2024
Wu and Feng 2024 Sarkar and Lausen 2023). exemplified by TableL.ama" 's Zhang et al. 2023b
generalist model trained on diverse weal-world tables. Several recent parallel studies [Long at 2025
Jin et al. 2025 Xu et al. 2025 Majee et al. 2025 Huang et al.] 2025) have independently explored modeling fine-grained table semantics (e.g., cell-level or column-level embeddings), typically in task-specific contexts such as text-to-SQL or table type classification. However, these methods generally rely on heavy pretraining or multi-stage training pipelines. In contrast, our method adopts plug-and-play design that provides global tabular representations and does not require any pretraining. detailed comparison provided in Appendix D.2]
The prompt engineering approach instead leverages carefully crafted prompts to enhance LLMs' reasoning over tables in specific scenarios
NNi et al. 2023 Wang et al. 2024b Jiang et al. 2023b Zhang et al. 2023b Cheng al. 2023)..
Representative works include Dater Ye et al.) 2023], which decomposes tables into subtables, and Chain-of Table (Wang et al.
2024b). which integrates chain-of- -f-thuuhttt reasoning with programmatic
methods.
Table Encoder. In recent years, numerous studies have explored effective methods for encoding and understanding tabular data. TaBert [Yin et al., 2020) adopts a dual-encoder framework that separately processes textual and structural elements of tables, improving table comprehension through masked language modeling. TabNet [Ark and Pfister 2021 utilizes a novel iterative masking
7 AAppendix C.2 presents the relatively low computational overhead introduced by the ToMO framework.
Appendix C.7 demonstrates the notable performance gain of TAMo in the multi-table QA setting.

---

attention mechanism to select important features. HyTrel [Chen et al. 2024 extends this concept by using hyperedges to capture richer interactions among simple flat table cells, resulting in enhanced representations for relational data. However, all these table encoders cannot handle joint text and table understanding tasks like table question answering. They are primarily used to encode raw tabular data into a low-dimensions vector space to get better table representation. As discussed in Section[] existing approaches often serialize tables into text, losing structural information. To address this, we propose TAMO, multimodal framework that integrates structural and textual semantics into LLMs. TAMO is compatible with various table encoders; in this work, we use HyTrel for its effectiveness. While above encoders focus on learning effective table representations, seminal works such as TAPAS [Herzig et al., 2020] (BERT-based) and TAPEX (Liu et al., 2022] (BART-based) successfully integrated structural awareness within the then-dominant encoder-only or encoder- decoder language model frameworks. Our work, TAMO, differs by proposing a modality interface specifically designed for integrating table structure into modern decoder-only LLMss. We provide empirical comparisons against TAPAS and TAPEX on our StructQA benchmark in Appendix CC.5 to highlight the advancements and specific challenges addressed within this newer architectural paradigm.
5 Limitations
While TAMO enhances Prozen-paramend L.Mss' table reasoning capabilities via hypergraph encoders and learnable features, it has several limitations. First, it relies on pre-structured tables following the TableQA paradigm [Pasupat and Liang, 2015]; for tables embedded in unstructured text, existing text-to-table techniques [Wu et al., 2022 Deng et al.) 2024) are required as a preprocessing step. Second, TAMO currently focuses on static table understanding in a single-turn setting. Extending to support dynamic multi-step reasoning, complex table editing, and multi-turn dialogue over tables remains an open challenge. Third, we differentiate our work from layout-aware models such as DocLLM [Wang al.) 2024a Liao et al., 2025). These models are designed to understand tables presented as images within documents, focusing on layout and OCR challenges. TAMO addresses different and complementary problem: reasoning over pre-structured tabular data. Fourth, experiments utilized a consistent serialization format. A systematic study of how different text serialization templates (e.g., markdown vs. SQL-based) interact with our structural modality remains an important direction for future work. Finally, extensive multimodal instruction data required to develop robust, out-of-the-box multimodal capabilities, which we leave for future work. These limitations highlight the early stage of our research and the need for further exploration to fully integrate table modalities with LLMs.
6 Conclusion
In this work, we introduced a novel framework, TAMo, which leverages bypergraph-enhanced table encoder to boost parrmm-rrrmmter L.LMs' generalization on tabular data. By adhering to the principle of table structure permutation invariance, TAMo effectively encodes table structures into LLL--coppeeeensible representations using learnable features. This enables the handling of tasks involving both text and table understanding, such as table QA. Additionally, we presented StructQA, a dataset focused on table structure understanding, and validated our framework's 's efficacy and versatility across four other public table QA benchmarks.
Acknowledgments
This paper is supported by the National Regional Innovationand Development Joint Fund (No. U24A20254). Haobo Wang is also supported by the Fundamental Research Funds for the Central Universities (No. 226-2025-00004) and Zhejiang Provincial Universities (No. 226-2025-00065), This work was supported by Ant Group.
10

---

References
S. Ô. Arik and Pfister. Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 6679-6687, 2021.
J. I., Ba, J. R. Kiros, and G. E. Hinton. Layer normalization, 2016.
P. Chen, S. Sarkar, L.. Lausen, B. Srinivasan, S. Zha, R. Huang, and G. Karypis. Hytrel: Hypergraph--enhnneed
tabular data representation learning. Advances in Neural Information Processing Systems, 36, 2024.
W. Chen, H. Wang, J. Chen, Y. Zhang. H. Wang, S. Li, X. Zhou, and W. Y. Wang. Tabfact: A large-scale datasct for table-based fact verification. In International Conference on Learning Representations, 2019.
Z. Cheng, H. Dong. Z. Wang, R. Jia, J. Guo, Y. Gao, S. Han, J.-G. Lou, and D. Zhang. HiTab: A hierarchical table dataset for question answering and natural language generation. In S. Muresan, P. Nakov, and A. Villavicencio, editors. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 1094-1110, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 1...8665/1//2222.dd acl.oo...cc--o..788 78. URL http:////acaaaahlloggggo org/ 002/2022..oo. acl sellllon.777 78
Z. Cheng. T. Xie, P. Shi, C. Li, R. Nadkarni, Y. Hu, C. Xiong. D. Radev, M. Ostendorf, L. Zettlemoyer. N. A. Smith, and T. Yu. Binding language models in symbolic languages. In The Eleventh International Conference on Learning Representations, 2023, URL htttht:///ppeereeeee
E. Chien, C. Pan, J. Peng. and O. Milenkovic. You are allset: A multiset function framework for hypergraph neural networks. In International Conference on Learning Representations, 2021.
Z. Deng, C. Chan, W. Wang, Y. Sun, W. Fan, T. Zheng, Y. Yim, and Y. Song, Text ext-ttulletable: Towards information integration in text-to-table generation via global tuple extraction. arXiv preprint 441:444441. 2024.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani. M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q Zhu, R. Xu, R. Zhang, S. Ma. X. Bi, et al. Deepscek-rl incentivizes reasoning in Ilms through reinforcement learning. Nature, 645(808 I ):633-638, 2025.
J. Herzig, P. K. Nowak, T. Müller, Piccinno, and J. Eisenschlos. TaPas: Weakly supervised table parsing via pre-training. In D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320 Online, July 2020. Association for Computational Linguistics. doi: 10. a200 URL htt.bttpps//accaatooloogg .org///gg
22200 aal.mminn...... /
E. J. Hu, P. Wallis, Z. Allen-Zhu, Y. Li. S. Wang, L. Wang, W. Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference Learning Representations, 2021.
L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng. X. Feng. B. Qin, et al. A survey on hallucination in large language models: Principles, taxonomy. challenges, and open questions. arXiv preprint arXiv:2311.05232. 2023.
S. Huang, H. Li, Y. Gu, X. Hu, Q Li, and G. Xu. Hyperg: Hypergraph-enhancel Ilms for structured knowledge. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '25, page 1218-12211 1228, New York, NY, USA. 2025. Association for Computing Machinery. ISBN 979840001 000-40071992.... 3730002
doi: 10. 011144/376602.377600
URL U htttp:://oii.rrg 10. 1145/3726302
A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, I.. Saulnier, I.. R. Lavaud, M.-A. Lachaux, Stock, L., Scao, T. Lavril, T. Wang, T. Lacroix, and W. F. Sayed. Mistral 7b, 2023a.
J. Jiang, K. Zhou, Z. Dong, K. Ye, X. Zhao, and J.-R. Wen. StructGPT: A general framework for large language model to reason over structured data. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9237-9251, Singapore, Dec. 2023b. Association for Computational Linguistics. doi: 10.18653/v12022..oom https://acllanthologg .wceoogg.oorrrrgg 2023/ emnlp- main. 574
574..574. URL
R. Jin. Y. Li. G. N. Hu, Y.F. Li, Chen, Wang, Chen, D. Min, and S. Bi. Hegta: Leveraging heterogeneous graph-enhanced large language models for few-shot complex table understanding. Proceedings of the AAAI Conference on Artificial Intelligence. 39(23):24294 24302, Apr. 2025. doi: 1669/aaai.v39/7 34606. URL http/ttps://///// //ojjs aaai. awai.org/ wtggindex... wwt.innwwwppteeeieeeeeetppttpptt AAAI/ attpa/atttccceeeeee http//eew////// /4666
11

---

J. Lee, Y. Lee. J. Kim. A. Kosiorek, S. Choi, and Y. W. Teh. Set transformer: A framework for sitention-based permutationn-nvariant neural networks. In International conference on machine learning, pages 3744-3753. PMLR, 2019.
B. Lester, R. AI-Rfou, and N. Constant. The power of scale for parameter-effiiim prompt tuning. In M.-F. Moens, Huang, Specia, and S. W.-t. Yih, editors. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059. Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics. doi: 10.18653/v 1/2021 1/2022..mnlppp......... main.243. URL https://aclaantholog. org org/2021.
emnlp main. 243
M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer. BART: sequence pre-training for natural language generation, translation, and comprehension. In D. Jurafsky, Chai, Schluter, and J. Tetreault, editors. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online, July 2020. Association for Computational Linguisties. doi: 10. 11/8653/1/2020..c// acl-main. URI. htt.hhttpa///// Dactpa://aclantho.... .rgg/ 22000. 10000acl-mannn main. 703
Denoising sequence-to- te-euueeee-ssuueece
L. Li, H. Wang. L. Zha, Q Huang, S. Wu, G. Chen, and J. Zhao. Learning data-driven policy network for pre-training automated feature engineering. In The Eleventh International Conference on Learning Representations, 2022.
W. Liao, J. Wang. H. Li, C. Wang, J. Huang, and L. Jin. Doclayllm: An efficient multi-modal extension of large language models for text-rich document understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
June 2025.
H. Liu, C. Li, Q Wu, and Y. J. Lee. Visual instruction tuning. Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL Uttpttpps//opperrviie weet/forumidddwwl2xooe
Q. Liu, B. Chen, Guo, Ziyadi, Lin, Chen, and J.-G. Lou. TAPEX: Table pre-training via learning a neural SQL executor. In International Conference on Learning Representations, 2022. URL https
L. Long, X. Gu, X. Sun, W. Ye, H. Wang, S. Wu, Chenn and J. Zhao. Bridging the semantic gap between text and table: A case study on NI 2SQL In The Thirteenth International Conference on Learning Representations, Apr. 2025. BuRLiattta://opeereeee
A. Majee, M. Xenochristou, and W.-P. Chen. Tabglm: Tabular graph language model for learning transferable representations through multi-modal consistency minimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 19387-19395, 2025.
Mcta. Llama https: :tttps::/aii htt.:/etttaa...mmmoo hoogm/llo/mmeet 1lama- 3/ 2024.
L. Nan, C. Hsieh, Z. Mao, X. V. Lin, N. Verma, R. Zhang, W. Kryšciáskki, H. Schoelkopf. R. Kong, X. Tang, M. Mutuma, B. Rosand, Trindade, Bandaru, Cunningham, C. Xiong, D. Radev, and Radev, FeTaQA: Free-form table question answering. Transactions of the Association for Computational Linguistics, 10:35-49, 2022. doi: 10. 22.1166/tac_a_0000 ___0646.. URL.,.ttRRRttt httpt:://acllnthhoooo
wwg.//ogg////.. 2022. 1.31-1.3
A. Ni, S. Iyer, D. Radev, V. Stoyanov, W.-. Yih, S. Wang, and X. V. Lin. Lever: Learning to verify language- to-code generation with execution. In International Conference on Machine Learning, pages 26106 26106-26128. 26128. PMLR, 2023.
OpenAl. Chatgpt. https://openai.com
chatgpt 2022.
OpenAL. Gpt-4 technical report, 2023.
V. Pal, A. Yates, E. Kanoulas, and M. de Rijke. Multitabqa: Generating tabular answers for multi-table question answering. In Proceedings of the 6 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages
2023.
K. Papineni. S. Roukos, T. Ward, and W.J. J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting the Association for Compusational Linguistics, pages 311 111-318. 2002.
P. Pasupat and P. Liang. Compositional semantic parsing on semi-structured tables. In C. Zong and M. Strube, editors, Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1470- 1480, Beijing, China, July 2015. Association for Computational Linguistics. doi: 10.3115/v1/P15 1 URL httpe://aclaanthooogg.ooo wttp://hhhohohoooor--/-1--1-
12

---

S. Patnaik, H. Changwal. M. Aggarwal, S. Bhatia, Y. Kumar, and B. Krishnamurthy. CABINET: Content relevance- noise reduction for table question answering. In The Twelfth International Conference on Learning Representations. 2024, URL httpts:/ wwwp:eewwwwwen.ee..eeeeeeeeeeeee t
A. Radfford. J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal. G. Sastry, A. Askell, P. Mishkin. J. Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning. pages 8748-8763. PMLR, 2021.
A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492-224222 28518, PMLR, 2023.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67, 67, 2020.
P. K. Rubenstein, C. Asawaroengchai. D. D. Nguyen, A. Bapna. Z. Borsos, F. d. C. Quitry, P. Chen, D. E. Badawy, W. Han, E. Kharitonov, et al. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306. 12925, 2023.
S. Sarkar and L. Lausen. Testing the limits of unified sequence to sequence Ilm pretraining diverse table data tasks. NeurlPS 2023 Second Table Representation Learning Workshop. 2023.
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykow. S. Batra, P. Bhargava, Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
B. Van Breugel and M. Van Der Schaar. Position: Why tabular foundation models should a research priority. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors. Proceedings of the 41st International Conference Machine Learning. volume 235 of Proceedings of Machine Learning Research, pages 48976-48993, PMLR, 21-27 Jul 2024. URL htt.ttts://prrrceeiiingn mlr. wlr.press//vee v235 van hatpa///aabbbuuuee220 html
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
D. Wang, N. Raman, M. Sibue, Z. Ma, P. Babkin, S. Kaur, Y. Pei, A. Nourbakhsh, and X. Liu. DocLLM: A layout-aware generative language model for multimodal document understanding. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Pupers). pages 8529-8548, Bangkok, Thailand, Aug. 2024a. Association for Computational Linguistics. doi: 10. 10.18653/11/2224.acco acc-ooo.44..--o..6...63 URL
org/
2024. acl long..ng..... 463/
Z. Wang, H. Zhang, C.-L. Li, J. M. Eisenschlos, V. Perot, Z. Wang, L. Miculicich, Y. Fujii, J. Shang, C.-Y. Lee, and T. Pfister. Chain-of-table: Evolving tables in the reasoning chain for table understanding. In The Twelfih International Conference on Learning Representations, 2024b. URL //tp://///e/..e...///// 1d=4LOxmS4GQM
X. Wu, Zhang, and H. Li. Text-ta-table; new way of information extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2518-2533, 2022.
Z. Wu and Y. Feng. Protrix: Building models for planning and reasoning over tables with sentence context, 2024.
K. Xu, Y. Wang, Y. Wang. Z. Wang, Z. Wen, and Y. Dong. Sead: End-to-end text-to-sql generation with schema-aware denoising. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1845-1853, 2022.
Y. Xu, S. He, J. Chen, 7. ZengXiangrong, B. Wang, G. Liu, J. Zhao, and K. Liu. LLASA: Large language and structured data assistant. In L. Chiruzzo, A. Ritter, and L. Wang. editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Techmologies (Volume 1: Long Pupers), pages 1935-1946, Albuquerque, New Mexico, Apr. 2025. Association for Computational Linguistics. ISBN 979-8-89176-1888 2025. naacl- long. 977
URL.httts://acclanthh org/
N. Yadati, M. Nimishakavi, P. Yadav, V. Nitin, A. Louis, and P. Talukdar. Hypergen: A new method for training graph convolutional networks on hypergraphs. Advances in neural information processing systems, 32, 2019.
13

---

J. Yang, A. Gupta, S. Upadhyay, L. He. R. Goel, and S. Paul. TableFormer: Robust transformer modeling for table-text encoding. In S. Muresan, Nakov, and A. Villavicencio, editors. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volame 1: Long Pupers), pages 528-537. Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.18655/41/2022.acc--oo Sstps://aclanthon
URL
org/2022 2022. acl 1ong.ong... 40
C. Ye. G. Lu, H. Wang. L. Li, S. Wu, G. Chen, and J. Zhao. Towards cross-table masked pretraining for web data mining. In Proceedings of the ACM on Web Conference 2024, pages 4449-4459, 2024a.
W. Ye, J. Hu, L. Li, H. Wang. G. Chen, and Zhao. Data contamination calibration for black-box 1lms. arXiv preprint arXiv:2405. 19330, 2024b.
Y. Ye, B. Hui, M. Yang. B. Li, F. Huang, and Y. Li. Large language models are versatile decomposers: Decomposing evidence and questions for table-based reasoning. In Proceedings of the 46th International ACM SIGIR Conference Research and Development in Information Retrieval, pages 174-184, 2023.
P. Yin. G. Neubig, W.-t. Yih, and S. Riedel. Tabert: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8413-8426, 2020.
T. Yu. R. Zhang. K. Yang. M. Yasunaga, D. Wang, Z. Li, J. Ma, Li, Q. Yao, S. Roman, Z. Zhang, and D. Radev. Spider: A large-scale human- labeled dataset for complex and cross-domain semantic parsing and text-to-SQI. task. F. Riloff, Chiang, Hockenmaier, and Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1911-1,,,, Brussels, Belgium, Ocw, 2018. Association for Computational Linguisties. doi: 18.18653/ 1/D18- 1425, URI. hctpe://aclanthhoogyyooo org/ Drg/DD8-142
D. Zhang, S. Li, X. Zhang. J. Zhan, P. Wang, Y. Zhou, and X. Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15757-15773, 2023a.
T. Zhang, X. Yue, Y. Li, and H. Sun. Tablellama: Towards open large generalist models for tables, 2023b.
X. Zhang, D. Wang, L. Dou, Q. Zhu, and W. Che. A survey of table reasoning with large language models, 2024.
V. Zhong, C. Xiong, and R. Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint (xXiv:1709000100 2017.
D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In The Twelfth International Conference on Learning Representations, 2023a.
K. Zhu, J. Wang, J. Zhou, Z. Wang. H. Chen, Y. Wang. L. Yang, W. Ye, Y. Zhang, N. Z. Gong, al. Prompt- bench: Towards evaluating the robustness of large language models adversarial prompts. arXiv preprint arXiv:2306.004: 04528, 2023b.
A. Zhuang, G. Zhang. T. Zheng, X. Du, J. Wang, W. Ren, W. Huang, Fu, X. Yue, and W. Chen. Structim: Towards building generalist models for structured knowledge grounding, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
The claims in the abstract and introduction caaioettionnmmmllyyyy the limitations of current LLMs in handling table structures and the proposal of TAMo multimodal solution are supported throughout the methodology and experimental results sections (Sections and The claimed improvements and the StructQA benchmark are also thoroughly evaluated.
Justification:
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
( ("Limitations") discusses several limitations, including dependency pre-structured tables, lack of support for multi-tun multi-turn reasoning, and the need for extensive instruction data for robust generalization.
Justification:
Section
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not present formal thcoretical theorems or proofs; instead, focuses on empirical and architectural contributions.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
Section 3 and Appendix [eenciiieeeeebbbee dataset details, hyperparameters, model training setups, and evaluation metrics. We also release code and datasets at lhttpa
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code and datasets are openly provided through an anonymous link included in the paper: https:: //tg://gtttttbbtt thub. com/liyyooi/TAMO
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 3.2 and Appendix C provide comprehensive details about training setups, hardware used, optimizer settings, and dataset splits.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
15

---

Due to the high computational cost associated with large-scale experiments involving LLMs, we did conduct repeated runs to report error bars. However, we used consistent random seeds and standardized settings across all experiments to ensure fairness and comparability.
Justification:
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix C.1 and C.2 reports usage of NVIDIA A100-800 80- GPUs, batch size, number of epochs, and training time per epoch, fulfilling reproducibility requirements.
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https htt////nuripp..ee hwtpp//bblcc...ccccccciiiiiaaaaa
Answer: [Yes]
Justification: The research uses publicly available datasets and does not involve human subjects sensitive data, conforming ethical standards.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section and the conclusion discuss the benefits of enhancing LLMs' capabil- ity for structured data reasoning. Potential misuse is implicitly addressed by not releasing any tuned LLM weights.
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The released code are research-focused and the datasets used are publicly available, posing no risk misuse.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The paper cites all datasets and models used (e.g., Llama2, WikiTQ, WikiSQL, etc.) with appropriate references and usage guidelines.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The StructQA benchmark introduced and documented in Section 3. and Appendix A with code and data made available for replication.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
16

---

Justification: The paper does not involve crowdsourcing research with human participants.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No research with human subjects conducted.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration not required.
Answer: [NA]
Justification: We used LLMs (e.g., ChatGPT) only for minor language refinement during manuscript preparation. No LLM was involved in the design, implementation, or evaluation of the core methodology.
17

---

A StructQA Dataset Details
As mentioned in Section we construct table structure understanding dataset StructQA, which has types of table structure tasks. Here, we provide the construct details. Specifically, we randomly select 500 tables from WikiTQ (Pasupat and Liang 2015), creating 3 question templates for each table per task, resulting 7500 question-anne answer pairs. We chose tables from WikiTQ as the source for StructQA due to its large, diverse collection of high-quality, real-world tables. Crucially, these tables are structurally simple for humans to understand. The fact that modern LLMs struggle with these seemingly simple structures makes their failure more pronounced and powerfully demonstrates the core problem we aim to solve. We split the data into training. validation, and test sets with a ratio of 60%, 20%, and 20%, respectively. The question templates for each task are as follows:
(1) Cell location
What is the value in the column (column name) of sample row (row number)?
Can you tell me the value of the column (column name) in sample row (row number)?
In sample row (row number), what is the value for the column [column name)?
(2) Column lookup
In sample row [row number), which columns contain the value |cell value
Can you identify the columns in sample (row number) that have the value (cell value)?
Which columns in sample row [row number] are associated with the value (cell value]?
(3) Row lookup
Which rows in the column (column name) have value of (cell value]?
Can you identify the sample rows where the column (column name] equals (cell value)? ?
In the column (column name), which rows contain the value (cell value}?
(4) Column comprehension
What the distinet values in the column (column name)
Could you list the unique values present the column (column namej?
In the column (column name). what various values can be found?
(5) Row comprehension
What are the values each cell in row (row number] of the sample?
Could you provide the cell values for each column in sample row (row number)?
In sample row (row number), what are the respective cell values?
B Public Datasets Details
HiTab [Cheng et al. 2022 is a table question answer dataset on hierarchical tables, which have multi-level structure in the table header. It comprises 10,672 questions over 3,597 hierarchical tables. We use execution accuracy as the evaluation metric. Experiments on HiTab can effectively demonstrate the superiority of hypergraphs in modeling arbitrary hierarchical tables.
WikiTableQuestions (WikiTQ) Pasupat and Liang 2015 a large-scale dataset for complex question answering over tables. It consists of 22,033 questions over 20108 Wikipedia tables. The questions often require complex reasoning and aggregation operations. The primary evaluation metric for WikiTQ is the accuracy of the predicted answers compared the ground truth.
WikiSQL [Zhong et al., 2017 a dataset designed for natural language to SQL query generation. contains 80,654 natural language questions paired with SQL queries and their corresponding answers over 24,241 tables from Wikipedia. We use the execution accuracy (the correctness of the query results) as the evaluation metric for WikiSQL.
18

---

FeTaQA Nan et al. 2022) is free-form table question answering dataset that emphasizes generating comprehensive, free free-text answers. It comprises 10,279 questions over 3,641 tables, primarily sourced from Wikipedia. BLEU metric [Papineni et al. 2002 is recommended officially to evaluate the similarity between generated and reference answers.
C Experiments
C.1 Implementation Settings
Experiments are conducted using NVIDIA A100-80G GPUs. All experiments are conducted using the same of random seeds for reproducibility.
The table encoder. We the hidden dimension of the table encoder to 768 and use 3-layer hypergraph transformer to model global table structure. To initialize each cell's semantic representation before hypergraph construction, we adopt the embedding of [CLS]" token output from the pretrained RoBERTa-base model applied the cell's textual content. The alignment projector is implemented as a single-layer MLP, which maps the encoder outputs to the LLM embedding space.
0.4
Hour
0.2
LLMs. We use the open-sourced Llama2-7h as the LLM backbone. In fine-tuning the LLM with LoRA, the lora parameter (dimension for LoRA update matrices) is set 8, and the lora _alpaaaphaaaaa (scaling factor) is set to 16. The dropout rate is set to 0.05. In prompt tuning, the L.L configured with virtual tokens. The number of max text length is 1024. The number of max new tokens, i.e., the maximum number of tokens to generate, is 128. We use Mistral-7B for some experiments.
PT
TAMO
TAMO-LoRD LORA
LORA
TAMO-SFT
SFT
Figure 7: Training time effi ciency comparison under dif- ferent settings for 1 epoch WikiTQ dataset.
Optimization. We use the AdamW optimizer. We the initial learning rate at le-5, with a weight decay of 0.05. The learning rate decays with a half-cyele cosine decay after the warm-up period. The batch size is 8, and the number of epochs is 10. To prevent overfitting and ensure training efficiency, an early stopping mechanism is implemented with a patience setting of epochs.
C.2 Efficiency Analysis of TAMO
To further demonstrate the practicality of TAMO, we evaluate its operational efficiency. In experiments, utilize server equipped with 2 A100 GPUs. Only SFT uses 2 GPUs while con- ducting all other experimental setups with single GPU training. We measure the time required to run 1 epoch on the WikiTQ dataset. The results (i)-TMMo has a faster runtime efficiency compared to LoRA; (i)-TAMo LoRA shows only slight increase in runtime com- pared to LoRA, as does TAMO SPT compared to SFT. There- fore, injecting learnable table features does not significantly add to the computational burden in practical applications.
Llama2-7B
Mistral-7B
0.65
Accuracy
0.55
shown in Figure) We found that
0.35
We clarify that our "cost-effectivee claim is relative to the computational cost of fine-tuning the whole LLM itself. As shown in Figure ToMo (in the frozen L.MM setting) requires significantly less training time than LoRA full SFT, only train the lightweight table encoder and alignment projector.
Figure 8: Analysis study of differ- ent numbers of table structure to- kens on the WikiTQ dataset.
httpp://wngiiggfee....//ttssssaa 2- hf httppp//wwgggggggoo...o//sssssts
19

---

C.3 Ablation of Table Token Number Parameter
We further explore the impact of the table structure token quantity parameter on the model's per- formance. Specifically, in the frozen LLM setting, we evaluate TAMo on the WikiTQ dataset with varying numbers of table structure tokens. Due to limited computational resources, we randomly select 6000 samples from the WikiTQ training set for the experiments, keeping the validation and test sets unchanged. The experimental results are shown in Figure The final performance of the model is consistently similar when the number of tokens is two more {2, 3, 5, 7,9), which indicates that a minimum one token is sufficient to explain the structural information in the table.
C.4 Ablation Study on Modality Components
To rigorously assess the contribution of each input modality in TAMo, discussed in Section 2.3
we conducted an ablation study comparing three configurations: (1) Graph-only, using only the structural embeddings X.t: (2) Text-only, using only the serialized text X and (3) TAMO, our full multimodal approach combining both X and X We evaluated these across all tuning settings (PT, LoRA, SFT) on the StructQA benchmark.
The results are presented in Table 4 The Graph-only pipeline consistently fails on this generative QA task (e.g., accuracy below 8% across settings), confirming that structural information alone insufficient without semantic grounding. The Text-only model performs substantially better but exhibits lower accuracy and robustness compared to the full model, especially in parameter-effciient settings (PT, LoRA). Only the complete ToMo framework achieves the highest performance and robustness, empirically validating the synergistic necessity of combining both structural and textual modalities.
Tuning
Modality
StructQA
Permute
Robustness
Graph-only
7.00
7.13
14.60
PT
Text-only
37.80
29.93
31.07
TAMo (Both)
59.07
43.47
43.80
Graph-only
7.93
7.00
14.80
LoRA
Text-only
45.93
35.87
39.67
TAMO (Both)
67.67
42.77
53.73
Graph-only
6.93
6.33
16.53
SFT
Text-only
62.00
42.86
51.53
ToMo (Both)
69.40
43.74
64.07
Table 4: Ablation Modality Components (StructQA- Accuracy Permutation Acc. Robustness)
C.5
Comparison with Structure-Aware Baselines (TAPAS/TAPEX)
To further situate TAMO relative prior paradigms focused table encoding, we conducted direct comparison against established structure-awar encoder models, TAPAS (Herzig et al.) 2020 and TAPEX [Liu et al.] 2022], on our StructQA benchmark (Table [5. TAMo dramatically outperforms these baselines on the core structural understanding tasks (StructQA and Permutation Accuracy), showcasing the advancements achieved by integrating structural information directly into modern LLMs.
We observe that while TAPAS and TAPEX show higher robustness scores, this is largely an artifact of their very low base accuracy (c.g., 4.67% for TAPAS). It is easier to maintain consistency when model is consistently incorrect. TAMo's performance demonstrates a far superior grasp of table structure.
C.6 Evaluation of Cross-Dataset Generalization of TAMO
In Table we demonstrated that TAMo, when trained individually on each dataset, achieves significant improvements the corresponding test sets. This raised the question of whether TAMo's
20

---

Model
StructQA (Ace.)
Permute (Acc.)
Robustness
TAPAS
4.67
4.67
65.47
TAPEX
12.33
9.60
54.73
TAMO (PT)
59.07
43.47
43.80
Table 5: Comparison with TAPAS/TAPEX on StructQA
table structure embeddings are generalizable to other datasets. To address this, we evaluated TAMO models trained on one dataset against the test sets of other datasets, as shown in Table 6
Theoretically, TAMO's s table structure embeddings are designed model general table structures. However, the training process also relies on task-specifi instruction data, and the loss for learning table structure representations is tied to QA objectives. This means the table embeddings can be influenced by the types of instructions used during training, introducing task-specific biases. For example, embeddings trained on StructQA, which involves simpler table structures, tend to perform well on structural recognition tasks but lack the complexity required for reasoning-heas WikiTQ. Consequently, while table structure embeddings trained on individual tasks consistently outperform baselines without structure embeddings, they fall short of matching the performance of embeddings trained directly on the target task. We also observed that datasets with significant differences, such as PetaQA-whice uses BLEU as evaluation metric for free-text answers show limited cross doosset transferabilityy. The model trained on FetaQA fail to provide improvements other datasets, and vice such as WikiTQ and WikiSQL, we observed some degree of transfcrability, suggesting that TAMo leverage shared patterns among related tasks. These observations consistent with findings in TableI.lama (Zhang et al., 2023b). where differences in task formats and reasoning complexity limited cross-task generalization.
tasks like
However, for QA datasets with similar formats and objectives,
Evaluation Dataset
StructQA Accuracy
HiTab Accuracy
WikiTQ Accuracy
WikisQL. Accuracy
FetaQA BLEU
Metric
Base
8.60
7.77
14.50
21.44
20.08
StructQA
59.07
16.73
18.74
32.57
8.38
HiTab
17.53
48.86
27.46
38.83
1.78
WikiTQ
16.40
29.29
37.06
38.74
0.95
WikiSQL
18.73
24.43
23.85
76.45
1.18
FetaQA
0.00
0.00
0.02
0.00
36.52
Table 6: Generalization results each TAMo separately trained on different dataset.
To isolate the effect of table structure representations from task-specific biases, we conducted additional experiments focusing solely table structure prediction tasks. As shown in Table 3 table encoder trained on one dataset achieved Fl scores above 60 % on structure prediction tasks from the other dataset. This demonstrates that TAMo's 's table encoder captures a unified representation of table structures and validates the generalizability of our approach.
A key factor is the absence of scale, task-agn pretraining for TAMO's table encoder. Similar how CLIP [Radford al., (202T] decouples modality-speciffe representations through extensive pretraining, dedicated pretraining phase for TAMO's table mncodr-ffccusig purely on table-related structural information- could mitigate task-specific biases. This remains an important direction for future work to enhance generalization across domains and datasets.
C.7 Effectiveness on Multiple- Halbee Scenarios
To validate TAMO in multiple-table scenarios, we have conducted additional experiments on MultiTabQQA-geoQues (Pal et al.) 2023] dataset. This dataset involves multiple-table queries with total token lengths reaching up to 4K, relatively larger than current TableQA benchmarks. Specifically, evaluated its cell selection task using precision, recall, and Fl score metrics. Due to the
21

---

Setting
Method
Precision
Recall
Fl score
Inference Only
One-shot
9.68
5.96
7.38
Prompt tuning TAMO
4.83
3,46
4.03
Frozen LLM
6.82
4.86
5.67
A Prompt tuning
t 41.20%
t 40.46%
t 40.69%
LoRA
30.56
10.30
15,41
Tuned LLM (LoRA)
TAMO + LoRA
28.32
10.67
15.50
LORA
-7.33%
t 3.59%
0.58%
SFT
30.55
11.04
16,22
Tuned LLM (SFT)
TAMO
49.36
25.46
33.59
SSFT
1 61..7%
1 130.62%
1 107.09%
Table 7: Effectiveness on MultiTaabQA-geoQueryy
unique output format requirements of this task, we adopted a one-shot setting across the following experiments while keeping other parameters unchanged. shown Table ToMo achieves over 40% and 100% improvements under frozen LLM and SFT LLM settings, respectively, demonstrating its effectiveness multi-table scenarios. While TAMo shows only marginal advantages in the LoRA setting, will investigate the detailed configurations in future work.
C.8 Effectiveness for Different LLMs
In our previous experiments, we choose Llama2-7B as the baseline for a fair comparison with prior works. However, is undeniable that, performance of various LLMs. In this section, we examine the issue of the baseline. Briefly, we introduce three advanced Cecelines-Tableliiee 8B-and conducted experiments under the frozen LLM setting.
modality, TAMO should theoretically enhance the
(Zhang et al. 2023b], Mistral-7B, and LLaMA 3.1
Table and Table 9 demonstrate that: (i)-The minimal gap (0.0016 acc.) between the base and prompt tuning on TableLlama indicates that the supervised fine-tuned LLMs already possess a strong capability follow tabular format instructions. Consequently, prompt tuning has a limited effect. However, incorporating global table structure information through TAMO further enhances table reasoning capabilities. (ii)-The ultimate performance of TAMO influenced by the capability of the LLMs. For instance, Llama3 shows significantly better performance than TableLlama (based on Llama2). (wil)-hiile LLaMA 3.1 8B achieves stronger baseline than LLaMA 2 7B, adding the table encoder consistently improved performance, with gains reaching over 10% on certain datasets. This further validates the unique benefits of hypergraph-bart based structural representation of tables across more advanced open-source LLMs.
Method
Llama2
TableLlama
Mistral
Inference Only (Base)
14.50
31.22
18.44
Prompt tuning
29.86
31.38
44.98
TAMo
37.06
39.85
47.33
A Prompt tuning
t 24.11%
+
26.99%
1 5.22%
Table 8: Evaluate the scalability for different LLMs of proposed TAMO the frozen LLM setting (prompt tuning) on the WikiTQ dataset.
22

---

Dataset
StructQA Structural QA Accuracy
HiTab Hierarchical QA
WikITQ Table QA
WikiSQL Table QA
FetaQA Free-form QA BLEU
Setting
Task Type Evaluation Metric
Accuracy
Accuracy
Accuracy
Inference Only
Ilama 3.1 8B
15.73
19.51
23.80
31.60
14.05
Prompt tuning
71.53
69.38
53.71
77.06
36.16
Frozen LLM
TAMO
78.00
73.73
56.93
85.44
38.09
Promut hewwing
9.05%
6.27%%
6.00%
10.87%
5.34%
Table 9: Results advanced LLM.
D Discussions
D.1
Positioning of TAMO
While both HyTre Trel [Chen et al. 2024 and TAMo adopt hypergraphh-based framework, there are significant distinctions. Hy Tirel focuses general tabular representation learning and, stated in its limitations, cannot handle joint text-table reasoning tasks like TableQA. In contrast, is non-trivial for TAMO to pioneer treating tables table representations with text representations to tackle complex reasoning tasks.
independent modality within LLMs, aligning hypergraph- based
This distinction parallels advancements in other domains. For example, in vision, ViT (Dosovitskiy et al., 2020] and CLIP [Radford et al. [02T] act as modality encoders, while GPT-4v [OpenAl 2023) and LLaVA [Liu et al.) [2023] integrate these encodings into multimodal frameworks. In the audio domain, there is a similar phenomenon, as shown in Table 10] For the first time, TAMo fills this gap in the table domain, going beyond table encoder to multimodal reasoning framework. This cross-modal fusion makes TAMo a significant advancement, not an incremental improvement. Notably, while TAMo and HyTreel share similar network architecture, their training tasks and optimization objectives are entirely different, further underscoring the contribution of our approach.
Domain
Modality Encoder
Multimodal LLMs
Vision Domain
ViT 2020]. CLIP 2021
GPT-4v [2023] LLaVA 22023, MiniGPT-4
2023a
Audio Domain
Whisper 2023
SpeechGPT 2023a AudioPaLM 2023
Table Domain
Hy Trel 2024
TAMo (Ours)
Modality alignment with LLMs obtain corresponding
Encoding domain-
Role
specific data
domain-speciffe multimodal models
Ability Generative Tasks (e.g.. QA)
No
Yes
Table 10: Positioning of TAMO the table domain.
D.2 Comparison with Potential Approaches
We acknowledge that there are several alternative approaches to incorporating table structure into LLM-based models. These include: (1) using 2D positional embeddings to capture row and column information, (2) data augmentation techniques to enforce permutation invariance, and (3) injecting fine-grained structural representations tailored specific downstream tasks. Below, we discuss the applicability and limitations of these strategies in contrast to our proposed approach.
Using 2D positional embeddings capture row and column information. Using 2D positional embeddings is indeed a natural approach, as it captures row and column information directly. However, implementing this method often requires intrusive modifications to the position encoding layer of LLMs (e.g., as in TableFormer (Yang et al. (022]), demanding extensive re- re-training of these position encodings. Such re-training is highly dependent on specific LLM architectures, and the learned modifications are not theoretically transferable to other L.MSs. In contrast, our proposed table encoder designed to operate as an external plugin of tabular modality, minimizing modifications to the LLM itself.
23

---

Data augmentation techniques to enforce permutation invariance. While data augmentation techniques to enforce permutation invariance are intuitive, they present practical challenges. For tables with dimensions Training on such large augmented dataset is computationally prohibitive, and the resulting models prone to overfitting due to the enormous training data requirements. TAMO is designed to be data-efficient, achieving structural permutation invariance without relying on large-scale data augmentation.
m. the number of possible permutations grows factorially as n! x m!.
Injecting task- -seciicc fine-grained structural representations. Several recent parallel studies have proposed injecting fine-gu rene-graindd table semantics into LLMs for specific structured reasoning tasks. For example, TNT [Long et al. 2025] targets the TEXT-TO- SQL task and models the database schema using column embeddings, enabling the LLM to generate accurate SQL queries. However, it ignores row-level semantics, making it less suitable for tasks like TABLEQA. HeGTa [Jin et al. 2025] adopts a heterogeneous graph encoder capture rich structural features in complex tables (e.g., merged cells), but its graph encoding depends on the original table layout and lacks permutation invariance, making it sensitive to row/column reordering. Recent LLaSA [Xu et al.] [225] introduces parameter- heavy G-Former module to unify various structured data formats (e.g.. knowledge graphs, databases), yet primarily tailored for clean, schematic data and struggles with the irregular formats in real-world tables such as HiTab. Moreover, these methods typically require extensive pretraining. Concurrently, HyperG [Huang al. 2025] mitigates sparsity via -Lggdddded contextual augmentation, followed by semantic hypergraph construction and prompt-attentive hypergraph learning, which typically entails substantial extra preparation beyond task-level supervision due the explicit augmentation pipeline. In contrast, our proposed TAMo framework is designed to be pretraining- and plug-and-play requiring only task-level training data for effective adaptation.
D.3 Beyond Row and Column Permutations
While row and column permutations are the most prominent cases in tabular data, other forms of order permutations can arise in more complex table structures. These include:
Nested Table Structures. In hierarchical or grouped tables, sub-tables are often nested within a broader table structure. Permutations can occur within these nested sub-tables, reflecting changes in the ordering of hierarchical levels. Such structures are common in multi-level reports and datasets with grouped summaries.
Composite Attributes. Tables may contain multi-column attributes where relationships or depen- dencies exist between columns. For instance, in a table representing geographic data, attributes such as latitude and longitude might form a composite structure. Permutations within such attributes could represent alternative orderings of these dependent fields, requiring specialized handling to maintain semantic coherence.
Cell-Level Permutations. In some cases, individual cells may contain structured or stuuturt data, such as lists, arrays, or key-value pairs. Order changes within these cell values represent another form of permutation, particularly relevant in domains where embedded structured data is prevalent (e.g., JSON-liki entries or lists of items within a cell).
While these forms of permutations observed in complex hierarchical datasets, such as HiTab [Cheng et al. 2022). In this study, we focus primarily on flat table structures from mainstream TableQA datasets, where row and column permutations are the predominant concerns. Addressing these additional forms of permutation is an important direction for future work, particularly for datasets with more complex organizational patterns.
significant in certain contexts, they are most commonly
24


Reviews:
[{"review_id": "41GyK4LR6Q", "reviewer": "Reviewer_TiPM", "summary": "This paper introduces TaMo, a framework that treats tables as a distinct modality to improve LLM performance on table reasoning tasks. Instead of flattening tables into text, TaMo uses a hypergraph encoder to capture structural relationships in the table. TaMo feeds these structure embeddings into the language model as additional input, improving its ability to reason over tables. Experiments across five benchmarks show that TaMo improves accuracy compared to standard LLM approaches. \n\nThe authors also introduce StructQA, a new benchmark to evaluate if permutation invariance is respected", "questions": "My main concern is the evaluation. \n- The term “Specialist SoTA” is used in Table 2, but the paper does not discuss which specific models were used for each dataset. Please provide a breakdown of which models are used for with brief descriptions of how they were trained or evaluated. \n- TaMo underperforms GPT-4 on WikiTQ, but this is not adequately discussed or analyzed - please include some minor comment on this\n-  The paper does not include comparisons to well-established structure-aware baselines such as TAPAS/TAPEX. TAPAS and TAPEX are more relevant baselines than GPT-3.5 for table QA tasks\n- The paper lacks ablations that show the contributions of TaMo’s components. Consider adding or discussing ablation experiments, this would clarify which parts of TaMo are important\n- While generally clear, some aspects (e.g., multiset functions and alignment details) could be simplified or clarified further - please do that\n- I would also like the authors to make more clear their contribution compared to prior works because it looks like the paper borrows a lot from existing approaches (e.g. hytrel)", "limitations": "The authors have addressed several limitations and I have no concerns regarding negative societal impact.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- Proposes treating tables as a separate modality, enhancing structural reasoning in LLMs.\n- Leverages hypergraph-based encoders to model hierarchy and permutation invariance — key properties of tabular data\n- The authors show large performance gains overall\n- Introduces StructQA, a new benchmark for testing if permutation invariance in tables is respected\n- Plug-and-play integration with LLMs\n- The paper is well-written and easy to follow\n\n**Weaknesses**\n- While the combination of parts is novel, many components (e.g., hypergraphs, structural encodings) are close extensions of existing work\n- While generally clear, some aspects (e.g., multiset functions and alignment details) could be simplified or clarified further\n- The evaluation does not seem thorough, the authors do not compare against other models that also incorporate table structure (e,g, TAPAS, TAPEX and others) and are baselines for table QA\n- Specialist SoTA scores are included but not clearly explained, making it hard to assess the fairness of the comparison\n- TaMo underperforms GPT-4 on WikiTQ, but this is not adequately discussed or analyzed\n- Lacks ablation studies to isolate the contribution of key architectural components", "was_revised": true, "final_justification": "The authors addressed most of my concerns and said they will update their manuscript accordingly. \n\nThey conducted additional experiments, included explanations/clarifications, and elaborated on previously vague statements.\n\nI updated my score to reflect this. \n\nHowever, I still have some concerns regarding originality/significance of the work.", "rebuttal": "We sincerely thank the reviewer for their valuable time and constructive feedback. We appreciate that the reviewer found our paper well-written and easy to follow. We address the concerns below.\n\n**1. W4 / Q1 (Clarification of \"Specialist SOTA\"):** \n\nThis is an excellent point. The term is indeed vague. In our revision, we will update the caption of Table 2 to explicitly name the specific model and provide its citation for each dataset. The specific models are:\n\n- For **HiTab**, the Specialist SOTA is **TableLlama**, which is based on Llama2-7B and underwent extensive supervised fine-tuning (SFT) on the HiTab training set along with other table QA datasets. The score reported in the original paper is 64.71, which we reproduced as 63.76.\n\n- For **WikiTQ** and **FetaQA**, the SOTA is **CABINET**, which improves LLM performance by using an external scorer to assign higher weights to question-relevant table cells, thereby filtering out noisy ones.\n\n- For **WikiSQL**, the SOTA is **SeaD**, which enhances text-to-SQL capabilities through a schema-aware denoising objective.\n\nIt is important to note that our purpose in including these specialist SOTAs is to highlight different methodological directions. Unlike our TAMO framework, which directly enhances the LLM's end-to-end understanding of the table as a modality, methods like CABINET and SeaD rely on external modules or task-specific pre-processing.\n\n**2. W5 / Q2 (Performance on WikiTQ vs. GPT-4):** \n\nWe thank the reviewer for this sharp observation. We will add a discussion of this point to Section 3.3. Our analysis is twofold. First, the WikiTQ dataset is sourced from early Wikipedia web pages, and its questions are often of a general nature. This creates a potential for data contamination, where a model as extensively trained as GPT-4 may have already encountered these tables or facts during its pre-training, allowing it to answer some questions without fully relying on the provided table.\n\nSecond, as we initially noted, the performance gap is also attributable to the immense scale and powerful general reasoning capabilities of the proprietary GPT-4 model. However, our key finding remains valid: TAMO provides a substantial boost over its base model (Llama2-7B SFT: 45.81 vs. Text-only SFT: 31.22), proving the effectiveness of our approach. Furthermore, as shown in Appendix C.6 (Table 7), when TAMO is applied to a more capable base model like LLaMA 3.1 8B, its performance on WikiTQ improves significantly, demonstrating that TAMO's benefits scale with the underlying LLM's power.\n\n**3. On Comparisons with TAPAS/TAPEX (W3, Q3):** \n\nWe agree that comparing with seminal structure-aware models like TAPAS and TAPEX is important. Our initial focus was on the modern **decoder-only LLM-based paradigm**, thus we prioritized comparisons with methods that also use serialized inputs (e.g., GPT-4, TableLlama). To further strengthen our evaluation, we have conducted additional experiments comparing TAMO with TAPAS and TAPEX on our StructQA benchmark. The results are as follows:\n\n| Model | StructQA (Accuracy) | Permute (Accuracy) | Robustness |\n|---|---|---|---|\n| **TAPAS** | 4.67 | 4.67 | 65.47 |\n| **TAPEX** | 12.33 | 9.6 | 54.73 |\n| TAMO | 59.07 | 43.47 | 43.8 |\n\nThe results clearly show that TAMO dramatically outperforms these baselines on the core structural understanding tasks (StructQA and Permutation Accuracy). It is crucial to consider the architectural differences: TAPAS is a BERT-based encoder, and TAPEX is a BART-based encoder-decoder model. In contrast, our TAMO framework is specifically designed as a multimodal extension for **decoder-only LLMs**, which represent the current dominant architecture. While TAPAS and TAPEX show higher robustness, this is largely an artifact of their very low base accuracy; it is easier to remain consistent when a model is consistently incorrect. TAMO's performance demonstrates a far superior grasp of table structure. We will add a new table and this analysis to the appendix in our revised manuscript.\n\n**4. On Ablation Studies (W6, Q4):** \n\nWe would like to respectfully clarify that our main experimental design in **Table 2** constitutes a core ablation study. Across four different settings (Inference-only, Prompt Tuning, LoRA, SFT), we directly compare the performance of using text-only inputs versus our TAMO approach, which adds the table structure modality. The consistent and significant gains demonstrated across all settings (e.g., +21.27 absolute accuracy on StructQA for Prompt Tuning vs. TAMO) serve as the primary evidence for our method's contribution. We also provide more fine-grained analyses in **Section 3.6 (encoder effectiveness)** and **Appendix C.3 (impact of token numbers)**.\n\n**5. On Novelty vs. HyTrel and Clarity (W2, Q5, Q6):** \n\nWe appreciate the request for clarification. We respectfully point the reviewer to Appendix D.1, \"Positioning of TAMO\", where we provide a detailed discussion and a summary table (Table 8) to articulate the fundamental difference. In short, HyTrel is a modality encoder for learning table representations, whereas TAMO is a complete multimodal reasoning framework that integrates such an encoder into an LLM to solve joint text-table tasks (e.g., QA). This is a non-trivial conceptual leap, analogous to the distinction between ViT (an encoder) and LLaVA (a multimodal LLM). To ensure this crucial point is not missed, we will add a forward-reference to Appendix D.1 in our main methodology section. We will also revise Section 2.2 to further clarify the mathematical notations to improve readability."}, {"review_id": "Fa4Uusi9ho", "reviewer": "Reviewer_v8d6", "summary": "This paper proposes a multimodal large language model called TAMO: by converting table contents into hypergraphs, then further transforming them into embeddings, which are fed into the LLM to enhance its overall understanding capabilities. Results show that this method can achieve significant improvements on multiple benchmarks. Besides, a novel tabular-structure-understanding benchmark is released for the community.", "questions": "Please refer to the Strengths and Weaknesses.", "limitations": "Please refer to the Strengths and Weaknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The approach is innovative: For table understanding scenarios, this paper proposes a special encoding scheme for table reasoning tasks, which introduces a graph-based approach into the LLM to enhance model performance.\n\n2. The logic is clear: The paper first highlights the importance of table representation, as shown in Fig. 2, then designs a specific table encoder to address issues in table understanding scenarios (Sec. 2), and demonstrates the benefits of this approach through experimental validation (Tab. 2).\n\n\nWeaknesses:\n1. As described in Sec. 2 & Figure 4, TAMO adds an additional multimodal processing component to represent the content of tables, while the original table content serialized as text is still preserved as input to the LLM. Based on experience, this approach essentially provides the LLM with detailed inputs from multiple perspectives, which naturally enables the model to achieve better performance. However, does this approach (keep the serialized table text) introduce redundancy in information representation? In other words, why does the parsed table text still need to be input into the LLM in text modality? Additionally, it is recommended to further clarify the statistics in Table 2 (Main results).\n\n2. The experimental conclusions are not sufficient enough: Since the addition of new modalities on LLM is not a completely innovative solution, but rather a situation where ongoing work is continuously innovating solutions on this topic. As a new modality bridging solution, has it been better compared with other multimodal solutions, such as DocLLM (Layout-Aware) and QWEN2-VL / TextMonkey (Image-Aware)? Can further analysis of this part be provided in Section 3.3?", "was_revised": true, "final_justification": "Thanks for solving my concerns and I hope this work is thoughtful for the community. I will raise my rating to 4", "rebuttal": "We thank the reviewer for their positive feedback on our work's innovation and clear logic. We address the thoughtful questions below.\n\n**1. W1 (Information Redundancy of Using Both Serialized Text and Structure Embeddings):**\n\nThis is an excellent question regarding our design philosophy. We retain the serialized text because it provides the **fine-grained semantic content** of the table, which LLMs are exceptionally skilled at processing. The hypergraph-based structure embedding, conversely, provides the **global relational structure** (e.g., row-column relationships, hierarchies) that is inherently lost during serialization.\n\nThe two modalities are complementary: the text provides the content (\"what\"), and the structure embedding provides the context and relations (\"where\"). As our case study in Figure 5 suggests, the structure token helps the LLM better focus its attention on the relevant parts of the serialized text to locate the correct answer. Therefore, this dual representation is not redundant but synergistic.\n\n**2. W2 (Comparison with Other Multimodal Solutions like DocLLM):**\n\nWe thank the reviewer for raising this point. We wish to clarify a fundamental distinction in the problem setting. Models like DocLLM or vision-language models (e.g., Qwen-VL) are designed to understand tables presented as images within documents, **focusing on layout and OCR challenges**. Our work, TAMO, addresses a different and complementary problem: reasoning over pre-structured tabular data (e.g., from databases, CSV files). As stated in our Limitations (Section 5), TAMO operates on this structured data format. The modalities are different (image+text vs. graph_structure+text), and the core tasks are distinct. Therefore, a direct comparison would not be appropriate."}, {"review_id": "MeLbirWaBz", "reviewer": "Reviewer_7uvU", "summary": "This paper addresses the topic of LLM-based question answering regarding tables.\nIt presents a new framework for this, TaMo, and shows that it outperforms previous methods, including using GPT4, GPT 3.5 and Llama 2 7B, and it is robust to permutations. The framework is based on a new method for representing complex tables as hypergraphs and encoding them. It then uses a fine-tuned Llama 2-7B model to answer questions based on the embeddings of both the structure and content of the table. The paper presents comparisons based on 5 question answering datasets, one of them, StructQA, was developed as a part of this work and includes 7500 examples based on 500 tables. The reported results are 42.65% average improvement in these comparisons (ranging from 4.5% to 86%), as well as outperforming the OpenAI models in 4 out of the 5 datasets. Further analysis in done on the changes in attention and the good generalization of the proposed method.", "questions": "Please perform a comparison with recent models, such as GPT4.1, O4-mini, Llama 3.3 and DeepSeek-R1.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper addresses an important problem and suggests a new approach for handling it. It includes significant experimentation, and it also provides a new dataset. It is clearly written and original.\nThe main quality weakness in my view is that it compares with old models (from over a year ago), which also raises concerns about its current significance. To be considered for NeurIPS, it would be important to show comparisons using the current state of the art models, including GPT4.1, O4-mini, Llama 3.3 and DeepSeek-R1, to clarify whether this method is still useful after the many improvements in LLMs that took place in the recent year.", "was_revised": true, "final_justification": "I feel that the authors have addressed my requests and the concerns of other reviewers. They provided comparisons with state-of-the-art models that they will include in their final version. Although the analysis of tables may seem like a narrow problem, I think it is important from a practical perspective. I raised my score since I think that the technical contribution and its practical value are significant enough for publication in NeurIPS.", "rebuttal": "We sincerely thank the reviewer for their valuable feedback and for acknowledging the importance of our work.\n\n**Comparison with Advanced Models:**\n\nWe completely agree with the reviewer that evaluating on the latest models is crucial in this fast-paced field. To thoroughly address this critical point, we allocated significant additional computational resources to conduct comprehensive new experiments, including on several models accessible only via paid APIs.\n\nFirst, to test whether the problem we identified still exists in the latest models, we evaluated several state-of-the-art LLMs you suggested on our StructQA benchmark. The results are as follows:\n\n| Model | StructQA | Permutation | Robustness |\n|---|---|---|---|\n| **GPT-4.1** | 60.33 | 49.86 | 60.07 |\n| **O4-Mini** | 55.93 | 53.27 | 59.40 |\n| **Llama-3.3-70B-Instruct** | 53.93 | 48.60 | 52.26 |\n| **DeepSeek-R1** | 57.46 | 52.07 | 63.2 |\n| GPT-4 (Original) | 51.40 | 47.13 | 50.53 |\n\n\nThese results are very telling. Even after more than a year of rapid progress, today's most powerful LLMs still struggle with table structure understanding, a task that is trivial for humans. While their performance on the original StructQA test set has improved (reaching ~60%), every single model shows a significant performance degradation after permutation. ***This confirms our central hypothesis: the serialization of tables into text sequences leads to a loss of critical structural information, a problem that persists even in the latest SOTA models.*** This finding significantly strengthens the motivation for our work.\n\nSecond, to demonstrate that our proposed method, TAMO, remains effective on newer models, we conducted further experiments. Our original choice of a 7B-scale model was to ensure a fair comparison with the established TableLlama baseline. In addition to the LLaMA 3.1 8B results already in our **Appendix C.6**, we have now evaluated TAMO on the recent Qwen2.5-7B-Instruct and Qwen3-8B models. The results are compelling:\n\n| Model | Setting | StructQA | Permutation | Robustness |\n|---|---|---|---|---|\n| Qwen2.5-7B | Inference Only | 38.20 | 31.80 | 40.87 |\n| Qwen2.5-7B | Prompt Tuning | 45.53 | 33.80 | 33.33 |\n| **Qwen2.5-7B** | **TAMO** | **72.27** | **61.33** | **62.73** |\n| Qwen3-8B | Inference Only | 46.86 | 41.73 | 48.87 |\n| Qwen3-8B | Prompt Tuning | 54.80 | 45.73 | 45.47 |\n| **Qwen3-8B** | **TAMO** | **70.20** | **53.93** | **55.6** |\n\n\nAs shown, TAMO provides a dramatic performance boost on these new models. For instance, on Qwen2.5-7B, TAMO improves the accuracy on StructQA from 45.53% (Prompt Tuning) to **72.27%**, an absolute improvement of nearly **27 points**. This clearly demonstrates that ***our method of treating tables as a distinct modality is a powerful and relevant solution for enhancing the structural reasoning capabilities of even the most recent LLMs.***\n\nWe will prominently feature these new findings in the revised manuscript. Thank you again for pushing us to strengthen our work in this crucial dimension."}, {"review_id": "goHkU4gYRy", "reviewer": "Reviewer_Qmvd", "summary": "The paper introduces TAMO which emphasizes on treating tables as an independent modality for tabular Question-Answering (QA) tasks. The paper presents a novel instruction tuning paradigm by combining structural information in tables and semantic information in text towards augmenting the capabilities of a pretrained LLM towards robust QA. TAMO further introduces the StructQA benchmark which evaluates various impending challenges like permutation invariance, structural variations etc. important for downstream tabular learning tasks.", "questions": "1. What is the importance of using only WikiTQ over other tabular datasets in StructQA ?\n\n2. Does the experimental results of TAMO depend on the serialization template used ? It would be good to know the variation in performance when using direct serialization as in TAPAS, SQL based serialization in TAPEX etc. \n\n3. It is not clear whether a graph-only pipeline would be sufficient for the improvements discussed in TAMO. Following Figure 6 where it is clear that text-only pipelines are insufficient the authors must include experiments comparing graph-only pipelines like GCondNet (Margeloiu et al., 2022), IGNNet (Alkhatib et al., 2024) etc. against TAMO. This is critical to establish the necessity of a multimodal model in TabularQA tasks.\n\n4. TAMO utilizes the serialized tables as additional input in their pipeline. Does this indicate that during the hypergraph transformation the learnt representation loses semantic understanding of the table ?\n\n**Minor Suggestions -**\n1. The section on Hyper-Trans (Chen et al., 2021) (lines 133-152) is not unique to TAMO and should definitely be moved to the appendix. The authors should use this space to highlight the core differences between TAMO and other existing methods in Tabular QA.\n\n2. The use of non-standardized mathematical notations in equations 1 through 3 makes the paper very hard to follow. I would request the authors to use meaningful notations and elucidate them in the text wherever possible. Eg. usage of $rFF$ in line 148 seems to be $r$ times $FF$, $X$ and $H$ in equation 3 was never defined etc.", "limitations": "Yes, Section 5 of the paper does mention several limitations of the paper which are valid and encourages future research in this space.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths -**\n\n1. TAMO introduces a novel instruction tuning architecture which combines text and hypergraph representations of tables towards Tabular QA tasks.\n\n2. The introduction of StructQA benchmark to evaluate permutation and structural invariance - a necessary evil in tabular QA tasks is a key contribution.\n\n**Weaknesses -**\n\n1. To the best of my knowledge, the introduction of Multi-Modal graph and text based tabular learning is not novel to TAMO. An example would be TabGLM (Majee et al., 2025) which combines graph and textual representations from tables for auxiliary tasks. However, the introduction of a multimodal instruction tuning model for Tabular QA would be a more appropriate positioning for TAMO.\n\n2. The claim in line 182 that TAMO is cost-effective may not be true or not clearly supported. The model in Figure 4 uses two encoders and an LLM instead of single encoders in unimodal pipelines like TAPAS (Herzig et al., 2020), TAPEX (Liu et al., 2022) ALTER (Zhang et al., 2024) and TaBERT (Yin et al., 2020) which might be more cost-effective than TAMO. \n\n3. The experiments seem to be insufficient - TAMO (a specialized multimodal model) is contrasted against general purpose LLMs like GPT-3.5, 4 etc. without contrasting against several existing tabular LLMs like TabLLM, TAPAS, TAPEX, TaBERT etc. which have already been established for Tabular QA and argue that representing a table as a natural language can largely mitigate permutation invariance (a major pitfall analyzed in this paper).", "was_revised": true, "final_justification": "From the paper contents and the discussion in the rebuttals I feel that the methodological contribution of TAMO is incremental and is better positioned as a multimodal instruction tuning strategy for Tabular QA. Nevertheless the authors have successfully demonstrated good empirical results in multimodal Tabular QA which is encouraging.", "rebuttal": "We thank the reviewer for their detailed feedback and for recognizing the value of our StructQA benchmark.\n\n**1. W1, W3 (Novelty and Insufficient Baselines):**\n\nWe thank the reviewer for the reference to TabGLM. While prior works have combined graphs and text, TAMO's novelty lies in pioneering the treatment of table structure as a distinct modality within the modern large language model instruction-tuning paradigm. Our framework is designed as a general, plug-and-play module for various LLMs, which is a significant advancement. We will revise our introduction and related work to discuss TabGLM and further sharpen our positioning.\n\nRegarding baselines like TAPAS/TAPEX, we have run these experiments on our StructQA benchmark. The results are as follows:\n\n| Model | StructQA (Accuracy) | Permute (Accuracy) | Robustness |\n|---|---|---|---|\n| **TAPAS** | 4.67 | 4.67 | 65.47 |\n| **TAPEX** | 12.33 | 9.6 | 54.73 |\n| TAMO | 59.07 | 43.47 | 43.8 |\n\nThe results clearly show that TAMO dramatically outperforms these baselines on the core structural understanding tasks (StructQA and Permutation Accuracy). It is crucial to consider the architectural differences: TAPAS is a BERT-based encoder, and TAPEX is a BART-based encoder-decoder model. In contrast, our TAMO framework is specifically designed as a multimodal extension for **decoder-only LLMs**, which represent the current dominant architecture. While TAPAS and TAPEX show higher robustness, this is largely an artifact of their very low base accuracy; it is easier to remain consistent when a model is consistently incorrect. TAMO's performance demonstrates a far superior grasp of table structure. We will add a new table and this analysis to the appendix in our revised manuscript.\n\n**2. W2 (Claim of \"Cost-Effective\"):**\n\nWe appreciate the request for clarification. The \"cost-effective\" claim is relative to the cost of **fine-tuning the LLM itself**. As shown in our efficiency analysis in **Appendix C.2 (Figure 7)**, TAMO's training time (in the frozen LLM setting) is significantly less than LoRA or full supervised fine-tuning (SFT) because we only train the small encoder and projector.\nFurthermore, to clarify the architecture in Figure 4, while there are two input streams, our framework only introduces **one new encoder (the table encoder)**. The textual input is processed directly by the native embedding layer of the **decoder-only LLM**, which does not require a separate text encoder. This is fundamentally different from models like TAPEX or TaBERT, which employ dual-encoder architectures and are thus computationally more demanding. We will clarify this context in the main text.\n\n**3. Q1 (Why only WikiTQ for StructQA?):**\n\nWe chose tables from WikiTQ because it offers a large and diverse collection of high-quality, real-world tables from Wikipedia. These tables are structurally simple for humans to understand. The fact that modern LLMs still struggle with these seemingly simple structures makes their failure more pronounced and powerfully demonstrates the core problem we aim to solve. This diversity and simplicity were essential for creating a benchmark that robustly tests a model's fundamental structural understanding. We will add this justification to Appendix A.\n\n**4. Q2 (Dependence on Serialization Template):** \n\nThis is an insightful point. While we used a consistent format for fair comparison, our core hypothesis is that any serialization format inevitably leads to a loss of structural information. TAMO is designed to compensate for this fundamental loss, regardless of the specific template. A systematic study of different templates is an excellent direction for future work, and we will add this as a discussion point in our limitations section.\n\n**5. Q3 (Sufficiency of a Graph-only Pipeline):** \n\nWe would like to clarify that a graph-only pipeline is not applicable for the tasks we address. Table question-answering requires understanding a **natural language question**. A pure graph-based model cannot process this textual query. The LLM is therefore an indispensable component for its language comprehension and reasoning abilities.\nFurthermore, we wish to clarify that the models mentioned by the reviewer, GCondNet and IGNNet, are designed for **tabular prediction tasks** (i.e., predicting a label for each row based on its features). This is a different task from **Table QA**, which involves generating a natural language answer based on a table and a question. Therefore, these models are not applicable baselines for our work.\n\n**6. Q4 (Hypergraph Losing Semantic Understanding):**\n\nThis observation is correct and highlights the synergy of our approach. The hypergraph encoder is primarily designed to capture **structural and relational properties**. The serialized text, processed directly by the LLM, remains crucial for a deep, contextual understanding of the cell contents and the question. The two modalities are thus complementary."}, {"review_id": "D92MIKWPuL", "reviewer": "Reviewer_vZ6u", "summary": "This paper reports an approach of treating tables as a separate modality - so tables are encoded using a hyper graph transformer with multi set semantics to handle permutation invariance,  and then 'fused' with the text representation of the table using standard multi modal architectures to perform better table QA.", "questions": "1.  Use of the inference only baseline to make all comparisons seemed a bit weak to me - was this a few shot prompt case?  Were some structural examples provided?\n2.  What exactly are the specialist SOTA methods - why do they not constitute a baseline and why is the fact that their performance better in those papers ignored?  Also, why aren't they evaluated on StructQA?  If the point is that these specialize SOTA methods (maybe refer to them by the citation in the table?) were specialized for the task - we do not know that do we we?  As in, those methods have not been evaluated on all the datasets mentioned here?\n\n\nThank you for addressing my questions - I believe you have addressed them satisfactorily.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1.  The use of hyper graphs as table encoders is interesting, as is the use of pretraining autoregressive tasks that teach the model about the structure of tables.\n2.  The results seem interesting although I do have some questions about the baselines.\n\nWeaknesses:\n1.  Use of the inference only baseline to make all comparisons seemed a bit weak to me - was this a few shot prompt case?  Were some structural examples provided?\n2.  What exactly are the specialist SOTA methods - why do they not constitute a baseline and why is the fact that their performance better in those papers ignored?  Also, why aren't they evaluated on StructQA?  If the point is that these specialize SOTA methods (maybe refer to them by the citation in the table?) were specialized for the task - we do not know that do we we?  As in, those methods have not been evaluated on all the datasets mentioned here?\n3.  If the point also is that this approach is more 'general' somehow - I'd question that as well because the table does need to be encoded by a specialized network.", "was_revised": true, "final_justification": "I considered the response and I find answers to my questions satisfactory.", "rebuttal": "We thank the reviewer for their feedback and for finding our hypergraph-based approach interesting.\n\n**1. W1 / Q1 (Weakness of \"Inference only\" baseline):**\n\nThe reviewer is correct: \"Inference only\" is a zero-shot baseline, chosen deliberately. First, there is no universally adopted few-shot methodology for the TableQA domain. Unlike other NLP tasks, each sample consists of a unique table and question. Constructing a few-shot prompt would require including multiple, often large, tables as examples, which is frequently impractical due to the context window limitations of most models.  \n\nSecond, the zero-shot baseline probes out-of-the-box LLMs’ intrinsic structural understanding, especially state-of-the-art table reasoners like TableLlama. Our results confirmed that even such models struggling here reveals a fundamental limitation, making this baseline a key diagnostic, not just a performance floor.  \n\nFinally, our core improvement claims rely on comparisons with stronger trained baselines (Prompt Tuning, LoRA, SFT), where TAMO shows significant advantages (Table 2). We will label \"Inference only\" as \"zero-shot\" in revisions for clarity.\n\n**2. W2 / Q2 (Clarification of \"Specialist SOTA\" and evaluation on StructQA):**\nThis is an excellent point. The term is indeed vague. In our revision, we will update the caption of Table 2 to explicitly name the specific model and provide its citation for each dataset. The specific models are:\n\n- For **HiTab**, the Specialist SOTA is **TableLlama**, which is based on Llama2-7B and underwent extensive supervised fine-tuning (SFT) on the HiTab training set along with other table QA datasets. The score reported in the original paper is 64.71, which we reproduced as 63.76.\n\n- For **WikiTQ** and **FetaQA**, the SOTA is **CABINET**, which improves LLM performance by using an external scorer to assign higher weights to question-relevant table cells, thereby filtering out noisy ones.\n\n- For **WikiSQL**, the SOTA is **SeaD**, which enhances text-to-SQL capabilities through a schema-aware denoising objective.\n\nIt is important to note that our purpose in including these specialist SOTAs is to highlight different methodological directions. Unlike our TAMO framework, which directly enhances the LLM's end-to-end understanding of the table as a modality, methods like CABINET and SeaD rely on external modules or task-specific pre-processing.\n\nThese models were not evaluated on our new StructQA benchmark because they often involve complex, dataset-specific pipelines that are not readily adaptable to a new task format without significant engineering effort. Our goal with StructQA was to test the core structural understanding of more generalist LLM-based approaches.\n\n**3. W3 (Generality of the Approach):**\n\nThis is a fair point about the term \"general.\" We use it in two senses: \n\n1) **Generalization across structural variations**, such as the row and column permutations tested in StructQA, where TAMO shows superior robustness (Figure 6) . \n\n2) **Architectural generality**, as TAMO is a \"plug-and-play\" module that can be integrated with various mainstream LLMs (e.g., Llama2, Mistral, Llama 3) without intrusive architectural modifications . This contrasts with methods that require re-training an LLM's internal position encodings. \n\nWe will refine our language in the paper to be more precise about these two aspects of generality."}]

Author rebuttal:

Meta-review: The authors study an interesting idea: rather serialize table as text input as existing works, they adapt a multimodal approach, by treating as a different modality. The reviewers found the paper novel, and the experiments are well conducted.
Decision: Accept (poster)