Exploring the Robustness of Language Models for Tabular Question Answering via Attention Analysis
Authors: Kushal Raj Bhandari
Sixue Xing
Soham Dan
Jianxi Gao
Venue: Accepted by TMLR ()

Abstract: Large Language Models (LLMs), already shown to ace various unstructured text comprehension tasks, have also remarkably been shown to tackle table (structured) comprehension tasks without specific training. Building on earlier studies of LLMs for tabular tasks, we probe how in-context learning (ICL), model scale, instruction tuning, and domain bias affect Tabular QA (TQA) robustness by testing LLMs, under diverse augmentations and perturbations, on diverse domains: Wikipedia-based $\textbf{WTQ}$, financial $\textbf{TAT-QA}$, and scientific $\textbf{SCITAB}$. Although instruction tuning and larger, newer LLMs deliver stronger, more robust TQA performance, data contamination and reliability issues, especially on $\textbf{WTQ}$, remain unresolved. Through an in-depth attention analysis, we reveal a strong correlation between perturbation-induced shifts in attention dispersion and the drops in performance, with sensitivity peaking in the model's middle layers. We highlight the need for improved interpretable methodologies to develop more reliable LLMs for table comprehension. Through an in-depth attention analysis, we reveal a strong correlation between perturbation-induced shifts in attention dispersion and performance drops, with sensitivity peaking in the model's middle layers. Based on these findings, we argue for the development of structure-aware self-attention mechanisms and domain-adaptive processing techniques to improve the transparency, generalization, and real-world reliability of LLMs on tabular data.

Published in Transactions on Machine Learning Research (08/2025)
Exploring the Robustness of Language Models for Tabular Question Answering via Attention Analysis
Kushal Raj Bhandari
ChandkOrpi.co edu
Deportment of Computer Science
Rensselaer Polytechnic Institute
Sixuc Xing
rings edu
Department of Computer Science Rensselaer Polytechnic Institute
Soham Dan
sohamdaan@msccrooff..com
Microsoft
Jianxi Gao
gaoj8@rppi.cde
Deportment of Computer Science
Rensselaer Polytochnic Institute
Reviewed OpenReview: http://///eeeeeeeeee///////////r
Abstract
Large Language Models (LLMs), already shown to ace various unstructured text compre- hension lasks, have also remarkably been shown to lackle table (structured) comprehension tasks without specific training. Building on earlier studies of LLMs for tabular tasks, we probe how in-context learning (ICL), model scale, instruction tuning, and domain bias affect Tabular QA (TQA) robustness by testing LLMs, under diverse augmentations and perturbations, diverse domains: Wikipedia-based WTQ, financial TAT-QA, and sci- entific SCITAB. Although instruction tuning and larger, newer LLMs deliver stronger, more robust TQA performance, data contamination and reliability issues, especially on WTQ, remain unresolved. Through an in-depth attention analysis, we reveal a strong corrolation between perturbation-inde performance, with sensitivity peaking in the model's middle layers. We highlight the need for improved interpretable methodologies develop more reliable LLMs for table compre- hension. Through an in-depth attention analysis, we reveal strong correlation between perturbation-indueer shifts in attention dispersion and performance drops, with sensitivity peaking in the model's middle layers. Based on these findings, argue for the development of atraeeur-awaare self-attention mechanisms and domain addativee processing techniques to improve the transparency, generalization, and real-world reliability of LLMs on tabular data.
shifts in attention dispersion and the drops in
1 Introduction
LLMs, despite being primarily trained on unstructured text, have demonstrated notable capabilities in structured data tasks, such Tabular Question Answering (TQA). TQA requires models to interpret data presented in tables, demanding strong structural reasoning. TQA specifically challenges models to discern relationships and hierarchies implicit within tabular data, making it an ideal benchmark for structural reasoning capabilities. Assessing how LLMs navigate structured comprehension challenges can provide valuable insights into their robustness and reasoning capabilities (Borisov al., 2023; Fang et al., 2024). Tabular QA offers clean, controllable structural perturbations (c.g., row/column swaps, transposes) that directly target schema alignment and spatial reasoning while holding topical content constant. In contrast,

---

Published in Transactions on Machine Learning Research (08/2025)
long-context QA evaluations introduce confounds from retrieval, chunking, and position bias that make it harder to attribute attention-patten changes to structure alone. Our focus on TQA thus isolates structure- sensitive behavior in way that complements long-context frameworks such as NoLiMa (Modarressi et al., 2025).
Recent studies emphasize the importance of robustness evaluations in understanding LLM behavior on structured tasks (Zhou et al., 2024). Specifically, perturbations in tabular structure and content significantly impact model performance, highlighting vulnerabilities that limit the practical reliability of models (Wang et al., 2022; Zhao et al., 2023). Furthermore, Liu al. (2023) argues that LLMs inherently struggle with structural manipulations, advocating for an integrated approach combining symbolic reasoning to enhance model robustness.
a)
Original Tuble
Randum Row Swip
Randen Column Swap
b)
Spearman p 0.90, 0.037
3
2
7
9
10
11
12
8
9
0.07
10
11
12
4
5
11
12
10
Random Transpose Row Swap
Raadom Transpose Swaap
AAttcation Entropy
0.06
Transposc
1
10
3
12
7
10
1
0.05
2
5
11
1
10
8
2
3
6
9
12
2
11
9
12
3
Perturbation Type
0.04
Swap
Data Type Preserving
Randemn Value Perurbation
Noll Value Pertarbario
Swap
Perturbation
Transpose
2
1
Transpose Swap
99
X
Transpose Swap
7
8
7
8
9
7
8
9
0.20
0.22
0.24
0.26
0.28
0.30
0.32
10
11
12
10
11
12
10
11
12
ΔEM
Figure (a) An example of the different possible table augmentation methods. (b) Scatter plot between change in attention entropy and EM score across perturbation types for average attention entropy change against EM scores across perturbation types for Llama-8B-Instruct on WTQ, averaged over all attention heads and data points.
WTQ. The
Although these studies identify vulnerabilities and suggest potential improvements, there remains limited understanding of how internal model mechanisms respond to perturbations in structured data. Attention mechanisms form the core of transsormer--bsedd LLMs, governing how models distribute focus across input elements during processing (Clark al., 2019). Prior work analyzing attention patterns in natural language tasks revealed that specific layers and attention heads critically influence model performance and robustness (Zhao et al., 2024; Barbero et al., 2025). These attention heads often serve specialized functions such syntactic parsing semantic alignment, highlighting the complexity of internal transformer mechanisms. However, detailed attention-level analyses for structured tasks, such as TQA, remain scarce.
To address this gap, we systematically investigate LLM robustness for TQA tasks across multiple dimensions, including in-contert learning, model scale, instruction luning, domain biases, value perturbations, and model size. Our experiments evaluate different perturbation types, highlighting how value-basod alterations influence reasoning fidelity and faithfulness (Table 1). We also compare performance domain biases and generalizability (detailed in Appendix A). Because robustness can be confounded by context length, we explicitly control for table size by restricting all inputs to < 150 cells (Appendix D). We also group-report performance and attention statistics across table-size bins to probe scaling effects in a rolled manner (Appendix F; Fig. 13). Together, these controls allow us to elarify differences primarily to structural understanding rather than to generic long-context limitations (Modarressi et al., 2025).
diverse dalasets assess
Extending beyond surface-level performance metrics, we quantify changes in attention entropy across different attention heads and layers under various perturbations, analyzing how these changes correlate
2

---

Published in Transactions on Machine Learning Research (08/2025)
with performance degradation. Attention entropy effectively quantifies the dispersion in model's attention distribution, providing a nuanced metric for understanding internal decision-making processes. With this, we aim to determine which attention heads are most sensitive to perturbations, leading to more substantial changes, and thus more distinct performance degradation.
Our study leverages diverse datasets: Wikipedia-based WTQ (Pasupat & Liang, 2015), financial report-based TAT-QA (Zhu et al., 2021), and scientific claims-based SCITAB (Lu et al., 2023). Each of these datasets provides unique context and complexity, allowing us assess the generalizability and domain-specific sensitivities of LLMs robustly. Through these diverse domains, findings give insights into the sensitivity and reliability of LLMs on TQA, highlighting the broader challenge of understanding how LLMs reason over structured
2
Various Perturbation Categories
Each perturbation is designed manipulate the table structure or content while preserving the inherent relational mcaning of the table, thereby measuring robustness to table comprehension, illustrated in Fig. 1.
2.1
Structural Perturbation (SP):
SP involves rearranging the columns and rows of the table to generate new examples. These perturbations simulate realistic scenarios where data presentation varies significantly, testing the model's ability to com- prehend tabular structure. This ensures flexibility in understanding tabular data without distorting the semantics of the table. SP involves column swap, row swap, transpose, transpose column swap, and transpose row swap. These provide diverse perspectives on table comprehension, allowing for thorough evaluation of the LLMs' ability to handle structural variations.
2.2 Value Perturbation (ValP):
ValP focuses on modifying the actual data values within tables, ensuring that the model accurately reflects the data. Value perturbations specifically challenge the semantic fidelity of the model by altering critical data points that directly affect the answer. We explore these types of ValP:
Data Type Preserving Perturbation (DVP): DVP involves altering the answer to the question and, respectively, the cell values within the table while maintaining their original data types. For instance, in Fig 9, given a question, "What was the first venue for the Asian games?", we modify the correct answer, "Bangkok, Thailand", to "Beijing". These counterfactual entitics test the faithfulness of the LLM to the tabular data. i, We utilize an automated counterfactual answer generation method that prompts GPT-3.55, ensuring the type correctness of the altered answer. Using large language model, such fake answer generation makes it possible to generate fake answers that adhere to the data type and make the table semantically correct. Examples of prompts and details of DVP dataset generation are present in the Appendix B and C.
Random Value Perturbation (RVP): RVP(an example is shown in Fig. 10) relaxes DVP where instead of a counterfactual entity, we have a fixed string, e.g., "rênD@m v@1u3". Performance on this perturbation correlates with whether the injection of random data into the table affects the accuracy. The comparison of random and data type-preserving perturbation also highlights whether models are influenced by the injection of abstract values for table comprehension.
Null Value Perturbation (NVP): NVP removes the correct answer from the table completely. Evaluating the performance on NVP highlights the influence of Wikipedia content solving the WTQ table anwwering task. LLMs that struggle more with the null value perturbation are likely to show consistent performance on TQA across different tabular datascts.
We filter the subset data points where the table does not contain the answer, e.g. many people stayed at least years office?"
3

---

Published in Transactions on Machine Learning Research (08/2025)
To understand the extent of bias in WTQ. evaluate LLMs the no-table baseline. This approach further emphasizes the reliance of LLMs on Wikipedia content. By analyzing the performance of LLMs in the absence of the table, we can better understand the extent of dependence on the particular tabular data. This method helps to reveal the intrinsic capabilities of LLMs for TQA and their generalizability across different tabular datasets.
No Table (NT):
To disentangle background memorization from table use, employ four value-centric probes. NT and NVP bound performance when factual recall is necessary but local evidence is absent; DVP and RVP stress reliance on value-level fidelity when structure is intact. While a full label-randomization protocol would further isolate memorization effects, it introduces nontrivial consisti- and schema- inieemiiintegttyy issues multi-column tables; we therefore leave systematic randomization study to futuure work and use NT/NVP/DNV DVP/RVP /RVP as practical, controlled approximations that foreground value-level dependency (related analyses in (Sakai et al., 2024)).
Collectively, these structural and value-based perturbations enable comprehensive analysis of the model's ability to reason over table structure and content. By introducing such constrained and adversarial transfor- mations, we can more precisely isolate the underlying factors that drive the performance in TQA.
3 Evaluation Metrics
Given the definition of different perturbations, we employ these three metrics to evaluate model performance under various perturbations in TQA tasks.
Exact Match Accuracy (EM): This metric calculates the proportion of instances for which the predicted answer exactly matches the ground truth answer. Formally, if N is the total mumber of instances, and correct(1) is an indicator function that if the prediction for instance is correct. and otherwise, then:
Liml correct(i)
EM
N
The Fl measure is a harmonic mean of precision and recall, commonly used to evaluate the similarity between two text sequences, such as predicted and ground truth answers. Given two strings, each is tokenized into sets of words, and the overlap between them is used to compute:
F1 Measure:
Number of overlapping tokens Number of tokens in predicted string"
Number of overlapping tokens
Procision
Rocall
Number of tokens in ground truth string'
The F1 score is then computed as:
F1 = 2- Precision Recall
Precision + Recall
A higher F1 score indicates greater similarity between the predicted and ground truth strings, while lower Fl score reflects greater divergence.
Exact Match Difference (Emd)(Zhao et al., 2023; Zhou et al., 2024): Let EMorig be the EM on the original (unperturbed) dataset, and EMperturbed the EM after applying a perturbation. The Emd quantifies the change in EM due to perturbations:
Emd EMpecturbed EMorig-
Negative values indicate performance degradation, while values close to zero imply robustness against perturbations.
Variation Percentage (VP) et al., 2022; Zhou al., 2024): This metric measures the extent to which predictions change after applying perturbations. Given that, C2W is the count of correct.
4

---

Published in Transactions on Machine Learning Research (08/2025)
Table 1: The average EM, F1, VP, and Emd small and large LLMs on Value Perturbations The
classification of large and small models is defined in Table 3.
ValP
Large Model
Small Model
Operation
EM
VP
Emd
EM
VP
Emd
WTQ
Original
0.37
0.45
0.26
0.31
NT
0.05
0.37
-0.32
0.02
0.25
DVP
0.22
0.29
0.40
-0.14
0..14
0.20
0.31
-0.12
0.10
0.30
NVP
0.06
0.09
0.37
-0.30
0.03
0.05
0.27
-0.23
TATQQA
Original
0.34
0.21
NT
0.01
0.04
0.31
-0.33
0.00
0.02
0.20
-0.20
SCITAB
Original
0.135
NT
0,000
0.00
0,120
-0.129
0.000
0.00
0,132
-0.132
before perturbation and wrong after, and W2C is the count of wrong before perturbation and correct. after. Given N as the total number of instances, the variation percentage is:
C2W + W2C
VP
N
higher VP indicates greater sensitivity of predictions to perturbations, while a lower VP signifies more stable predictions.
In addition to performance-based metrics, we also examine internal model behavior through the analysis of atlention patterns.
Attention Entropy: We analyze attention entropy to capture the dispersion of attention within different attention heads. Entropy serves meastre for structural awareness, where high entropy indicates more evenly distributed attention, while low entropy reflects concentrated focus on few lokens.
H; =
Σ Aig log(Ais)
where Aij is the attention weight to the j-th token.
4 Evaluation Performance
4.1 Effects of ICL examples TQA
Does instruction prompling assisl LLM for belter table comprehension for queslion-answering tasks? The heat maps in Figure 2 illustrate the performance of various LLMs, demonstrating that models fine-tuned for instructions or conversation exhibit improved performance. For instance, the 1aamma-70B-Insttee model significantly outperforms its original version across all table transformations, indicating that instruction-based fine-tuning enhances the model's 's ability handle complex reasoning tasks. Similarly, conversation-focuse fine-tuning also yields better scores, albeit with less pronounced improvement compared instruction- focused tuning. This suggests that fine-tuning models specific tasks like following instructions conversing effectively enhances their capability to interpret and manipulate tabular data, making such approaches valuable for improving performance in structured data tasks. For Llama2 models, the gains from instruction-tuning are present but less pronounced compared to Llama3, especially on TAT-QA where the performance gap between base and instruction-tunn tuned variants is narrower. Qwen2.5 models exhibit a similar improvement trend to Llama3, but their relative advantage most visible on SCITAB, where instruction-tuned variants close a larger fraction of the base-to-top model gap. Qwen3 shows decent performance on WTQ, and also similar performance with the Base variant. iuggesting that its base configuration already incorporates stronger
5

---

Published in Transactions on Machine Learning Research (08/2025)
WTQ
TAT-QA
SCITAB
8)
16
0:348
E111
8.21
0,754
0,812
b)
0.184
0.16
0.154
R113
C333
TM
RJ00333 R.8033
K.125
OUNDS?
40
0.716
0.558
4B
8.119
Minra
Mirra
10
Ovee25
0.559
0,53
RARLAT
Qreed
1.297
6.384
8.07
0,513
1.39
0L554
8.229
0.218
6.36
0.20
6.16
6.327
0.29
6:311
0.312
0.26
0.27
3.273
0.291
0.36
0.335
1.29
017
0.443
0,314
8.345
0.275
8365
0.233
8.67
8.8544
0.44
08831
8.
3:000
0.96
8.11
0.44
1309
9.29
0.004
18k
0.004
0
0.962
03997
0,423
0331
9.14
0.36
0.590
0.95
9.997
6.44
8.556
6:10
1.211
0
1.42
0.285
1255
0.219
0.843
09:58
0L317
8.279
0,215
6.75
GATRS
BCT9
Qvea25
QHI225
126
Ovee3
Qnael
Unee?
8.208
6.333
8.1%6
0.881
4.42
0.12
0.854
0.156
0.099
0.168
6.155
0124
6.135
6:384
0.207
6.383
0.775
0.347
1
8.333
0.34
033
0.355
0314
0,342
0
0.778
8.354
0.253
0.25
8.47
0
0,31
0.313
0,32
0.315
0,334
8331
8.6
3.129
0.24
8.300
03944
8.75
0.099
0.114
0.324
8.393
0.199
0395
6.911
00664
0:384
6:21
0.26
922
6319
8.245
78
8.325
0,312
3.92
0
120
1.9
7B
8.24
8.222
0.219
1B
83352
0.323
8353
0.344
6.44
8.399
Qwea3
Qnap?
18
Ulaee]
Lkoest
Liaad
1lxea3
6:53
1.47
0:211
E477
G442
8.477
0.42
EA71
B314
0.41
6.67
3.401
C404
3:451
Mard
6311
8.74
6/81
1379
Mbirl
4.341
6.777
6.51
6
0.80
(UR25
Owea25
129
3489
6:41
1442
6/772
E407
0,45
0.50
83353
0
0.54
0,30
0.33
00075
1281
0,03
3.57
657)
3.22
CUTS
0.5
64
MR
1261
6,33
1234
924
0,173
1R
CL:44
8.385
0.39
9.2
0.36
889973
6/016
BR113
88807
RA8714
Oripiaal
Calose
Drnsgam
Tronaamm
Seiphaagpn Col
Seap
Original
Crioss Sw
Transase
Tranngoie
Stre Cil
Seup
Original
Colense
Traspens
0,0
0.2
0.4
0.6
Figure 2: The average FM scores of the original/base(dnnool base(deeoted by B) models and cnnversatioondeenoaae by C) Cl/instructinn(dewt by I) models across various table augmentation techniques for WTQ, TAT-QA and SCITAB dataset with three fewshot examples. More individual results with additional metric are included in Appendix G.
0.6
Few-Shot
O-shot
Avenge Exact Match
I-shot.
2-thot
3-shot
0.3
0.2
0.1
0.0
1. 2.amm PhirralQwwa..50mmam 2ama2 lamad MistralQwen2.6
S0wen3 1.ama2 MistralQuen2.5 Iaama Ilamad MistralQwam2.e 50wem3 C. 70B
KB
7B
7B
8B
70B
70B
72B
30B
Sx7B
728 30B
Figure 3: Comparison of average EM scores across models under varying few-shot settings (0-shot, 1-shot, 2-shot, and 3-shot). Here, B denotes the Base variant of the model, C denotes the Chat variant, and I denotes the Instruction variant.
reasoning priors for table tasks. It also distinctly indicates that LLMs that have undergone instruction or conversation based fine-tuning outperform their base counterparts in TQA tasks for SCITAB dataset. Although SCITAB is inherently a classification task presented in a TQA format, using few-shot prompting provides valuable context, ultimately leading to more accurate and relevant responses.
4.2 Effects of the Model Type on TQA
Do newer models have belter TQA abilities? Figure 4 shows that Llama3 models generally outporform the Llama2 and Mistral models across different configurations, indicating that newer architectures like Llama3 are more effective at table reasoning tasks. The 70B versions of these models generally perform better than their 7B variants, indicating that larger model sizes enhance reasoning capabilities. Overall, the advancements in model architecture and increased model size significantly contribute to better TQA abilities. While Llama3
6

---

Published in Transactions on Machine Learning Research (08/2025)
models outperform their predecessors, the margin over Llama2 is largest WTQ and narrows TAT-QA and SCITAB. Notably, Qwen2.5 models, especially the larger variants, perform competitively with Llama3 on TAT-QA, often surpassing Mistral. For smaller variant models, Qwen3 shows comparable performance to Qwen2.5 model; however, the larger Qwen2.5 model retains an absolute performance edge in most cases. Larger models (e.g.: Llama3-70B, Mixtral-8x7B, Qwen2.5-72B) generally show higher performance than
Avcrage Exact Malch
0.4
0.0
1.aemx2
1.bmma2
LJama33
I.lemxt
Mietral 7B
Mistral 3x7B
Qwen2.5 Qwen25
Qwent
Qwen3
Llama 7
Llamal
Liama3
Mistral
Mistral
Qwan2.5 7B
Qwen2.5
Qwent
Qwent
Llamaz
Mistral
Qweni-Base Base
Llema3-Instrust
Qwen2,5-Insttue S-Instrult
Llama3
Qwen2.5
Llama2-Chatt
Mistral-lnstruct
Qwea3
Figure 4: The average FM for different models across the WTQ dataset with 3- 3-fewshot settings over all structural perturbation. The models are categorized by their types (Llama2, Llama3, Mistral, Qwen2.5 and Qwen3) and separated by their sizes (7B, 70B, etc.)
smaller models (e.g., Llama3-8B, Mistral-7B, Qwen2.5-8B), as seen in both the bar plot (Figure4) and heat maps (Figure2). For instance, Llama3-70B and Mixtral-8x7B have higher EM scores than Llama3-8B and Mistral-7B. However, we do see that average EM between the Qwen3 variant small and large variants is not significantly different. This could the result of an architectural difference between a dense model and MOE model. Although large, the active parameters in the 30B parameter model are only 3B parameter model. This suggests that model size has significant impact TQA performance.
How do performances vary with value perturbations? 2 Table 1 shows the performance of various LLMs, as defined on Table 3, when subjected to different value-based perturbations on WTQ using different evaluation metrics: Exact Match(EM)), Variation percentage (VP), and Exact Match Difference (EMD). We observe that LLMs experience a decline in EM scores across various operations compared to the original setup. The performance with RVP results in significant performance drop, more so than DVP. This suggests sharp decrease in the model's ability process and comprehend tables when the insertion of arbitrary, non-contextual values compromises the table comprehension ability of LLMs. Converscly, the DVP result indicates that while the model struggles with content that deviates from the original data structure, maintaining data type consistency helps. VP increases significantly across all perturbations, indicating considerable changes in predictions due to the perturbations. EMD consistently shows negative values, with the most substantial performance drops occurring in the NT and NVP scenarios. The observed discrepancies in performance, particularly pronounced DVP and RVP, underline a fundamental challenge: these models do not consistently apply their tabular comprehension capabilities when faced with perturbed tables.
4.3 Effects of Domain Specificity on TQA
How biased are LLMs towards Wiki-tables? Table 1 shows that when no table provided, LLMs show a notable performance decline, emphasizing their dependence on labular data to generate correct answers. Interestingly, the models still manage to answer about 25% of queries correctly, indicating a potential bias Wiki-data. In the NVP scenario, where table values relevant to queries are nullified, there significant drop in performance, yet less severe compared to the complete absence of table. This suggests that the models are biased by the contextual structured format cue even in the absence of relevant data.
How do out-of-box: LLMs perform on specialized domains: TAT-QA and SCITAB? As shown in Table 1, LLMs exhibit moderate performance across various table augmentations on the TAT-QA dataset and SCITAB dataset. For TAT-QA, large models consistently outperform smaller models, underscoring their superior TQA abilitics, but for SCITAB, the small modcls have better performance in comparison larger
More details in Appendices

---

Published in Transactions on Machine Learning Research (08/2025)
models. The overall inconsistent scores still denote significant challenges inherent in niche domains, but instruction-tuned models are better here. From Table 1, the comparison among WTQ, TAT-QA, and SCITAB datasets reveals an interesting accuracy-robustness tradeoff: while models with higher EM on WTQ suffer larger EMD and higher VP, indicating higher sensitivity to perturbations, their relatively smaller EMD and lower VP on both TAT-QA and SCITAB reflect greater robustness, despite SCITAB 's overall lower bascline EM. Also notable the sharp contrast in the NT performance, where on TAT-QA and SCITAB datasets, models have approximately 0% accuracy, contrast from the WTQ performance. This suggests that performance WTQ might be inflated due to biases favoring familiarity with Wikipedia, compared to niche domains like TAT-QA and SCITAB. This highlights the need for better benchmark design for tabular understanding. We also observe trends consistent with prior work: performance improves with few-shot prompting, instruction-luned models generally out perform their base counterparts, and larger models tend to exhibit stronger TQA (Wei et al., 2022; Fang et al., 2024).
5 Attention Analysis
Understanding how structural perturbations in labular inputs affect a model is critical for assessing robustness and diagnosing potential failure modes in TQA. As attention mechanisms regulate how language models allocate focus across table elements, examining their sensitivity to perturbations provides insights into representational stability. Here, we analyze this sensitivity by quantifying how perturbation-indee inducced changes in attention dispersion correlate with performance degradation.
We utilize attention entropy quantitative measure of how attention distributions respond to structural perturbations. Attention entropy reflects the dispersion of attention weights across input tokens, serving as an indicator of how focused or diffuse the model's attention becomes under disruption. In the context. of tabular data, where semantic understanding hinges on precisc structural alignment, operations like row and column swaps inherently distort positional cues. These distortions lead to measurable shifts in entropy, making particularly effective diagnostic for attention misalignment. By tracking how entropy changes in response to perturbations, we capture the extent to which structural instability propagates through the attention mechanism. This perspective adds prior work showing the role of entropy in capturing context encoding diversity Zhang et al. (2025), stabilizing attention dynamics during training Zhai et al. (2023), and improving generalization under limited data conditions Araabi et al. (2024), and extends these insights to structured, table-based inputs.
5.1 Effect of Perturbations on Attention Maps
We examine how varying severities of structural perturbations influence allention weights using the Llama3-88-Instruct model on the WTQ dataset. For each attention head across all layers, we mea- sure the change in attention entropy between the original and perturbed table inputs. In parallel, we compute the corresponding change in Exact Match (EM) scores, capturing the performance impact of each perturbation.
Atlention entropy captures the distribution uniformity of attention across tokens; higher entropy indicates diffuse attention, while lower entropy signifies focused attention. Thus, significant entropy changes reflect considerable shifts in the model's internal attentional focus due to perturbations.
Our analysis (Figure 1) reveals significant positive correlation (Spearman p 0.90, p 0.037) between changes in attention entropy and EM degradation across perturbation types. This shows that more severe perturbations significantly disrupt the model's attention distribution, consequently diminishing its ability for TQA.
These findings highlight the sensitivity of attention mechanisms to structural integrity within tabular inputs. Perturbations affecting relational semantics induce greater attention dispersion and misalignment, directly impairing model accuracy. Thus, attention dispersion is a crucial indicator linking input perturbations with performance.
8

---

Published in Transactions on Machine Learning Research (08/2025)
5.2
Analysis of Individual Attention Head Sensitivity to Perturbations
87
1,00
7.00
1.00
0.25
0.25
0.25
Layers
0100
9.00
1000
20
-0335
-5.0
70
-0.55
-0.75
-.75
1.00
-1000
-.00
Headitertee))
Headasurted)
Hosbtponted
WTQ
TAT-QA
SCITAB
Figure 5: Hcatmap showing Spearman correlation between changes in attention entropy and EM difference across all attention heads and layers in the -lama3-8B-Instruee model on the a) WTQ, b) TAT-QA, and c) SCITAB dataset.
While averaging attention metrics across heads provides general understanding of model behavior, prior research emphasizes that individual attention heads and specific layers, particularly the middle layers, significantly contribute to encoding task-relevant information (Barbero et al., 2025; Zhao et al., 2024). To investigate this at a finer granularity, we perform a layer-wise analysis of how perturbation-indueee changes in attention entropy correlate with EM score degradation.
Specifically, we use the 1lama3-8B-Instreeee model on the WTQ dataset to compute the Spearman correlation between entropy changes (original vs. perturbed inputs) and corresponding EM differences for each attention head across all layers. Figure 5(a) demonstrates that correlations peak predominantly in the middle layers and remain consistently elevated through these central layers. This indicates that perturbation-indeer shifts in attention distribution within middle layers are strongly predictive of performance degradation. The sharp correlation peak in WTQ suggests that the model relies heavily on mid-layer representations to align tabular structures with natural language queries, particularly when interpreting entity references and table schema. Distinct, though narrower, spikes at the input layer (0) and the output layers (30-31) further reveal that both the earliest loken-encoding stage and the final representational consolidation phase are also vulnerable to structural perturbations.
Figure 5(b) and 5(c) extend the analysis to TAT-QA and SCITAB, respectively. The results reveal distinct patterns tied to domain characteristics. For TAT-QA, correlations are elevated not only in the middle layers but also extend into the upper layers of the model. In contrast, SCITAB exhibits a more sharply localized pattern, with peak correlations tightly concentrated within the middle layers. Although we observe a high correlation in these middle layers, the contrastive negative correlations are primarily due the significantly poor FM performance on the SCITAB dataset. These findings reinforce the centrality of middle-layer attention mechanisms in tabular reasoning tasks, while also highlighting domain-specific variations in the vertical distribution of sensitivily. Additional experiments are included within the Appendix H for other models(L1ama33-88, Mistral-7B, and
5.3 Correlation within Individual Attention
We find that structural perturbations in tabular inputs affect the internal attention dynamics of LLMs, specifically the LLama3-88-Instruct model on WTQ, and that these dynamics differentially expressed across the attention heads and layers. Figure 6(a) presents an unsorted Spearman correlation heatmap between per-head changes in atlention entropy and the corresponding EM score differences, revealing substantial heterogeneity across all 32 layers and 32 heads, The heatmap captures a wide range of correlation strengths, from ncar-zero to values approaching p = 0.9, illustrating that not all attention mechanisms contribute equally to robustness. In particular, the concentration of high-correlatie values in lower layers suggests that early-stage attention heads may play foundational role in establishing reliable structural interpretations.
9

---

Published in Transactions on Machine Learning Research (08/2025)
3)
1,00
b)
Layer Head 10)
(Layer 19, Ulead 23)
0.05
0.0000
Perlurbation Type
0.50
0.685
Row
Colorss Stop
Mettrtoo Eittogy
0.080
Iraauppoe
+0.25
Lramppoc Colama
15
0.00
0,075
Tainsisoo Rew Swaz
Perturbation Type
0.670
soup
0.005
Bov Snop
-0.75
-0.50
0.01
Thoosposs Uotaraa
Transpse
Timmpponc
0.060
0.685
0.050
1,00
0.20
0.22
0.26
0.28
0.32
0.24
0.26
Hesdaummiotta)
AEM
Figure 6: (a) Heatmap showing Spearman correlation between changes in attention entropy and EM difference across all attention heads and layers in the depict actual correlation over each attention head. (b-c) Scaller plot showing the correlation between changes in attention entropy and EM scores across five structural perturbations for the WTQ dataset. (b) The plot highlights& a strong positive correlation for Layer 0, Head 10, indicating high sensitivity of this head to structural perturbations.
model on the WTQ dataset, unsorted to
(c) In contrast, Layer 19, Head 23 shows the least correlation. Note that each point in plots and e averages only across data points for each specific head-layer pair.
To unpack this variability, Figures 6(b) and 6(c) zoom in on two extreme heads identified from the heatmap. Layer O, Head 10 (Figure 6(b)) exhibits strong positive relationship: higher shifts in attention entropy due to structural perturbations reliably predict larger drops in EM scores. This implies that this head is critical for encoding spatial consistency or row-column alignment in tabular inputs, and disruptions to this aligument directly degrade model accuracy. The correlation is visually evident as a tight clustering of points around a positively sloped trend line across all five structural perturbation types (Row Swap, Column Swap, Transpose, Transpose Row Swap, and Transpose Column Swap).
In contrast, Layer 19, Head 23 (Figure 6(c)) demonstrates minimal correlation between entropy change and EM variation. This indicates form of functional redundancy or robustness in this head's role; it either performs task unrelated to structural parsing or maintains stable attention regardless of structural distortions. This wide range in sensitivity reinforces the notion that attention heads are functionally diverse and that only a subset contributes significantly to robustness under labular perturbation.
Altogether, these findings suggest the possibility of identifying and selectively reinforcing robustness-criticn attention heads through architectural tuning or fine-tuning objectives. By mapping correlation strengths across the entire model, we can isolate those mechanisms most affected by structural shifts and polentially develop strategies, such attention regularization selective re-weighting, that mitigate their susceptibility to disruption. This also opens avenues for pruning or interpretability studies focused on attention heads with minimal impact, offering insights into model compression or simplification without significant performance loss.
5.4 Cross-Model and Dataset Analysis of Perturbation Sensitivity
To evaluate the generality of our findings beyond the LLama3-8B-Instree model on WTQ, we extend our corrclation analysis between attention entropy change and EM performance degradation across a diverse sot of model-datasel combinations. These include all the small models defined in 3, for which each models are tested on the WTQ, TAT-QA, and SCITAB datasets. For each configuration, compute the Spearman correlation between the perturbation-indueed changes in attention entropy and the corresponding drop in EM scores.
As shown in Figure 7, the positive correlation trend persists across nearly all settings. In particular, Instruct and chat variants of the models consistently show the strongest correlation values across datasets, reinforcing their heightened sensitivity to attention dispersion caused by structural perturbations. Notably, even non- instruct variants like Mistral-70 exhibit moderate to strong correlations, although the magnitude tends to be
10

---

Published in Transactions on Machine Learning Research (08/2025)
1.0
0.8
0.6
Spcarmnn
0.4
0.2
0.0
-.2
SCITAB
TATQA
WTQ
SCITAB
TATQA
WTQ
SCITAB
TATQA
WTQ
SCITAB
TATQA
WTQ
SCITAB
TATQA
WTQ
SCITAB
TATQA
WTQ
SCITAB
TATQA
WTQ
SCITAB
TATQA
WTQ
SCITAB
TATQA
WTQ SCITAB TATQA
WTQ
Llama2-7B
Llama3-8B
Mistral-7B Instruct
Llama2-7B
Llama3-8B
Mistral-7B
Chat
Instruct
Qwen2.5-7B Instruct
Qwen3-8B
Qwen2.5-7B
Qwen3-8B
Base
Figure 7: Spearman correlation between perturbation-induued attention-entropg change and EM degradation
across 30 model-datasel pairs.
lower and more variable. We also find that the SCITAB dataset exhibits the greatest variation in correlation values, specifically due to the particularly poor performance of the Llama2-7B model on this dataset, where the baseline EM with the original table is already low enough substantially affect the correlation analysis.
These consistent patterns across models and datasets confirm the robustness of our main claim: the severity of perturbation that induces greater shifts in attention entropy is reliably associated with decline in model performance. This suggests that the change in attention entropy serves as broadly applicable proxy for evaluating robustness in table- based QA models.
Morcover, this analysis demonstrates that the observed dynamics are not specific to any single dataset or model archilecture. Instead, they reflect a more general representational vulnerability within current. attention- based architectures when handling perturbed structured inputs.
6 Conclusion
Our study provides comprehensive and robust analysis of LLMs under various perturbations for TQA. While larger, instruction-tuned models show improved performance, they remain highly sensitive to structural and value-based disruption. These disruptions notably manifest as perturbations such as random row swaps, column swaps, and transpositions, highlighting vulnerabilities in their structural reasoning capabilities. We also uncover domain biases, where models perform well on WTQ without tables but struggle on more specialized datasets, such TAT-QA and SCITAB. Specifically, the performance on WTQ even in the absence of tables indicates reliance memorized textual patterns from pretraining, rather than genuine tabular reasoning. In contrast, specialized domains such financial and scientific tables pose more significant. challenges due to their complexity and unique domain specificity.
We demonstrate that shifts in attention entropy, particularly in middle layers, are correlated with performance dogradation. This observation was supported by detailed attention-level analyses, which revealed that attention heads in middle layers are particularly critical in encoding structural information, and their instability directly contributes to errors in comprehension. Furthermore, layer-wise attention analysis reveals
11

---

Published in Transactions on Machine Learning Research (08/2025)
that certain attention heads exhibit greater sensitivity, suggesting that targeted improvements in these areas could enhance robust.ess.
Our findings underline the critical need for improving both the interpretability and robustness of LLMs in TQA. Building on these insights, future research should focus on developing fine-grained attention-interpe intrrrettlt methods and domain-adaptive processing strategies to enhance transparency, cross-domain generalization, and reliability in practical applications.
7 Limitation
Although we provide extensive evaluation of LLMs on WTQ, TAT-QA, and SCITAB datasets, it is possible include a broader range of datasets for a more comprehensive comparison that would highlight. the generalizability of our method for both domain-specific datasets and Wikipedia-basse baaedd datasets. While we anticipate that similar performance could be achieved with other lasks, such as table summarization, future work should include extensive analysis across various tasks and datasets to validate the assumption. It would also be insightful to expand the analysis to make a comparison with long-context QA. Moreover, our study did not involve any detailed analysis on structurally aware or fine-tuned models for tabular datasets. It is plausible that fine-tuning and structurally enhanced models could significantly impact the performance of different models. Additionally, our evaluation relied on exact match accuracy for assessing the text generation model's performance. This metric, while useful, limits the scope of evaluation for the question answering task. Future studies should employ more nuanced evaluation metrics to better assess the robust ness of the models in TQA tasks. Moreover, only conduct case study of attention analysis with small models because of the computation cost.
References
Ali Araabi, Vlad Niculae, and Christof Monz. Entropy- and distancc-regullrizzed attention improves low- resource neural machine translation. In Rebecca Knowles, Akiko Friguchi, and Shivali Goel (eds.), Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pp. 140-153, Chicago, USA, September 2024. Association for Machine Translation in the Americas. URL https: / hwwpa//olaaaaaaa.cc.oooo hww.oollgyooggg////// 20g//024... resaare. 13/.
Federico Barbero, Álvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veličkovi é, and Razvan Pascanu. Why do LLMs attend to the first token?, April 2025.
Vadim Borisov, Kathrin Sessler, Tobias Leemann, Martin Pawelczyk, and Gjergii Kasneci. Language models are realistic tabular data generators. In The Eleventh International Conference on Learning Representations, 2023. URL
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What Does BERT Look at? An Analysis of BERT's Attention. In Proceedings of the 2019 ACL Workshop BlacktozNLP: Analyzing and Interpreting Newral Networks for NLP, pp. 276-286, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1 11993/4888.9
Xi Fang, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun Qi, Scott Nickleach, Diego Socolinsky, Srinivasan Sengamedu, and Christos Faloutsos. Large Language (LLMs) on Tabular Data: Prediction, Generation, and Understanding Survey, 2024. CRL http:://axxivvoooo herrg/abs/24222 17944.
Tianyang Liu, Fei Wang, and Muhao Chen. Rethinking Tabular Data Understanding with Large Language Models, 2023. URL http://arxiv.. http:///xxxvv.oo/abb////// w3tp://///.......////// 16702..
16702.
Xinyuan Lu, Liangming Pan, Qian Liu, Preslav Nakov, and Min- Yen Kan. SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7787-7813, Singapore, December 2023, Association for Computational Linguistics. doi: 10. http:///2222...mmmmmmem
12

---

Published in Transactions on Machine Learning Research (08/2025)
Ali Modarressi, Hanich Deilamsalchy, Franck Dernoncourt, Trung Bui, Ryan A. Rossi, Seunghyun Yoon, and Hinrich Schuetze. Nolima: Long-context evaluation beyond literal matching. In Forty-second International Conference on Machine Learning, 2025. URL https////eepeeeeeeeeeeeee////oo/ nett/forummid-00tthiiiis
Panupong Pasupat and Percy Liang. Compositional Semantic Parsing on SemiiStructured Tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1470-1480. Association for Computational Linguisties, 2015. doi: 10.3115/v1/P15-1144 1142. URL http: / http//awllwb.rrrr////eee/ee anthology/P15-114..
Yusuke Sakai, Hidctaka Kamigaito, Katsuhiko Hayashi, and Taro Watanabe. Does Pre-trained Language Model Actually Infer Unseen Links in Knowledge Graph Completion? In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 8091 8106, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 40.18653/v1. /t////////2222//// naacl- maacalloan..44ee44- long.447.
Fei Wang, Zhewei Xu, Pedro Szekely, and Muhao Chen. Robust (Controlled) Table-to- Text Generation with Structure-Aware Equivariance Learning. In Proceedings of the 2022 Conference of the North American Chapler of the Associalion for Compulational Linguislies: Human Language Technologies, pp. 5037 5048. Association for Computational Linguistics, 2022. doi: 10.18653/v1 (18653/v1// / m022..aaccmmiin.331. URL https://acllanthooogg .rgg org/2022 naa.oa-aaaccmmmi..... 371.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent Abilitics of Large Language Models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856.
URL https://openrevieer www.:neetfoommmime het//forum?ddeyzkuu52c....
Jingfeng Yang, Aditya Gupta, Shyam Upadhyay, Luheng He, Rahul Goel, and Shachi Paul. TableFormer: Robust transformer modeling for table-text encoding. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Avnual Meeling of the Associalion for Computational Linguislics (Volume 1: Long Papers), pp. 528 537, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/ //22//1..222222aloo 40. URL pttps://aclanttholle wawe:.rg//20 acl 1onllong.44 40.
Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Josh Susskind. Stabilizing transformer training by preventing attention entropy collapse. In Proceedings of the 40th International Conference on Machine Learning, ICML'22 JMLR.org, 2023.
Zhisong Zhang, Yan Wang, Xinting Huang, Tianqing Fang, Hongming Zhang, Chenlong Deng, Shuaiyi Li, and Dong Yu. Attention entropy is a key factor: An analysis of parallel context encoding with full-attention-baseed pre-trained language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9840 9855, Vienna, Austria, July 2025. Association for Computational Linguisties. ISBN 979-8-89166-251-00 doi: 10.18653/v1/2000 11/0222.22c// acl- lollloon.... URL org/2925. acl- long. 485/-
https:///cclanthology htt.:/anttoooooooog//2....
Yilun Zhao, Chen Zhao, Linyong Nan, Zhenting Qi, Wenlin Zhang, Xiangru Tang, Boyu Mi, and Dragomir Radev. RobuT: Systematic Study of Table QA Robustness Against Annotteed Adversarial Perturbations. In Proceedings of the 61st Anaual Meeting of the Association for Computational Linguisties (Votume 1: Long Papers), pp. 6064 6081. Association for Computational Linguisties, 2023. doi: 10. 10.18653/ acl-2...ll-oo.g.3. URL https: /atass///cllaahooggeoe . org/2023 acl-long. 1l...lln..33-- 334.
Zheng Zhao, Yftah Ziser, and Shay B Cohen. Layer by Layer: Uncovering Where Multi-Tast Learning Happens in Instruction-Tun Tuned Large Language Models. In Proceedings of the 2024 Conference on Empérical Methods in Natural Language Processing, pp. 15195-15214, Miami, Florida, USA, 2024, Association for Computational Linguistics. doi: 10. 10.18653/v1/ /2022222.mmmmmmmaiiii
13

---

Published in Transactions on Machine Learning Research (08/2025)
Wei Zhou, Mohsen Mesgar, Heike Adel, and Annemarie Friedrich. FREB-TQA: A fine-grained robustness evaluation benchmark for table question answering. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 2479-2497, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/ / /2224. 20a2000aacl-oomm. neacl-long.137 URL https://aclanthologgy org/2024 naacl-long. 137.
Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. TAT-QA: Question Answering Benchmark on a Hybrid of Tabular and Textual Content Finance. In Proceedings of the 59th Annual Meeling of the Association for Compulational Lónguistics and the 11th International Joint Conference on Natural Language Processing (Volume Long Papers), pp. 3277 3287. Association for Computational Linguistics, 2021. doi: 10.18653/vl, 12022.aallloongg 254. URL https://aclaantholgg. org/ 2021. acl- 1onlllog..222 254.
14

---

Published in Transactions on Machine Learning Research (08/2025)
A
Dataset
We study three tabular QA datasets chosen for complementary domains and reasoning demands. WTQ (Wikipedia tables) emphasizes broad-coverage knowledge with mixed textual numeric cells and multi-step, compositional queries over semi-structured schemas. TAT-QA (financial reports) combines dense numeric content with accompanying textual descriptions, stressing arithmetic reasoning and text table fusion. SC- ITAB (scientific tables) features claims supported by measurements and categorical attributes, prioritizing structured logical inference under domain-specific vocabulary. To limit length-induced degradation and focus the analysis on table comprehension, we cap all tables at < 150 cells (Appendix D). We further stratify results by table-size bins (Appendix F; Fig. 13) to examine how robustness and attention behavior evolve with increasing structural complexity.
WTQ dataset
WTQ features tables taken from Wikipodia and paired with natural langnage questions that test a model's ability to reason across semi-structured data. This dataset especially useful for evaluating two key aspects of question answering: the breadth of domain knowledge and the depth of logical reasoning. Unlike datasets that focus on only one of these, WTQ challenges models on both fronts. Because the tables come from a wide range of topics on Wikipedia, WTQ introduces semantic variability, which tests model's ability to generalize beyond surface-level patterns.
TAT-QA dataset
TAT-QA blends structured financial tables with associated unstructured textual explanations, challenging models to align table content with surrounding text. The questions frequently require arithmetic computations (e.g., difference, sum, ratio) and integration of multiple modalities. This dataset simulates real-world information extraction tasks where decision-making depends on both tabular facts and contextual narrative, making it ideal for evaluating text-table fusion capabilities and numerical robustness. Errors in TAT-QA tend to reveal how well models can track quantitative facts across modalities.
SCITAB dataset
SCITAB introduces TQA grounded in scientific inquiry, where tables often encode structured results from controlled experiments or observational studies. The dataset features complex relational structures, such as columnar trends, controlled variables, and statistical comparisons. Unlike WTQ or TAT-QA, SCITAB emphasizes scientific reasoning, requiring models to interpret claims, draw evidence-backed conclusions, and perform multi-hop inferences within a structured format. The nuanced scientific language and evidence-based justification make SCITAB strong beuchmark for lesting logical rigor and attention to detail in LLMs.
B
Example Prompt
Example of the prompts:
C Generating Data Type Preserving Perturbation Dataset
We utilize ChatGPT-3.5 to generate type-preserving counterfactull answers using the prompt illustrated in Fig. 12. The prompt is designed to ensure that any generated answer aligns with the data type of the original answer while delibcrately introducing perturbation. For instance, given question like "What was the first venue for the Asian games?" with the correct answer "Bangkok, Thailand,' the model is prompted lo output a different but type-consistent response, such as "Beijing." Similarly, for numerical data, a value like "22" might be replacod with another plausible number, such as '50,' ensuring that the altered answer retains the original datatype constraints. This is achieved by explicitly defining the model's role as a generator of "fake' answers that preserve the data type of the original value. To ensure the validity of value perturbations,
15

---

Published in Transactions on Machine Learning Research (08/2025)
Based on the information shown in the Table, answer the fo lowing Test Question.
Ensure the final answer format is only 'Final Answer: Answe namee,, AnswerName2... form, no other form.
Test:
Table
Year
Competition
Venue
Position
Notes
1996
Olympic Games
At United States
36th (q)
5.55 m
1998
Asian Games
Bangkok, Thailand
8th
6.07 m
1999
World Championships
Seville, Spain
23rd (q)
6.40 m
2000
Olympic Games
Sydney, Australia
14th (q)
6.57 m
2001
World Championships
Edmonton, Canada
13th (q)
6.46 m
2002
Asian Championships
Co Sri Lanka
1st
6.61 m
2002
Asian Games
Busan, South Korea
3rd
6.30 m
2003
World Championships
Paris, France Manila, Philippines Athens, Greece
23rd (q)
6.13 m
2003
Asian Championships
6th
6.23 m
2004
Olympic Games
11th
6.53 m
Question: What was the first venue for the Asian Games?
Final Answer: Bangkok, Thailand
Figure Example of a prompt with answer for WTQ dataset without Few Shot Prompt
Based on the information shown the Table, answer the following Test Question.
Ensure the final answer format is only 'Final Answer: Answe naame,,
N form, no other form.
Test:
Table
Year
Competition
Venue At lanta, United States Beijing
Posit
Notes
1996
Olympic Games
36th (q)
5.55 m
1998
Asian Games
8th
6.07 m
1999
World Championships
Seville, Spain Sydney, Australia
23rd (q)
6.40 m
2000
Olympic Games
14th (q)
6.57 m
2001
World Conspionships
Edmonton, Canada Co Sri Lanka
13th (q)
6.46 m
2002
Asian Championships
1st
6.61 m
2002
Asian Games
Busan, South Korea Paris, France Manila, Philippines Athens, Greece
3rd
6.30 m
2003
World Championships
23rd (q)
6.13 m
2003
Asian Championships
6th
6.23 m
2004
Olympic Games
11th
6.53 m
Question: What was the first venue for the Asian Games?
Final Answer: Beijing
Figure 9: Example of a prompt with answer for WTQ dataset. for Data Type Preserving Perturbation. In comparison to Fig. 8, we replace the correct aBsner(Bangkok, Thailand) with fake answer (Bejjing).
select only tables that contain the exact answer to the given question. These constraints guarantee that data type preserving perturbations are applied exclusively to table-question-answe triples where the table explicitly contains the correct answer required to solve the question.
16

---

Busan, South Korea Paris, France Manila, Philippines Athens, Greece Published in Transactions on Machine Learning Research (08/2025)
Based on the information shown the Table, answer the following Test Question.
Ensure the final answer format is only 'Final Answer: NNmmel, AnswerName2... form, no other form.
Test:
Table
Year
Competition
Venue At lalata United States r@nD@m v@lu3 Seville, Spain Sydney, Australia Edmonton, Canada Colombo, Sri Lanka
Position
Notes
1996
Olympic Games
36th (q)
5.55 m
1998 Asian Games
6.07 m
1999
World Championships
23rd (q)
6.40 m
2000
Olympic Games
14th (q)
6.57 m
2001
World Championships
13th (q)
6.45 m
2002
Asian Championships
1st
6.61 m
2002
Asian Games
Busan, South Korea
3rd
6.30 m
2003
World Championships
Paris, France Manila, Philippines Athens, Greece
23rd (q)
6.13 m
2003
Asian Championships
6th
6.23 m
2004
Olympic Games
11th
6.53 m
Question: What was the first venue for the Asian Games?
Final Answer: r@nD0m v@lu3
Example prompt with answer for WTQ dataset for Random Value Perturbation. Here, we replace the correct answer (Bankkok, Thailand) with an abstract random value (r@nDOm v@lu3).
Figure 10:
Based on information shown the Table, answer the following Test Question.
Ensure the final answer format is only 'Final Answer: AnswerName1, AnswerName2... form, no other form.
Test:
Table
Year
Competition
Venue
Posi tion
Notes
1996
Olympic Games
At lanta, United States
36th (q)
5.55 m
1998
Asian Games
6.07 m
1999
World Championships
Seville, Spain Sydney, Australia Edmonton, Canada Co Colombo, Sri Lanka
23rd (q)
6.40 m
2000
Olympic Games
14th (q)
6.57 m
2001
World Championships
13th (q)
6.46 m
2002
Asian Championships
1st
6.61 m
2002
Asian Games
3rd
6.30 m
2003
World Championships
23rd (q)
6.13 m
2003
Asian Championships
6th
6.23 m
2004
Olympic Games
11th
6.53 m
Question: What was the first venue for the Asian Games?
Final Answer: Bangkok, Thailand
Figure 11: Example of a prompt with answer for WTQ dataset for Random Value Perturbation. We remove the correct answer(Bangkok, Thailand).
17

---

Published in Transactions on Machine Learning Research (08/2025)
Role [System]
You are a fake answer generator that outputs fake answer to a given question. You will only provide a one word answer but match the datatype.
Role [User]:
Provide fake answer by matching the datatype, if it is a number provide similar number, if it is location provide a fake location, and if it a word then provide a different word. Given the Question: {Question} Provide a one word incorrect answer:
Figure 12: Prompt designed for generating one-word, datatype-matching fake answers to questions from the
WTQ dataset.
18

---

Published in Transactions on Machine Learning Research (08/2025)
D Evaluation Dataset Size
We select three different datasets for comparison, WTQ, TAT- QA, and SCITAB datascts, with different numbers of evalu- ation datasets as described in Table 2. Each dataset provides distinct characteristics, WTQ is composed of Wikipedia tables, TAT-QA centers on financial reports, and SCITAB focuses scientific claims, which together enable comprehensive and robust assessment. For fair comparison, we limit the number of cell elements (< 150) within the table for both datascts. This constraint ensures that models are not disproportionately af- fected by excessively large tables, which could skew performance duc to context window limitations. Morcover, koeping table size bounded allows for more consistent mcasurement of the effects of perturbations across datasets.
Operation
Number of Pairs
WTQ dataset
Row Swap
204
Column Swap
204
Transpose
204
Transpose Row Swap Transpose Column Swap
204
204
Data Type Preserving
141
Random Value
Null Value No Table
141
204
TAT-QA dataset
Row Swap Column Swap
1668
1668
Similarly, for Value Perturbation, some queries relate to the overall structure of the table. Hence, we filter only those ta- bles that contain the answer value for the given query. This filtering ensures semantic alignment between the question and the modified table, avoiding misleading evaluations where no correct answer exists in the perturbed version. In the case of the WTQ dataset, for instance, not all tablo-question pairs are amenable to value alterations, particularly when the table structure is insufficiently informative or lacks direct answer candidates. Altogether, this preprocessing pipeline yields a cu- rated benchmark that well-suited for analyzing perturbation sensitivity while controlling for confounding factors related to table size and answerability.
Transpose
1668
Transpose Row Swap Transpose Column Swap
1668
1668
No Table
1668
SCITAB dataset
Row Swap
1225
Column Swap
1225
Transpose
1225
Transpose Row Swap Transpose Column Swap No Table
1225
1225
1225
Table Size of the Evaluation datasize
E
Models
We selected recent open-source models that have been exten- sively studied and analyzed. Table 3 lists all the models considered with their parameter size and their date of release. We include both base and instruction-tuned variants, allowing us to explore not only the effect of scale but also the impact of task specialization on labular reasoning. The models span three major families: L1ama-2, Llama-3, Mistral, Qwen2. 5 and Qwen3, which together represent some of the most widely adopted transformer architectures in the open-source ecosystem. The inclusion of instruction-tuned variants is particularly important, as these models are optimized for following natural language in- structions. This ability has been shown to influence performance on tasks that require a strctureed understanding significantly.
Model
Date Released
Small Model than 10B
parameter)
Llana-2-70-hf
6.74B
July 2023
Llana-2-15 Llane-2-7b-chat-ff
6.74B
July 2023
Mistral-78-vo.1
7.21B
Sept 2023
Mistral-78-Insttrutt-ee.
7.21B
Sept
Heta-Llama-3-80
8.03B
April 2024
Meta-Llama-3-88 800--------Inuutttt
8.03B
April 2024
Qwen2, 5-78
7.62B
Sept 20/1
5- 5-7--nrrtuuct
Sepl 20/24
Duen3-88-BBas
8.19B
April 2025
Qwen3-88
8.19B
April 2025
Large Larger than 2a0B parameter)
Llana-2-70b-hf
699
July 2023
July 2023
Mixtral-Bx7B--v
46.7B
Doc 2023
46.7B
Doc 2023
Moreover, by categorizing the models into 'small' and 'large' based on parameter count, aim to systematically assess how model scale interacts with robustness and accuracy under
Meta-Llana-3-768
70.6B
April 2024
Heta-LLama-3-70-IInn
April 2024
Quen2.5-726
72.7B
Sept 2024
Qwen2. 1om22557226 5-728-Instruct
72.7B
Sept 2024
various perturbation regimes. Recent models, such as Llama-3 and Mistral, demonstrate architectural innovations, including improved token representations and mixture-of-expert routing, which provide a richer set of inductive biases for our evaluation. Such a comprehensive suite enables a detailed comparison of robustness across architecture, scale, and fine-tuning strate-
Qwen3-308-AAB 3-----88---BB
30.5B
April 2025
Quen3-308-A38
April
Table 3: All the models with their parameter size and their date relcased. Large Models are defined as models with parameters larger than 40 billion parameters, and Small Mod- els are models with parameters smaller than 10 billion parameters.
19

---

Published in Transactions on Machine Learning Research (08/2025)
gies, thereby supporting more generalizable insights into LLM behavior on TQA tasks.
F Performance over Table Complexity
WTQ
TAT-QA
SCITAB
0.25
0.20
0.150
Avenge Score
Aveago Soore
0.4
0.15
0.125
Averige
0.100
0.10
0.2
0.075
0.05
0.0
0.00
100
200
50
100
50
100
150
Table Size (Midpoint of Bins)
Table Size (Midpoint Bins)
Table Size (Midpoint of Bins)
Original
Column Swap
Row Swap
Transpose
Transpose Row Swap
Transpose Col Swap
Figure 13: The average EM score for the different structural perturbations over the different sizes of tables
WTQ, TAT-QA, and SCITAB datasets
Figure 13 presents average model performance across varying
table sizes for three tabular question-answering datasets- WTQ. TAT-QA, and SCITAB under several table structure perturbations. Each plot compares the original condition with different table augmentation operations, such as column swapping, row swapping, and transpose-based modifications.
For WTQ, performance is low overall and highly variable across increasing table sizes, with no single condition consistently outperforming others. In contrast, TAT-QA shows generally higher and more stable baseline, though still affected by growing table size and perturbations; performance tends to decline as tables grow larger, suggesting sensitivity to complexity. SCITAB results are somewhat mixed, with fluctuations at different size intervals, but the performance remains closer among the different conditions.
Across all three datasots, these results highlight that large language models, though capable, display uneven robustness to structural manipulations of tables. Although instruction tuning and larger model scales improve performance, structural changes continue to pose challenges, underscoring the need for more robust, structure-aware approaches to ensure reliable table comprehension.
G
Additional Robustness Result
Across all three datasets WTQ, TAT-QA, and SCITAB the EM results in Table 4 elaborate on the consistent trends supporting the main findings. In addition to standard deviation for statistical significance, it also inchudes the TAPAS models(ranging from Tiny to Large) that are fine-tuned on the WTQ dataset and TAPEX morres(T.aree and Base) fine-tuned on WTQ dataset. The TAPAS models exhibit a distinctive performance profile that contrasts sharply with the general-purpose LLMs evaluated in the main text. On WTQ, TAPAS achieves notably high EM scores the original setting (0.562-0.7) 0.. 720) and retains relatively strong performance under most structural perturbations, with only modest declines for operations such column swap and row swap. However, its robustness sharply deteriorates under value-based perturbations such as DVP, RVP, and NVP, with EM drops similar to those seen for many instruccion LLMs. On TAT-QA, TAPAS's performance is substantially lower than on WTQ, with EM scores often under 0.05 for most structural perturbations an wrder-of-magnitude drop that reflects limited domain transfcrability from its trained dataset. Overall, TAPAS models show better structural robustness within their training domain (WTQ) but exhibit steep cross-domain degradation and heightened vulnerability to semantic content.
20

---

Published in Transactions on Machine Learning Research (08/2025)
Table 4: EM results across all datascts (WTQ, TAT-QA, and SCITAB) and model families. Note: The
labels B, C, and I refer to the Base, Chat, and Instruct variants of the model.
Stndel
Original Calcann Shoap
How Shop
Tiatapooe
Transpoen How Shap
TIrstapion Cal Shoap
ST
DVF
NV
WTQ
13ama3 671
0.97 0.035
0,535
0.000
0:221
4-3.0
0:316
0.181 4 4.0.015
0.167 0.014
0.016
6.149
0.03
0.085
0.0m1
Maatral 70
0.281 0.000
0.531
0.000
0.201
0.036
0.251
0.015
0.211 4 0.003
0.101 0.005
0.605 0.00
6:18
0.0000
0.066
0.018
0.031
0.004
Quam2.
0.191 0.013
5.15
0,185
0.012
0.384
0.10 2 0.006
0.30
0.011
0.064
0.014
0.026
Llame2 708
0.273 0.037
9,209
0,815
0.25
0.015
0.253
0,012
0,229 0.01
0.178 0.077
0,364 2: 0.015
0.558
2 0.002
0.011
0.204
0.018
0.000
Tlamad 700
0,364 0:111
0.36 0..01
0.311
0.018
0:366
0.318 0.071
0.25 0.04
0.045 0.00
0-0.00
0195
0.085
0.010
Quee2 5 731
6:37 0:021
0.300 0.000
0.010
0.399
5.011 0.252
0.375
0.016
0033
0.012 0.15
033
0.005
0.043
0.000
Qwas 30B B
0.268 = 0.01
0.237 = 00025
0.223
4 0.0
#: 3.014
0,200 2 0.014
0,105 $ 0.011
0,015 4 0.005
1115=+00 0.02
0.070
0,011
0.025
0.000
Liuma2
0.223 1.014
0,218
10,00
0.375
0.181
0.242 0.17T
0.123
0007
0,029
0.006
0.121
0,004 0.01
C.U13
0.000
Mtttral TB
0.522 0.022
0,524
0.248
0.284
2,015
0,271 0.035
0.159 = 0,31
0.030 0.0
0.248
0.007
0.229
0.208
0.019
0.004
Quee 5 75.1
0.531 0.013
0.01 0.313
0.290
0.00
0.273
5.075
0.343 0.01
010 0012
0.002
000
6. 1.33
0.05
0
0.005
0.657
L1ama2 700
0:306 0.30
0.316
0.31
5.005
0.011
0.18
6.00
100
0.45
Llamas 70B
0.544 0.01
0.535
10000
0.447
14815
0.328
0.407 4 0.012
0.5K7 = 0.033
0.063
0.005
0.40G 0.015
0.316
0.034
0
0.011
Matral 8cTB
0.805 1.015
0.000
0.273
0.015
0.327
0.277 2 0.005
0.225
0.035
0,040
0.57
0.061
Queen 30n
0.361
0.019
3.935
0.033
0:334
0.00
0.04
3.017
0.000 + 0.1
0.773 6 0.00
0.018 + 0.010
0. 37
4 8.005
0.50
0.005
0.036
0 0.004
TAPAS Thy (WTT))
0.569
0.509
0.230
3.075
0,044
0.78
0.555
0.418
064
TAPAS Mini (WTTQ)
6.075
0.50
0.505
TAPAS Mediums (W (14
14720
11.776
5.074
0.059
0000
0.811
0.554
0.057
TAPAS Bass (22
0.200
0.444
0088
0.805
0.211
0.250)
TAPEX Basee (WTTQ)
0.030
0.000
0.000
5.000
00000
0.000
0.000
0.000
0.000
TAPEX Large (WTTQ)
0.098
0.078
0.076
000
0.044
0.055
0.075
0.007
0.000
TATQ-A
Llama2 71
0:184 14007
0.35
0.351
0.387
5.003
0.213 4: 0.1NT
0:1113 A 0.000
0.001 1 0.003
1lama5
0,206 0.000
0.00%
0:279
0.014
0.000
0.000
Qwa2 5 TB
6.226 0.033
3.100
0.000
0.107
0.077
0.310
5.005
0.181 + 0.0.5
0.306 + 0.04
0,003 + 0.035
Quon3
0.254 11000
5.219
11037
0.214
0.250
5.107
0.204 0.012
0.216 0020
Liama3 700
0:204 11011
0:35
110322
0.362
0.357
5,015
0.322
0.064
0.30 0,30
0.001
Matnol 8x7B
0.317 0.015
0.279
0.000
0.273
0.005
0,264
0.01
0.315- + 0.011
0.975 + 0.006
0.01
Qwee2
0.305
0012
B
0.038
Llmus 73 C
0.185 0.034
0.160
0.003
0.108
0.007
0.155
5.005
0.124 + 0.005
0.185 + 0.000
0,006
0.004
Hamas
0.276 1100-
0.251
5.001
5.005
0.215 1 0.617
0.247
0.001
0.991
quv22. 5 76
0:24 11007
0,232
1351
327
5.005
0.390 1 0.013
0.21%
Dil 0.0
Quas 59
0.513 8.000
0.241
0/000
0200
8.000
0.240
3.005
0.216 + 0.001
0.226 + 0018
0.007 + 0.004
0:01
0.00
0.0%
0.211
0.112
0012
0.001
Ststral BxTB 1
0:177
1:006
0.865
0221
1.55
0.27
5.005
0.22 6013
0,271
L 0001
0.030 0.0
quee2
1:00 0.303
5:55
0.00
0.377
0.54
5.012
0.303 0.066
023
0.000
6.0
TARAS They (WTTQ)
0017
0.11
0.000
5,013
0.025
0.35
TAPAS Small (WT(2)
00032
00000
0U022
2.005
0.032
0,052
0.945
0.015
0.105
0.03
0.097
TAPAS Mediums (WT)Q)
TATAS Tase (W/T0)
0.011
5.071
0.0
0005
TAPAS Luage (WTTQ))
TAPEX Laras (WT(Q)
0.012
0.008
0.006
5,002
0.005
0.000
1lama2 271
0.001 0.001
5.005
0.001
0.200
0.00
0.000-1
0.001
0.007 4 6.003
0.001 0.001
0.0 4.0.0
Llamad A71
0.208 0.025
0.300
0.00
0.32
0.016
0.313
3.011
0.352:1 - 0.3
0.310 0.077
Dd 0.0
Xitrra
0,0 +
0.0
0.0 +
0.0
0.0
B. 5.0
7B
Quan 59 B
0:527 0.030
0,208
0.019
0,503
0.019
0.312
9.011
0.29 2 0.007
0.275 0017
0002 0.0
Tlumak 700
0.001 0.007
0.030
0.025 0.00
5.009
0.006 1 0.013
0008
01 -00
0.361
0:306
Mistral 8x7n
0.6 o.0
00 1.0.
C.0
0.0
Qua22 5 728
0,545 10010
0.5
0,203 =: 10045
0.341
5.005
0,311 = 0.005
0,580 = 00088
D00 0.0
Queuā 50B B
0,003 0.012
5.065
0.011
0.04 = 0.01
0.082
0.009
0,078 0.007
0.000 0.01
0.0
Llunad
0.362
BA
Llama3 $3
0.92 0.035
0,513
0.523 4: 8.026
0.313
9.014
0.324 0.004
0.511 0.020
BU) 0.0
Matral 71 T
0:381 0.010
0.28
0.016
0015
0.250-2
0.015
0:270 4 0.000
0.385
0011
06 0.0
Qaveg 57
0.324 0.016
0.13
0.321
0.374
5.011
0.356
0015
0.392 0.000
00 6.00
Llumaz 70B
0.004 1.004
0.072 2 0.01
0.075
0,001
2: 5.015
0,677 2 0.005
0.075 2 0.014
DJ0 0.0
Llmaa
0:475 0.007
5.602
00014
0,407
0006
0,461
2.014
0.464 - 0.017
0.04
00
5c7B
Queid 5 7911 1
06111 0015
0.57%
8611
0.57 4 0.01
0.885
1 3.07%
0.373
4 01
0.6.1 6 0.14
00 0.0
Queed 301
0.051 0.03
3,032
0.013
0.312
0:031
0000 0.5
0.005
000
0.007
0.006
86 0.0
21

---

Published in Transactions on Machine Learning Research (08/2025)
changes, reinforcing the main-text conclusion that while specialized tabular models can excel in-domain, instruction-tuned LLMs demonstrate superior adaptability and broader robustness across perturbations and domains. Across all datasets, TAPEX performs substantially worse than both TAPAS and modern LLM- models, with EM scores often near zero regardless of perturbation. TAPEX models - even the WTQ-sppecfiic large variant, peak at only 0.098 EM and collapse under perturbations, indicating a lack of generalization, likely due to training, evaluation mismatches, or overfitting On TAT-QA, LLM-basod models again lead (e.g., Llama3-70B 0.441), whereas TAPAS drops to the 0.017 0.046 range and TAPEX remains near zero.
G.1 Score with F1 Measure
Table 5: Fl measure results across all datasets (WTQ, TAT-QA, and SCITAB) and model families. Note: The labels B, C, and refer to the Base, Chat, and Instruct variants of the model.
Original Calcan Stop
Swop
Thirspooe
Thamspoe Row Snap Trecspone Cul Stou
Liam2 73
0.173 0.028
0.100 +: 0.01
0.194
0.014
0.327
5,009
0,12) 2 0.014
0.05 2 0.021
0,582 d: 0.004
0,003
0.03
0.022 = 0,212
0.028
0.005
1lama3 23
0.516 B1021
0.234
0.012
0.202 2 2.0.0
0.230 2 0..0
0.212 = 0.22
0.105 2 0.016
0.007
0.002
0.087
0.000
0.112 0.0
0,027 6.00
0.024
0:241 0.005
0.214
0.350
0.0004
0.036
0.271
0.551
0.61
0.205
0-200
5.005
0.19
0.011
0.06:
0.001
0.000
Queed 60 n
0.35 0.01
0.302
0.00
0.252
0.021
0.318
5.021
0.286 1 0.015
0.07 0.181
0.00
6.041
0.032
0:77
0.000
0.001
1.000
6.036
0.006
1imma2 700
0.018 8:006
3:31
0.033
0.292
0:011
0.363
5,002
0.340 0.01
0.00x
1-0000
6.15
0.77
0.007
0.054
0.014
Mistra RxTB
0.503 0.025
0.815
0.012
0.293
0.004
0.291
0,012
0,275 6: 0.004
0,209
0.000
0.04 2 0.001
0,302
0.005
0.052
0,205
0.051
0.000
Quaz a 728
0.422
0.019
0.026 3.626
0.564
0.018
0.415 2: 0.04
0.481 + 0.011
0.27 $ 0.008
0,067 + 0.000
0.25
0.0.05
0,300 + 6.02
0.069
0.00
Llana2 273
0.305 0.076
0.201
005
0.0.1
0.007
0.003
0.166
0.008 0.00
000
Laaadd A71
0.40 0.014
0-334
0.008
0:41
0.077
0-366
3.016
0.363 4 0.007
0.292 0.016
0.003
0.016
0.356
003
0:335
0.001
0.075
60G
Matral
0.117 0.016
5-00
0.004
0:313
0.01
0.371
5.015
0.351 0.006
0.857
0.16
0.095
0.325
0.01
0.065
Quanä 53
0:547
0.011
0:302
0.00
0.292
0.021
0.318
5.021
0,296 2: 0.015
0.191
0015
0,082
0.014
0,278
0.015
0,001 2 0.00
0.019
0.000
Hlume 70B
0.500 00011
0.341
0.01
0.281
JU812
0.272
2.005
0.281 4 0.005
0.215
2 0003
0,024
0.0
0,214 + 0002
0.154
0.200
0.036 0.007
Mittral Be7 1
0.406
0.0%
0.008
0.318
0.00
0.386
0.02
0.351
0005
0.12
0.057
0.01
0.13
0.201
0.079
Quent 5 231 9
0.585
00013
0.58
0.01
0.511
0.055
0.351
5,012
0.563
1 0.01
0.032
0.04
0:00:
0.005
0.475
0.001
0.565
0.01
6. 0.3
0.011
Quea8
0,965
0.025
0.543
0.025
0.501
0.029
0,304
0.005
0.246
0.011
0.041
0.027
0.24
0.00
0,287
6.0L
0.264
C.OL
TAFAS
0.390
TAPAS Small (WTTQ)
0.717
0.700
0.422
5.067
0.058
0.51
0.965
0.521
0.000
TAPAS Mint (WTQ:
0.425
0.120
0.067
0.07
0.835
0.551
0.082
(WT)))
TAPAS Base (WTQ))
0.55
0.706
0.076
TAPAS Large (WTTQ)
0.255
0.7
0.51
110
090
0.709
0.511
0.050
(WTQQ)
0.020
00019
0.014
0022
0.022
0,001
0.204
TAPEX
TATQQ-
0.103
8J
0.548 110000
5,294
1.000
0.305
1.002
01.205
5.0015
0.264
0.014
0,015
0.002
Statral 70
0.320 140002
0,232
10.01
0.305
0.200
5.005
0.230 4 0.004
0.272 à 0.004
01021 0.0
Quom22 5 70
0.202 110000
0.254
11:000
0,251
10005
0.27
0.00%
0.231
0.014
0.264
0000
0.02
0.092
Llama8 708
0,975 0.015
0.910
0.030
0.34
0.036
0.35 6 0.01
0.285 = 0.03
0.505
0011
0.013 + 0.004
1lanah 708
0.45 0.011
0.033
0.024
0.01
0.491
5.015
0.389 + 0.000
0.055
0011
0.064 6.0
Quen2. 2 723
0.588 0.032
0.308
10000
0.361
0.3633
5.00
0.327 0.22
0.364
0038
0.02
0.007
Quens 30B B
0.394 14014
0.351
00021
0.355
0.37L
2. 5.U0T
0.21 4 0.005
0.325
2 0.006
0.00:
1 0.002
Liuma2
11006
5.134
11221
SUDON
0.309
5.004
0.356 0.115
0.175
0.008
0.012
Matrol TB
0.512 0.030
0.261
0.003
0.21
0.010
0.250 + 0.0
0.21 + 0002
0.207 > 0.06
0.624 6.0
Qua22 a 7B I
0.50 0.506-1
0.000
0.285
0006
0.281
0.007
0.294
3.005
0.20 + 0.055
+ 0.000
0.015
0.00
Liama2 700
0.307
11011
0.266
11000
0.
0.27
5.007
0.221 0.0444
0.245
1.256
0.004 1 0.055
Llsmas 700
0.523 14005
0.009
10000
0.00
10825
0.314
2.001
0,422 4 0.006
0.472 L 1100
0.254 0.0
0.972 0.016
0.010
0.00%
0,300
0,505
0.280
Qwen2 P2B
0.01
0.006
Qwem S0B
6 40400-4
0.0.0
0.373 2 0.00
0.573
0.035
0.37L
9.021
0.0L0 = 0.01
0,500 6 0.01
0,684 6.0
TAPAS Tiy (WTTQ)
0.031
0.030
0.026
2.047
0.025
0.440
TARASS Minl (WT1Q)
0.025
TARAS Mediomm (W 71)
14077
0.155
Basc
3,855
TAPAS (WTTQ)
0.11
TAPEX Box (W77Q)
0.08
0.000
5.015
COLL
0.020
TAPEX Large (W (1)
00020
5.005
0.03
1003
SCITAB
1lama2 73
0.001
B1000
21005
0,2002
+: 0.0
0.002
0.001
0,002 # 0.005
0,001 5 0.002
BX0 0.0
0.541
0:501
0,051
0.523
0.010
0.345
0.011
0.514
0:32 0018
0.0
Mistral 70
0.00
0.00
1.0.0
0.2
QuweR.5 70
0.033 0.004
0.01
0.001
0.207
0.001
0.000
3.003
0.000 1 0.004
0.00 11 (100)
bdl on
Queed 501
0.200 11036
0.301
0.311
3.012
0.381 0.005
0.80 11030
Llama 70B
0:30 0.020
0.960
0,007
0.502
0.000
0.800
5.004
0,365 + 0.007
0.565 - 0905
000 0.0
MEsrrl kxTB
0.0 + 0.0
0,0
0.0
+ 0.0
0.0
= 0.0
= C.0
01 > 000
D0 0.0
Quend 30th D
0.307 0.015
5.171
0.000
0.283
0.001
0.001
01006 0.00T
0,005
00
Tlana2 271 C
0.581 0.007
0.247
0.008
0.353
0.0
0.376
3.005
0.385 4 0.008
0.387 + 0.004
DO 0.0
0.011 1.00
0.016
0.331
0.00%
0.183
5.015
0.333
0.013
0:22 8008
86 0.0
Quan2 a 7B
0.572 0.036
3,515
0,007
0.383
0.012
0,576 2 0.01
0.300 2 0.004
0,511 0.004
BJ 0.0
Quaa 53
0.583 (.016
03545
0011
0,532
0.383 2: 2.034
0,364 2 0.000
035 5 0.007
D00 0.0
Lamad 70n
0.178 0.03
0-965
0.666
0.05
105
5.011
0.01%
0.151 0.14
00
Mhttrl BcTIL I
01881 0.001
0.0
8.000
0/661
3.007
0.087
1 0.00%
0.083 0.000
00 0.0
Quaa2 a
0,042
0.014
0.605
0011
0.585
0012
0.522
0.025
0.579
2 0.007
0.15
00
0.002
22

---

Published in Transactions on Machine Learning Research (08/2025)
The extended evaluation Table 5 expands the comparison to include the TAPAS family (Tiny to Large) model trained on the WTQ dataset. These results reaffirm and sharpen several robustness patterns identified in the main paper. First, newer architectures such as Llama3 and Qwen3 consistently achieve higher average Fl scores across all datascts and perturbations, with instruction-tuned variants showing the most consistent robustness gains. The Llama2 and Qwen2.5 families, while competilive on WTQ, exhibit larger relative performance drops when evaluated on out-of-domain datasets (TAT-QA, SCITAB), suggosting weaker cross-domain generalization.
Second, TAPAS demonstrates markedly different profile. As structure-aware table-specific model, TAPAS achieves the highest in-domain (WTQ) performance under nearly all perturbations, particularly in column swap scenarios. However, its accuracy declines sharply on TAT-QA, confirming a robustness generalization tradeoff: specialized models excel within their training domain but are more sensitive to schema and domain shifts. Across all model classes, column swap emerges as the least destructive structural perturbation, whereas transpose-basee based operations, especially transpose column swap, yield the steepest declines, consistent with the trends reported in the main study.
Since TAPAS is originally trained to solve synthetic SQL queries and is fine-tuned for table-based question answering and fact verification. Given that the SCITAB dataset can be formulated as a question-answering lask, without fine-tuning on it, it is not possible to generate a scientific claims verification problem; hence, omitted for comparison.
TAPEX shows markedly better results under the relaxed F1 metric compared to its near-zero EM acores(Table 4) indicating that while it rarely produces exact. matches, it often generates partially correct answers. On WTQ, the WTQ finetuned TAPEX Large variant, which achieved only 0.098 EM, improves substantially under F1, reflecting better token-level overlap despite structural or formatting mismatches. Similar patterns hold in TAT-QA, where EM performance was effectively zero, but F1 reveals non-trivial overlap with the gold answers, suggesting that TAPEX captures fragments of the correct information even when failing to match exactly. This gain under F1 highlights that TAPEX's weaknesses in these evaluations stem less from a complete lack of understanding and more from its inability to align outputs precisely with the expected format, reinforcing the importance of using both exact and relaxed metrics when assessing its capabilities.
23

---

Published in Transactions on Machine Learning Research (08/2025)
H
Attention Matrix Analysis
H.1
Spearman Correlation within All Attention Heads
WTQ
TAT-QA
SCITAB
0.55
415
0.50
0.59
130
Llama2-7B
1030
0.00
3.0
-0.50
4.50
-1.00
1.00
1.00
0.5
0.75
475
025
0.35
425
Llama2-7B Chat
HasepeteD
Figure 14: Heatmap showing Spearman correlation between changes in attention entropy and EM difference across all attention heads and layers in the Llama2-7B model(a, and c) and Llama2-7B-chhe model e and f).
WTQ
TAT-QA
SCITAB
075
0.5
475
422
0.25
125
Llama3-8B
-1.08
2.00
Radivers
4.5
0.59
4.0
0025
0.25
425
Llama3-8B Instruct
0.22
75
Figure 15: Heatmap showing Spearman correlation between changes in atlention entropy and EM difference across all attention heads and layers in the Llama3-88 model(a, b and e) and and f).
model(d,
Across both the model families Llama3 and Mistral, the middle layers consistently exhibit the strongest positive Spearman correlations between perturbation-indut changes attention entropy and EM degradation. This recurring 'hot spot' in the mid-layer shows general architectural property; middle transformer blocks
24

---

Published in Transactions on Machine Learning Research (08/2025)
WTQ
TAT-QA
SCITAB
0.55
015
0.90
450
425
Mistral-7B
1100
0..00
0.00
-0.30
-0.52
4.50
-1.00
-1.00
1.00
Heabioanai
5.75
0.75
0.75
4.25
0.25
425
Mistral-7B Instruct
8.50
-100
Figure 16: Heatmap showing Spearman correlation between changes attention entropy and EM difference model(d,
across all attention heads and layers in the Mistral-7B model(a, b and e) and and f).
WTQ
TAT-QA
SCITAB
1.00
3.06
1.00
000
920
0.90
0.55
Qwen2.5-7B
0.25
-0.5
-0.05
-0.75
(laate/ortee)
(taasoored)
9
0.75
8.5
6.75
0.23
9.23
4.25
Qwen2.5-7B Instruct
0.50
BodiGaatd
Figure 17: Heatmap showing Spearman correlation between changes in attention entropy and EM difference across all attention heads and layers in the Qwen2. 5-7B model(a, b and e) and Qwen2. 5-788iinstrrct model(d, and f).
appear to serve as critical junctions where structural perturbations most directly translate into downstream performance loss. Such robustness vulnerabilities likely stem from these layers' dual role in integrating lower- level token interactions and preparing higher-level semantic abstractions, making them both information-rich and sensitive to distributional shifts.
However, differences emerge when contrasting base versus chat- or instruction-tuned variants. the base models (Llama3-8B, Mistral-7B; Figure 15, and 16) subfigures a-c), the correlation patterns outside the
25

---

Published in Transactions on Machine Learning Research (08/2025)
WTQ
TAT-QA
SCITAB
1:09
1.22
0.3
0.75
4.25
9.25
6.25
Sportno
Qwen3-8R Base
14
6.75
0.73
0075
Meacs(antod)
Roadsported
Hcsib
5.75
0.75
6.75
1.25
4223
6225
Qwen3-8B
11
4.50
1.00
1.00
1.20
leaca(ertod)
(crtt(
Figure 18: Heatmap showing Spearman correlation between changes in attention entropy and EM difference across all attention heads and layers the Qwen3-8B base model(a, b and c) and Qwen3-8B model(d, e and f).
middle layers fluctuate markedly, with mixture of weak even negative correlation, especially apparent. TAT-QA and SCITAB tasks. Specifically, for Figure (o(c)),Mistra-777 does not correlate because for all the questions EM was O. These oscillations align with the base models' generally lower EM scores on these datasets, suggesting that noisy or unreliable predictions can obscure the coherent relationships between entropy and performance.
By contrast, the chat and instruct-adapted models (Llama3-8--Instrue Mistral-7B-Instrucc; Figure (15 and 16) subfigures d-f) display stronger positive correlation, with some extending beyond the middle layers. On WTQ in particular, both the initial encoding layers (layers 0-2) and the highest layers (uppermost 29 31 blocks) contribute positively to the entropy -M relation, suggesting that conversational and instruction luning bolsters the model's resilience to perturbations at both the token-embedkding stage and the final consolidation stage. This extended positive band likely reflects enhanced parameter alignment across the architecture, cnabling more stable information propagation even under input disruptions.
In general, these results yield two primary implications. First, conversational and instructtion-Lumngg systematically extends the alignment between attention entropy instabilities and performance degradation across broader portion of the transformer hierarchy, thereby allowing perturbation robustness not only in the mid layers but also at its boundaries. Second, task domain complexity governs which layers most critically underpin model stability: general-domain benchmarks (e.g.. WTQ) draw on broad spectrum of transformer depths, whereas specialized datasets remain predominantly reliant on central layers.
Across the Qwen model families, the correlation heatmaps in Figures 17 and 18 reveal patterns broadly consistent with those reported for Llama3 and Mistral in the main text. For both Qwen 2.5 and Qwen 3, the middle transformer layers consistently exhibit the strongest positive Spearman correlations between perturbation-ing indueed changes in attention entropy and EM degradation, reinforcing the interpretation of these layers structural "choke points" where disruptions to schema alignment most directly translate into performance loss. Base variants (Figures 17a-c, 18a-c) show more fluctuation outside the mid-layer band, including weak or negalive correlations on TAT-QA and SCITAB, mirroring the instability seen in Llama3-88 8B and Mistral-78. This instability often coincides with lower overall EM, as in Qwen 2. 2-5-7B on SCITAB. contrast, the instruction-tuned counterparts (Figures 17d-f, 18d-f) display stronger and more spatially extended positive correlations, with WTQ in particular showing elevated values not only in the middle layers but also in the initial embedding layers (0-2) and uppermost layers, suggesting improved perturbation
26

---

Published in Transactions on Machine Learning Research (08/2025)
robustness at both early encoding and final integration stages. These trends indicate that, for Qwen for Llama3 and Mistral, instruction tuning enhances the architectural alignment of cntention-pefformance relationships, while the dataset domain continues to determine how localized or distributed the layerwise sensitivity appears.
27


Reviews:
[{"review_id": "6RlnBCOsgF", "reviewer": "Reviewer_gQzM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nUnderstanding biases and preferences in LLMs is an important topic, empirical studies of that data are needed\nThe paper is thorough in its core approach, authors document their work and experiments\nThe paper is generally well written and the overall argument is intuitive\n\nWeaknesses\n\nThe paper overstates its claims, such as \"newer architectures like Llama3 are more\neffective at table reasoning tasks\" -- authors only evaluate Llama3 and Mistral models (incl instruction-tuned variants), and no more recent \"architecture\" (or training corpus) has been tested\nI find the tasks, especially \"value perturbation,\" ill-posed: one should not use an LLM to explicitly parse a table (but of course it can be done) -- because the LLM will mingle its internalized knowledge with the table information. \nThe paper is purely empirical and results are likely to be different for every new LLM, every new data type, etc -- so very little generalizable results are found.\nThe study on attention entropy is interesting but mostly descriptive.\nI find it difficult to understand the main findings of the paper. For example, I cannot follow \"Table 1 also distinctly indicates that LLMs that have undergone instruction or conversation-based fine-tuning outperform their base counterparts in TQA tasks for SCITAB dataset.\" -- Table 1 is huge and I don't see instruction-tuned models mentioned there.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an analysis of how LLMs' performance on \"structured comprehension tasks\" depends on how information is presented -- essentially, what makes it easier or harder for LLMs to extract information from tables. Authors create \"perturbed\" variants of three test sets (WTQ, TAT-QA, SCITAB) by (1) rearranging the columns and rows of the table (structural perturbation, SP) and (2) applying value perturbation (ValP) in 4 different ways: changing the actual value while preserving the data type, using a random value, removing the correct answer, and not providing a table at all. Authors also provide an analysis of attention entropy in internal layers and show how it correlates with task performance.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please provide results with more recent models.\n\nPlease provide a better presentation of results so that readers can follow the main argument. E.g. in Table 1, maybe provide averages of the different EM/VP/EMD \"scores\" across the different perturbations, so that a \"big picture\" can emerge. Similarly, in Figures 1 and 6, provide an aggregate analysis that shows that there is some correlation between attention entropy and task performance beyond individual heads.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DXsuo3yi7c", "reviewer": "Reviewer_sAwL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* Analysis on the tabular question answering datasets is systematically conducted by measuring the relation of perturbation of the tabular information, performance differences and changes in attention entropy.\n\nWeaknesses\n* This work split the experiments by three datasets, but it is not clear the impact of difficulties of the tasks. First, no discussion exists regarding the characteristics of the three datasets. Second, this work needs further split by the size of table and information in each cell, e.g., numeric values or textual information, for a more comprehensive studies to measure the impact of difficulties.\n* The impact of knowledge memorized in LLMs is not clear. Given the drop of performance when adding noise to cell, LLMs might heavily rely on the memorized knowledge and this might have an impact to the attention pattern.\n* Several clarity issues exist in this work. For example, the scatter plot in Figure 1 (b) is not clear what is plotted, whether it is showing only averages of layers and heads.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "It is an analysis paper measuring the relation of attention weights and tabular question answering capacity of large language models (LLMs). Basic idea is to measure the dispersion of attention pattern when looking at tabular information in order to retrieve information by differentiating the difficulties of tables and perturbation.  Experiments show that the attention pattern shifts, measured by entropy, are correlated with degradation in performance on three datasets coming from Wikipedia, financial domain and scientific domain.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "* This work needs discussion on why focusing on tabular question answering task is suitable to measure the changes in attention patterns and performance. For example, it is possible to run comparison with other tasks, e.g., QAs with long context [1].\n* Further discussion regarding the use of entropy metric will be necessary to justify the use of the metrics with related studies [1, 2, 3].\n* This work needs detail explanation for the three tasks settings, e.g, what kind of challenges shared or specific to each dataset.\n* Experiments should be controlled by several factors, e.g., size of a table, information in each cell, that might affect the difficulties of the tabular understanding.\n* Further discussion and/or experiment is necessary to separate the knowledge memorization impact to see whether an LLM can answer without looking at any tabular information. For example, it is possible to systematically randomize the labels in order to completely remove the impact of knowledge [4], not partly adding noises.\n* It is not what is plotted in Figure 1 (b), Figure 6 (b) and (c), whether they are showing average of all the information or not. \n\nReferences\n\n[1] Zhang et al. 2024. Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Models.\n\n[2] Zhai et al., 2023. Stabilizing Transformer Training by Preventing Attention Entropy Collapse.\n\n[3] Araabi et al., 2024. Entropy– and Distance-Regularized Attention Improves Low-Resource Neural Machine Translation.\n\n[4] Sakai et al., 2024. Does Pre-trained Language Model Actually Infer Unseen Links in Knowledge Graph Completion?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OwADJ0Qrom", "reviewer": "Reviewer_JP73", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The authors conducted comprehensive and rigorous robustness testing of multiple large language models, applying rich tabular perturbations—including structural transformations and value modifications—across diverse domains such as Wikipedia, financial reports, and scientific tables, and systematically revealed how in-context learning, model scale, and instruction tuning affect Tabular QA performance and stability.\n- Secondly, through fine-grained analysis of model attention distributions, the paper finds that perturbation-induced attention dispersion is significantly correlated with performance degradation, and further points out that attention in the middle layers is most sensitive to table-structure changes, providing new insights into the models’ internal working mechanisms.\n\nWeaknesses\n- The study is limited to prompt-based Tabular QA with general-purpose LLMs and lacks comparison with models specially trained for table tasks or equipped with structure-aware optimizations, so it is unclear whether such specialized methods could mitigate the problems revealed or alter the conclusions.\n- The paper uses attention weight changes as the main explanatory tool; prior work shows that attention distributions are not necessarily strictly aligned with true feature importance, therefore, evaluating model robustness solely by attention dispersion may be insufficient to uncover the deeper causes of errors.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors point out an interesting phenomenon: although large language models have already excelled in unstructured tasks, their structural reasoning mechanisms and robustness in Tabular QA (TQA) remain a “black box.” In real-world scenarios, row/column misalignment or numeric noise can distort answers, and existing work lacks systematic evaluations and explanations of internal attention dynamics. The authors design a benchmark covering row/column swaps, transposition, and various value perturbations, systematically evaluating the ICL robustness of LLMs of different scales and instruction tuning on WTQ, TAT-QA, and SCITAB, and quantifying performance degradation and sensitivity with Emd and VP. Through layer- and head-wise attention entropy analysis, they find that perturbation-induced entropy increases are highly correlated with EM drops and are most sensitive in the middle layers, and on this basis they call for developing structure-aware self-attention and domain-adaptive techniques.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.\tMethod level (critical)\n\n    Add comparative experiments with structure-aware or models specifically trained for tabular tasks (e.g., TAPEX, TableFormer) (the authors may choose models that are easier to implement; the methods I suggest are only for reference and do not necessarily have to be compared with the ones I mention) to verify the generality of the paper’s findings and eliminate model-selection bias.\n\n2.\tExperimental metrics (improvement)\n\n    In the evaluation, supplement semantic consistency metrics (such as F1 or Set EM) and report significance testing, to avoid underestimating the model’s true capability by relying solely on Exact Match.\n\n3.\tTable-size analysis (improvement)\n\n    Extend experiments to larger or more complex tables and group-report performance and attention-entropy changes to examine whether the middle-layer sensitivity conclusion remains stable under different context-window lengths (my consideration here is that current large language models already show very strong understanding of short, small tables; for example, users can first convert a table into markdown format for better text comprehension by the model).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)