Interpretable LLM-based Table Question Answering
Authors: Giang Nguyen
Ivan Brugere
Shubham Sharma
Sanjay Kariyappa
Anh Totti Nguyen
Freddy Lecue
Venue: Accepted by TMLR ()

Abstract: Interpretability in Table Question Answering (Table QA) is critical, especially in high-stakes domains like finance and healthcare. While recent Table QA approaches based on Large Language Models (LLMs) achieve high accuracy, they often produce ambiguous explanations of how answers are derived. We propose Plan-of-SQLs (POS), a new Table QA method that makes the model's decision-making process interpretable. POS decomposes a question into a sequence of atomic steps, each directly translated into an executable SQL command on the table, thereby ensuring that every intermediate result is transparent. Through extensive experiments, we show that: First, POS generates the highest-quality explanations among compared methods, which markedly improves the users' ability to simulate and verify the model’s decisions. Second, when evaluated on standard Table QA benchmarks (TabFact, WikiTQ, and FeTaQA), POS achieves QA accuracy that is competitive to existing methods, while also offering greater efficiency—requiring significantly fewer LLM calls and table database queries (up to 25x fewer)—and more robust performance on large-sized tables. Finally, we observe high agreement (up to 90.59% in forward simulation) between LLMs and human users when making decisions based on the same explanations, suggesting that LLMs could serve as an effective proxy for humans in evaluating Table QA explanations.

Published in Transactions on Machine Learning Research (06/2025)
Interpretable LLM-based Table Question Answering
Giang Nguyen
nguyennginngkhan@@mmal.ccmmm
Auburn University
Ivan Brugere
svan, boan.brugere www.bbrgere@iimcchss.co.
J.P. Morgan
Shubham Sharma
shubham.2..harmaa88802
J.P. Morgan
Sanjay Kariyappa"
skarigarppasnuiidiaccmm
NVIDIA
Anh Totti Nguyen Auburn University
voh.ngg@@mmaai com
Freddy Lecue
sccue.fredyy@nnee com
J.P. Morgan
Reviewed on OpenReview: https: // openreview. net/ forum? id= 2eTsZBoU2
Abstract
Interpretability in Table Question Answering (Table QA) is critical, especially in high-stakes domains like finance and healtheare. While recent Table QA approaches based on Large Language Models (LLMss) achieve high accuracy, they often produce ambiguous explanations of how answers are derived. We propose Plan-of-SQLs (POS), new Table QA method that makes the model's decision-making process interpretable. POS decomposcs a question into sequence of atomic steps, each directly translated into an executable SQL command on the table, thereby ensuring that every intermediate result is transparent. Through extensive experiments, we show that: First, POS generates the highest-quality explanations among compared methods, which markedly improves the users' ability to simulate and verify the model's decisions. Second, when evaluated standard Table QA benchmarks (TabFact, WikiTQ, and FeTaQA), POS achieves QA accuracy that is competitive existing meth- ods, while also offering greater efficiency requiring significantly fewer LLM calls and table database queries (up to 25x fewer)-and and more robust performance on large-sized tables. Finally, we observe high agrcement (up to 90.59% forward simulation) between LLMs and human users when making decisions based on the same explanations, suggesting that L.MSs could serve as an effective proxy for humans evaluating Table QA explanations. Code and data available at: http://tthub..cooannggeeeee
1 Introduction
An estimated 38% office tasks involve working with tables, often using Excel (Richardson) 2022 highlight- ing the need for advanced tools for tabular data analysis. LLM-powered Table QA models (those in Fig, D address this gap by enabling users quickly extract insights answer questions for tables, making them invaluable in various industries. For example, financial analysts leverage these models to predict trends from tabular market data (Lo & Ross 2024h. Similarly, medical professionals use them to analyze tabular medical records of patients, facilitating accurate and timely treatment decisions (Bardhan et al. 2022
"Work doue J.P. Morgan Al Research.
Equal advising.

---

Published in Transactions on Machine Learning Research (06/2025)
However, the value of these systems comes with high risks. Errors in financial decision making have led to calastrophic outcomes, such as the billion-dollar loss Citigroup faced in 2022 (Jane) 2024). In healthcare, the stakes are even fatal, considering a model that misjudges a man's 's health by overlooking his family history, resulting in his death from cardiac arrest weeks later (Stanford HAT 2024). These examples underscore the pressing need for interpretability in Table QA to ensure sale and accountable use of AT (Fang et al. 2024).
Despite its importance, interpretability remains an underexplored dimension in Table QA literature. Recent approaches have significantly increased accuracy and often present themselves as interpretable solutions (Ye ct al. 2023 Cheng et al. 2023 Wang et al. 2024), but this interpretability is unsubstantiated by empirical evidence. In practice, the explanations provided by these models can be unclear. For instance, as shown in Fig. 1 user cannot discern why certain rows were selected by function f select contttrro(o___ or how an operation like simple_query( produced the final answer. In other words, current Table QA methods do not adequately explain their reasoning to users.
GT: True
Answer
gomoc
osponent
resuft WeL_
opponents
game
pln sppoowentc
ospanents
dinchnnl
wih
20
0
2
20
(c) Hybrid
(a) End-to-end
2
dircinnt
Whh
20
zavler
Wh
LLM
soiler
win
peorgla
garrgaa
vardertal
With
0
14
G
5
vandentel
win
14
metigan
win
Iaput Talk P. widdcao bettall loam
anatarma
13
1_aelect 3..55)
opponents
viggiia
win
15
8
Leeeect columnigarraw wT, (ik.opporeer)))
Answer
CW mix
Ioas
14
evonsiille
Min
19
Jennesseo
13
2
20
U
[_eeect_row[223344
Plan
LLM
guorga
vin
28
2. [select__olluue wwc.loolllggeee.... httpeeec__ppppppppeee plp.ooptpooetssss
LLM
36
simple_query))
michigan
simple, query()
dittoro
1056
13
erginta
Order
by 'opponennts' ascending order Select rows where 'opponente'' is 0
Plan
evbrevil
win
26
LLM
Jatraccsee
3. Use a CASE statement lo return TRUE if the no. of rows is equal 4, otherwise relurn FALSE
True Fiebe? The on ldoata lasp the
oppcoing tean scordiess in garees.
pame
oppenent
resui
pbi ospomerta
gane
opponemi
resull lec obi aspenents
osponeri
resut, we sta
opporiens
candtradi
cancrrsss
Win
zinchnadi
LLM
3
xivee
wir
20
Hancarts
98
14
xandert
wh
14
0
goorgia
CHarS=ll:
excroville
5
vendersil
SELECT CARS
6
ITAht gin
wer
15
WHEII COUBT (+)
SELECT
THIN 'PAOR"
5
atcarme
Ioea
13
"PASE!
WARN COUNT(") THRU
EED AS cemult
FROM cable PoL
eransellc wir
36
0
10
101005500
loss
13
2I.5E 'FALSE'
3
WHERE opponecte 0:
egg-
SELaCT
1
SELECT
2
Answer
table aqgi
FROH table ved
Answer (d) Plan-of-SQLs
(b) Text-to-SQL
opposents
opporents
Figure 1: (a) End-to-End relies entirely on an LLM to answer the question directly, leaving no room for users to understand the prediction. (b) Text-to-SQL: generates an SQL command to solve the question, requiring domain expertise to understand and becoming unintelligible when the question becomes complex. (c) Mfai--of-Table or CoTable: performs planning with abstract functions and exccutes soquentially to arrive al the final answer. However, function arguments are not justified, and the final answer depends on the LLM's opaque reasoning. (d) Plan-of-SQLss or POS (Ours): plans in natural language, making each step simple and understandable. Each step is then converted into an SQL command, sequentially transforming the input table end-to-end to produce the final answer. We provide a public interface to compare explanations.
To address this gap, we propose Plan-of-SQLs (or POS), an LLM-based Table QA approach that places interpretability at its core. POS decomposes each question into a sequence of atomic steps, where each step is a simple sub-problem that can be translated into a corresponding SQL command and executed on the lable. By design, each transformation is self-contained and limited in scope, forcing the model's decision decision-making to be broken down into transparent and verifiable steps. This has two key benefits. First, by requiring simple step-by-stey by-step table transformations via SQL, we avoid the model arbitrarily pulling in irrelevant data for answering. For example, existing methods that select a large subtable in one shot (Ye et al. 2023
Wang
2

---

Published in Transactions on Machine Learning Research (06/2025)
et al. 2024 often include spurious entries due to the black-box LLM reasoning (see the irrelevant selection of row3 in Fig. [e). By contrast, POS uses programmatic conditions (e.g.. "WHERE oppoments = 0²) to ensure only relevant entries are chosen (Step 2 of Fig. Dt). Second, POS largely avoids the opaque answer-generation step commonly found in LLM-based Table QA literature (Ye et al. 2023 Wang et al. 2024 Nahid & Rafici 2024b; Zhao el. al. 2024] (illustrated in Fig. [.c). Rather than sending the entire table to an LLM, POS produces its final answer via a deterministic SQL query exccution (Step of Fig. 11,, making it entirely clear how the answer derived from the data.
We thoroughly evaluate the interpretability and performance of POS. We compare against existing Table QA methods that provide explanations, including Texi-lo-SQI. (Rajkumar al. 2022), DATER @Ye et al. 2023
Chain-of-Table (CoTable) (Wang al., 2024h, and self-xxplaining (Madsen et al.) 2024p. Our interpretability
evaluation spans three benchmarkk
explanation quality preference ranking (Ramaswamy et al.) 2023
Yang et al. 2024), (2) forward simulation of the model's behavior (Doshi- Velez & Kim |017] Hase & Bansal
2020 Chen et al. 2022: Mills et al.) 2023 and (3) model prediction verification (Nguyen et al.) 2021 Taesiri
it al. 2022 using both human participants and LLM-based judges. all settings, POS usually yields the
best results, showing that its explanations help users (human or AT) simulate and verily the model output far more effectively than the existing interpretable competitors. Furthermore, when tested on standard datasets (TabFact (Chen et al., 2020). WikiTQ (Pasupat & Liang 2015), and FeTaQA (Nan ct al. 2022) POS achieves accuracy par (within 1 2 poinls)) with existing methods, while requiring drastically lewer LLM calls and database queries. Notably, POS scales robustly to large tables, where current methods Cheng et al. (2023); Ye et al. (2023): Wang et al. (2024) often struggle. Finally, we observe a high agreement between human evaluators and LLM-based evaluators (up to 90%) in forward simulation, suggesting that LLM-basod proxies can reliably stand in for human judgment. In summary, our contributions are as follows.
We introduce POS, new Table QA method that is designed for interpretability. We carry out thor- ough study of explanation effectiveness, and show that our explanations substantially improve users' understanding of the model's decision-making over existing interpretable methods (see Sec. 4.1.3).
Compared to existing Table QA approaches, POS is more robust on large tables and significantly more efficient in its use of LLM calls and database queries (25x fewer LLM calls than DATER, and 25x fewer database queries than Binder), while maintaining competitive QA accuracy (see Sec.
Our experiments reveal high agreement between human users and LLM judges in evaluating Table QA explanations (up to 90.59% agreement in forward simulation). This suggests that LLMs could effectively proxy for human evaluators in evaluating Table QA explanations (see Sec. 4.1.5).
2 Related Work
2.1 Atomic Table Transformations for Table QA
LLM-based Table QA models have improved performance by decomposing complex input queries into smaller
problems (Ye et al. 2023 Nahid & Rafiei 2024b Zhao et al. 2024h or by employing step-by-step reason-
ing (Wang et al.J 2024 Wu & Feng 2024 Abhyankar et al.] 2024). However, these approaches often rely on highly compler table transformations for instance, selecting a question-relevant subtable from a large
input table via the opaque reasoning of LLMs (Ye ct al.) 2023 Nahid & Rafici 2024b
Wu & Feng. 2024
Abhyankar el al. 2024) which can lead to errors in retrieving the correct dala (see details in Appendix Often, the resulting sub-tables contain entries that are either irrelevant or logically incorrect with respect to the input question, due to the long-context challenges inherent in LLM reasoning.
In contrast, POS constrains a transformation of the table to an atomic SQL operation, that is, a single clause with at most one condition and one variable (e.g., "Select rows where opponents = 0"). This design has two main advantages. First, it improves accuracy and comprehensibility: because each step is minimal and focused, there is less room for LLMs to make mistakes, and the operation is casy for users to digest. Second, it enables fine- grained attribution: by executing each step with an atomic SQL query, we can pinpoint exactly which cells were used at that step, yielding detailed trace of how the final answer is derived.
3

---

Published in Transactions on Machine Learning Research (06/2025)
2.2 Program- based Table Transformations
Using programming languages such as SQL Nahiid Rafici, 2024b Ye et al. 2023 Stoisser et al. 2025 or Python (Cheng et al. 2023 Chen et al. 2020) to manipulate tables is prelerable for two reasons. First, these languages perform rule- based operations with explicit references to table cells, offering much greater traceability than the implicit, black-box transformations of LLMs (see contrastive examples between Fig. & d). Second, programmatic transformations can handle large or complex table operations more reliably and efficiently, since they do not suffer from the context length and inconsistency issues that LLM-based
methods encounter when processing entire tables (Chen 2023 Wang et al. 2024 Nahid & Rafiei 2024b).
POS builds on this line of work by leveraging program-based transformations specifically, SQL to solve Ta- ble QA. Notably, POS uses SQL exclusively for executing reasoning steps. To our knowledge, only two methods in the Table QA literature, LPA (Chen et al., (2200) (using Python-Pandas) and Text-to-SQL, (Rajkumar let al.) 2022), handle queries end-to-end through program- aagdam--based operations. However, since Text-to-SQL generates single complex SQL command for an input question (Fig. [D), it requires highly powerful Text-to-SQL model and often generates Prro--prone commands (Shi et al. 2020). Similarly, LPA's one-pass program synthosis can be brittle, as generating a correct multi-step program in one go is inherently challeng- ing (Chen et al. 2020). In contrast, POS breaks the problem into multiple simpler SQL queries corresponding to atomic sub-steps initially expressed in natural language. This stepwise use of SQL removes the need for an advanced program generator and, as our results will show in Sec. leads to higher overall accuracy and interpretability than Text-to-SQL or LPA.
2.3 Evaluating Interpretability
Interpretability a critical aspect of AT models, and there a rich body of work on evaluating explanations
with human users (Adebayo al. 2020 Nguyen et al.
2021;
Kim et al. 2022 Taesiri et al. 2022; Colin
ett al. 2022 Steyvers & Kumar 2023 Chen et al. 2023a Nguyen al.) 2024a| b Zhang et al. 2025). In the
2024 had users subjectively rate the faithfulness and comprehensivenees of explanations on a Likert scale. However, such ratings serve only as proxies for explanation quality and do not directly measure the explanations" utility to users. POS guarantees 100% faithfulness (the explanation steps exactly produce the answer) and comprehensiveness (no reasoning step is hidden) because its explanations explicitly represent the model's reasoning through actual executed SQL operations. Therefore, in our experiments, we prioritize direct evaluation of explanations' offoctiveness such as how well users can simulate or verify the model's behaviors over faithfulness or completeness metries.
context of LLM-based Table QA, TAPERA (Zhao et al.
3 POS: Interpretable Table QA with Atomic Table Transformations
Problem Formulation. In Table Question Answering (Table QA), each sample is represented as triplet (T,Q, A), where T is table, Q is question about the table, and A is the answer. The task is to predict the answer A given the question Q and the table T. POS decomposes Q into steps converts each step into an SQL command - applics these commands sequentially to T to arrive at A.
Atomicity in Table QA Reasoning. We define an alomic step as a simple, minimal natural-language table operation that can be reliably translated into an SQL query. Specifically, each operation is restricted to: (i) at most one condition (e.g., =), and (ii) at most one column or variable in that condition (e.g., opponents). Enforcing this atomicity keeps each step's translation and execution reliable and lowers the chance of errors. It also makes the model's reasoning more interpretable: because each step addresses only a small part of the problem, a human can easily understand the purpose and effect of that step.
3.1 Planning in Natural Language
Rather than planning in a space of abstraci functions for Table QA like Chen et al. (2020) and Wang et al. 2024), we perform the reasoning-step generation in natural language. Leveraging an LLM to plan in natural language takes advantage of the model's strong priors from its language training (Hunng et al.) (022). It also makes the plan inherently understandable: each step is described in natural language, which far more interpretable than, say, a sequence of function names (e.g., simple query()) whose purpose is not
4

---

Published in Transactions on Machine Learning Research (06/2025)
[GT] True
St. Order the table by 'opponents" in ascending order.
32. rows opponents" 8.
wildeats
Si. Use "AASE" statement to return TRUE if the number rows equal 4,
game
opponent
result we_ w opponents
Step
de miss
loss
NL Atomic Planner
SQL
Engine
step-to-SQL
cincinnati
3
xavier win
20
georgia win
26
Table 77
Table 73 answer
vanderbilt
game
opponent
wo__ts opponents
michigan win
2
cincinnat
win
20
7
alabama loss
13
georgia win
26
virginia
vanderbiltt
game
opponent result pls opponents
evansville
36
9
evansville
2
cincinnati win
20
10
tennessee
loss
13
6
michigan win
6
4
georgia win
(0)
kept
w virginia
win
5
vanderbilt
opposing scoreless games.
..
9
evansville
36
Illustration of Plan-of-Sooe (POS). I The Natural Language (NL) Atomic Planner takes (T, () as input and generates a step-by-step plan in plain language. and converts the first step Si into an SQL query, which is executed on T to produce an intermediate table Ti. Step-to-SQL takes (2) produce and execute the next SQL query, yielding T2. The final Step-to-SQL uses (I2.S3, to generate an SQL query that returns the final answer. We provide an interactive demo interface of POS.
Figure 2:
2 Step-to-SQL then takes (T.S1)
immediately clear (Wang al. 2024). In Fig. 1 the NL Atomic Planner takes the input (T)Q) and produces plan of atomic steps in natural language that outlines how to derive the answer A. POS prompts LLM with examples and explicit instructions to incorporate the atomic decomposition. Specifically, we use prompt of this form:
Generate plan of natural-language, atomic steps
## In-context Planning Examples
## Input Table
## Question Q
## Planning Instructions
Here the plan to solve the question Q
3.2 Converting Step to SQL
Once we have the plan in natural language, the next stage is to translate each step into an executable SQL query. We leverage the LLMs' capability as a Text-to-SQL converter (Hong et al. 2024 to perform this conversion for each step. There are two main reasons to use a dedicated Step-to-SQL module (Fig. 2 First, exccuting the reasoning steps via SQL commands greatly improves the correctness of each step compared to letting LLMs manipulate the table implicitly in its hidden state (see our analysis in Appendix C In particular, offloading the computation to an SQL query ensures each operation is carried out exactly and climinates errors from the LLMs' reasoning. Second, using SQL makes the sequence of transformations fully trackable: for every step, we know preciscly which table entries were accessed or modified, since the SQL query specifies those conditions. This allows us to identify what information each step used.
In Fig. 2 the Step-to-SQL module takes the current intermediate table (initially the original table T') and one atomic step as input, and outputs the corresponding SQL query. We steer an LLM to generate an SQL with a specialized prompt template, as illustrated below:
5

---

Published in Transactions on Machine Learning Research (06/2025)
Convert an atomic step into SQL
## In-context Step-to-SQI. Examples
## Current Intermediate Table
## Natural-Language Atomic Step
## Step-to-SQL. Instructions
Here is the SQL to execute the step
After converting all steps, POS executes them sequentially using an SQL engine (we use SQLite3 (Gaffney et al. 2022) in our implementation). The intermediate result of each SQL command becomes the input for the next step, faithfully carrying out the transformations specified by the plan (see the chain Fig. 2 In contrast to end-to-end latent reasoning to jump to the final answer (as in Fig. [p, c), POS maintains interpretability throughout the process: the answer is obtained through another SQL operation whose effects can be understood.
"chain-of-thought" Table QA methods that rely on an LLM's
3.3 Generating Explanations for Table QA
Despile the need for interpretability, prior work has not established a formal method to generate human- understandable explanations for LLM-based Table QA models. Most studies focus on improving accuracy and leave the model's decision-making process opaque (Wu & Feng, 2024 Kong et al. 2024 Cheng et al. 2025 To bridge this gap, we propose an approach to generate explanations for Table QA using attribu- tion maps the main medium for explaining AT decisions to humans in various domains, including image classification (Colin al., 2022). text analysis (Hase & Bansal 2020), , and time series (Theissler et al. 2022).
For each step, we create an attribution map on the intermodiate table that highlights the information used in that step. Specifically, as we apply a transformation, we mark the rows and columns that were selected affected in yellow and we highlight the cells that satisfy the step's condition in green By doing this for every step (see Fig. 2), we obtain intermediate tables annotated with highlights indicating which data contributed to the final answer. Finally, we present the explanation as a chain of attribution maps: each step is shown alongside its highlighted intermediate table. This allows a user to visually follow the process of rcasoning. In practice, the explanation shows which cells were used when, making it clear why, for example, the final answer is True in the rumning example of Fig. 2 We also apply this visualization ion approach to other Table QA methods, such as CoTabbe (Wang et al. 2024 and DATER Ye et al. 2023). with additional examples provided in Appendix A.
4 Experiments
We conduct experiments using three popular and standard Table QA benchmarks: TabFact (Chen et al. 2020), WikiTQ (Pasupat & Liang] 2015), and FeTaQQ Nan et al. (2022). TabFact a fact verification dataset which cach statement associated with a table is labeled TRUE or FALSE. We use the cleaned TabIacc dataset from Wang et al. (2024) and evaluate Table QA methods with binary classification accuracy the 2,024-sample test-small set. WikiTQ is a question-annseringg dataset where the goal is to answer human-written written questions using an input table. Using the dataset and evaluation scripts from Ye et al. (2023). we assess model denotation accuracy (whether the predicted answer is equal to the ground-truth answer) on the 4,344-sample standard test. set. FeTaQA is a free-form Table QA dataset where the lask is to generate free-form natural language responses based on information retrieved or inferred from a table. We evaluate models on the 2, 2,003-sample standard test set using BLEU and ROUGE.
4.1 Evaluating Explanations in Table QA
Basclines. We select Text-to-SQL. (Rajkumar et al. 2022), DATER (Ye al. 2023) and Chain-of-Table
(CoTable) (Wang al., 2024) benchmark the interpretability of POS. They are chosen for their high performance, interpretability, and reproducibility (see details of baseline methods in Appendix A). Later, we
6

---

Published in Transactions on Machine Learning Research (06/2025)
compare POS with self-cxplanations (aadsen et al. 2024), where LLMs are prompted to provide explanations for their own answers.
Evaluation measures. To evaluate the quality of explanations, we involve both human judgments and LLM-based judgments (which we denote as LLM-as-XAI-juuye). We follow the two complementary evalua- tion perspectives (qualitative and quantitative) proposed by Velez & Kim 2017
In the qualitative evaluation, the judge is shown the table, the question; the model's answer, and the model's explanation. The task is to rank the explanations from different methods by overall perceived preference (to identify which explanations users are most likely to use in practice). This preference rating is based on clearly defined rubric comprising three criteria: clarity, coherence, and usefulness in understanding the modol's reasoning. We refer to this as a Preference Ranking task, following prior works on comparative explanation evaluation (Ramaswamy et al. 2023 Yang et al. 2024 Zhang el al., 2025 Wazzan et al, 2025 For each table question, we collect a ranking of the four methods' explanations (1 best, 4 worst). To ensure fairness, we only consider test samples where all methods being compared got the question either correct or incorrect (so that judges comparing explanations for answers of the same correctness). We aggregate rankings over 707 TabFact questions meeting this criterion, computing the average rank for each method (lower is better 4).
In the puantitative evaluation, the judge is asked to perform tasks that objectively measure how well the explanation informs them about the model's behaviors. We use two standard tasks for evaluating explana-
tions: Forward Simulation (Doshi-Velez & Kim 2017 Hase & Bansal
2020 Chen et al.] 2022 Mills et al.
2023 and Model Prediction Verification (Nguyen et al. 2021 Taesiri et al. 2022 Chen et al.) 2023a]..
In Forward Simulation, the judge is given the table, question, and explanation without the model's answer, and must predict what answer the model's output would be. This evaluates how clearly the explanation communicates the model's decision boundary or reasoning process to users.
In Prediction Verification, the judge is given the table, question, model's answer, and explanation, then must decide whether the model's answer is correct or not based on the explanation. This measures how well the explanation justifies the model's prediction (e.g., can the judge catch model errors using explanations?).
We compute performance for these lasks as the percentage of samples in which the judge makes a correct decision, reported as Simulation Accuracy and Verification Accuracy, respectively.
4.1.1 Evaluating explanations with human users
Motivation. Human evaluation is considered the gold standard for evaluating AT explanations, as hu- mans are the ultimate users who work with Al models (Doshi- Velez & Kim 2017j. We aim to study how explanations help humans in understanding then predicting model behaviors via Forward Simulation.
Participants and Data. We recruit 32 volunteers, all of whom are undergraduate, master, or Ph.D. students in Computer Science. In each session, users select one of four explanation methods and complete 10 samples, with an option to participate in multiple sessions. We collect 800 responses ( 200 per method).
4.1.2 Evaluating explanations with LLMs
Motivation.
The use of LLMs trained to align with human preference OOyaag et al.
2022 as judges has
been gaining attention due to their strong correlation with human judgments (Dubois et al. 2024
Zheng
ct al.
2023 Liu et al.) 2023 Mills et al., 2023 Fernándoz-
et al. 2024 Poché et al.. 2025
This
makes LLM judge a promising, scalable solution for evaluating explanations, particularly in tasks like Table QA, where the information still text-based yet. structured. Thus, we are motivated to leverage LLM judges for all three tasks: Preference Ranking, Forward Simulation, and Model Prediction Verification.
LLM judge. Inspired by recent works showing the effectiveness of OpenAl's instruction-tuned GPT models as reliable judges Zheng et al. 2023 Liu et al. 2023 Dubois et al. 2024), we utilize 3 OpenAl's LLMs: not-tuubo-222404--111
gpt-4o, and gpt-4o-mint mini to evaluate Table QA explanations. We ensure the prompts encourage the model to follow the given instructions strictly (see Appendix G for the exact prompt templates and calibration procedures). We also explore the use of open-source LLMs as judges in Sec. 4.1.4
7

---

Published in Transactions on Machine Learning Research (06/2025)
Table 1: (a) Preference Rankings for explanation methods given by LLM judges on TabFact. Lower values
indicate better rankings 4. (b) Simulation Accuracy t (%) of LLM and human judges on TabFact.
(a) Preference Ranking (1 4)
(b) Simulation Accuracy (%)
XAI method
Text-to-SQL
DATER
CoTTable
POS
Texl-to-SQL
DATER
CoTTable
POS
GPT-4
3.33
3.36
1.98
1.33
75.15
80.04
79.99
84.89
GPT-lo-mini
3.95
2.75
1.75
1.55
65.67
73.57
76.53
81.61
GPT-1o
3.60
3.35
2.01
1.01
73.73
78.21
79.55
85.25
Human
83.68
86.50
84.29
93.00
Table 2: Verification Accuracy (%) of LLM judges TabFact and WikiTQ.
TabFact
WikiTQ
XAI method
Text-to-SQL
DATER
CoTable
POS
DATER
POS
GPT-4
49.93
57.56
60.38
72.08
73.50
72.38
GPT-1o-mini
55.37
55.43
61.36
76.74
64.58
71.93
GPT-4o
55.97
70.95
67.34
72.85
73.31
74.45
4.1.3 Findings from the evaluation of explanations
POS is ranked highest in quality. In Tab. [a), POS explanations consistently reccive the bost ranks from all LLM judges. Specifically, our explanations achieve average ranks of 1.33, 1.55, and 1.01 from GPT-4, GPT-4o-mini, and GPT-4o; respectively, substantially outperforming CoTable, DATER, and Text-to-SQL.
This shows that POS explanations are regarded by LLM judges as the clearest, most coherent, and most helpful for understanding the Table QA model's reasoning process (see the rubrics in Appendix G In practice, this preference could translate into increased trust- key factor for Al adoption in high-stakes domains (Doshi-Vee Velez & Kim 2017
POS is most effective for predicting model behaviors. Tab. [b) shows that POS effectively helps human and LLM judges predict the model behaviors. Specifically, human judges achieve 93.00% with POS explanations, outperforming other methods such DATER (86.50%) and CoTable (84.29%). Similarly, across all LLM judges, POS consistently yields the highest accuracy, with improvements ranging from 5% 6% over the second best methods.
POS is most effective for model prediction verification. We perform this experiment on TabFact and WikiTQ, comparing the verification accuracy with different explanation methods in Tab. 2 We find that POS is the best method in five out of six settings. In addition, the improvements between DATER and POS are more pronounced in TabFact compared to WikiTQ, suggesting that the effectiveness of explanations is influenced by the nature of the task. As more complex dataset, WikiTQ makes it inherently more difficult for the judges to verify the predictions. Please note that we do not compare POS with Text-to-SQI. and CoTable on WikiTQ, as these methods have not been publicly released for this benchmark.
Later in Sec. 4.1.6 we present an ablation study that further studies the factors driving POS's improved interpretability.
Qualitative rankings strongly correlate with quantitative measures. Using Tab. & Tab. we perform correlation analysis to study whether qualitative preference rankings inform quantitative measures. Since lower rankings in Tab. Tab. indicate higher-quality explanations, we invert the rankings to align higher comulation/verifiaant accuracy with higher preference.
Interestingly, we find slatistically significant posilive Pearson correlations between preference rankings vs. simulation accuracy (r= 0.7865, = 0.0024) and vs. verification accuracy (r =r=0) 0.7035, p = 0.0107). These high corrclations suggest that Table QA, our proposed rubrics based on perceived quality (sce Appendix G can effectively identify high-utility explanations. This allows for efficient pro-selection of explanations present users, significantly reducing the need for expensive and time-consuming user studies.
8

---

Published in Transactions on Machine Learning Research (06/2025)
Comparison with post-hoc self-explanations. Finally, we investigate whether simple, post-hoc self- explaining approach could rival POS in interpretability. Tab. T0 compares POS with widely used "self- explanation" method, in which the LLM first generates an answer and then retrospectively explains its answer (sec an example in Fig. S
We find that self-explanation remains substantially less effective than POS in all three interpretability bench- marks, largely due to the lack of faithfulness in post-hoc explanations (Madsen et al., 2024 Chen et al. 2023b Agarwal et al. 2024). By contrast, POS grounds its explanations in offline transformation steps (via SQL exccution), yielding absolute faithfulness and comprehensiveness, which are especially valuable for model simulation verification. We provide more details in Appendix A.3
4.1.4 Evaluating explanations with open-source LLMs
So far, our evaluation relies on GPT-family models. This can restrict the generalizability of our findings in Soc. 4.1.3. To address this, we use two additional model families Qwen and Llama to confirm whether the observed trends in explanation quality persist across multiple LLM families.
We use TabFact as the benchmark dataset, focusing on two lasks: Forward Simulation and Model Prediction Verification. In both tasks, we test four methods Text-to-SQL, DATER, CoTable, and POS -under open- source Qwen2.5-72B-Inss. and
hosted by SambaNova
Table 3: Evaluating Table QA explanations (%) on TabFact with open-source models.
XAI Judge
Text-to-SQL DATER CoTable
POS
Forward Simulation Accuracy
Qwen2.5-7213
84.31%
91.38%
84.18%
90.59%
Llama-3, 1.mma-3.1405BB 1.44BBBB
79.08%
86.63%
79.10%
90.59%
Human
83.68%
86.50%
84.29%
93.00%
Model Prediction Verification Accuracy
Qweu2.5-7213
71.57%
75.49%
79.25%
80.01%
LJama-3.1-4055
75.30%
77.57%
77.08%
78.49%
We report the evaluation results on open-source LLMs in Tab. 3 In Verification, POS consistently delivers top performance, while Text-to-SQL remains the worst across all model familios. In Forward Simulation, POS out perforrsm other baselines on Llama, although DATER slightly exceeds POS with Qwen2.5 (91.38% vs. 90.59%). In general, find that the interprefability provided by POS translates well to open-source LLMs.
4.1.5 LLM-Human agreement in XAI evaluation
In Tab. we report the instance-level agreement between LLM-based and human forward simulations on the same samples for four explanation methods. This metric differs from accuracy; it measures how often LLM judges and humans arrive al the same decision when given identical information (i.e., the input and an explanation). We find that: First, POS consistently yields the highest LLM-human agreement across all tosted LLM judges, reaching up to 90.59%. This suggests that when explanations groundedee.g. e.g., via atomic SQL steps-bott LLMs and humans converge similar decisions in evaluating explanations. Second, baseline methods exhibit lower agreement (71-83%), likely because their explanations are less informative or less faithful, leading LLM judges and humans to follow different reasoning paths. Practically, these high agreements (71-90%) suggest that LLMs may serve as effective proxies for carly- phase evaluation of Table QA explanations, reducing the high cost of user studies while still approximating human judgments.
faithful and
4.1.6 Ablation study on POS interpretability
To better understand which components of POS (Sec. most strongly contribute to its interpretability, we perform an ablation study on both TabFact and WikiTQ, We focus on the model prediction verification in which an XAT judge (here, GPT-4o-mini) is shown POS's explanations and asked to verify the final answer.
https://sambanova.al

---

Published in Transactions on Machine Learning Research (06/2025)
Table 4: HLM--hmmn agreement (%) on forward simulation for TabFact. Fach cell indicates how often an
LLM's docisions align with the human decisions on the same subset of samples (listed Human Samples). Higher values suggest. stronger agreement between LLMs and humans.
Method GPT-4o-mini Qwen2.5-72B-Inn
Human Samples
Text-to-SQI.
71.24%
79.08%
75.16%
153
DATER
81.98%
83.14%
76. 74%
172
CoTTable 70.10%
82.49%
80.7%%
177
POS
88.30%
90.59%
89.41% %
171
We remove (i.e., ablate out) one of the following three core components in POS and observe the resultant drop (or change) in verification accuracy: (i) Atomic operations: Instead of enforcing single-condition trans- formations, we allow complex multi-condition steps. (ii) NL planning: We replace the natural-language planning with dircet prompt that asks the LLM to generate a sequence of SQL commands to solve the question. (iii) SQL execution: We replace the SQI-based Iransformations with direct black-box LLM-based transformations.
Tab. 5 shows the verification accuracy in each ablation setting. We find that removing SQL execution leads to the biggest interpretability drop (from 76.74 to 64.19% on TabFact), while removing the natural-Languago planning also leads to substantial decrease. Atomic operations, though beneficial, exhibit the smallest impact when removed.
Table 5: Verification Accuracy (%) on TabFact and WikiTQ. Removing each component from POS reveals its impact on interpretability; SQL execution is most critical, followed by the plan, and then atomic operations. Variant
TabFact
WikiTQ
POS (Full)
76.74%
71.93%
Atomic operations
76.65% (-0.09)
71.74% (-0.19)
NL planning
67.96% (-8.78)
66.20% (-5.73)
SQL execution
64.19% (-12.55)
66.54% (-5.39)
By translating each atomic step into an explicit SQL query, POS naturally produces step-by-stey attribution maps that pinpoint exactly which table cells influence the final answer. This clarity is especially valuable for verification: if highlighted entry in the explanation irrelevant or obviously mismatched to the ques- tion, a user can readily infer that the model's final answer is suspect. The natural-language plan itself also matters significantly, presumably because it makes the ofain-of-reasonin much easier follow. Al- though ablating atomicity canses only minor drop-likely because modern LLMs already employ strong argue that explicit atomic steps remain meaningful for D for qualitative examples.
step-by-step reasoning priors (Wei et al.) 2022)
human understanding. Please refer to Appendix
4.2 Evaluating Table QA Performance
Baselines.
We compare POS with several baseline methods, categorizing them into three groups based on how table transformation and answer generation are performed: LLM-only, program-only, and hybrid approaches. Unless otherwise noted, we use a temp = and top-p for LLM generation.
4.2.1 Table QA for TabFact and WikiTQ
As shown in Tab. POS achieves strong performance on both TabFact and WikiTQ on two different LLM backbones. When paired with GPT-3.5, POS achieves 78.31% accuracy TabFact and 54.80% on WikiTQ, yielding substantial gains over LLM-only methods such as End-to-End QA, Few-Shot QA, and Chain-of- Thought. It also surpasses other program-only basclines by wide margins: for instance, POS outperforms Text-to-SLL by +13.6 points and LPA by +9.41 points on TabFact. Compared to hybrid approaches, which combine LLM and program- based operations, POS offers a compelling alternative. Although POS scores lower accuracy on both benchmarks than these hybrid mothods, its exclusive use of program-basod transformations ensures complete transparency of each reasoning step, making much casier for users verify and understand the underlying decision-making process.
10

---

Published in Transactions on Machine Learning Research (06/2025)
Table 6: Accuracy (%) for TabFact and WikiTQ using GPT-3.5 and GPT-4o-mini. "Decomposed" indicates
whether queries are decomposed into sub-problems (Fig. 2 "Transformed by" refers whether inter-
mediate tables transformed by an LLM or program (Fig
2
"Answered by" specifies whether
the final answer is generated by LLM a program (Fig.
LLM-only approaches provide the final
answer without table transformations. The best performance for each model and dataset is shown in bold.
Accuracy (%)
Tables
Final answer
Method
Decomposed
TabFact
WikiTQ
transformed by by
GPT-3.5 (gpt-3. 3.5 5-turbo- - 1oeooo-66k00600 16k-0613)
End-to-End QA
70.45
51.84
X
LLM
Few-Shot QA
71.54
52.56
X
LLM
Chain-of-Thougee
Wei et
2022
65.37
53.48
X
LLM
Binder Cheng
2023
79.17
56.74
V
LLM Program
Program
DATER Ye et al.
2023)
78.01
52.81
V
Program
LLM
CoTable
Wang et
2024
80.20
59.90
V
Program
LLM
Text-Lo-SQL
(Rajkumaa kumar et al.
2022
64.71
52.90
X
Program
Program
LPA
Chen et al.
2020
68.90
Program
Program
POS (ours)
78.31
54.80
Program
Program
GP'''44--m
1oot--4-minn
2022-077-8)
Binder
Cheng et al.
2023
84.63
58.86
V
LLM Program
Program
DATER
Ye et al.
2023)
80.98
58.83
Program
LLM
CoTable
Wang el
2024
84.24
55.60
/
Program
LLM
POS (ours)
82.70
59.32
Program
Program
Using GPT--o-mini, POS reaches 82.70% on TabFact and 59.32% on WikiTQ. While hybrid approaches like Binder and CoTable score higher on TabFact, POS achieves the best performance on WikiTQ. This demonstrates that with a more advanced language model, POS remains competitive while offering great interpretability, as shown in Sec. 4.1.3
Table 7: Accuracy of POS across varying table sizes. The Pearson corrolation reveals nogligible relationships
between table size and accuracy.
TabFact
WikIQQ
Size
Small 30-109 79.1% (533/674)
Medium
Small
Medium
Large
Token Range
109-188
188-804 804
135 135-638
638-1307
1307-33675
Accuracy
(575/675)
(550/675)
(713/1448)
(574/1448)
(558/144)
Correlation
-0.006
-0.023
4.2.2 Performance vs. table size analysis
A natural concern for Table QA systems is whether the performance degrades as input tables grow larger and more complex. To answer this, we provide a quantitative evaluation of POS accuracy on TabFact and WikiTQ, stratified by table size. Specifically, following the methodology Wang et al. (2024). we sort cach table by token count (our measure for table size) and split datasets into three bins of equal size (small, medium, and large). We use GPT-4o-mini as the backbone LLM and report the QA accuracy in Tab. 7
We observe that POS maintains stable accuracy on TabFact (79.1 85.2 + 81.5%, r= 0.006), though on WikiTQ performance drops from 56.1% (small) 16.7% (medium) before recovering slightly to 47.8% (large) (<10 percentage points; r= 0.023). This suggests POS is resilient on some benchmarks but can suffer on tables of intermediate size in more complex tasks.
We find that POS is more robust than existing methods-sui suffer significant accuracy drops with large tables (i.c., tables longer than 4K tokens; see Table 3 in Wang et al. (2024)). For instance, Binder and DATER experience accuracy declines of 30 50 percentage points, dropping to 6.41% and 34.62%, respectively, while Cobabl degrades to 44.87% on WikiTQ, This degradation is primarily due to the challenges LLMs face when processing long contexts. In contrast, POS's SQL-basod executions remain resilient variations in table length, making well-suited for real-world scenarios where table sizes may scale to millions of tokens.
Binder, DATER, and ColTable which all
11

---

Published in Transactions on Machine Learning Research (06/2025)
4.3 Efficiency Analysis
Table 8: Efficiency analysis on WikiTQ. SC denotes self-consistency usage, LLM represents the total
number of LLM calls (detailed in Breakdown), and DB is the number of database queries. Notably, POS requires up to 25x fewer calls/querias than other methods.
Method
SC
LLM
Breakdown
DB
Binder
Cheng et al. 2023
50
GenerateSQT: 50
50
Decompose: 40;
Cloze: 20;
DATER Ye et al. 2023)
v
100
20
GenerateSQL: 20;
GenerateAnswer: 20
Planning: S 5;
CoTable
(Wang et al. 2024)
V
S
25
GenerateArgs: S 19;
5
GenerateAnswer: 1
Planning:
POS (ours)
x
4
2
GenerateSQL: 2
Efficiency in Table QA is crucial because reducing the number of LLM calls and database queries directly low- ers computational costs and improves scalability. We leverage the efficiency benchmark introduced by Wang et al. 2024), which measures the number of LLM calls required to answer a WiiiiQQ question. Additionally, we propose using the number of database queries (i.e., table transformatiins) that reflects the computational workload on the table database.
As POS employs highly deterministic and atomic steps, it eliminates the need for the costly self-consistency prompting [Wang et al. 2022) required by Binder, DATER, and CoTable. As result, POS requires only four LLM calls per question significantly fewer than CoTable (25), Binder (50), and DATER (100). Regarding database queries, POS is also more efficient than others with only two queries per question, compared to five of CoTable, 50 of Binder, or 20 of DATER.
4.4 POS for Free-form Table QA
POS processes Table QA queries end-to-end using SQL commands, performing table transformations exclu- sively on the input table and without accessing to external knowledge. This SQL-only pipeline restricts POS's applicability to tasks requiring creativity, such as generating paragraph-like answers. To address this, we extend POS to the frec-form Table QA task (FeTaQA) (Nan et al.) 2022 by integrating an LLM call in the final step to generate free- form natural language answers, following the algorithm illustrated Fig. 2
Table 9: Results on FeTaQA, a free-form Table QA task (using GPT-4o-mini as the base LLM). POS out- performs both 0-shot end-to-end and few-shot QA.
Method
BLEU
ROUGE-1
ROUGE-L
Sind-toEEnd QA
18.99
51.92
46.44
Few-Shot QA
19.18
53.32
46.86
POS
20.16
54.70
48.69
We compare POS with End-to-End and Few-Shot QA and find that POS consistently outperforms both methods (see Tab. 9 This improved performance (+0.98 points in BLEU and +1.83 points in ROUGE-L compared to Few-Shot QA) is attributed to intermediate SQL executions of POS, which retrieve fine-grained and relevant information to generate final answers. In contrast, End-to-End and Few-Shot QA process the entire input table once, making challenging for the model to pinpoint and make use of the correct data.
Please note that we compare POS against these two LLM-only baselines selected specifically to showcase the adaptability of POS generating free-form natural language responses while preserving its interpretability advantages (in intermediate table transformations). As we prioritize interpretability, we have not optimized POS accuracy in free-form Table QA, and therefore do not include many hybrid QA baselines in Tab.
12

---

Published in Transactions on Machine Learning Research (06/2025)
5 Discussion and Future Works
We address pertinent questions regarding the robustness and interpretability of POS; more are in Appendix
How does POS handle real-world "messy" tables? We have so far evaluated on benchmark tables that are semi-structured and relatively clean (as preprocessed by Wang et al. (2024) and Ye et al. (2023) In practice, tables "in the wild" may have merged cells, nested headers, irregular formats, or other noise. POS in its current form does not explicitly tackle such noise. A practical extension would be to integrate a preprocessing step that normalizes and restructures messy tables (e.g., using a tool like NormTab (Nhhid & Rañiei 2024a)) before applying POS. This could allow POS to maintain high accuracy and interpretability even on more complex, non-canonical tables.
How can POS detect errors in its own SQL generation? POS uses a fallback that could diminish the program-only nature: if SQL execution fails, LLM answers the question end-to-end triggered in 3.16% of TabFact and 13.58% of WikiTQ samples, similar to other Table QA methods (Ye et al., 2023 Cheng ct al. 2023 Wang et al.) 2024). Tab. G reports POS accuracy using this fallback mechanism. To minimize such fallbacks, we propose implementing a proactive error delection and correction mechanism that validates each generated SQL query against the table schema. When an error is detected, the model regenerates the query using the execution history log as context.
Is POS completely interprctable? No; no AI system is entirely interpretable. In computer vision, for example, methods such as concept bottleneck models provide luman--ndderslandaabbe concepts (Ismail et al.] 2023) for model decisions, but the reasoning underlying the network generate the concepts remains unintelligible. Likewise, LLMs can generate thhouht rationales, but the internal hidden states driving these thoughts cannot be directly understood (Wei et al. 2022 Madsen et al., 2024). POS bottlenecks the decision-making into human-interpretabue operations- namely, natural-language steps that are translated into SQLs. They deterministically produce the answer, allowing users to follow the logical chain of table transformations. Although the underlying LLM may involve black-box internal processes (in generating natural language steps), our focus is on making model's decision-making interpretable, aligning with prior and current interpretability research (Tacsiii et al. 2022 Ismail et al., 2023 Zhao et al. 2024).
Can every based question be decomposed into a set of atomic steps? Yes; by design POS breaks any question over table into a sequence of atomic operations-
select, group, aggregate, sort, limit, join, set operations, and so on on- on-eeah implemented by a single, standalone SQL statement. Because SQL is relationally complete (i.e., it can express every relational-algebra query), this set of primitives (or atomic operations) can express any query, from simple counts to complex multi-table analytic
The only limitation arises for truly free-form queries that demand narrative or summary generation (e.g.. "Summarize the basic information of the football clubs in Saint Petersburg."). In such cases, POS still applies atomic SQL steps to extract and prepare the data, then defers only the final writing step to language model, preserving full transparency of all intermediate transformations.
What are the most common failure modes of POS? We observed that errors in POS mostly originate from the planning stage rather than from the Step-to-SQI. For example, the planner often omits necessary condition checks in its atomic steps or performs steps in the orders that lead to the wrong answers (refer to Appendix H.1.3 for the importance of step order). As LLMs continue to improve their planning capabil- ities (Huang et al. 2022), we expect POS to benefit accordingly in terms of performance while retaining its strong interpretability. Please refer to Appendix J for examples and analysis of POS errors.
Why POS explanations do not contain SQLs? We designed POS present natural- language steps rather than raw SQL because, in our user studies, including SQL in the explanations tended to overwhelm users. The natural language descriptions are equivalent to the SQL commands -hanmands-hhanks to the atomicity constraint, which guarantees minimal-to-no discrepancy between the two-yet they are far more readable and accessible. This design choice strikes a balance between interpretability and clarity, ensuring that users grasp the model's reasoning without being burdened by technical SQL syntax.
13

---

Published in Transactions on Machine Learning Research (06/2025)
6 Conclusion
We introduce Plan-of-SQTs (POS), an interpretable, effective, and efficient approach to Table QA with large language models. POS decomposes a table question into simple atomic steps and executes each step using SQL, thereby ensuring that each transformation in the reasoning process is transparent. This design fills critical gap in current LLM-based Table QA models, which often produce answers without clear explanations. Our experiments demonstrate that prioritizing interpretability does not come at the expense of performance: POS achieves explanation quality superior to existing methods (as confirmed by both human evaluations and LLM judges) and competitive accuracy on Table QA benchmarks. Moreover, POS accomplishes this with an order-of- mfggntuue fewer LLM calls and database queries than prior approaches, making it a more efficient and scalable solution for rcal-world use. We also find that human evaluators and LLM judges largely agree their evaluations of explanations, indicating that LLM-based automated evaluation is feasible. The only significant limitation observe is rooted in the LLM's planning capability if the LLM produces a suboptimal plan, POS can err, which aligns with recent observations by ZZhao et al. (2024). Encouragingly, as LLMs become more powerful in planning, we expect the performance of POS to naturally improve while preserving its inherent interpretability. In summary, POS represents a step toward Table QA models that not only deliver accurate answers but also provide human-uundersaandabbl explanations.
Acknowledgement
GN was supported by an Auburn University Presidential Graduate Research Fellowship. AN was supported by NSF Grant #2145767, NAIRR award #240116, donations from NaphCarc Foundation, and research gifts from Adobe Research.
Disclaimer
This paper was prepared for informational purposes by the Artificial Intelligence Research group of JPMorgan Chase & Co. and its affiliates ("JP Morgan") and is not a product of the Research Department of JP Morgan. JP Morgan makes no representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This document is not intended as investment research or investment advice, or recommendation, offer solicitation for the purchase sale of any security, financial instrument, financial product service, to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful.
References
Nikhil Abhyankar, Vivek Gupta, Dan Roth, and Chandan K Reddy. H-star: Llm-driven hybrid sql-text adaptive reasoning on tables. arXiv preprint or wxXi:2407.059522 2024.
Julius Adebayo, Michael Muelly, Taria Liccardi, and Been Kim. Debugging tests for model explanations. Tn Proceedings of the 34th International Conference on Neural Information Processing Systems, pp. 700 712, 2020.
Chirag Agarwal, Sree Harsha. Tanneru, and Himabindu Lakkaraju. Faithfulness plausibility: On the (un)
reliability of explanations from large language models. arXie preprint orXiv:2402.0461 04614, 2024.
Jayetri Bardhan, Anthony Colas, Kirk Roberts, and Daisy Zhe Wang. Drugehrqa: A question answering
dataset on structured and unstructured electronic health records for medicine related queries. In Proceed- ings of the Thirleenth Language Resourees and Evaluation Conference, pp. 1083-1097, 2022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voce Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Fric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack
14

---

Published in Transactions on Machine Learning Research (06/2025)
Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Chacha Chen, Shi Feng, Amit Sharma, and Chenhao Tan. Machine explanations and human understanding, In Proceedings of the 2023 ACM Conference on Foirness, Accountability, and Transparency, pp. 1-1, 2023a.
Valerie Chen, Nari Johnson, Nicholay Topin, Gregory Plumb, and Ameet Talwalkar. Use-case- -eraaeegroondddd simulations for explanation evaluation. Advances in neural information processing systems, 35:1764-1775, 2022.
Wenhu Chen. Large language models are few (1)-shot table reasoners. In Findings of the Association for Computational Linguistics: EACL 2023, pp. 1120-1130, 2023.
Wenhu Chen, Hongmin Wang. Jianshu Chen, Yunkaí Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact A largo-scale dataset for table-based fact verification. In International Conference Learning Represenlations (ICLR), Addis Ababa, Ethiopia, April 2020.
Yanda Chen, Ruiqi Zhong, Narutatsu Ri, Chen Zhao, He He, Jacob Steinhardt, Zhou Yu, and Kathleen McKeown. Do models explain themselves? counterfactual simulatability of natural language explanations. In Forty-first International Conference Machine Learning, 2023b.
Mingyue Cheng, Qingyang Mao, Qi Liu, Yitong Zhou, Yupeng Li, Jiahao Wang, Jiaying Lin, Jiawei Cao, and Enhong Chen. A survey on table mining with large language models: Challenges, advancements and prospects. Authorea Preprinls, 2025.
Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. Binding language models in symbolic languages. ICLR, 2023.
Julien Colin, Thomas Fel, Rémi Cadène, and Thomas Serre. What cannot predict, do not understand: A human-centered evaluation framework for explainability methods. Advonces in neural information processing systems, 35:2832-286 2845, 2022.
Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint :Xfx:1702.08600 2017.
Yann Dubois, Percy Liang, and Tatsunori Hashimoto. Length- alpacaeval: A simple debiasing of automatic evaluators. In First Conference on Language Modeling, 2024. URL httpp:///operrevier. net/
Forum?1d=CybBm22WBx
Xi Fang, Waijic Xu, Fiona Anting Tan, Ziqing Hu, Jiani Zhang, Yanjun Qi, Srinivasan H. Sengamodu, and Christos Faloutsos. Large language models (LLMs)) on tabular data: Prediction, generation, and understanding survey. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL
https://openreviee http:://ppppreeeoe.e.e////rtrttt
Laura Somanddez-Becervs Miguel Ángel González-Santamarta,, Ángel Manuel Guerrero-Higueras, Fran- cisco Javier Rodríguez-Lere and Vicente Matellán Olivera. Enhancing trust autonomous agents: An architecture for accountability and explainability through blockchain and large language models. arXie preprint arXie:2403.090 0956677 2024.
Kevin P Gaffney, Martin Prammer, Larry Brasfield, D Richard Hipp, Dan Kennedy, and Jignesh M Patel. Sqlite: past, present, and future. Proceedings of the VLDB Endowment, 15(12))3535-3555 2022.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Penglei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pp. 10764 10799. PMLR, 2023.
Peter Hase and Mohit Bansal. Evaluating explainable ai: Which algorithmic explanations help users pre- diet model behavior? In Proceedings of the 58th Annuol Meeting of the Association for Computational Linguistics, pp. 5540-5552 2020.
15

---

Published in Transactions on Machine Learning Research (06/2025)
Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, and Zeynep Akata. Grounding visual explanations. In Proceedings of the European conference on computer vision (ECCV), pp. 264-279, 2018.
Zijin Hong, Zheng Yuan, Qinggang Zhang, Hao Chen, Junnan Dong, Feiran Huang, and Xiao Huang. Next- generation database interfaces: A survey lim-based text-to-sql. arXie preprint arXiv:2406. 08426, 2024.
Wenlong Huang, Picter Abbecl, Deepak Pathak, and Igor Mordatch. Language models as zcro-shot planners: Extracting actionable knowledge for embodied agents. In International conference on machine learning, pp. 9118 9147.9917..71 PMLR, 2022.
Aya. Abdelsalam Ismail, Julius Adebayo, Hector Corrada. Bravo, Stephen Ra, and Kyunghyun Cho. Concept bottleneck generative models. In The Twelfth International Conference on Learning Representations, 2023.
Croft Jane. Citigroup fined over 'fat finger' enror that led to 11.1bn of mistaken orders | citi- group | the guardian, 2024. URL httt::://www.theeuureee //tpa////aaaaa///////o////////// citigroup-fined-oree www.ra-innne-roorr-ii-eeeee orders
Sunnie SY Kim, Nicole Meister, Vikram V Ramaswamy, Ruth Fong, and Olga Russakovsky. Hive: Evaluating the human interpretability of visual explanations. In European Conference on Computer Vision, pp. 280- 298. Springer, 2022.
Kezhi Kong, Jiani Zhang, Zhengyuan Shen, Balasubramaniam Srinivasan, Chuan Lei, Christos Faloutsos, Huzefa Rangwala, and George Karypis. Opentab: Advancing large language models as open-domain table reasoners. In The Twelfth International Conference on Learning Representations, 2024. URL https: //openreview.co http://wwweeeeoorrmrreeeeooooue
Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-contex( llms struggle with long in-context learning. arXiv preprint arXiv:2404. 02060, 2024.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu, G-eval: Nlg evaluation using gpt-4 with better human aligument. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2511-2522, 2023.
Andrew W Lo and Jillian Ross. Generative ai from theory practice: A case study of financial advice. An MIT Exploration of Generative AT, 2024.
Andreas Madsen, Sarath Chandar, and Siva Reddy. Are self-explanations from large language models faithful? In Findings of the Association for Computational Linguistics ACL 2024, pp. 293-337, 2024.
Edmund Mills, Shiye Su, Stuart Russell, and Scott Emmons. Almanacs: A simulatability benchmark for language model explainability. arXiv preprinl arXiv:2312 12747, 2023.
Md Nahid and Davood Rafiei. Normtab: Improving symbolic reasoning in llms through tabular data nor- malization. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 3569-3585. 2024a.
Md Nahid and Davood Rafici. Tabsqlify: Enhancing reasoning capabilities of 1lms through table decom- position. In Proceedings of the 2024 Conference of the North American Chapler of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), pp. 5725 5737, 2024b.
Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojeiech Kryő- ciñski, Hailey Schoelkopf, Riley Kong, Xiangru Tang, et al. Fetaqa: Free-form table question answering. Transactions of the Association for Computational Linguistics, 10:35-49, 2022.
Giang Nguyen, Dacyoung Kim, and Anh Nguyen. The effectiveness of feature attribution methods and its correlation with automatic evaluation scores. Advances in Neural Information Processing Systems, 34: 26422-26448 2021.
16

---

Published in Transactions on Machine Learning Research (06/2025)
Giang Nguyen, Valerie Chen, Mohammad Reza Taesiri, and Anh Nguyen. PCNN:
Probable-class nearest-
neighbor explanations improve fine-grained image classification accuracy for AIs and humans. Transactions
on Machine Learning Research, 2024a. ISSN 2835-8856. URL httpss:/oppnrevvis... http::/pppeareeeeeeeee/ffrorrrir www.:eeeeffffiiiiioo
DcF DcFjq1J98b
Giang Nguyen, Mohammad Reza Taesiri, Sunnie S. Y. Kim, and Anh Nguyen. Allowing humans to interac- tively guide machines where to look does not always improve human-ai team's classification accuracy. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR) Work- shops, pp. 8169 8175, June 2024b.
Long Ouyang, Jessrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730 27744, 2022.
Panupong Pasupat and Percy Liang, Compositional semantic parsing semi-structured tables. In Proceed- ings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pp. 1470 1480, 2015.
Antonin Poché, Alon Jacovi, Agustin Martin Picard, Victor Boutin, and Fanny Jourdan. Consim: Measuring concept-based explanations' effectiveness with automated simulatability. arXiv preprint orXiv:2501.056 055855 2025.
Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. Evaluating the text-to-sql capabilities of large language models. arXiv preprint arXiv:2204.00450 2022.
Vikram V Ramaswamy, Sunnie SY Kim, Ruth Fong, and Olga Russakovsky. Overlooked factors in concept- based explanations: Dataset choice, concept learnability, and human capability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1092-1 2023.
Ben Richardson. Excel facts & statistics: Original rescarch acuity training, 3 2022. URL https::/ /ww.
acuitytraining.om co. httpaaaeestpppeeeeeeeeee-eaeeeee
Tianz Shi, Chen Zhao, Jordan Boyd-Graber, Hal Daumé III, and Lillian Lee. On the potential of lexico- logical alignments for semantic parsing to sql queries. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1849-1864, 2020.
Stanford University; Stanford HAI. Who's at fault when ai fails in health care?, 2024. URL https:
7/hai stanford. www./ewwe/eeeeeeeeeee/eee hoot///eww//hoo---atte
com com
M Steyvers and A Kumar. Three challenges for ai-assisted decision-making. Perspectives on Psychological Sci- ence: Journal of the Associalion for Psychologicul Science, pp. 77456916231181001 11762221111111221111111111199999 182118110,, 2023.
Josefa Lia Stoisser, Mare Boubnovski Martell, and Julien Fauqueur. Sparks of tabular reasoning via text2sql reinforcement learning. arXiv preprint or Xiv:22500 000016, 2025.
Mohammad Reza Taesiri, Giang Nguyen, and Anh Nguyen. Visual correspondennce--aseed explanations improve ai robustness and human-ai team accuracy. Advances in Neural Information Processing Systems, 35:34287-34301, 2022.
Andreas Theissler, Francesco Spinnato, Udo Schlegel, and Riccardo Guidotti. Explainable ai for time series classification: a review, taxonomy and research directions. leee Access, 11100700-1077220 2022.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint
SxXiv:2208.11177 2022.
Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Miculicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lec, and Tomas Pfister. Chain-of-table: Evolving tables in the reasoning chain for table understanding. In The Twelfth International Conference on Learning Representations, 2024. Utt.https:// http://pwwaeeieeee.eeeorormaaddd
17

---

Published in Transactions on Machine Learning Research (06/2025)
Albatool Wazzan, Marcus Wright, Stephen MacNeil, and Richard Souvenir. Evaluating the impact of ai- generated visual explanations on decision-making for image matching. In Proceedings of the 30th Interna- tional Conference on Intelligent User Interfaces, pp. 672 684, 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Fdd Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting clicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
Zirui Wu and Yansong Feng. Protrix: Building models for planning and reasoning over tables with sentence context. Xi preprinl arXiv:2433.02117, 2024.
Yuqing Yang, Boris Joukovsky, José Oramas Mogrovejo, Tinne Tuytelaars, and Nikos Deligiannis. Snip- pet: framework for subjective evaluation of visual explanations applied to deepfake detection. ACM Transactions on Multimedia Computing, Communications and Applications, 20(8):1-29, 2024.
Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. Large language models are versatile decomposers: Decomposing evidence and questions for table-based reasoning. In Proceedings of the 46th International ACM SIGIR Conference on Reseorch and Development in Information Retrieval,
pp. 174-184, 2023.
Tong Zhang, Mengao Zhang, Wei Yan Low, X Jessie Yang, and Boyang Albert Li. Conversational explana- tions: Discussing explainable ai with non-ai experts. In Proceedings of the 30th International Conference on Intelligent User Interfaces, pp. 409-424, 2025.
Yilun Zhao, Lyuhao Chen, Arman Cohan, and Chen Zhao. Tapera: Enhancing faithfulness and inter- pretability in long-form table qa by content planning and execution-based reasoning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12824-12840, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Newral Information Processing Systems, 36:46595-4666 46623,, 2023.
18

---

Published in Transactions on Machine Learning Research (06/2025)
Appendix for:
Interpretable LLM-based Table Question Answering
A Explanation Methods for Table QA
In this section, we present visual explanations for Table QA models, which help bridge the gap between model behaviors and human understanding, Each visualization provides insights into how the model leverages the imput table, highlighting the key information used its reasoning process.
For our experiments, we use TabFact (Chen et al. 2020 and WikiTQ (Pasupat & Liang 2015). running each method on the test sets of 2,024 and 4344 samples. POS, CoTable, and DATER all use the same visualization format described in Sec. 3.3
We use four different methods for explaining Table QA answers: Text-to-SOL, DATER, CoTable, and POS (ours). Each method offers a unique set of information follows:
Text-to-SQL (Rajkumar et al. 2022
This method translates the question into a single SQL com- mand, which is then executed on the input table produce an answer. Erplanation Generation: We present the generated SQL command along with the question and the input table show how the final answer is dcrived. Although the SQL itself can be clear to experts, it may require additional domain knowledge to interpret (see Fig. 4 for example). Therefore, the we recruited Computer Science students (who have SQL expertise) for our human study.
DATER Ye ct al., 2023). DATER solves natural language question via extracting relevant subtable and uses SQLs to verify partial facts (Fig. 5). Explanation Generation: We extract (1) the subtable selected by DATER from the input table, (2) the verified facls, and (3) altribution maps that highlight which table cells DATER considers relevant. Furthermore, we show step descriptions (e.g.. "Select rows") from DATER's working logs, allowing users to track subtable extraction (sce
Fig. 5).
Co'Table (Wang et al. 2024). Colabbl processes queries by planning sequence of abstract function calls (e.g., f_sort, f f_select_ron _roo, etc.), cach responsible for table transformation. Explanation Generation: Similar to POS, we visualize each intermediate table along with aldribution maps, but the transformations are represented by function names and their arguments (e.g. f_select_rosl (1)). Although this sy-seb approach can be informative, arguments are not well-justified and the final answer relies on LLM-driven "black-box" reasoning (sce Fig. 0
POS (Ours). POS decomposes a natural language question into a sequence of alomic reasoning steps, where each step explicitly translated into an executable SQL command. Explanation Generation: For POS, we present the natural language description of cach step-these are functionally equivalent to the underlying SQL commands, but far more accessible and readable to users. Users can inspect each transformation through the clear and concise natural language steps, supported by attribution maps highlighting the relevant table cells for every operation (see Fig. 7)
Comparison of DATER and CoTable. Empirically, we find no major difference in their overall "infor- mativeness" (Sec. 4.1.3.. although CoTable displays intermediate tables more explicitly. Both methods are still considered bybrid they rely on partial SQLs or function calls combined with LLM-driven reasoning. For tasks demanding deeper inspection of intermediate steps (c.g., verification complex filters), CoTable's step-by-step interface may be more revealing than DATER's subtable-based approach.
In general, by generating method-specific explanations under a unified visualization format, we can fairly compare how each Table QA methods explain reasoning and arrives its final answer. This shared framework allows us examine strengths, weaknesses, and interpretability trade-offs across different approaches in earlier sections.
19

---

Published in Transactions on Machine Learning Research (06/2025)
game
opponent result
opponents
2
cincinnati
win
20
4
georgia win
26
Select rows where 'opponents' is
5
vanderbiitt win
14
(e)
9
evansville win
36
SELECT
(d)
6
michigan win
7
6
FROM table _qble_sql
cols=[opponents]
8
w virginia
win
15
6
WHERE opponents ;
3
xavier
win
20
7
7
alabama loss
0
13
game
opponent
result wc_ pts opponents
idk
game
opponent
result
wc__ss
opponents
10
tennessee
loss
6
13
2
cincinnati win
20
0
cincinnati win
20
ale miss
loss
7
14
georgia win
26
0
georgia
win
(e)
vanderbill win
14
vanderbill
win
rows= 1,2,3]
evansville win 36
evansville win
michigan win
michigan win
(c)
virginia win
virginia
win
idx
game
opponent
result
wc__pt pts opponents
xavier win
xavier win
0
2
cincinnati win
20
0
(a)
(b)
alabama loss
13
alabama
georgia win
0
lennessee
tennessee
loss
vanderbilt
win
loss
14
miss
loss
9
evansvvlle
win
Figure 3: Generating attributions maps for POS. Column
idx is added to track row attribution.
A.1
Attribution Maps
During the execution of each SQL command, we perform the following steps:
Adding the tracking index column: Before executing an SQL, we add a tracking index column
to the current table. This column contains the original row indices from the initial table- Fig. 3 a).
Exccuting the SQL command: An SQL command is executed on the table with the tracking index column, producing a modified lable Fig. (b).
Identifying selected rows: After execution, we use the tracking index column to identify which rows have been selected or filtered by the SQL command- Fig. (
Identifying selected columns: We parse the SQL command extract the columns involved in the operation- Fig. (d) (see Appendix
E
Visualizing an attribution map: The prior information allow us to generate an attribution map
for the intermodiate tables Fig. 3(c). The index column is also removed at this step.
Since both rows and columns can be attributed within an operation, POS offers a distinct advantage over previous works (Ye et al., 2023 Wang et al. 2 accurately attributing responsible cells for cach trans- formation. For example, when SQL command includes condition that requires cell to match specific value range (e-g., WHERE opponents 0), we can determine which cells in the opponents column satisfy this condition and are thus responsible for the answer Fig. e)).
A.2 Explanations as Chains of Attribution Maps
At each step of the table transformation process, we visualize attribution maps on the input table for that step, highlighting the data selected or filtered the current operation. Rows and columns that contain relevant data for the operation are yellow- higloo-hgghighhtdd, while cells that match the specific condition in this step are green-highlighted.
Using the information obtained from the plan execution and attribution maps, we combine the three com- ponents: (1) intermediate tables; (2) attribution maps; and (3) step description; to create explanation.
20

---

Published in Transactions on Machine Learning Research (06/2025)
We present the explanation in chain of attribution maps, helping users visually follow the sequence of transformations and understand how each table cell contributes to the final answer. We show representative examples of explanation methods in our study below.
Statement: the wildcats kept the opposing team scoreless in four games
Input Table: 1947 kentucky wildcats football team
date
opponent
result
wildccats_piinge
opponents
record
9999-09-20
ole miss
loss
7
14
0-1
9999-09-27
cincinnati
win
0
1-1
9899-10-04
xavier
7
2-1
9999-10-11
9 georgia
win
26
0
3-1,20
9999-10-18
10 vanderbilt
win
0
4-1, 14
9999-10-25
michigan state
5-1,
9999-11-01
18 alabama
loss
0
13
5-2
8
9999-11-08
west virginia
Win
15
6
6-2
9999-11-15
evansville
win
0
7-2
9899-11-22
SQL Command:
SELECT
table_sql opponents
Figure 4: Text-to-SQL explanations provide only the SQL command, which is intuitive for SQL users.
21

---

Published in Transactions on Machine Learning Research (06/2025)
the wildcats kept the opposing team scoreless in four games in the table: the wildcats kept the opposing team scoreless in 4 games.
Statement:
Input Table: 1947 kentucky wildcats football team
Step 1: Select Rows (row 4, row 5, row 3, row 2, row 9) and Select Columns (opponents, wildcats points, game)
game
date
opponent
result
wildcats points
opponents
record
sept
miss
2
sept
cincinnati
win
3
xavier
win
4
georgia
win
3-1,20
5
18
vanderbilt
win
4-1,14
6
oct 25
michigan state
win
7
5-1,13 13
7
nov
alabama
loss
8
nov
west virginia
win
9
nov
evansville
win
7-2
nov tennessee
loss
7-3
Sub-table Selection
opponents
wildcats points
game
0
20
2
7
20
3
0
26
4
0
14
5
0
36
9
Contextual information: the wildcats kept the opposing team scoreless 4 games.
Prompting LLM for the final answer... >>>
Prediction: TRUE
Figure 5: DATER explanations contain sub-table selection, contextual information (or verified facts), and highlights that reveal which input features influence the final answer.
22

---

Published in Transactions on Machine Learning Research (06/2025)
Statement: the wildcats kept the opposing team scoreless in four games
Input Table: 1947 kentucky wildcats football team
Step 1: _Lweeet_rowoweee row)row 1, row 2, row 3, row 4, row 8)
garee
date
opponent
resut
widcate points
opponents
1e00r0
sept.22
dindinaa
5
0
Kavier
Win
20
georgia
oct.16 18
vanderbitt
win
54
4x1,14
michigan
nov
alibama
loss
0
13
5-2
Bov
evansville
win
7-2
tennessee
13
7-3
Step 2: [ seleect column(game, wildcals points, opponents)
game
date
opponent
result
wildcats points
opponents
record
cincknnati
oct
xavier
win
26
oct
10 vanderbilt
win
4-1,
nov 15
evaravillo
38
7-2
Step 3: f sort collumn(opponents
game
wildcats points
opponents
28
Step [umppe_
wilcatt
28
36
0
Prompting LLM for the final answer... >>>
Prediction: TRUE
Figure 6: CoTable explanations present intermediate tables and highlights, showing key steps in data transformation. Additionally, the steps are presented through function names and their arguments.
23

---

Published in Transactions on Machine Learning Research (06/2025)
kept opposing
games
Table: kentucky wildcats football
Order the by 'appanents' ascending
reau.
widcats_ppo
rwcord
0989-09-20
Glc
9999.06-27
diddrnad
W0h
20
9890-10-11
gedrgia
Wih
20
3-1.1
9090-10:25
michigan otane
win
5-:1.
50088-11-08
MEGE airiini
688
15
B-
9909.11
11-15
evansvill
10
3689-11-22
lannessoe
loss
13
Step Select rowe where 'opponente is
pame
opponent
wildcals_pooft
opponents
9899 09-22 27
drndrnnt
648
20
1:-1
gecrgia
3=11.20
9098: 10:18
501 saaddwbbl
668
3.6
A-11.11
9000-11-15
5099-1044
michigan abaae
win
1 13
9999-11-08
9069 10
Xover
win
20
9899-11-01
13
10
9668-111 11-22-1--22
lenneoone
9089
14
Step statement return
rows equal
opporent
widoattsppnnts
opponents
9009-09-21
sincinnali
WA5
1=:1
3959-10:11
3959-10:18
10 vandertbi
W85
4m
9959/11:16
evansoill
wefiicatoo.resul
Prediction: TRUE
Figure 7: POS explanations contain intermediate tables and highlights. The green- higennhighlihhted cells indicate where the information in the table matches the conditions specified in the natural language steps.
24

---

Published in Transactions on Machine Learning Research (06/2025)
A.3
Self- Celf-Explanation of Table QA Models
An interesting question is whether simple, post-hoc explanation method might achieve similar interpretabil- ily. To explore this, we adopt widely used "sell-xxplanation" baseline (Madsen al., 2024 Chen et al.
2023b Agarwal et al. 2024). Specifically, the LLM is first prompted to produce an answer and then asked to retrospectively explain how it arrived at that answer (sce Fig. 8) We compare this post-hoc approach with our proposed POS and other XAI baselines (Text-to-SQL, DATER, and CoTable) in Tab. 10 focusing
on three XAI benchmarks in the TabFact dataset using GPT-4o-mini.
Table 10:
Comparison of post-hoc self-explanation vs. other explanation methods. Lower is better for Preference; higher better for Forward Simulation and Verification.
Method
Preference (4)
Forward Sim. (1)
Verification (f)
Sell-explanation 5.00
65.98%
68.03%
Text-to-SQL.
3.99
65.67%
55.37%
DATER
2.71
73.57%
55.43%
CoTable
1.77
76.53%
61.36%
POS (Ours)
1.53
81.61%
76.74%
Our main finding is that post-hoe self-explanation outperforms Text-to-SQLi (always the worst) but remains significantly less effective than our proposed POS method. This may attributed its lack of faithfulness, extensively identified in previous studies (Chen et al. 2023b Madsen et al. 2024 Agarwal et al. 2024), where the explanation often fails to align with the modcl's true reasoning process. In contrast, POS explanations are generated and execuled at each step, inherently tying them to the actual transformations applied to the data. This design, as demonstrated in our earlier ablation and user studies, results in greater understanding and more actionable interpretability for downstream tasks like verification and forward simulation. We show an example of self-explanation below.
25

---

Published in Transactions on Machine Learning Research (06/2025)
Prompt:
You are given Statement and Input Table.
Your Task:
Verify if Statement True or False agaisnt the Imput Table. Explain your verification process based on the provided info. Your answer must be TRUE or FALSE only.
Statement: the wildcats kept the opposing team scoreless in four games Input Table Caption: 1947 kentucky wildcats football team
game
date
opponent
result
wildcats_ points
opponents
record
9999-09-20
miss
loss
14
2
9999-09-27
cincinnati
win
20
3
9999-10-04
xavier
win
20
2
4
9999-10-11
goorgia
win
26
3 1, 20
5
9999-10-IP 18
10 vanderbilt
win
14
0
4 1, 14
6
10-9-10-25
michigan state
win
6
5 1, 13
7
9999-11-01
18 alabama west virginia
losss
13
5-
8
1199-11-08
win
15
6
6- 2
9
11-1
evansville tennessee
win
36
7- 2
10
9999-11-22
loss
13
7- 3
LLM Self-explanation: Verification: TRUE
To verify the statement "the wildcats kept the opposing toam scoreless in four games", we need to caamine the input table, which lists the results of the Kentucky Wildcats football games. The key information here that the Wildcats are considered to have kept the opposing team scoreless if the column "opponents points" shows value of "0" for that game.
We can through the table and count how many games the Wildouts kept the opposing team scoreless (opponents 0):
pointss points
Game 1: Ole Miss Opponents' points 14 (not scoreless)
Game Cincinnati Opponents' points (scoreless)
Game Xavier Opponents' points 7 (not scoreleas)
Game Georgia Opponents points 0 (scorcless)
Game Vanderbilt Opponents' points 0 (scorcless)
Game Michigan State Opponents' points (not scoreless)
Game Alabama Opponents" points 13 (not scoreless)
Game West Virginia Opponentss points (not scoreless)
Game Evansville- Opponents" points (scoveless)
Game Tennessce Opponents' points 13 (not scorcless)
From table, we see that the Wildents kept opposing team scoreless in four games (games 2, 4, 5, and 9).
An example of post-hoc self-explanation. The LLM first produces an answer (TRUE) and then justifies it by referencing the table entries that support the claim.
Figure 8:
26

---

Published in Transactions on Machine Learning Research (06/2025)
B Baselines and Related Works
LLM-only. These approaches rely solely on LLMs to generate answers without explicitly performing table transformations. End-to-End QA prompts the LLM to generate answers diroctly from the input table and question. Similarly, Few-Shot QA (Brown al., 2020) includes few-shot examples (T, A) as the context to aid the LLM. In contrast, process stspeby-step before delivering the final answer.
Thoughh (Wei et al., 2022) prompts the LLM to explain its reasoning
Program-only. Program-based approaches generale explicit programs to perform table transformation and answer the question. Latent Program Algorithm (I.PA) (Chen et al. 2020) frames TabFact verification as program synthesis task, converting input queries into sequential operations (e.g., min, max, count, filter) executed via Sython-Pandas. On the other hand, Text-to-SQL (Rajkumar et al. 2022) translates a natural language question directly into a single SQL command, which is then applied to the input table to generate the answer.
Hybrid. Hybrid approaches combine the strengths of LLM reasoning and programs to perform Table QA and achieve state-of-the-art accuracy, DATER (Ye et al., 2023) uses an LLM to extract relevant sub- tables, while breaking queries into sub-queries and executing SQL commands retrieve factual information. Binder (Cheng et al., 2023 takes different approach by converting natural language questions into exe- cutable programs. It blends API calls with symbolic language interpreters like SQL or Python to address reasoning gaps that cannot be handled through offline methods alone. Lastly, CoTable (Wang et al., 2024) as selecting rows or adding columns, al-
dynamically plans a sequence of predefined table operations-sun lowing it to iteratively transform the table based on the intermediate information. Despite their differences, they all share a common strategy: they input the final simplified table along with the original question into LLM to produce the final answer.
Program-aided language models vs. Plan-of-SQLs. Although both PAL (Gao et al. 2023 and POS break questions into executable, program-based steps, they differ fundamentally in focus. PAL targets general math and symbolic problems by emitting a single Python script- comments and code intertwined making it hard for non-programmery to trace each operation. In contrast, POS is built specifically for Table QA: it forces every reasoning step to be "atomic" (one condition and one variable at time), translating each into trivial SQL query. This guarantees that users can inspect and verify each intermediate result before moving on. Moreover, POS produces fine-grained attribution maps over the actual tables at every step, so users see exactly which rows and columns were involved whereas PAL's logic remains buried in Python code, opaque to those without coding expertise.
C More Ablation, Discussion, and Future Works
What is the impact of removing SQL execution from POS? We investigate this impact in Sec. 4.1.6. Removing SQL execution not only diminishes interpretability, but also affects accuracy on benchmark datasets. In this ablation, the LLM is tasked with directly transforming the table rather than execut- ing SQL commands. While this leads to a negligible decrease in Table QA accuracy on TabFact, it causes a substantial drop on WikiTQ. This discrepancy suggests that relying solely on the black-box reasoning of LLMs for table transfornations can severely impact model accuracy likely due LLM hallucinations or errors when handling complex tables (Chen 2023 Wang et al.. 2024). Morcover, bypassing SQL removes layer of transparency, as the table transformations are no longer traceable, a reduction in interpretability that is further quantified by Tab. [5]
Have we evaluated explanation effectiveness in free-form Table QA? No; our work has not yet investigated the interpretability in free-form Table QA (e.g., FeTaQA), where the final answer is generated through a black-box operation of LLMs making POS explanations no longer 100% faithful or compre- hensive. A promising explanation approach is to develop hybrid explanations that blend POS-generated intermediato-stop explanations with groundings (Hendricks et al. (2088) for the final answer. In this hy- brid format, intermediate steps generated by POS provide a transparent view into the intermediate reasoning process while grounding algorithms are employed to anchor the final response to the intermediate tables.
27

---

Published in Transactions on Machine Learning Research (06/2025)
What tabular questions are not SQL-decomposable? There are questions that cannot be broken down into Plans of SQLs (or sequential SQL queries). To our knowledge, they are:
Free-form narrative queries: egg,, "Summarize the basic information of the football clubs in Saint Petersburg". They require surface-level composition or paraphrasing, not relational result set.
World-knowledge or multi-table reasoning outside the given database: e.g., "Which of these universities the oldest in the country"? Suppose that the table omitss lounding years. They require lookup beyond the input table; POS atomic SQL over the local table is, by construction, blind to external facts.
To automatically detect questions that are not suitable for POS, there are simple auto deto-detection strategies:
Rule-based heuristics: This is the most naive solution. We flag questions containing key verbs such as "summarize", "describe", or "compare" as likely free-form or analytic.
Execution-fallus signals: Run the Step-to-SQL execution; if step converts to invalid SQL (Step- to-SQL raises error), flag non-decomposable. This uses the same exccution-validati hooks we have already described for error handling in Sec.
LLM-based classification: We prompt the NI. planner to emit Boolean "SQL- decomposable?" tag or fine-tune lightweight classifier small labeled dataset of decomposable vs. non-decomposable questions.
SQL or Python; which is better for Table QA?
To see the difference, consider the query "top 3 departments by average salary among active employees over 40". A Python solution might look like:
data load table ("employees.com ceV")
step1 [r for r in data if r["age"] > 40]
step2 = [r for r in step1 if r r["status"] == "active"]
# custom code to group by department and compute averages
# then sort and slice the top 3
Using Python here has few drawbacks:
Memory & speed: We must load the entire table into Python and loop over every row, which is
slow and memory-intensive for large datasots.
Grouping, aggregating, and sorting require handwritten code (counters, loops, temporary structures) instead of built-in operations ions.
Custom logic:
No indexing: Python loops cannot leverage database indexes or optimized query planners, leading to poor scalability.
SQL is more preferable for Table QA because:
Declarative clarity: We specify what we want (c.g., "Seloct rows where age > 40"), instead of using steps sequentially: loop, filter 40, and put to the list. This makes program more concise, self-documenting, and readable than using Python.
Engine optimizations: Modern database engines have extensively optimized query planning, and indexing; they can process large tables far faster and more memory-efficienlly than Python loops
Consistency:
SQL's syntax and semantics are stable across databases (e.g., Oracle, MySQL, Mi- crosoft SQL Server, and PostgreSQL), so the same query runs everywhere. Python script, by con- trast, require custom connectors, custom layers, and may vary with library versions.
28

---

Published in Transactions on Machine Learning Research (06/2025)
Ablation study POS accuracy
In addition to the ablation study we conducted on interpretability (in Sec. 4.1.6), we run another ablation study on the accuracy of POS on both TabFact and WikiTQ to pinpoint modules which may not be necessary for either interpretability or performance.
Table 11: Accuracy (%) on TabFact and WikiTQ using GPT-4o-mini.
Variant
TabFact (%)
WikiTQ (%)
POS (Full)
82.70
59.32
Atomic operations
78.11
(-4.59)
50.02 (-9.30)
NL Planning
77.90
4.800
48.27 ( 11.05)
SQL execution
79.84
(-2.86)
27.05 (-32.37)
We find that:
Replacing POS's SQL transformations with end-to-end LLM calls causes the biggest accuracy loss especially on WikiTQ (59.32 + 27.05%, -32.37 pp) and smaller drop TabFact (82.70
+ 79.84%, -286 pp).
Removing either the natural- laanuaggg planning or atomic-step enforcement each costs approximately 4 pp TabFact and 9 11 pp on WikiTQ.
While all three components are important, robust SQL execution has the largest impact on accuracy, followed by step-by-step NL planning, and then atomic decomposition.
29

---

Published in Transactions on Machine Learning Research (06/2025)
D Qualitative Examples for POS Explanations without Atomicity
In this section, we provide qualitative examples of POS explanations with and without atomicity enforcement in NL Planning. Removing atomicity from the plan steps can negatively impact interpretability, as the added complexity makes it harder for users to understand the model's reasoning process.
Talae
TRB PiLit
Statament: the time value for the rider brian finch, tearn suzuki and rank greater than i 2:14.50.0
Input Table: 1970 isle
Step Select wheee 'ider' Eindf AND Nearn' "suzuki AND "rank'a than
mikk
Ader
speed
whitmuay
manh
3:5.00.0
67
206:15.0
66 min
Jeads
Suokk
15.31 man
21144..0
fetch
mph
214.60
rark
Stop 3: Use "AAEE statement to rolurn TRUE if the number of rows equal to othenise rotum FALSE.
Ver Reriictoon, 196R
Prediction: FALSE
Figure 9: Upper: POS explanation with atomicity. Lower: POS explanation without atomicity.
30

---

Published in Transactions on Machine Learning Research (06/2025)
Susement: ofter 1985, the united states contributed two players with rafoel araãjo being the most recent
Impurt Table: utah jazz all time rostee
Step Select nowr ofhene tratorns Lunted straet and yearse for juzz afher 15E1S
paleer
john amacch
Gamerr Sarwedd
200100000
parn 4046
BIMME STAME
200200001
BOMI
2066:0001
Cakk anTIp
200200011
Rurda
Boo Use CASE stalemen: to rotari TRUE The number (ONO issosssee oqual to ochorwao rotur FALSE
yoer_/or_jez
2007:01:01
0000000:11
100100001
Prediction: FALSE
Statement: galina vookobouva played total of games on hard tannis court, and on clay
Table: gatina vaskoboeva
gemmer played
2302/1040
mbeds
2005-15:06
Elaoodecss
2305-00:08
Stop 2: Solect rows whone 'surfioce 'aaay and count the number of such rows determine the total gomes played city courts.
619
Inuraamett
rurbce
oрpoнent
rcom
hard, poart_paees
Fammno up
2305-11-06
plibblugg
Ford
Miu oslbeihh
5- 7.
Stop 3: Use CASE statement to retum TRUE if the count of hard count gamea is equal to AND the count clay ccult gamea is equal to othanwise nitum
TALSE
Prediction: FALSE
Figure 10: Two POS explanations without atomicity, The steps are compound and the attribution maps are non-trivial to comprehend.
31

---

Published in Transactions on Machine Learning Research (06/2025)
E
Extracting Columns from SQL Commands
In this section, we detail the algorithm to analyze SQL queries and identify the columns used within them.
game
opponent
result
(WC, pls apponents
aincnmalii
georgia
Select rows where 'opponents' is
vanderbilt
win
14
0
(e)
evansville
SELECT
(d)
michigan
FROM table table_sql sql
cols=lopponents]
virginia win
15
6
WHERE opponents
slabema
loes
13
game
opponent
resul
pts
opponents
idx
game
opponent result jwc pts opponents
10
tonnessee
loss
13
4
georgia win
26
4
georgia win
(e)
vanderbilt win
14
2
5
vanderbit win
rows=(0, 1,2,3]
evansville win
evansvile win
6
michigan win
7
4
6
6 michigan win
(c)
virginia
vinginia win
gamee opponent we_ppt opponents
xavier win
cincinnati
wn
(a)
(b)
7
elebera
loea
0
13
7
7
alsbama
loes
13
georgia
10
loss
10
ole miss
loea
miss
l06s
ovansvill wn
Figure 11: Data-attribution tracking algorithm for POS.
E.1 Algorithm Overview
The algorithm follows these main steps:
1. Preprocessing: Remove comments and normalize whitespace in the SQL query.
Column Extraction: Parse different clauses of the SQL query to identify column names:
SELECT clause: Extract both regular columns and those used in functions.
WHERE clause: Identify columns used in conditions.
ORDER BY clause: Extract columns used for sorting.
Filtering: Compare extracted columns against a list of original columns to ensure validity.
E.2
Implementation Details
The algorithm is implemented using regular expressions to parse the SQL query. Key implementation details include:
Use of re. sub for comment removal and whitespace normalization.
Application of search( and re. findall () for extracting column names from different parts of
the query.
Special treatment for columns usod within functions in the SELECT, WHERE, ORDER BY clauses.
E.3 An example of data-attribution tracking for Table QA
Here, we use the table transformation in Fig. as an example to illustrate our data-attribution tracking algorithm (Fig.
(a) Adding the Tracking Index Column
32

---

Published in Transactions on Machine Learning Research (06/2025)
(b) Exccuting the SQL Command
(c) Identifying Selected Rows
(d) Parsing SQL Commands to Identify Sclected Columns
(e) Mapping to Original Indices
F Hallucinations in Sub-table Selection
Methods like CoTable and DATER aim to answer questions by performing complex table transformations specifically, selecting sub-tables from the input table based reasoning steps. However, these methods are prone to errors regarding which table entries to select, leading to irrational or irrelevant information being considered in the final answer.
As illustrated in Fig. 12 although Chain-of- Table correctly answers the question Q: True or False? In four different baseball games, the final score was 9-2, it irrationally selects unrelated information (game 3) from the input table. Similarly, DATER, shown in Fig. 13 selects rows 2, 3, 4, 5, and 9 to answer the same question. However, the inclusion of row illogical and does not contribute a valid answer.
wilictss kept opponing bas aconiss infour garse
Fgeaage
12
12
Prompting LLM for final answer...
Figure 12: Although CoTable correctly answers the question Q: True or False? In four different baseball games, the final score was 9-2, it irrationally selects unrelated information (game 3) from the input table.
33

---

Published in Transactions on Machine Learning Research (06/2025)
the opposing games
Input Table: 1947 kentucky wildcata football team
Select Rows
Select
george
vanaeerttll
25
michégan dtale
ROW
alacerna
10s
west virgina 1ia
15
6-2
35
22
txenessce
7-8
Sub-betre Selection
Contextual Information: wilddots kept the opposing toam sooreless in games.
Prompting LLM for final answer...
Prediction: TRUE
Figure 13: DATER selects rows 2, 3, 4, 5, and 9 to answer the question. However, the inclusion row 3 is illogical and does contribute to a valid answer.
34

---

Published in Transactions on Machine Learning Research (06/2025)
G
Details for LLM-as---Juddee Experiments
In this section, describe how we prompt LLMs work judges for evaluating explanation methods.
G.1
Prompt to LLM Judges in Prediction Verification
Prompt for LLM-as-a-Judge in Prediction Verification
prompt = ["
The Table Fact Verification (TabFact) model is working on verifying if a given Statement is TRUE or FALSE based on given input Table.
You are given HTML file containing a Statement, Input Table, Prediction, and Explanation clarifying the Prediction.
Your task carefully analyze the Explanation and determine whether the Prediction is Explanation Method: [method]
mmethod_specific, info]
content: [
Answer with 'optionl' 'option2' only.
You MUST ignore the order the options and answer based the correctness of the Prediction!
G.2
Prompt LLM Judges in Forward Simulation
Prompt for LLM-as-a-Judge Forward Simulation
prompt ""
Given an input statement, Artificial Intelligence (AI) model will output either TRUE FALSE, Your job in this Sim- ulation task is to use the Al's explanation to guess the machine response. Specifically, please choose which response (TRUE/FALSE) model would output regardless of whether you think that response correct not.
Explanation: [text_content]
Based on this explanation, guess what the model will predict on the Statement based on the provided explanation. Answer with only 'TRUE' or "FALSE':
G.3 Prompt to LLM Judges in Preference Ranking
It is well known that LLM-as-a-Judge exhibits a strong bias toward the position of the options presented it (Dubois et al. 2024 To climinate this bias in our prompt, shuffie the order of the four methods four times and compute the average ranking.
35

---

Published in Transactions on Machine Learning Research (06/2025)
Prompt for LLM-as-a-Judge Preference Ranking
prompts []
num methods Len (methoda)
Create dictionary mapping methods their descriptions
method method_desscripine descriptions
"DATER": """DATER method that focuses selecting relevant information from the input table and providing contex- tual information support the statement verification process. The explanation contains:
1. Sub-table Selection: DATER selects sub-table from the original input Table that is relevant to the Statement.
2. Contextual Information: DATER provides contextual information that is fact-checked against the Table."" "able.""..
"COT": """COT is method that breaks down the crestion-answering process into series of intermediate tables. Each step
in the chain represents a specific operation on the table, leading to the final answer. The explanation contains:
1. Step Descriptions: Each step is accompanied by a function with arguments, providing context for the transformation.
2. Intermediate Tables: We display the intermediate tables resulting from each function, showing the state the data at cach
step.
3. Row and Column Highlighting: Rows and Columns used in the current step are highlighted with background- color:yellow.""!
"Texi2SQL."; """"ext2QQL. is method that translates the natural language question into single SQL query. The SQL query itself serves as the explanation for how the system arrives at its answer. The explanation contains: The generated SQL command that will directly applied onto the table to generate the final answer." """
"POS": """POS Table QA method that breaks down the question question-answering process into a series natural-languse steps. Each step represents specific operation the table, Icading to the final answer. The explanation contains:
1. Step Descriptions: Each step is accompanied by natural-lanuuage description of the atomic step performed, providing context for the transformation.
2. Intermediate Tables: We display the intermediate tables resulting from each step, showing the state of the data at cach step.
3. Attribution Maps: highlight rows, columns, and cells involved each table transformation over intermediate tables. Row and Column Highlighting: Rows and Columns used in the current step highlighted with background- color:yellow, Cell Highlighting: Cells that directly match the conditions in the current step are highlighted with background- color:90BE90."" """
for i in range (num_na _am____thhds::
shuffled
methods ehuffled_method methods [i:] methods [:i]
prompt f You are given explanations from four different methods for the same table fact verification task. Please rank these explanations based on their clarity, coherence, and helpfulness in understanding the model's d reasoning.
Clarity Definition: How easy is the explanation to understand? Is the language clear and straightforward?
Coherence Definition: Does the explanation logically flow and make sense whole? Are the ideas well-connected?
Helpfulness in Understanding the Model's Reasoning Definition: How effectively does the explanation help understand why the model made its decision? Does reveal the reasoning process?
Provide the ranking from best to worst.
Explanations:
36

---

Published in Transactions on Machine Learning Research (06/2025)
H Prompt Engineering
H.1 Prompt for Atomic Planning
H.1.1 Decomposition of Question Q
The decomposition process breaks down the complex question Q into a sequence of atomic steps. This is achieved through carefully crafted prompt provided to the LLM. The prompt includes:
Instructional Guidelines: We instruct the LLM "Develop a step-by-step plan to answer the question given the input table".
The LLM is instructed that "Each step in your plan should be very atomic and straightforward, ensuring they can be casily executed or converted into SQL".
Emphasis on Atomicity:
In-context Examples: We provide example inputs (T, Q) along with their corresponding plans to serve in-context examples for planning (see Appendix D
H.1.2 Sequencing of Steps
Correct sequencing is crucial because each step depends on the output of the previous one. We ensure proper sequencing by:
Explicit Instructions: The LLM is instructed that "The order of steps is crucial! You must ensure the orders support the correct information retrieval and verification!".
Dependencies: Clarifying that "The next step will be executed on the output table of the previous step. The first step will be executed on the given Table".
Handling Comparatives and Superlatives: Instructing the LLM on how to handle slatements involving terms like 'highest,, 'lowest', etc., by ordering the table before selecting rows.
Prompt for atomic planning
[In-context examples]
### Here come to your task!
Table caption: [caption)
/* paatrrinnttaaeeeifooooaaaaaa
+/ Convert Table into markdown format
This Table has (num_rove (_nsrroww) rows.
Statement: (umplee(statemment]]
Let's develop step-by-step plan to verify given Statement TRUE or FALSE on the given Table!
You MUST think carefully analyze Statement and comprehend it before writing plan!
Plan Steps: Each step in your plan should very atomic and straightforward, ensuring they can be easily executed or converted into SQL.
You MUST make sure all conditions (except those mentioned in the table caption) checked properly in the steps.
Step order: The order of steps crucial! You must ensure the orders support the correct information retrieval and verifica- tion!
The next step will be executed on the output table of the previous step. The first step will be executed on the given Table.
For comparative or superlative Statement involving "highest," "lowest," "carliest," "latest," "better," "faster," "carlier," etc., you should order the table accordingly before selecting rows. This ensures that the desired comparative or superlative data is correctly retrieved.
Plan:
37

---

Published in Transactions on Machine Learning Research (06/2025)
H.1.3 The Importance of the Step Order
In this example, step 1 is crucial. If the table is not ordered by 'rank' first, selecting row number 1 (step 2) or filtering by 'athlete' (step 3) will return the wrong result. Only by ensuring that the table is correctly ordered beforehand can we reliably select the top-ranked athlete. Thus, the sequence of steps must be followed precisely to avoid logical errors.
A plan where the step order determines the correctness
Table: Olympic 2018; Table Tennis
/+
col
rank athlete
time
row 1
manjeet kaur (ind)
52. 17
row 2 2 olga tereshkova (kaz)
51.86
row +/
pinki pramanik (ind)
53.06
Statement: manject had the highest rank the competition.
Plan:
1. Order the table by 'rank' in ascending order.
2. Select number 1.
Select rows where 'athlete' 'manject' using the LIKE function.
4. Use CASE statement to return TRUE if the number of rows is equal to 1, otherwise return FALSE.
H.2 Prompt for Step-to-SQL
Prompt for Step-to-SQL.
[In-context examples]
Given this table:
/* (uabeessring(intemmeiitee | */
Data types of columns:
[coL_]): (dtype_str_1)
[coL_]]: [dtype_str_2]
Write an SQL command that: (natural nhauurallanguagc__topp The original table has [num_rows] rows.
Constraints for your SQL:
using SELECT COUNT(*), SUM, MAX, AVG, you MUST use name new column. adding new columns, they should be different than columns (existing_cols).
Your SQL command MUST compatible and executable by Python sqlite3 and pandas.
3. using FROM, the table to be selected MUST be name)).
38

---

Published in Transactions on Machine Learning Research (06/2025)
In-context Examples
1.1 In-context Examples for Atomic Planning
In-context examples for atomic planning
TabFact
Table: 2005 tournament results
col id name
hometown
score
row 1 : 1
alice
new york 85
row charlie chicago 75
los angeles 90
row 4 4 dave
new york 88
row 5 5 eve los angeles
92
+/
Statement: 2005 tournament, charlic are both from chicago.
Plan: # Natural-language step
1. Select rows where the 'name' is 'bob' 'charlic'.
Select rows where 'hometown' is 'chicago".
3. Use a CASE statement to return TRUE if the number of rows is equal to 2, otherwise return FALSE.
WikiTQ
Table: 2005 tournament results
/*
col id
hometown
row 1 1
alice new york 85
row 2 2 bob los angeles
90
row +/
charlie
chicago
75
Question: which players are from chicago?
Plan: # Natural-language step
1. Select rows where the 'hometown' is 'chicago'.
2. Select the 'name" column.
39

---

Published in Transactions on Machine Learning Research (06/2025)
1.2
In-context Examples for Step-to-SQL
In-context examples for Step-to-SQL
Given this table:
/*
col
name
| department salary years
alice | it
95000
row 2 2 bob finance
105000
row 3
charlie
marketing
88000
+/
Write SQL. command that: Select where the "salary" is greater than 100000.
SQLis:
SELECT
FROM [qll
WHERE salary > 100000;
Select rows where the 'salary" is greater than 100000.
40

---

Published in Transactions on Machine Learning Research (06/2025)
J Error Analysis of POS and Improved Planning Algorithm
J.1 Error Analysis of POS
We nolice that many errors in POS are due to the planning stage rather than the Step-to-SQL process. In particular, the Planner misses condition checks (see Fig. 14 Fig. Fig. 15 Fig,
Fig. 18 in atomic Arguably, this error can be attributed to the planner, as it has access to both the query and the input table and should therefore generate the correct step. In this case, the Step-to-SQL component is functioning as intended. We also offer alternative perspective on this interesting failure in Appendix J.2
17
steps. Another interesting (and inherently unavoidable) error presented Fig. 19
J.2 Quantifying contributions of Planning and Step-to-SQL to POS errors
In the example shown in Fig. 19 the failure to retrieve the correct rows is not due to any flaw in our Step- to-SQT. converter. Instead, it stems from mismatch between the query token and the table's content-ant and, ultimately, from an inconsistent ground-truth label. Specifically, the NL Planner faithfully mirrors the input string "bjørn" when it generates its first step ("Select rows where player = 'thomas bjørn""), but the actual table only contains "thomas bjarn," so no rows are returned. Under standard Table-QA semantics, "bjørn" and "bjarn" denote two distinct entities, so the correct label for this query should be Falsc, not Truc. Consequently, neither the Step-to-SQL module nor the NL Planner has malfunctioned; the discrepancy arises from the dataset's labeling and the lack of accent normalization in the query definition.
Putting this example in context, we observe that across all of where the Step-to-SQL module itself dircetly produces incorrect SQL query. Instead, errors occur two other points in the pipeline:
analyses there no instances
NL Planner errors: the natural-language plan deviates from the correct reasoning (even though every resulting SQL statement remains executable).
Fallback errors: ad least one SQL in the plan is invalid, causing the system to defer to the end-to-end QA model, which then also fails to produce the right answer.
To quantify how much each of these failure modes contributes to our overall error rate on TabFact and WikiTQ, we labeled every incorrect sample in our evaluation set as cither a Planning error or a Fallback error. The breakdown of their relative frequencies is reported Tab. 12
Table 12: Contribution to POS errors of Planning and Fallback.
Dataset
Model
Planning (%)
Fallback (%)
TabFact
GPT-4o-mini
95.4
4.6
TabFact
GPT-3.5
89.7
10.3
WikiQQ
GPT-4o-mini
75.5
24.5
WikiTQ
GPT-3.5
70.7
29.3
We observe that planning mistakes constitute the vast majority of errors: over 95% for GPT-4o-mini on TabFact (and nearly 90% for GPT-3.5), and around 70-75% on the more complex WikiTQ benchmark. This confirms that planning is the primary bottleneck in POS.
J.3 Planning one step at at time improves Table QA accuracy
Based on this observation, we implement an improved planning algorithm in which only one step is generated a time, rather than generating all steps upfront, as shown in Fig. [ This approach encourages the L.M to think one step at time and reduces the complexity of the planning task. The input of NL Planner is the previous steps and the current intermediate table.
As shown in Tab. 13 planning one step at a time leads to substantial improvements in accuracy for both TabFact and WikiTQ with GPT-4o-min:. There are leading hypotheses for these improvements:
41

---

Published in Transactions on Machine Learning Research (06/2025)
Grounding in real data. Before generating each new step, one-step planning provides the actual intermediate table (the result of the previous SQL.) into the prompt. This lets the LLM see exactly which rows and values it's operating on (condition on the actual intermediate table state), mitigat- ing hallucinations and mis-alignments. By contrast, generating an entire multi-step plan up front requires the model to imagine how each intermediale table will look and those imagined tables don't 't match reality, later steps can go wrong-
Focused context window. By feeding the LLM only the rows and columns it needs for cach step-rather rather than the entire table- we keep the input prompt concise and free of irrelevant data. Since LLMs can struggle when their context grows too large Li et al. 2024), this focused context helps them generate the next stop more accurately.
Table 13: Planning step time (one-step planning) with GPT-4o-mini
Method
TabFact (%)
WikiTQ (%)
End-to-end QA
71.17
49.24
POS one-time planning
77.22
48.90
POS one-step planning
83.45
59.32
pádraig harringtion only player from
Input Table: 2006 (galf)
Step Select 'country'
place
payarr
courtry
steve striker
70 -0-
oolin monggoneli
sootion)
79-71-140
fomneith Ixcte
71 71- 141
guoll agihy
P1 70-1111
Eurp
stakes
72
70
Step Use CAAE staloment retum TRUE the number of rows equal 1. othewwsee retum FALSE.
place
player
coUrTy
acore
tx par
Wolthem RMAETE
21 72 163
verfiiation resut
TRUE
Prediction: TRUE
Figure 14: POS predicts TRUE but the groundtruth FALSE (False Positive). planning, POS misses checking the player name.
42

---

Published in Transactions on Machine Learning Research (06/2025)
Statement: frank zimbatowe
Input aga champlonshii
loourtyy
Port trnkks
UMTIC ROAM
28. 70+ 20-
430300
kinny paryy
5Xee cltnglan
+00+ 73 20--2
pumeyk
Juy angh
53 00 72-2777
-0
30057
Jeuan
71
recseiion
20
UKO COANS
75 70x 68 20
(0530
York rote
13* 72
+5
5/530
CAAE'
TRUE
Plose
player
countty
Accre
noW
(81 75
(T50)
THUe
Figure 15: POS predicts TRUE but the groundtruth is FALSE, (False Positive). In planning, POS misses checking the player name.
43

---

Published in Transactions on Machine Learning Research (06/2025)
Statement: as porto novo scored three points against the victoria club mokanda
Input Table: african cup champions
Slep Select rows whene Team I' laa porta neve!.
agg
6_1el_
0_2nd_
8 mentel
2-2 (5- pen)
Iele stmora
9689-01-02
0668-01-01
R00RUPOa unied
1-3
great dtympios
6899-01-01
0099-01-01
mollen
9889-09-01
8959-01-01
85 p00000000000 none
0.33
victoll dlub mokands
0589/01:01
0058:01:00
0onon sooundds
0.4
solcomt
8899:07:03
9998 00
escArance
(bonghazi
9889-01.01
9999
A
orugu rongers
younng offians
2-0
lavori pub
9889-01-0
0988-01.0 01
Select
Victoria
1 1
bean
c_
<_0vd_/gg
porto RoND
D::3
Webora dMD mosands
8999 01
5809/01102
Step
rows is greater equai
Ieam_1
tean
s_19t_bg
3rd
Webon dMC mosands
5095/001001
5899/01102
TRUE
Prediction: TRUE
Figure 16: POS predicts TRUE but the groundtruth is FALSE (False Positive). In planning, POS misses checking the score.
44

---

Published in Transactions on Machine Learning Research (06/2025)
soaddigg the spor lesar dan 2007 Toole come by Seret sodes
2001200
Figure 17: POS predicts FALSE but the groundtruth is TRUE (False Negative). In planning, POS misses checking the year.
45

---

Published in Transactions on Machine Learning Research (06/2025)
Statement: crick walder and bob beamon had the same wind
long jump
1921-00-30
D00 Tosnnio
1949-10-19
8.07 (200Ti))
car Mtdde
Ind st30e
1909
1901-00-30
8:09 11 2
1.5
1967-00-22
in (himsaa
201/48-92
Stop Belect rowa whore Vihicce is bob bearion.
wind
VEELE
Stop
whhate
Step At Usei CASE statement to riturn IItLE: the number rows equai to othenee sn relum FALSE
axtivally
FlL SE
Prediction: FALSE
Figure 18: POS predicts FALSE but the groundtruth is TRUE (False Negative). In planning, POS should select two rows at the same step.
46

---

Published in Transactions on Machine Learning Research (06/2025)
Statement: thomas bjam, denmark, has score of 68 71 771+770 215
Table: champlenahip
Slop Select where playe' Thomao bjant.
plpper
Kayl
70
mart cirose
12 12--8--22-21 72-212
Jouze
72
71
Faxt froo
16+20-77-20
193
7 -1-
Slep vou when 'ouutry tevmmaek
Skop Select wwo whore Score la 10 21 78- 210.
Skop Use CASE stalomont to rolum TRUE the number of rowa & czuel othewiie rotum FALSE.
Prediction: FALSE
Figure 19: POS predicts FALSE but the groundtruth is TRUE (False Negative). Arguably, this error can be altributed to the planner, as sees both the query and the input table and should therefore generate the correct step (i.e., Select rows where 'player' is 'thomas bjarn'). We observe that the Step-to-SQL component always functions correctly. We also offer an alternative perspective on this interesting failure in Appendix J.2.
47


Reviews:
[{"review_id": "qXUsWGxv1n", "reviewer": "Reviewer_LDtj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nThe paper is well written and it is easy to follow the arguments being made. Thorough evaluation using multiple datasets and interpretability measures, including both human and LLM-based assessments. The ablation study in Section 4.1.6 is very welcome and gives a lot of insight into breaking down the nature of the improvements. The efficiency analysis presented in the paper strengthens the novelty of the framework.\n\n### Weaknesses\n1) “POS uses a fallback that could diminish the program-only nature: if SQL execution fails, an LLM answers the question end-to-end—triggered in 3.16% of TabFact and 13.58% of WikiTQ samples, similar to other Table QA methods” - I think this significantly undermines the interpretability claim given in the paper. Although 13.58% already seems high, there is no guarantees on how high of a fraction will this be in real world datasets - which might prevent the use of this technique at scale.\n2) The method is not able to handle tables where we cannot reasonably apply SQL like multi level nested tables etc. Although the authors do acknowledge this, most of the real world tables are of this format - and clearly end-to-end LLM reasoning will be strongly preferable over POS in those cases.\n\n-----\nOverall, although the applicability of the framework seems quite limited, I appreciate the strong analysis done by the authors and think that the TMLR community will find this interesting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Plan-of-SQLs (POS), a new approach for making Table Question Answering (Table QA) more understandable. Instead of treating each question as a single, complex task, POS breaks it down into a series of simple, atomic steps which are converted into a straightforward SQL command, and run on the table one step at a time, making it easy to see how the final answer is reached. The key focus is on transparency. Unlike many other methods that rely on opaque, black-box models, every step in POS is clear and traceable, ensuring users can follow the reasoning behind each answer. Despite being simpler and more interpretable, POS still performs competitively on popular benchmarks like TabFact and WikiTQ.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) The paper will also benefit from a deeper discussion around the space of questions which cannot be broken down into sequential SQL queries and if we can auto-detect such cases. \n2) Ablation study should also be carried out on performance on the framework so that we can pinpoint modules which are not necessary for neither interpretability nor performance.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vfgRt1AJ0G", "reviewer": "Reviewer_8AWs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n\nThe paper is well-written and easy to follow.\n\nThe paper conducts thorough experiments against an extensive set of baselines including Dater and chain-of-table.\n\nPOS enhances solution interpretability, significantly reduces the number of required LLM calls while maintaining competitive accuracy. \n\n**Weaknesses**\n\nConceptually the framework lacks novelty, for example, PaL [1] breaks a question down into steps which are represented by python operations, the final solution is then obtained by executing the python program. \n\nThe type of questions POS can answer may be limited by the SQL commands, as the answer must be directly derived from SQL execution. Can every Table-based question be decomposed into a set of atomic steps?\n\n[1] Gao, Luyu, et al. \"Pal: Program-aided language models.\" International Conference on Machine Learning. PMLR, 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Plan-of-SQLs (POS) for Table Question Answering. POS uses LLM to decompose a Table QA problem into atomic steps which are then converted to SQL commands. The SQL commands are executed to retrieve the answer. POS also derives a chain of attribution maps as explanation for the produced solution. Experiments show that POS improves solution interpretability while maintaining accuracy compared to existing approaches. POS further reduces the number of required LLM calls, reducing cost.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Can the authors comment on \n- Why a DSL like SQL is chosen instead of a general language like python. \n- Limitation (or the lack thereof) on the questions POS can answer due to its reliance on SQL.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4zBx9z75Tw", "reviewer": "Reviewer_Rtym", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nWell-Designed Method (POS): The POS method is intuitive and well-motivated. Enforcing atomicity (single condition, single variable per SQL step) and using natural language for planning before SQL conversion makes the process inherently more understandable and less prone to complex errors compared to end-to-end Text-to-SQL or opaque LLM reasoning steps.\n\nEfficiency and Robustness: POS demonstrates significant improvements in computational efficiency (fewer LLM calls and DB queries) and robustness to large table sizes compared to several state-of-the-art methods. This makes it more practical for real-world applications.\n\n\nWeaknesses:\nDependence on LLM Planner Quality: The paper acknowledges (Sec 5, Conclusion, Appendix J) that POS's accuracy is limited by the LLM's ability to generate a correct plan of atomic steps. Errors in the initial natural language plan (e.g., missing conditions, incorrect logic) directly lead to incorrect final answers, even if the subsequent Step-to-SQL translation is perfect. While the one-step-at-a-time planning shows improvement (Appendix J), the fundamental reliance on the LLM's planning capability remains a potential bottleneck.\n\nHandling \"Messy\" Real-World Tables: The evaluation is primarily on benchmark datasets that are relatively clean. The paper briefly discusses (Sec 5) extending POS to handle \"messy\" tables (merged cells, irregular formats) using preprocessing tools like NormTab, but this is presented as future work and not empirically validated. The practical applicability to truly unstructured or noisy real-world tables is not fully demonstrated.\n\nLimited Scope of \"Atomicity\": While the defined \"atomic\" steps (single clause, max one condition/variable) enhance interpretability, they might be overly restrictive for certain complex reasoning patterns that could potentially be expressed clearly in slightly more complex SQL (e.g., joins, nested queries, more complex WHERE clauses). The trade-off between strict atomicity and expressive power could be explored further.\n\nSQL Generation Errors: Although POS aims to simplify SQL generation, errors can still occur, especially with exact matching issues (e.g., 'bjørn' vs 'bjon', Fig 19). The paper mentions a fallback mechanism but doesn't deeply analyze the frequency or types of Step-to-SQL errors.\n\nInterpretability in Free-form QA: The extension to FeTaQA involves a final LLM call to generate the free-form answer, which re-introduces a black-box component, slightly compromising the end-to-end interpretability claim for this specific task type. The paper notes this limitation (Sec C).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Plan-of-SQLs (POS), a novel LLM-based Table Question Answering (Table QA) method focused on interpretability. The core idea is to decompose a natural language question into a sequence of atomic, natural language steps. Each step is then translated into a simple, executable SQL query that transforms the table incrementally. This process ensures transparency as each intermediate table state and the corresponding SQL operation are explicitly shown.\nThe contributions of the paper are:\nIntroduction of POS: A new interpretable Table QA method that breaks down complex questions into atomic, understandable steps translated into SQL queries. This contrasts with methods that generate complex single SQL queries or use opaque LLM reasoning steps (like abstract function calls).\nEnhanced Interpretability: Through experiments involving both human participants and LLM-based judges, the authors demonstrate that POS explanations are superior in quality (preference ranking) and significantly improve users' ability to simulate the model's behavior (forward simulation) and verify its predictions compared to existing interpretable methods (Text-to-SQL, DATER, CoTable, self-explanation).\nCompetitive Performance and Efficiency: POS achieves QA accuracy comparable to state-of-the-art methods on standard benchmarks (TabFact, WikiTQ, FeTaQA) while being significantly more efficient, requiring substantially fewer LLM calls (up to 25x fewer than DATER) and database queries (up to 25x fewer than Binder). It also shows better robustness on large tables where other methods often struggle.\nLLM-as-Human-Proxy Validation: The study reveals high agreement (up to 90.59% in forward simulation) between LLM judges and human evaluators when assessing POS explanations, suggesting that LLMs can serve as reliable proxies for human judgment in evaluating Table QA explanation quality, potentially reducing the cost of user studies.\nGeneration of Attribution Maps: POS inherently generates step-by-step attribution maps by highlighting the rows/columns used and cells satisfying conditions in each SQL step, providing a clear trace of the reasoning process.\nAblation Study: An ablation study confirms the importance of SQL execution and natural language planning steps for the interpretability benefits of POS.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Expand on Planner Error Analysis (Appendix J / Sec 5): While Appendix J shows examples of planning errors (missed conditions), it would be beneficial to provide a more quantitative analysis if possible. For instance, what percentage of errors on TabFact/WikiTQ are attributable to planning vs. Step-to-SQL errors? This would give a clearer picture of the main bottleneck. Discussing why the one-step planning approach (Table 11) yields such significant accuracy improvements (+6% on TabFact, +10% on WikiTQ) beyond just \"reducing complexity\" would also be valuable. Does it allow the LLM to condition on the actual intermediate table state more effectively?\n\nDiscuss Handling More Complex SQL Needs: Briefly discuss the limitations of the current atomic SQL definition. Are there common Table QA reasoning patterns that POS cannot currently handle due to the restriction on SQL complexity (e.g., queries requiring explicit JOINs, complex aggregations with GROUP BY and HAVING, window functions)? Acknowledging these potential limitations and outlining how POS might be extended (even if as future work) would strengthen the discussion.\n\nClarify Fallback Mechanism Usage: Section 5 mentions a fallback mechanism (end-to-end LLM) triggered in 3.16% (TabFact) and 13.58% (WikiTQ) of cases due to SQL execution failures. Could the authors clarify if the main accuracy results reported in Table 6 include results from this fallback? If so, it slightly nuances the \"program-only\" nature for those specific samples. It's a minor point but worth clarifying for full transparency. If the fallback is used, how does its accuracy compare to the main POS approach?", "broader_impact_concerns": "The paper focuses on improving the interpretability of Table Question Answering systems, which is generally seen as a positive step towards more responsible and trustworthy AI. By making the reasoning process transparent through atomic SQL steps, the work directly aims to mitigate risks associated with opaque AI decision-making, particularly in high-stakes domains like finance and healthcare, which the authors explicitly mention.\nTherefore, the work itself does not appear to introduce new ethical concerns or negative societal impacts", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)