Triples as the Key: Structuring Makes Decomposition and Verification Easier in LLM-based TableQA
Authors: Zhen Yang
Ziwei Du
Minghan Zhang
Wei Du
Jie Chen
Zhen Duan
Shu Zhao
Venue: ICLR 2025 Poster (2025)
Keywords: TableQA
Triples
Primary area: generative models
TL;DR: We propose Triples-Inspired Decomposition and Verification strtegy to improve LLM in TableQA.

Abstract: As the mainstream approach, LLMs have been widely applied and researched in TableQA tasks. Currently, the core of LLM-based TableQA methods typically include three phases: question decomposition, sub-question TableQA reasoning, and answer verification. However, several challenges remain in this process: i) Sub-questions generated by these methods often exhibit significant gaps with the original question due to critical information overlooked during the LLM's direct decomposition; ii) Verification of answers is typically challenging because LLMs tend to generate optimal responses during self-correct. To address these challenges, we propose a Triple-Inspired Decomposition and vErification (TIDE) strategy, which leverages the structural properties of triples to assist in decomposition and verification in TableQA. The inherent structure of triples (head entity, relation, tail entity) requires the LLM to extract as many entities and relations from the question as possible. Unlike direct decomposition methods that may overlook key information, our transformed sub-questions using triples encompass more critical details. Additionally, this explicit structure facilitates verification. By comparing the triples derived from the answers with those from the question decomposition, we can achieve easier and more straightforward validation than when relying on the LLM's self-correct tendencies. By employing triples alongside established LLM modes, Direct Prompting and Agent modes, TIDE achieves state-of-the-art performance across multiple TableQA datasets, demonstrating the effectiveness of our method.

Published as a conference paper at ICLR 2025
TRIPLES AS THE KEY: STRUCTURING MAKES DECOM- POSITION AND VERIFICATION EASIER IN LLM-BASED TABLEQA
Zhen Yangl.23, Ziwei D2. Minghan Zhangl23, Wei Du Jie Chen Zhen Duan Shu Zhao
Schooo of Computer Science and Technology, Anhui University
Innfrrmaiinn Materials and Intelligent Sensing Laboratory of Anhui Province, Anhui University
3 Annui Provincial Key Laboratory of Security Artificial Intelligence, Anhui University
wch.oshuss200220 www.hauwwwso200hooooooo com
ABSTRACT
As the mainstream approach, LLMs have been widely applied and researched TableQA tasks. Currently, the core of LLM--ased TableQA methods typically include three phases: question decomposition, sub-question TableQA reasoning. and answer verification. However, several challenges remain in this process: i) Sub-questions generated by these methods often exhibit significant gaps with the original question due to critical information overlooked during the LLM's direct decomposition; ii) Verification of answers is typically challenging because LLMs tend to generate optimal responses during self-correct. To address these chal- lenges, we propose a Triple-Inspired Decomposition and VErification (TIDE) strategy, which leverages the structural properties of triples assist in decompo- sition and verification in TableQA. The inherent structure of triples (head entity, relation, tail entity) requires the LLM to extract many entities and relations from the question as possible. Unlike direct decomposition methods that may overlook key information, our transformed sub-questions using triples encompass more critical details. Additionally, this explicit structure facilitates verification. By comparing the triples derived from the answers with those from the question decomposition, we can achieve easier and more straightforward verification than when relying on the LLM's self-correct tendencies. By employing triples along- side established LLM modes, Direct Prompting and Agent modes, TIDE achieves state-of-the-an art performance across multiple TableQA datasets, demonstrating the effectiveness of our method. We release code here
INTRODUCTION
Tabular data is ubiquitous in daily life and plays a crucial role in fields such as finance and education. Table Question Answering (TableQA) assists in judgment, analysis, and decision-making based on this data (Wang et al. 2023] Wu et al. 2024 Cao et al. 20033] Jin et al.) 2023 Lee et al.] 2023). TableQA involves comprehending and analyzing table content to infer answers to specific questions.
With the development of large language models (LLMs) (Zhao et al. 2023b Yang et al. 2024 Xiao
Lu et al. 2024 Brown al. 2020), the of LLMs for TableQA has become main- stream research direction (Zhang et al. 2024bb Ye et al. 2024 Zhao et al. 2023a). Most LLM-based
al. 2024
TableQA methods can be divided into three phases: question decomposition (Lekkoyyzz et al.)
2022 Liu et al. 2022 Fu et al. 2022: Pasupat & Liang 2015), sub-question TableQA reasoning
and answer verification Madaan al. 2024 Ni al. 2023).
"Corresponding authors

---

Published as a conference paper at ICLR 2025
In recent years, under decomposition--easonii verifiiaatinn phases, LLM-based TableQA tasks have mainly employed two modes for solving complex questions: Direct Prompting (DP), a textual reasoning mode, and Agent, symbolic reasoning mode. In the DP mode, the decomposition- reasoning-verificane process primarily utilizes a few examples to prompt L.Mss. Specifically, during the decomposition phase, the L.M is prompted to break down the question using manu- ally crafted Moain-of-Thoughle (CoT) examples (Zhang et al.] 2023b; Guan et al.) [024]] Sarkar & Lausen 2023), SQL statements (Mouravieff et al. 2024), or even zero-shot requests leverage its capabilities directly (Zhang et al. [223a]] Gemmell & Dalton 2023; Ye et al. 2023a Kojima et al., 2022). In the reasoning phase, the LLM is prompted to reason through the sub-questions, while the verification phase is guided by manually constructed verification examples or by asking the LLM to re-answer for confirmation. In the Agent mode, the recoopositionn-rasoniii caasooiinn-vriiiattion process is carried out automatically by the L.M (Zhang et al.) 2024a Li et al.) 2024a Gong et al. 2020: Li
ict al., 2024b). Specifically, during the decomposition phase, the model first observes the current and previous states plan the next steps, designing the question incrementally. In the reasoning phase, it employs symbolic code, such as Python or SQL,, to take action on the sub-questions. Finally, in the verification phase, the model performs self-correct, prompting the LLM to reassess its results and make necessary corrections.
Question Destrppraition
Sul-quection Breoorinng
Answer Verifrattonn
Question How eopplestapel at lost yearcin in efficr?
wleeday iouth on Symurs" utufcfe nanave?
Ncauet aupplstay ofell 3ymn.
Vas caca alppog year.
Joneer
X
(Wiliin Hedree) [ntoriatt) [bejeniilhoorr
lus M.Srery).
whosiaed in lean yaund whaatederreeliaeee
X
(Piter usik
Wilan Fribyy
Mart 1255
AX1 1815
[Beqjanda Bleward
100 1/2
[EL stgadis afta)
(Alcunder fer MuSiin tixx, years (Miduu Pakaary stay loa than. puan)
(M
(nn bas Metzeurs lik mere thes yandd
[Jnnae ticktin rop sare jeart] [Aoc Hovardalyy man tlua. poyrr))
Table
Syant (1)
prard)
[Boce Howandaayy
Direct decomposition by large models can overlook critical information, resulting in a semantic gap between sub-questions and the original question. (b) Self-correct Methods: The LLM's self-correct tends to generate responses that align with the prompt, contradict the actual context, making difficult to verification answers correctly due to this bias. (c) Our TIDE: The structured nature of triples prompts the LLM to extract many entities and relations from the question as possible, thereby covering more critical information. Additionally, the triple structure enables clear verification process by comparing the triples from question decomposition with those derived from the answer.
Figure 1: (a) Direct Decomposition Methods:
However, current LLM-based methods encounter several challenges in the decomposition- reasoning-verrfiiation process. First, the limited understanding of complex queries by LLMs often leads to fragmented sub-questions during direct decomposition, creating significant semantic gap with the original question due to overlooked critical information. As illustrated in Figure I(a), both Direct Prompting and Agent's 's direct decomposition methods frequently miss essential details, exac- erbating this gap. Second, during the verification phase, simply prompting the L.MM to self-correct or bypass verification hinders the accurate identification of errors in intermediate answers. Research (Huang et al.] 2024) indicates that this approach may bias the model away from generating optimal responses to the initial prompt, resulting in performance decline. As shown in Figure 1(b), direct verification of answers by LLMs often fails to detect such errors. Therefore, more explicit method needed to enable LLMs to generate as many sub-questions as necessary to comprehensively ad- dress the initial query while mitigating the tendency for self-correct.
To address the aforementioned challenges, we propose Triple-Inspired Decomposition and VErification (TIDE) method that leverages the structured properties of triples for decomposition and verification, as illustrated in Figure 1(c). First, LLMs directly decomposing sub-questions often overlook critical information. In contrast, we use triples, with their fixed structure of (head entity, relation, tail entity), compels LLMs to extract as many entities and relations from the question as possible, thereby covering more key information. Second, LLMs' self-correction tends to generate responses related to the prompt, which can introduce biases in answer verification. By converting answers into triples and comparing their structures with those of the decomposed question triples,
2

---

Published as a conference paper at ICLR 2025
utilize the explicit nature of triples to determine the correctness of the answers by checking the equivalence of entities and relations. Since our triples are applied during both the decomposition and verification, and are compatible with both DP and Agent modes, our method can be utilized independently or in a joint approach with either DP or Agent.
veasoning-verfffiattin cetnmpoosiiioo-eesoninnn verification
We evaluated our method on the WikiTableeQustionss (Pasupat & Liang, 2015) and TabFact (Chen et al. 2020) datasets. Our method can be applied independently to either DP or Agent methods, or joint reasoning framework, enhance accuracy. Integrating our TIDE into these modes led to significant accuracy improvements, achieving the latest state-of-the-arl (SOTA) results. The main contributions of this study are summarized follows:
We leverage the structured characteristics of triples (head entity, relation, tail entity) to prompt the LLM extract as many entities and relations from the question possible, thereby covering more key information. Consequently, the sub-questions derived from these triples help minimize the semantic gap with the original question.
Utilizing the clear structure of triples allows for more easier and straightforward answer verification. By converting the answer into triple format and comparing the entities and relations with those from the question decomposition, we can directly validate the answer, thereby avoiding the biases inherent in LLMs when directly verifying responses.
The use of triples for decomposition and verification aligns well with both DP and Agent modes, allowing our TIDE approach to be employed in either standalone or joint reasoning. Through joint reasoning, TIDE achieves state-of-theaa results in TableQA, demonstrating its effectiveness and accuracy.
2 RELATED WORK
2.1 TABLEQA
As erucial task in table-related research, TableQA involves understanding table to infer answers and has gained significant attention in the NLP (Cheng et al. 2024 Li et al.] 2023 Chen et al. 2024]. With the development of LLMs, leveraging their knowleds ledge and capabilities for TableQA become the mainstream approach (Pal et al.) 2023 Lee et al. 2024 Zhong et al.] 2017 Xia et al.) 2023 LLM-based TableQA can divided into three phases: question decomposition, sub-question TableQA reasoning, and answer verification. First, LLMs decompose complex questions, then rea- son for the sub-questions, and finally verify the intermediate answers. Under this decomposition- reasoning-vente verifiictio framework. LLM-based TableQA tasks primarily employ two modes: Direct
Prompting (DP) (Sui et al. 2023 Chen 2023)
and Agent modes (Li et al. 2024b
Lei et al. 2023
2.2 DIRECT PROMPTING IN TABLEQA
In the DP mode, the decomposition- cnnompooitiio-rcaoonnnnn reasoning-verificatiinn verification process heavily relies on manually con- structed Chain-of-Thouge
(CoT) prompts (Kong et al. 2024 7hao et al. 2023d: Deng et al. 2024) to guide L.MSs in gradual decomposition, reasoning, and verification. Co'l was originally utilized in LLM--ased text inference, incorporating prompts like think step by step to facilitate gradual reason- ing toward the final answer. During the critical question decomposition phase, early DP (Zhao et al.) 2023c Sui al., 2023 Chemmengath al. 202T employed few-shot examples, SQL statements. or shot prompts to assist LLMs in breaking down complex questions. For instance, (Luo et al., 2023) used manually constructed CoTs and retrieval-basee reconstruction simulate human- like reasoning. BINDER (Cheng et al. 2023) guided models to reason through decomposed sub-queries using SQI. statements with embedded sub-queries. Similarly, DATER (Ye et al.) 2023b) employed SQL to parse, fill, and reasoning answers after extracting relevant rows and columns from large ta- bles. (Liu et al. 2024a) added the zero-shot prompt think step by step for question decomposition and inference. In answer verification phase, the DP mode frequently neglects the verification process or asks the LLM to re-check answers directly. Studies have shown (Stechly et al.) 2023) Valmeekam et al. 2023 that it is challenging for LLMs to identify issues in answers they generated themselves.
Although these LLM-based TableQA methods utilize LLMs to generate intermediate sub-questions, relying solely on a few examples for direct decomposition usually result in fragmented sub-questions
3

---

Published as a conference paper at ICLR 2025
that overlook critical information. This leads to significant semantic gap between the decomposed sub-questions and the original question. Additionally, due to LLM's tendency, the constructed veri- fication examples prompt the LLM to direct self-correct, leading to a bias toward generating contra- dictory viewpoints, which hampers the accurate verification of answers in TableQA.
2.3 AGENT IN TABLEQA
With the continuous development of agents, their application in TableQA has been widely studied (Li et al.) 2024a Gong et al. 2020). Agents automate the decomposition-reassone reantoinnn reasoning- verification process using L.Mss. Typical LL.M-based Agent approaches either iteratively generate intermedi- ate tables during question decomposition or utilize L.LMs to automatically break down questions. CHAIN-OF-TABLE (Wang et al. (024) decomposes to generate intermediate tables and introduces functions to derive the answer. Similarly, ReAccTable (Zhang et al., (024c) continuously generates intermediate tables to answer questions. (Liu et al., 2024b) used SQL statements to construct an- swers from newly created tables, while (Mouravieff et al. (2224) converted intermediate tables into computational graphs for reasoning. During the sub- sub-question TableQA reasoning phase, agents typ- ically generate code using languages like Python to obtain answers. In the answer verification phase, they often require the LLM to observe the previous and current states to perform self-correct.
The above methods typically focus on specific parts, such as decomposition or verification. To wttpoooittooo-rsonnnnneeoo veriffcatini process more intelligent, subsequent research has
make the developed various automated agent frameworks. StructGPT (Jiang et al. 2023) provided a unified framework for LLMs to conduct QA on structured data by invoking functions and serializing re- sponses. However, its effectiveness
limited by the absence of integrated symbolic reasoning. To address this, SheetCopilot (Li et al. 2024a) and DataCopilot (Zhang et al.] 2024a), inspired by AutoGPT, proposed solutions that traditional programming finds difficult to achieve. Nonetheless, these frameworks still require rigorous evaluation across various scenarios. To adapt to different contexts, LLu et 2024a) explored table structures and introduced an agent for flexible reasoning.
Although Agent mode can handle TableQA tasks, it primarily relies on the LLM's self- decomposition and self-correct. The LLM's limited understanding of complex queries often leads to the omission of key information, resulting in a semantic gap between sub-questions and the orig- inal question. Additionally, relying on the LLM to automatically self-correct makes it difficult to detect errors in the answers. Therefore, we need more comprehensive decomposition approach to narrow the gap between sub-questions and the original question, while also facilitating the correct verification of sub-question answers.
To address this, we leverage the structured nature of triples to assist LLMs in TableQA decompo- sition and verification. By extracting many entities and relations possible from the question to form triples, we can capture more critical information, reducing the semantic gap. Additionally, the clear structure of triples allows for easier verification by comparing the consistency between the triples formed during question decomposition and those generated from the answers. This enables more straightffrward and accurate answer verification.
3
TASK DEFINITION
3.1 TABLEQA
TableQA aims infer the answer a question based on table content. Given question Q and a table context C, the goal of TableQA is to determine the final answer A A This task can be formally represented by Eq. where Ci structured data composed of rows and columns. Currently, most TableQA methods utilizing LLMs follow three phases: question O decomposition into sub- questions, reasoning based on table content C, and verification to obtain the final answer Afinat In our approach, we erialize the table content C. by separating each cell with a 'I', resulting in the format: valuey [uduuy/waluee) (...uuue)..///uue,, Here, Q is a natural language question, and Afinal can be calculated number, a specific value from the table, or any other possible outcome, as seen in the WikiTableQuestionns dataset (Pasupat & Liang 2015). Additionally, there a specific type of TableQA known as fact verification, where @ is a natural language statement about the table's 's content, and A Ainat indicates whether the statement is correct, with possible values of Yes or No, as shown in the TabFact dataset (Chen et al. 2020).
4

---

Published as a conference paper at ICLR 2025
LLM(Q,C) + Afsnat
(I)
3.2 TRIPLE
A triple is commonly used in graph-related tasks and typically takes the form of Eq.2. Here, the head and tail are entities, while the relation represents the connection between them. When the structure is correct and link exists, knowing any two parts of the triple allows for the inference of the unknown part based on the graph's 's links, For example, if the head entity and the relation are known, the unknown tail entity can be inferred by traversing the graph.
th triple (head entity, relation, tail entity) + (e.1 Tay ex)
In our approach, we leverage the structured nature of triples in both the question decomposition and answer verification. The fixed structure of triples encourages the LLM extract as many entities and relations from the question as possible, thereby covering more critical information. Therefore the sub-questions derived from these triples are better aligned with the original question, reducing the semantic gap. Additionally, the structured format of triples makes answer verification more easier. By comparing the entities and relations between the triples generated from the answer and those extracted during question decomposition, verification becomes more straightforward and reliable.
4 METHOD
Overview, Our method, illustrated in Figure 2, consists of four components: TIDE- DDDE-Decomoositonn DP/Agent Reasoning, TIDE- Verification, and Joint Reasoning. This aligns with the decomposition reasoning- verification process found in most approaches. We utilize triples to assist the LLM in both decomposition and verification, leading the development of the Deooppocconooiiionn and TIDE- Verification modules. the sub-question reasoning phase, we integrate both Direct Prompting and Agent modes to get the answers of the sub-questions. Ultimately in the joint reasoning phase, we apply these two modes answers for joint reasoning.
4.1 QUESTION DECOMPOSITION
We leverage triples to narrow the gap between sub-questions and original question. As shown in Figure 2 TIDE-Decomposition, we first utilize the LLM to extract many entities and relations from the question as possible form triples. Compared to direct decomposition, these triples capture more critical information. Consequently, transforming these triples into natural language sub-questions helps to bridge the semantic gap with the original query. The process in Eq.3-4.
(L, S,Q,,,,A Promptrriptee generate) + T
(3)
Vi (I,.... T. (c ( ET. LT. Promppddccmppsstiion S
(4)
Here, S (S S. ................................ Sx) is the set of sub-questions, S, denotes the i-th of the sub-questions where T. is the i-th triple: T.0 (e r: eg)
set. 9 (T ,...,, I, TT)
As illustrated in Figure 2, we first utilize the LLM to identify the entities and relations within the question. For the example shown, identify the explicit entities: people, office, and 3 years, along with the relations: stay and at least 3 years. Additionally. certain words that require prediction serve as implicit entities; for instance, how many can be transformed into the entity number for prediction, while people acts as the entity to be predicted. After combining all identified entities and relations into triples, we utilize the LLM to convert these triples into natural language sub-questions, facilitating further inference. For example, we form the triple (S, stay in, office), leading to Sub- question 1: "Who stayed the office?" Next, we construct the triple (S, stay more than, years) to generate Sub-question 2: "Who stayed for more than 3 years?" Finally, to align with the original question, we create the triple (number, stay at least 3 years in office, S), resulting in Sub-question 3: "What's the total number of people who stayed at least years in the office?"
5

---

Published as a conference paper at ICLR 2025
By using these sub-questions, aim to cover much critical information from the original ques- tion as possible. For each question, we utilize the LLM to extract entities and relations, construct triples, and transform them into natural language sub- sub-questions part of the decomposition process. This decomposition breaks down the question step-by-step. allowing the L.M to refine its reasoning process and enhance accuracy.
Question: How cuury people stayed at least years in office?
Earre thap
May
yeart office
From sibwerd
por
Triples
Triples
Verification
stayaed
years?
what's the mital number people stayed least 3 years office?
Diroct Promotting 61)
Agent (i)
Bertirnn stayed
El del geLnnmeettlle
sillaa Bertarin staped more than years
B del get, name searrs (ook office oeleel
Seibe number
lead
Azawear
B del sintname Syans))
DP Agent Ressoning
Decomposition
Joint Reasoning
Answer: William, Alexander, Seaieer,,eeeee Peter and Benjamin stay time more than 3 years. So the final answer is
Figure 2: Question Decomposition: Our method first extracts entities and relations from the question to form triples, then divides them into sub-questions based these triples. DP/Agent Reasoning: Next, we use either the DP or Agent mode to perform reasoning on the sub-questions. Answer Verification: In the verification phase, the inferred answers verified based on the structure of the triples. Joint Reasoning: Finally, answers are combined using majority voting for joint reasoning.
4.2 DP/AGENT REASONING
Both the DP method, which utilizes CoT-related prompts, and the Agent method, which employs external tools, have proven effective for TableQA tasks. After TIDE-Decomposition, sub-questions that are processed separately through the DP and Agent approaches for inference.
obtain
In the DP mode, prompt the LLM infer answers for each sub-question in conjunction with the table content, resulting in intermediate answers, as shown in the Direct Prompting section of Figure 2. For the Agent method, we configure the LLM interact with Python functions for each sub-question. Initially, it prompts the LLM to observe the current state and, using the table content, generates Python code or other symbolic languages to obtain intermediate answers, as depicted in the Agent section of Figure 2. The process can be represented as Eq.5.
Vi kk. LSM Prompt APP, LLM(S,, Prompt Agent) A Agent
Here, DP (APP... A.....APP.... ..... APP) is the set of DP sub-answers, APP denotes the i-th of the DP sub-answers set. A {A, Agent A Agent ...., A Ak Agentt is the set of Agent sub-
A
answers, A, Agent denotes the i-th of the Agent sub-answers set.
4.3 ANSWER VERIFICATION
To mitigate the tendency of LLM self-correct towards contradictory responses, do not directly use LLM for answer verification. Instead, we provide the triples from the question decomposition as examples, prompting the LLM to generate triples for the answers as shown in Eq.6. By comparing the relationships the two triples and ensuring that one entity (either the head or tail entity) matches, validate the correctness of the answer. The process be represented as 7-17-10.

---

Published as a conference paper at ICLR 2025
Vi E .,,...,,), LLM(A Prompt answer triple) T.P
(6)
LAM(A, Agent Prompt triple) - T.Agent
Here, TDP
TDP). }. is the set of sub-answer triples. The i- {TPP... T.PP. .....IT
DP triple T.P = {e : 2 ) E {l....,k}. Moreover, T'
(T,ARent 1.... T.Agent y..., Tk
Tdgeet i. the set of sub-answer triples. The i-th Agent triple T, T.Agent
Agent Agent Agent
222 } (1.....k)) k).
APP LS, promptpp)
e = =e e 6,21 r!p ri
W (L,...,k), k).
A
otherwise.
vi E {1..... k).
Agent
if Agent eil otherwise.
= =e Agent
Q 6/2 7i
Agent
=r (8)
A Agent
e,2
(S, Prompt Agent)
APP
if E (1,.....),.4..
e eP ei1
DP
6,2;
Q
DP
=r:
Final
(Sk, Promptpp))
otherwise.
(9)
Agent
if (
A Agent
Q Agent e 2 622 r
Agent
Agent ri (10)
fnnt
(S, Prompt PromptAgent) Agent)
otherwise.
For example, for the intermediate answer: William McCreery, Alexander McKim, William Pinkney, Peter Little, Benjamin Howard stayed in office, transforms this into the triples: (William McCreery, stay in, office), (Alexander McKim, stay in, office), (William Pinkney, stay office), (Peter Little, stay in, office), and (Benjamin Howard, stay in, office). Comparing with triple (S, stay in, office) which generate in question, we find both the relation and tail entity match, confirming correctness.
However, when the converted triples do not align with the initial triples, it indicates an error in the intermediate answer, prompting LLM to regenerate the answer. For instance, the transformed triple (William Pinkney, stay less than, years) compared with the triple ($, stay more than, 3 years) shows the same tail entity but different relation. Therefore, this sub-question must be re-answered. Once all intermediate answers have been verified as correct, we can derive the final answer.
4.4 JOINT REASONING
After reasoning and verifying both the DP and Agent modes, we obtain the final answers for each mode that pass verification. To mitigate errors from single-pass LLM reasoning, we use majority voting to combine the results. For flexibility, we first generate five answers for each mode and then randomly select specified number based set hyperparameter. Further results and analysis of different selection combinations are provided in Section Eq.11 Afsmat is the final answer.
Afinal = Majority Vote(ADP finals Agent
(11)
EXPERIMENT
5.1 DATASETS AND EVALUATION
Dataset. We used two widely recognized TableQA datasets: WikiTableQuestionss (Pasupat & Liang 2015) and FabFact (Chen al. 2020). WikiTaalleQuestions is commonly used dataset for complex

---

Published as a conference paper at ICLR 2025
TableQA tasks, involving operations such aggregation, comparison, and arithmetic calculations. Our method was evaluated on its test set, containing 4,344 table-related questions. The TabFact dataset focuses on fact verification in TableQA, determining whether a statement is correct based on the table's content, with answers being yes or no. TabFact test set includes 2,024 samples.
Evaluation. Following previous work (Cheng et al. 2023 Liu et al. 2024a),
exact match accuracy as the evaluation metric, comparing whether the final answer matches the gold answer.
5.2 IMPLEMENTATION DETAILS
We followed previous work (Cheng et al. 2023 Ye et al. 2023b Liu ct al. 2024a and used GPT-3.5 as the LLM, with temperature of 0.8. To reduce errors from single-pass reasoning, we generated five answers for both the DP and Agent. The combination of answer selections is detailed in Section 5.5. The specific prompts used for the DP and Agent methods are provided in the Appendix I.
5.3 BASELINES
For LLM-based TableQA, we first directly compared several language models, including SASP (Ou
& Liu 2022), TAPAS-large (Eisenschlos et al.) 2020). T5-3B (Xie et al.) 2022). TAPEX-large (Liu
ct al. 2021), ) and Codex (Cheng et al. 2023). Additionally, to better evaluate TIDE in DP and Agent,
selected methods under both frameworks, including BINDER (Cheng et al. 2023). ) DATER (Ye
et al. 2023b), StructGPT (Jiang et al. 2023). DTE (Wang et al. 2023). TACR (Wu al.
2023), ITR
(Lin et al.) 2023), (Liu al. (2024b). ToT (Xiao al. 2024), -HAIN--FFTABLE
Wang al.
2024). ReAcTable (Zhang et al. 2024c), Cabinet (Patnaik et al. 2022), and (Liu et al.
2024a). For
detailed descriptions of the baseline method, please refer Appendix H.
5.4
MAIN RESULTS
Table presents the results on the WikiTaalleQessions. Both standalone DP and agent deliver strong results. Notably, when our method combines both modes, it achieves SOTA, surpassing all other methods by 1.0. Table shows the results on the TabFact. Similarly, our method effective in both standalone DP and agent, outperforming most approaches. The combination of our method with both modes achieves SOTA performance, improving over all methods by about 1%- 16%. Since best performance on TabFact is already high, the gains of 1.32% should be interpreted as a proportion of scope further improvement possible, 1.32/ (.02/(100- 88.5), which is R 11.48%.
TIDE's success largely stems from its effective guidance during decomposition and verification. In decomposition, we identify all entities and relations in the question, forming triples that ensure sub- questions align with the original semantics. This step-by-step reasoning mitigates errors from direct answering. During verification, explicit standards enable the L.M to make consistent judgments, reducing errors each step and ensuring accurate final inferences. By seamlessly integrating DP and Agent approaches, our method leverages joint reasoning across both modes, minimizing biases from single-pass predictions.
Additionally, TIDE-Agent consistently outperforms TIDE-DP on both datasets, primarily due to two factors. First, the agent generates Python code interact with tables, making better suited for handling structured tabular data. Second, its global observation capability allows it to select the next action based on the current state, enabling more accurate global reasoning.
5.5 ABLATION STUDY
To examine TIDE's impact during the decomposition and verification phases both DP and Agent modes, conducted ablation experiments the WikiTableeQestions and TabFact datasets. We evaluated the effect of using only TIDE's decomposition and only TIDE's verification in both modes, with results shown in Table 3.
As Table illustrates, TIDE's decomposition has a significant impact on accuracy. This is primarily due the pipeline's 's sequential nature: first, the question is decomposed, then responses vali- dated. the decomposition is incorrect leading to erroneous sub-questions or failing to capture the original question's semantics the final answer will be incorrect, even if intermediate responses
8

---

Published as a conference paper at ICLR 2025
Table 1: Results on WikiTableeuestionn. TIDE-DP and TIDE-Agent show results for DP and Agent modes separately. TIDE- DP&Agent shows the joint reasoning result.
Table Results on TabFact dataset. TIDE- DP and TIDE-Agent show results for DP and Agent modes separately. &ggen shows the joint reasoning result.
Method
Acc.
Method
Acc.
TAPEX-large (Lu et al.
2021
59.10
TAPAS-large [Eiensshhlss al.
2020
81.00
T5-3B (Xie al. 2022
50.60
TAPEX-large [Lu et al. 2021)
84.20
BINDER (Cheng et al.
2023
64.60
SASP OOu & Liu
2022)
77.00
DATER Ye et al 2023b)
65.90
T5-3B (Xie et al.
2022
83.68
StructGPT Jiang et al. 2023)
57.00
Codex end-to-end (Cheng al.
2023
72.60
DTE Wang et al. 2023
54.20
Codex SQL (Cheng et al.
2023
80.70
TACR (Wu et 2023)
60.20
BINDER (Cheng al. 2023
85.10
ITR (Lin et al. 2023)
63.40
DATER Ye et al. 2023bb
85.60
Liu ct al 20246)
55.80
StructGPI (Jang al. 2023)
87.30
CHAIN-OP FFMIN--F-TABE TABLE WWang et al.
2024)
59.94
CHAIN-OFTABLE
Wang et al
2024
80.20
ReAcTable (Zhang et 2024c
68.00
ReAcTable (Zhang et al. 2024c)
86.10
Cabinet (Patnark et al. 2024)
69.10
Tab-PoT (Xiao et al. 2024)
85.77
(Liu et al.
73.65
(Liu et al.
88.50
TIDE-DP
66.51
TIDE-DP
81.32
TIDE-Agent
68.72
TIDE-Agent
88. 19
TIDE-DP&Agent
75.00
TIDE-DP&Agent
89.82
accurate. Conversely, when decomposition is correct, the LLM's inherent reasoning ability still maintain a certain level of accuracy in intermediate responses, even without explicit verification.
Table 3:
Ablation results on WikiTableQuestions (WTQ) and TabFact.
Table 4: Impact of answer selection in TIDE-DP and TIDE-Agent.
Method
WTQ
TabFact
Agent
DP
WTQ
TabFact
TIDE-DP
66.51
81.32
1
65.06
81.52
w/o Decomposition 60.33 (L 6.18) 77.22 22 (L 4.10)
3
3
73.73
89.33
w/o verification
64.89 (L 1.62)
79.64
1.68)
5
5
75.00
89.82
TIDE-Agent
68.72
88.19
3
67.11
81.99
w/o Decomposition
61.21 (L 7.51)
81.27 (1 6.92)
3
70.28
83.27
w/o verification
67.12 (4 1.60)
86.86 1.33)
5
69.38
82.16
5
1
71.04
87.35
TIDE-DP&Agen
75.00
89.82
w/o Decomposition
68.61 (1 6.39)
85.47 (L 4.35)
3
5
73.02
84.98
w/o verification
71.22 (1 3.78) 87.06 (4 1.76)
5
3
74.15
89.23
5.6 PERFORMANCE ANALYSIS UNDER DIFFERENT FACTORS
Number of Answer Selections in TIDE-DP and TIDE-Agent. Our method can be applied inde- pendently in either DP or Agent, or used for joint reasoning across both modes. To examine the effect of different selection combinations, Table 4 presents the results. For single mode, increasing the number of selected answers improves performance, suggesting that majority voting mitigates bias from single-pass L.MM reasoning. In joint mode, performance is better when the number of Agent-generated answers exceeds DP-generated answers, indicating that the agent's results are more reliable and its reasoning capability is stronger, consistent with our analysis in Section 5.4.
Number of Triples Decomposed in TableQA. Generally, more triples indicate more complex question, requiring additional inference steps. To evaluate our method's adaptability to complex- ity and decomposition efficiency, report the frequency and accuracy of decomposed triples in both datasets. As shown in Figure 3. in WTQ, inference accuracy declines as question complex- ity increases. We speculate this because complex questions require more evidence, straining the

---

Published as a conference paper at ICLR 2025
LLM's ability to reason over longer texts. In contrast, TabFact exhibits fluctuating accuracy. Ex- amining its decomposition, we found that due to limited prompt examples, TabFact questions are broken down more granularly than in WTQ. This sometimes leads to overly simple questions being split into multiple sub-questions, increasing their count. Any error in inferring a sub-question can propagate, resulting in accuracy fluctuations.
TasFad
-O- V77
TabFac
100
40
40
20
2
Number of Eacompuoiiio Sub-cusstors
Nf Decomppsiiionn
Number of Trigles
Number of Triles
Subvqaadtonn
Figure 3: Number of Triples Decomposed in WTQ and TabFact.
Figure 4: Number of Sub-questions Decom- posed in WTQ and TabFact.
Number of Sub-questions Decomposed in TableQA. We further analyze the sub-questions derived from triples. To evaluate TIDE' across both datasets, we report accuracy for different numbers of decomposed sub-questions. As shown in Figure 4, the number of sub-questions varies, with the LLM automatically breaking complex triples into additional sub-questions during inference. This is especially noticeable in the Agent. While the number of triples reflects the problem's complexity, the number of sub-questions indicates reasoning difficulty. In Figure 4, aside from fluctuations at the 4-step decomposition in WTQ, the overall trend aligns with the decline seen Figure 3 for WTQ. Similarly, TabFact shows notable fluctuations, supporting our previous hypothesis.
TIDE-DP aTDE-Agent
TIDE-DP TIDE-Agant
120
100
50
80
Ascuracy
80
Accuracy
60
20
8.
RR15)
(12 16)
t Mgj
(15,616)
re)
(18,21)
Gat .ag
(36.519)
(16
WTQ Row Count
Count
Figure 5: Impact of table size TableQA performance.
Table Size in TableQA. To assess the inference capabilities of TIDE-DP and TIDE-Agent across different table sizes, we categorized tables based on previous work. Each interval contained approxi- mately 430 data points, and we calculated average accuracy within these ranges. As shown in Figure 5. TIDE-Agent generally outperforms TIDE-DP across both datasets, consistent with our analysis in Section 5.4. With its ability to process tables via code and observe data globally, TIDE-Agent proves more effective in most cases. In WTQ, we observe that table size increases, the performance gap between TIDE-Agent and TIDE-DP widens. Larger tables result in longer serialized text, making inference harder for TIDE-DP, whereas TIDE-Agent efficiently extracts data through code. This highlights the need for future research to focus on handling larger tables more effectively.
CONCLUSION
To address issues like overlooked information and contradictory self-corrections, we propose TIDE, which leverages the structured nature of triples. The fixed structure ensures LLMs extract more entities and relations, covering critical information. It also simplifies answer verification by directly comparing decomposed triples with those from the answer.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENTS
Our work is supported by the National Natural Science Foundation of China (62476003), Anhui Province Excellent Scientific Research and Innovation Team (2024AH010004), Anhui Provincial Natural Science Foundation Water Science Joint Fund (2408055US006), the University Synergy Innovation Program of Anhui Province (GXXT-2023-050) and SMP-Zhipu.. Large Model Cross- Disciplinary Fund (SMP-ZE) JMMP-Zhipu202402111 University Joint Research Center, and the High-Performance Computing Platform of Anhui Univer- sity.
We also acknowledge the support from Zhipu AI-Anhui
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamerr. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
Yihan Cao, Shuyi Chen, Ryan Liu, Zhiruo Wang, and Daniel Fried. Api-assisted code generation for question answering on varied table structures. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 14536-14544 2023.
Saneem Chemmengath, Vishwajeet Kumar, Samarth Bharadwaj. Jaydeep Sen, Mustafa Canim, Soumen Chakrabarti, Alfio Gliozzo, and Karthik Sankaranarayanan. Topic transferable table question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 4159 4172, 2021.
Wenhu Chen. Large language models are few (1) (1)-shot table reasoners. In Findings of the Association for Computational Linguistics: EACL 2023, pp. 1 120-11330 2023.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: large-scale dataset for table-based fact verification. In The 8th International Conference Learning Representations, 2020.
Yongrui Chen, Shenyu Zhang, Guilin Qi, and Xinnan Guo. Parameterizing context: Unleashing the power of parameter-efficien fine-tune tuning and in-context tuning for continual table semantic parsing. Advances in Neural Information P Processing Systems, 36, 2024.
Sitao Cheng, Ziyuan Zhuang, Yong Xu, Fangkai Yang, Chaoyun Zhang, Xiaoting Qin, Xiang Huang, Ling Chen, Qingwei Lin, Dongmei Zhang, et al. Call me when necessary: Lims can efficiently and faithfully reason structured environments. arXiv preprint arXiv:2403.cm 085593 2024.
Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, et al. Binding language models symbolic languages. In The 1/th International Conference on Learning Representations, 2023.
Naihao Deng, Zhenjie Sun, Ruiqi He, Aman Sikka, Yulong Chen, Lin Ma, Yue Zhang, and Rada Mihalcea. Tables as images? exploring the strengths and limitations of Ilms on multimodal representations of tabular data. arXiv preprint a0Xiv:2402. 12424, 2024.
Julian Eisenschlos, Syrine Krichene, and Thomas Mueller. Understanding tables with intermediate pre-training. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp- 281-296. 2020.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity- prompting for multi-step reasoning. In The 11th International Conference on Learning Representations, 2022.
Carlos Gemmell and Jeff Dalton. Toolwriter: Question specific tool synthesis for tabular data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp- 16137-16148, 2023.
11

---

Published as a conference paper at ICLR 2025
Heng Gong, Yawei Sun, Xiaocheng Feng, Bing Qin, Wei Bi, Xiaojiang Liu, and Ting Liu. Tablegpt: Few-shot table-to-text generation with table structure reconstruction and content matching. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 1988, 2020.
Che Guan, Mengyu Huang, and Peng Zhang. Mfo Multi-hop few-shot open rich table question answering. arXiv preprint arXiv:2403.1 1100.191666. 2024.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In The 12th Interna- tional Conference on Learning Representations, 2024.
Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Wayne Xin Zhao, and Ji- Ji-Rong Wen. Structgpt: A general framework for large language model to reason over structured data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9237- 9237-9251, 2023.
Zhengbao Jiang, Yi Mao, Pengcheng He, Graham Neubig, and Weizhu Chen. Omnitab: Pretraining with natural and synthetic data for few-shot table-based question answering. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 932 9429942,47 2022.
Nengzheng Jin, Dongfang Li, Junying Chen, Joanna Siebert, and Qingcai Chen. Enhancing open- domain table question answering via syntax-and structure-aware dense retrieval. In Proceedings of the 13th International Joint Conference Natural Language Processing and the 3rd Con- ference of the Pac--ae Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 157-165, 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems, 35:22199-22213. 2022.
Kezhi Kong, Jiani Zhang, Zhengyuan Shen, Balasubramaniam Srinivasan, Chuan Lei, Christos Faloutsos, Huzefa Rangwala, and George Karypis. Opentab: Advancing large language models as open-domain table reasoners. The 12th International Conference on Learning Representations, 2024.
Kang-il Lee, Segwang Kim, and Kyomin Jung. Weakly supervised semantic parsing with execution- based spurious program filtering. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 6884-6894, 2023.
Younghun Lee, Sungchul Kim, Ryan A Rossi, Tong Yu, and Xiang Chen. Learning reduce: Towards improving performance of large language models on structured data. In First Workshop on Long-Conter Foundation Models@ ICMI. 2024, 2024.
Fangyu Lei, Tongxu I Luo, Pengqi Yang, Weihao Liu, Hanwen Liu, Jiahe Lei, Yiming Huang, Yifan Wei, Shizhu Hc, Jun Zhao, et al. Tableqakit: A comprehensive and practical toolkit for table-based question answering. arXiv preprint 11Xi::23100.0055 2023.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag. Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Hongxin Li, Jingran Su, Yuntao Chen, Qing Li, and ZHAO-XIANG ZHANG. Sheetcopilot: Bring- ing software productivity to the next level through large language models. Advances in Neural Information Processing Systems, 36, 2024a.
Jinyang Li, Binyuan Hui, Reynold Cheng. Bowen Qin, Chenhao Ma, Nan Huo, Fei Huang, Wenyu Du, Luo Si, and Yongbin Li. Graphix-t5: Mixing pre-trained transformers with graph-aware layers for text-to-sql parsing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 13076- 1776--1084 13084, 2023.
12

---

Published as a conference paper at ICLR 2025
Peng Li, Yeye He, Dror Yashar, Weiwei Cui, Song Ge, Haidong Zhang, Danielle Rifinski Fainman, Dongmei Zhang, and Surajit Chaudhuri. Table-gpt: Table tunet gpt for diverse table tasks. Proceedings of the ACM Management of Data, 2(3):1-28, 2024b.
Weizhe Lin, Rexhina Blloshmi, Bill Byrne, Adrià de Gispert, and Gonzalo Iglesias. An inner table retriever for robust table question answering. In Proceedings of the 61 Anmual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pp. 9909-9926, 2023.
Jiacheng Liu, Skyler Hallinan, Ximing Lu, Pengfei He, Sean Welleck, Hannaneh Hajishirzi, and Yejin Choi. Rainier: Reinforced knowledge introspector for commonsense question answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 8938-8958, 2022.
Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. Tapex: Table pre-training via learning neural sql executor. arXiv preprint arXiv:2107.0763: 2021.
Tianyang Liu, Fei Wang, and Muhao Chen. Rethinking tabular data understanding with large lan- guage models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 450 482, 2024a.
Yujian Liu, Jiabao Ji, Tong Yu, Ryan Rossi, Sungchul Kim, Handong Zhao, Ritwik Sinha, Yang Zhang, and Shiyu Chang, Augment before you try: nholeedge-enhanced table question answer- ing via table expansion. arXiv preprint arXiv:2401. /5555, 2024b.
Weizheng Lu, Jiaming Zhang, Jing Zhang, and Yueguo Chen. Large language model for table processing: A survey, arXiv preprint 202.02240..05121, 2024.
Tongxu Luo, Fangyu Lei, Jiahe Lei, Weihao Liu, Shihu He, Jun Zhao, and Kang Liu. Hrot: Hybrid prompt strategy and retrieval of thought for table-text hybrid question answering. arXiv preprint arXiv:2309. 12669, 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
Raphaël Mouravieff, Benjamin Piwowarkki, and Sylvain Lamprier. Training table question answer- ing via sql query decomposition. arXiv preprint arXiv:2402.13288. 2024.
Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang. and Xi Victoria Lin. Lever: Learning to verify language-to-code generation with execution. In International Conference on Machine Learning, pp. 26106 26128. PMLR, 2023.
Suixin Ou and Yongmei Liu. Learning to generate programs for table fact verification via structure- aware semantic parsing. In Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Pupers), pp. 7624-7638, 2022.
Vaishali Pal, Andrew Yates, Evangelos Kanoulas, and Maarten de Rijke. Multitabqa: Generating
tabular answers for multi-table question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 6322-6334, 2023.
Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1470-1480, 2015.
Sohan Patnaik, Heril Changwal, Milan Aggarwal, Sumit Bhatia, Yaman Kumar, and Balaji Krishna- murthy. Cabinet: Content relevance-ber based noise reduction for table question answering. In The 12th International Conference on Learning Representations, 2024.
Soumajyoti Sarkar and Leonard Lausen. Testing the limits of unified sequence to sequence Ilm pretraining on diverse table data tasks. In NeurIPS 2023 Second Table Representation Learning Workshop, 2023.
13

---

Published as a conference paper at ICLR 2025
Kaya Stechly, Matthew Marquez, and Subbarao Kambhampati. Gpt-4 doesn't ' know it's wrong. An analysis of iterative prompting for reasoning problems. In NeurlPS 2023 Foundation Models for Decision Making Workshop, 2023.
Yuan Sui, Jiaru Zou, Mengyu Zhou, Xinyi He, Lun Du, Shi Han, and Dongmei Zhang. Tap4Ilm: Table provider sampling, augmenting, and packing semi stmitstruttured data for large language model reasoning. arXiv preprint arXiv:2312.0903,, 2023.
Karthik Valmeekam, Matthew Marquez, and Subbarao Kambhampati. Can large language models really improve by self-critiquing their own plans? In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023.
Bailin Wang, Ivan Titov, and Mirella Lapata. Learning semantic parsers from denotations with latent structured alignments and abstract programs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3774-3785, 2019.
Bing Wang, Yan Gao, Zhoujun Li, and Jian- GuanG Lou. Know what i don't know: Handling am- biguous and unknown questions for text-to-sql. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 5701-5714, 2023.
Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Miculicich, Yasuhisa Fujii, Jingbo Shang, Chen- Buen-Yu Lee, et al. Chain-of-table: Evolving tables in the reasoning chain for table understanding. In The 12th International Conference Learning Representations, 2024.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Ohain-of--though prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837. 2022.
Jian Wu, Yicheng Xu, Yan Gao, Jian HannGuang Lou, Börje Karlsson, and Manabu Okumura. Tacr: A table-alignment- based cell-selection and reasoning model for hybrid preprint arXiv:2305. 14682, 2023.
arXiv
Jian Wu, Yicheng Xu, Börje F Karlsson, and Manabu Okumura. A table question alignment based cell-selection method for table-text qa. Journal of Natural Language Processing, (1(1):189 211, 2024.
Renqiu Xia, Bo Zhang, Haoyang Peng, Ning Liao, Peng Ye, Botian Shi, Junchi Yan, and Yu Qiao. Structchart: Perception, structuring, reasoning for visual chart understanding. arXiv preprint arXiv:2309.11266 2023.
Bin Xiao, Burak Kantarci, Jiawen Kang, Dusit Niyato, and Mohsen Guizani. Efficient prompting for Ilm-based generative internet of things. arXiv preprint arXiv:2406. 10382, 2024.
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien- Sheng Wu, Ming Zhong, Pengcheng Yin, Sida Wang, al. Unifiedskg: Unifying and multi- tasking structured knowledge grounding with text-to-text language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 602-631, 2022.
Bohao Yang, Chen Tang, Kun Zhao, Chenghao Xiao, and Chenghua Lin. Effective distillation of table-based reasoning ability from 1lms. In Proceedings of the 2024 Joint International Confer- ence on Computational Linguistics, Language Resources and Evaluation (Lronnn(RReee pp. 5538-5550, 2024.
2024),
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, al. Ureader: Universal ocr-free risually-sitatte language understanding with multimodal large language model. In Findings of the Association for Computational Lin- guistics: EMNLP 2023, pp. 2841-2858. 2023a.
Junyi Ye, Mengnan Du and Guiling Wang. Dataframe qa: A universal Ilm framework on dataframe question answering without data exposure. arXiv preprint orXiv:2401.154pp 2024.
14

---

Published as a conference paper at ICLR 2025
Yunhu Ye, Binyuan Hui, Min Yang. Binhua Li, Fei Huang, and Yongbin Li. Large language models are versatile decomposers: Decomposing evidence and questions for table-based reasoning. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 174-184, 2023b.
Le Zhang, Yihong Wu, Fengran Mo, Jian-Yun Nie, and Aishwarya Agrawal. Moqagpt: Zero-shot multi-modal open-domain question answering with large language model. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1195-1210, 2023a.
Wenqi Zhang, Yongliang Shen, Weiming Lu, and Yueting Zhuang. Data-copilot: Bridging billions of data and humans with autonomous workflow. ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024a.
Xuanliang Zhang, Dingzirui Wang, Longxu Dou, Qingfu Zhu, and Wanxiang Che. A survey of table reasoning with large language models. arXiv preprini arXiv:2402.08259 2024b.
Yunjia Zhang, Jordan Henkel, Avrilia Floratou, Joyce Cahoon, Shaleen Deep, and Jignesh M Patel. Reactable: Enhancing react for table question answering. Proceedings of the VLDB Endowment, 17(8):1981-1994 2024c.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. In The 11th International Conference on Learning Representations, 2023b.
Bowen Zhao, Changkai Ji, Yuejie Zhang, Wen He, Yingwen Wang, Qing Wang, Rui Feng, and Xiaobo Zhang. Large language models are complex table parsers. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 14802, 2023a.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint ar Xrvi::2303. 18.223333.,2,,, 2023b.
Wenting Zhao, Ye Liu, Yao Wan, Yibo Wang. Zhongfen Deng, and Yu Philip. Localize, retrieve and fuse: A generalized framework for free-form question answering over tables. In Findings of the Association for Computational Linguistics: IJCNLP-AACI, 2023 (Findings), pp. 1-12, 2023c.
Yilun Zhao, Linyong Nan, Zhenting Qi, Rui Zhang, and Dragomir Radev. Reastap: Injecting table reasoning skills during pre-training via synthetic reasoning examples. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9006-9018, 2022.
Yilun Zhao, Haowei Zhang, Shengyun Si, Linyong Nan, Xiangru Tang, and Arman Cohan. Inves- tigating suble-to-text generation capabilities of llms in real-world information seeking scenarios. arXiv preprint 2rXiv:2305.14987, 2023d.
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint 17Xiv:7709.00103, 2017.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
A ABLATED IMPLEMENT
This section mainly provides a formal description of the implementation details of the ablation experiments.
1) w/o DiDo-eccomooiition
This method is similar to Chain of Thought (CoT) ct al. 2022). where the prompt think step by step and few examples used. Sub-questions Decomposition (S1 ....... Sk} can be represented as the equation.
LL, C, Promplcor) + S
2) w/o TIDE-Verification
This approach resembles providing any explicit standards. Instead. the LLM is prompted to rely on oef-correct et al., 2024) using the instruction: Review your previous answer and find problems with your answer. Based the problems you found, improve your answer. Please reiterate your answer and few examples. The Sub-answers A, Verification can be represented as the equation.
[A (S, Promptself)
if (LM otherwise.
Prompteel,) PAompppeeel
True,
4;
B OTHER LLMS
1) Open-source and Closed-source LLM Models
Table 5 results demonstrate that our method improves the performance of multiple open-source and closed-source models, validating its effectiveness.
Model
Acc.
Llama 2-7b
42.23
Llama 2-7b + Ours
45.00 (2.77 1)
Llama 3-70b
53.25
Llama 3-70b Ours
58.64 (5.39 1))
GLM
60.17
GLM 4+ Ours
64.00 (3.83 T)
Gemini 1.5
54.23
Gemini 1.5 + Ours
59.84 (5.61 ))
Claude 3.5
70.38
Claude 3.5 + Ours
75.60 (5.22 T)
Table 5: Comparison of models with and without our method
Size
Acc.
Llama 2-7b + Ours
45.00
Llama 3-8b Ours
50.27
Llama 3-70b + Ours 58.64
Table 6: Model Size and Accuracy Comparison
16

---

Published as a conference paper at ICLR 2025
2) Model Size Affect
We conducted verification on different sizes of the open-source Llama model, and the results are as in Table 6. The results indicate that the model size has significant impact performance. Larger models consistently achieve better results, demonstrating improved effectiveness.
3) Non-LLM methods
Some baseline methods our paper are non-LLM-basee approaches. Table comparison shows substantial performance gap, with our method achieving improvements ranging from 11.7% to 30.5%. This demonstrates the effectiveness our approach.
Model
Acc.
Structured Attention (Wang al.
2019
44.50
ReasTAP-Larre (Zhao et al. 2022)
58.70
TAPEX-large (Liu et al. 2021
59.10
OmniTab-Large (Jiang et al. 2022)
63.30
Ours
75.00
Table 7: Comparison Non-LLM methods
4) Data Contamination
In order to mitigate the impact of LLM data contamination, we address this issue by comparing the performance of directly using the same LLM for question answering versus applying our pro- posed method in Table From the comparison result, it is evident that our method improves the model's 's accuracy by approximately 27%, demonstrating that our approach itself brings significant performance gains.
Model
Acc.
GPT Direct QA (Cheng et al. 2023
48.70
Ours
75.00
Table 8: Comparison of Direct QA.
Method
DP
Agent
Result
(Liu et al. -DP&
5
73.65
Ours
5
75.00
Table 9: Comparison of SOTA.
C VERIFICATION FAILS
This section primarily discusses cases verification failure. In our implementation, we limit the number of iterations maximum 4. Additionally, our statistics show that nearly 90% of the data reaches the final answer within single iteration. Furthermore, less than 2% of cases fail to obtain an answer after iterations, for which we default value of error. To mitigate the impact of these error cases, we follow the SOTA approach by obtaining multiple results from the DP and Agent modes, respectively, and applying majority voting. In Table we also present the results of different answer combinations based on the two modes.
D DP AND AGENT JOINT
This section primarily explores the joint effect of DP and Agent. our method, majority voting is employed for their integration. First, we compare it with methods that also utilize majority voting.
17

---

Published as a conference paper at ICLR 2025
Method
Result
ReAcTable Zhang et al. 2024c
68.00
Ours
75.00
Table 10: Comparison with majority voting method.
as shown in Table 10, where can be observed that combining DP and Agent indeed achieves excellent performance. Subsequently, we compare it with the results of using DP Agent alone in SOTA methods, as presented in Table 11.
Our method follows the SOTA approach, which combines DP and Agent. We introduced triples building upon the SOTA framework. As shown in the table 13, while the performance in the DP mode roughly the same, significant improvements were observed in the Agent and joint modes. This demonstrates the effectiveness of incorporating triples.
Method
Result
Liu et
2024a [6]
66.99
Durs-DP
66.51 (0.48), almost same)
Liu et al.
1124a--Agen [6]
63.77
Ours-Agent
68.72 (4.951)
Liu et
2024a DP&Agent [6]
73.65
Durs-DP&Agenn
75.00 (1.351)
Table 11: Comparison with SOTA DP and Agent.
For the comparable results in the DP mode, we speculate that this is due DP relying on text- based reasoning, which has limited understanding of the structured nature of tables. Agent, using Python-basee bashen-based methods, has a stronger understanding table structure.
E TIME AND TOKEN USED
In practical implementation, we combine the triple prompt with the decomposition prompt and inte- grate answer generation with triple verification. We analyzed the average time and token usage for the decomposition and verification LLM calls in Table 12. Besides, compared to other LLM-basee methods, approach reduces the number of API calls while achieving better results, shown in Table 13.
Operation
Time
Token
Decomposition
2.190s
Prompt length 1159 + table length question length Prompt length 970 table length question length + triple and sub-question length
Verification
3.211s
Table 12: Operation time and token details
Methods
Result
Number of API calls
(Next operation Argument 1 Transform table 1) * Iter N 3.V
CHAIN-OF-TABLE
Wang et al. 2024)
59.94
Decompose triple and sub-questions
Ours
75.00
Answer and verification 2
Table 13: Comparison of Methods with Results and API Calls
18

---

Published as a conference paper at ICLR 2025
F QUESTION TYPE
Our method does not impose limitations, as it structures questions and answers into triples, mak- ing it broadly applicable table question-answering tasks. Additionally, triples are generated specifically based on the questions and are designed to aid understanding the table context.
The WTQ dataset covers variety of operations and question types, including where, which, what, who, how many, average, and is, among others.
To illustrate this, we have provided actual results from the test dataset.
"Average" Type
Question: cycle austria's next top model, what is average of all the contestants' ages?
Triples: (in cycle of austria's next lop model, $1).
(for each contestant, what is their age).
($1, is the average of, contestants ages)
Decompose: 1. what is in cycle 4 of austria's 's next top model?
2. for each contostant, what is their age?
3. what the average of all the contestants' ages?
"Who" Type
Question:
who did the team play after the law vegas legends on november
Triples: ($1, did the team play ay, after the law vegas legends).
($1. played against, $2).
($2, played on, november 10)
Decompose: 1. who did the ream play after the law vegas legends?
when did the team play against the law vegas legends?
3. what was the date when the team played against the law vegas legends?
"Which" Type
Question:
which country had the most cyclists finish within the top 10?
Triples:
($1, had, cyclists finish within, top 10),
(the top 10, total, oyclists),
($2, country, most)
Decompose:
which country had cyclists finish within the top 10?
every country, what's the total number of eyclists who finished within the top 10?
which country had the mast cyclists finish within the top 10?
"Was" Type
Question: 18th chronologically after moggina manasu?
Triples: (18th cross. film, $1).
(moggina manasu, tilm, S7).
(18th cross, was (chronologically). after)
Decomppose: 1. what film was 18th cross?
2. what film moggina manasu?
3. was 18th cross chronclogically after maggina menasu?
Figure 6: Question type example.
19

---

Published as a conference paper at ICLR 2025
G ERROR ANALYSIS
Here, we randomly selected 100 test samples and categorized the errors for analysis.
G.1 ANSWER FORMATING
Description Losses
1939/40
1940/41
1941/42
1942/43
1943/44
1944/45
Total
Direct War Losses
360.000
183.000
Murdered
75.000
100.000
116.000
133.000
82,000
506,000
Deaths In Prisons & Camps
69.000
210.000
220,000
266.000
381,000
1.146.000
Deaths Outside of Prisons Camps
42,000
71.000
142,000
218.000
473,000
Murdered Eastern Regions
100,000
100,000
Deaths other countries
2,000
Total
504.000
352.000
407.000
541.000
681,000
270.000
2,770.000
Question:
how many people were murdered 1940/41?
Triples: ($1, were murdered in, year),
($2, number of people, $3)
Decompose: 1. who were murdered in 1940/41?
what's the number of people who were murdered in 1940/41?
Reasoning:
The people who were murdered are described in row 1 of the table.
They were murdered in various years, including 1940/41, as indicated in the table.
The number of people murdered 1940/41 100000.
Final Answer: 100000
Gold Answer: 100,000
Error Analysis 100000 and 100,000 format different
Figure 7: Answer formating emror.
20

---

Published as a conference paper at ICLR 2025
G.2
INCORRECT NUMERICAL COMPARISON
Description Losses
1939/40
1940/41
1941/42
1942/43
1943/44
1944/45
Total
Direct War Losses
360.000
183,000
543,000
Murdered
75,000
100,000
82,000
506,000
Deaths In Prisons & Camps
69.000
210,000
220.000
266,000
381,000
1.146,000
Deaths Outside of Prisons Camps
42,000
71,000
142.000
218,000
473,000
Murdered Eastern Regions
100.000
100,000
Deaths other countries
2,000
504.000
352,000
407,000
541.000
681,000
270.000
2,770,000
Question: what is the only years that total losses were below 350,000?
Triples: ($1, total losses, below 350,000),
($2, were the only. years)
Decompose: 1. what the total losses each year?
which years total below 350,000? 000?
3. what is the only years that total losses were below 350,0007
Reasoning: The total losses in each year are as follows: 1939/40: 504,000; 1940/41: 352, 1941/42:
407,000 1942/43: 541,000; 1943/44: 681,000; 1944/45: 270,000.
The years that had total losses below 350,000 is: 1940/41.
Final Answer: 1940/41
Gold Answer: 1944/45
Error Analysis: Incorrectly assumed that 352,000 350,000
Figure 8: Incorrect numerical comparison error.
21

---

Published as a conference paper at ICLR 2025
G.3
SPECIAL Row
Description Losses
1939/40
1940/41
1941/42
1942/43
1943/44
1944/45
Total
Direct War Losses
360,000
183.000
543,000
Murdered
75.000
100.000
116.000
133.000
82,000
506,000
Deaths In Prisons & Camps
69.000
210,000
220,000
266,000
381,000
1.146,000
Deaths Outside of Prisons & Camps
42,000
71,000
142,000
218,000
473,000
Murdered in Eastern Regions
100,000
100.000
Deaths other countries
2,000
Total
504,000
352.000
407.000
541.000
681.000
270,000
2.770.000
Question:
what the last description losses on this chart? ?
Triples: ($1. had, most cyclists),
($2, finish within, the top 10),
(S3, is, country)
Decompose:
who finished within the top 10?
2. what's the country for each cyclist?
3. which country had most cyclists finish within the top 10?
Reasoning: Thought: Extract the unique values from the "Description Losses" column. Select the last value to answer the second sub-que qub--uustion.
Action: python_repl..o Action Input: df[ Desserrcii josses')unique)
Observation: ("Direct War Losses' "Murdered' 'Deaths In Prisons & Camps' 'Deaths Outside
of Prisons & Camps' 'Murdered Eastern Regions' 'Deaths other countries' 'Total"]
Thought: The unique values of the "Description Losses" column ['Direct War Losses',
'Murdered; 'Deaths In Prisons & Camps', 'Deaths Outside of Prisons & Camps', 'Murdered in Eastem Regions', 'Deaths other countries', 'Total]. Select the last value.
Action: python_replase Action Input: df[ Descrrpppto Losses'] [unique[[[]]
Observation: Total
Final Answer: Total
Gold Answer:
Deaths other countries
Error Analysis : Unable to identify special rows, such total
Figure 9: Special row error.
22

---

Published as a conference paper at ICLR 2025
G.4 COUNTING
Placing 31 October 2008
Rider
Location
Country
Event
Placing
Rider
Nationality
2008-09 World Cup
Manchester
United Kingdom
Keirin
Jason Kenny
GBR
Victoria
a1 October 2008
2008-09 world Cup
Manchester
United kingdom
Sprint
GBR
Pendleton
Victoria
1 November 2008
2008-09 World Cup
Manchester
United kingdom
500 m time trial
GBR
Pendleton
November 2008
2008-09 World Cup
Manchester
United kingdom
Sprint
Jason Kenny
November 2008
2008-09 World Cup
Manchester
United kingdom
Team sprint
Jason Kenny
Sth International
International
November 2008
Manchester
United kingdom
Ross Edgar
Keirin Event
keirin
November 2008-09 World Cup
Manchester
United Kingdom
sprint
Jamie
Victoria
November 2008-09 World Cup
Manchester
United Kingdom
Keirin
Pendleton
November 2008
2008-09 World Cup
Manchester
United Kingdom
sprint
Ross Edgar
February 2009
2008-09 World Cup
Copenhagen
Denmark
Team sprint
Chris Hoy
February 2009
2008-09 World Cup Copenhagen
Denmark
Team sprint
Jason Kenny
GBR
February 2009
2008-09 World Cup
Copenhagen
Denmark
Team sprint
Jamie Staff
GBR
Victoria
13 February 2009
2009-09 World Cup
Copenhagen
Denmark
Sprint
GBR
Pendleton
Victoria
30 October 2009
2009-10 World Cup
Manchester
United Kingdom
Sprint
GBR
Pendleton
30 October 2009
2009-10 World Cup
Manchester
United Kingdom
Sprint
Chris Hoy
GBR
20 October 2009
2009-10 World Cup
Manchester
United kingdom
Keirin
1
Chris Hoy
GBR
Victoria
20 October 2009
2009-10 World Cup
Manchester
United kingdom
500 m time trial
2
GBR
Pendleton
1 November 2009
2009-10 World Cup
Manchester
United kingdom
Team sprint
1
Chris Hoy
GBR
1 November 2009
2009-10 world Cup Manchester
United kingdom
Team sprint
Ross Edgar
GBR
1 November 2009
2009-10 World Cup
Manchester
United Kingdom
Team sprint
Jamie Staff
GBR
Question: what is the number of 1st place finishes across events?
Triples: ($1, the number of, 1st place finishes).
(across all events, total, $2)
Decompose: for each event, what is placing?
what the total number of 1st place finishes all events?
Reasoning: event Jason Kenny placed 2nd. event
Pendleton placed 1st event Victoria Pendleton placed Kenny placed 1st. event Kenny placed event 5, Edgar placed 2nd, event 6. Jamie Staff placed 1st. In event Victoria Pendleton placed 1st. event 8, Ross Edgar placed 1st. In event 9, Chris Hay placed 1st In event 10, Jason Kenny placed 1st. in event 11, Jamie Staff placed 1st. In event 12, Victoria Pendleton placed Chris Hay placed 1st. In event 16, Victoria Pendleton placed 2nd. event Chris Hoy placed event Ross Edgar placed 1st. event Jamie Staff placed 1st.
event Victoria Pendleton placed 1st In event 14, Hoy placed 1st In event 15,
The total number of 1st place finishes across all events is 14
Final Answer: 14
Gold Answer:
Error Analysis: count error, total 17
Figure 10: Counting error.
H BASELINES
SASP (Ou & Liu 2022 uses lexical and structural features to generate programs for solving pseudo- programs. TAPEX-large (Liu al. 2021) learns neural SQL executor synthetic corpus of executable SQL queries and their outputs. T5-3B (Xie et al., (022) within the UnifiedSKG frame- work unifies 21 SKG tasks into text-to-text format for comprehensive SKG research. TAPAS-large
23

---

Published as a conference paper at ICLR 2025
[Eisenschlos et al. 2020) creates balanced dataset of millions of automatically generated training
examples for intermediate learning before fine-tuning. Codex (Cheng et al. 2023), as an OpenAl
API, can generate SQL Python statements and perform end-to-end QA.
BINDER (Cheng et al. 2023) iteratively refines ascudo-SQU queries for final answers. DATER (Ye
al.
2023b extracts sub-tables and decomposes questions for joint reasoning. StructGPT (Jang
et al
2023 enhances zero-shot reasoning with specialized interfaces for structured data.
DTE
Wang et al. 2023) refines text-to-SQL QA using counterfactual examples. TACR (Wu al.
2023
aligns multi-hop questions with various modalities for evidence retrieval. ITR (Lin et al.
2023
selects relevant rows and columns for compact sub-tables. (Liu et al., (202bb creates new tables with external information for SQL-based answers. Tab-PoT (Xiao et al. (202) enhances open-
source LLMSs with prompt management and post-processing modules. CHAIN-OF-TABLE: Wang
et al. (2024) plans operation chains dynamically based on table structure and questions. ReAcTable
(Zhang et al. 2024c) iteratively generates intermediate tables using L.LMs and external code. Cabi-
net (Patnaik et al. (2024) removes noise from tables to improve LLM reasoning accuracy. (Liu al.
2024a) explore combining DP and Agent to address LLM sensitivity to table structure.
I DP AND AGENT PROMPT
1.1 DP PROMPT
Direct Prompt
Accendin the decorooceet cuectton and labto content ttenk sbout answers inoo fisilty finsl grendtenn entty namers as short on possible without any eoplenation.
canssitaero nun mare treserved (eolsshone) diared num of tecionates (2000)
Warkgatt posse hhar garg
45 cowa mone (Boaggahh 141 205
44 matarruur noce pfr
idmnatepurr 152 605
(otal: total lotal lotel Hotel 1.007 000
Question:
Question
groatest
wh datici
number eleconatas?
(kangar coionarr dia uss, nusih dif diaticcs
numberat
145, 133, 161 545 D1 number of 162.400
Ukamgan haa the greatest lotal number etecooaaee match (S2, hal the greatest sotal number of, stectorales)
caction: (TITLE)
Quentoon tredets:
Figure 11: The prompt of Direct Prompting mode.
24

---

Published as a conference paper at ICLR 2025
1.2
AGENT PROMPT
Agent Prompt
"python,
the quartion pound you.
Tool descrption:
sython rel Python shell Use to nano nythen commaniss Inpur should be uslid puthoo command When using this
"AAggingted Rowe" Bo cautous of rows than aggregate dats such as fotaa. Numnt 'averager Ensum these mos do not influence
your reauft arspropprabell.
"Datt Verti Ventication" ": Before concluding final question
veriy that observations with original tabls and
Stricly follow the given format raspond:
Quastio::
cuesten must
Queston Trikes: Sub-quectine:
drson acconding quastion
the quattio based quastion
Thought you should s'ways fulow the desompowe sub wib-quesiuun,, and alwuys think ubout what do to internct witt sylhuun ruspasi Acton Inpui the imput code to the attion
Clorevvtions
of every suc-queation's sciinn whathr conforma structure of tiple and the unsteer whither right. ThoughtAAgton/kccee IepuCttseeeelenn Thought vedtying
imea)
abcanationn, and quection contident
answer (lotww Mommet Knowerkamee..
the
Ensure the frai anower foraar is only "Sinal Answer Ansuerhamet, Antamrraame? form, no other form Ensure the frnil anawer number or ently names, as possible, without verifies shservatoon questton before the final answer.
concluting
cala the table should be considered an 'ojject data tyoe, regardiass thair appearance.
Bugir
Quaetin Tripee: [T/phe]]
(PPcoompsse)
Figure 12: The prompt of Agent mode.
25


Reviews:
[{"review_id": "ndOleBe0GG", "reviewer": "Reviewer_yMYg", "summary": "The authors use Triples in different phases of TableQA. The generation of triples during Question decomposition phase helps in extracting key subquestions and conversion of answers during answer verification phase helps remove LLM bias in verification and enables better answer verification.", "questions": "1. Given that this is an additional step along the pipeline of TableQA steps, what does this step do to the average response time and number of tokens used? \n2. Does this method of generating triples work for mathematical questions such as average age/salary etc? Does this mean that the subquestions would require triples for every entry in the table?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Incorporating triples in multiple stages of TableQA is a novel idea and as shown by the evidence in the paper, helps improve the overall QA performance. \n2. The authors demonstrate the effectiveness of TIDE by performing ablation study which is critical in identifying the impact of this method.\n3. This method works alongside DP and adds an additional step of generating triples, so this keeps all the benefits of DP as well.", "weaknesses": "1. Inorder to demonstrate the impact of TIDE, it could have been useful to see TIDE with other standard LLMs as well. The ablation study shows the impact of TIDE but is this method tailored to work with GPT 3.5 in any specific way?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6nZ3E74jdL", "reviewer": "Reviewer_BvQy", "summary": "This paper proposed a decomposition and verification strategy based on triple to address TableQA tasks. Specifically, it decomposes and transforms questions using triples to capture more key information, while utilizes explicit structure for better verification. By combining LLM-based reasoning and symbolic reasoning based on structured representations, TIDE improves the performance in the TableQA task", "questions": "The appendix lacks cases, including successful cases compared to other methods, error cases analysis, etc. I think showing some cases can better explain the working mechanism of the entire pipeline.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of the work is clear. It leverages triple extraction as a tool to enhance understanding and validation of TableQA questions.\n  \n2. The experiment is sufficient, which compares TIDE with many TableQA baselines. Further study can reveal the properties of the proposed method.", "weaknesses": "1. The contribution is limited; it only proposes a framework based on LLMs with triple extraction. It would be better if it could be demonstrated on some open-source models like Llama3.\n  \n2. The conclusion drawn from the experimental results that TIDE improves the effectiveness of TableQA problems is questionable. The results of DP and Agent baseline for both datasets are not SOTA. So, can it be considered that the gain in this result comes from additional reasoning integration of DP and Agen rather than the introduction of triplet?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MP6GSi7k5V", "reviewer": "Reviewer_uGLH", "summary": "The paper proposed an inference time approach, TIDE, to address table question answering. Their method builds on a Decomposition-Reasoning-Verification approach for Table QA using LLM, and they proposed to use semantic triples as intermediate representations instead of natural text. First, during question decomposition, they extract triples from the question then generate subquestions based on those triples. Then, they apply CoT (for DP mode)/generating python code (for agent mode) to answer each subquestions. Finally, in the answer verification stage, they compare the triples from stage 1 with triples from stage 2 to filter correctly answered triples. \n\nTheir contribution is using triples as intermediate representations in order to facilitates decomposition and verification, and showed that it leads to improvements over SOTA.", "questions": "This goes back to the clarity of the verification stage.The example authors provided seems to be a ‘COUNT’ type questions. How does the system handle other types of numeric aggregation? For questions that are not numeric in nature (e.g. compose-style, multi-hop questions), how does the final stage work after verification? Does this approach only work on aggregation type questions? What algorithms are you using to unify the triples for each subquestion?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Incorporating triples as intermediate steps is an interesting approach, and it intuitively makes sense as a natural intermediate representation to tackle the verification of subquestions. The authors effectively combine existing ideas from different problems and apply it in the settings of table QA, showing SOTA results on challenging table qa datasets.", "weaknesses": "My main concern is clarity and contribution.\nFor clarity\n1.  Details of how each module is implemented is confusing. The formulation really only has one equation (eq 1), and it is about the Table QA problem. Formal explanation (instead of examples only) for each submodule would help improve clarity. Furthermore, I think the prompt for extracting triples are not provided?\n\n2. It’s unclear to me how the final answer is synthesized after the verification stage. Again, providing formulation for this stage would be more helpful. \n\n3. In Table 3 of your ablation studies, you have TIDE vs TIDE-decomposition vs TIDE-verification. It is confusing to me how the ablated versions are implemented. Again, I think having formulation would help improve clarity.\n\nIn terms of contribution, since this is combining different existing ideas from other tasks (see below for examples), I would appreciate more in-depth analysis of the proposed method.\n\n1.  Would other language models be equally effective with generating triples? When does this assumption breakdown? How does model size affect the result and error propagation through each stage ... etc. Have you tested this with open source large/small sized language models or other information extraction approaches for triple extraction? Is it possible to substitute some LLM API calls with other non-LLM methods without hindering performance?\n\n2. The strong performance could potentially come from data contamination. How do you attribute the success of TIDE vs GPT having seen those tables in pretraining? \n\n3. In verification stage, you mentioned the LLM is prompted to regenerate if verification failed. I would appreciate more analysis on this part such as showing how often verification fails or how many times max you limit the LLM to regenerate.\n\n\nPS by combining existing ideas I mean\n1. Question decomposition is not new. See [1] [2] for example\n2. using LLM for information extraction is not new. See [3] for example\n\nI think there are some merits in combining these insights and seeing that triples could apply to this problem, so I think more analysis could help improve this paper.\n\n[1] Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. Multi-hop Reading Comprehension through Question Decomposition and Rescoring. ACL 2019.\n\n[2] Xiaoyu Yang and Xiaodan Zhu.  Exploring Decomposition for Table-based Fact Verification. EMNLP Findings 2021\n\n[3] Zhiyuan Fan and Shizhu He. 2023. Efficient Data Learning for Open Information Extraction with Pre-trained Language Models.  EMNLP Findings 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)