Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding
Authors: Zilong Wang
Hao Zhang
Chun-Liang Li
Julian Martin Eisenschlos
Vincent Perot
Zifeng Wang
Lesly Miculicich
Yasuhisa Fujii
Jingbo Shang
Chen-Yu Lee
Tomas Pfister
Venue: ICLR 2024 poster (2024)
Keywords: Table Understanding
In-context Learning
Large Language Model
Primary area: generative models

Abstract: Table-based reasoning with large language models (LLMs) is a promising direction to tackle many table understanding tasks, such as table-based question answering and fact verification. Compared with generic reasoning, table-based reasoning requires the extraction of underlying semantics from both free-form questions and semi-structured tabular data. Chain-of-Thought and its similar approaches incorporate the reasoning chain in the form of textual context, but it is still an open question how to effectively leverage tabular data in the reasoning chain. We propose the Chain-of-Table framework, where tabular data is explicitly used in the reasoning chain as a proxy for intermediate thoughts. Specifically, we guide LLMs using in-context learning to iteratively generate operations and update the table to represent a tabular reasoning chain. LLMs can therefore dynamically plan the next operation based on the results of the previous ones. This continuous evolution of the table forms a chain, showing the reasoning process for a given tabular problem. The chain carries structured information of the intermediate results, enabling more accurate and reliable predictions. Chain-of-Table achieves new state-of-the-art performance on WikiTQ, FeTaQA, and TabFact benchmarks across multiple LLM choices.

Published as a conference paper at ICLR 2024
CHAIN-OF-TABLE EVOLVING TABLES IN THE REASONING CHAIN FOR TABLE UNDERSTANDING
Zilong Wang",
Hao Zhang
Chun-Liang Li
Julian Martin Eisenschlos
Vincent Perot
Zifeng Wang?
Lesty Miculicich²
Yasuhisa Fujii
Jingho Shang
Cuen-Yu Lee²
Tomas Pfister
University of California, San Diego
GGooge Cloud AI Research
Google Research
ABSTRACT
Table-based reasoning with large language models (LLMs) is a promising di- rection to tackle many table understanding tasks, such table-based question answering and fact verification. Compared with generic reasoning, table-based reasoning requires the extraction of underlying semanties from both free-form questions and semi-structured tabular data. Chain-of- Thouugh and its similar ap- proaches incorporate the reasoning chain in the form of textual context, but it is still an open question how to effectively leverage tabular data in the reason- ing chain. We propose the CHAIN-OF-TARIE explicitly used in the reasoning chain as a proxy for intermediate thoughts. Specif- ically, we guide LLMs using in-context learning to iteratively generate operations and update the table to represent tabular reasoning chain. LLMs can therefore dynamically plan the next operation based on the results of the previous ones. This continuous evolution of the table forms chain, showing the reasoning pro- cess for a given tabular problem. The chain carries structured information of the intermediate results, enabling more accurate and reliable predictions. CHAIN- OF- TABLE achieves new statc-of-the-an performance on WikiTQ, FeTaQA, and TabFact benchmarks across multiple LLM choices.
framework, where tabular data is
INTRODUCTION
Tables are a popular data format and widely used in daily life (Cafarella et al., 2008). Understanding tabular data with language models can benefit various downstream tasks, such table-based fact verification (Chen et al., 2019), and table-based question answering (Jin et al., 2022). Distinct from pure text, tables deliver rich information through the interaction between rows and columns in the tabular structure, which enhances the data capacity but also increases the difficulty for language models to understand them. Thus, reasoning over the tabular data important direction in natural language processing and attracts increasing attention from both academia and industry.
In recent years, several approaches have been suggested tackle the problem of table understand- ing by training language models. One common direction is to add specialized embedding layers or attention mechanisms into language models and pre-train the models by recovering table cells or segments (Herzig et al.. 2020; Wang et al., 2021; Gu et al., 2022; Andrejezuk et al., 2022). In this way, the pre-trained models are aware of the tabular structure. Another direction is synthesize SQL query-response pairs and pre-train an encoder-decoder model as neural SQL executor (Eisen- schlos al., 2020; Liu et al., 2021; Jiang et al., 2022).
Recently, large language models (LLMs) achieve outstanding performance across diverse tasks solely by prompting, thanks to the massive scale of pre-training (Brown et al., 2020: Kojima et al., 2022). As series of works on prompting techniques have further improved the reliability of LLMs by designing reasoning chains, such as Chain-of- Chain-of-Thought (Wei et al., 2022), Least-to-Most (Zhou (Yao et al., 2023). Dif-
et al., 2022), Program-of-Thorn Thouughht (Chen et al.. 2022) and ferent works have also explored the possibility of using LLMS to solve table-based problems (Chen,
Work done while the author was student researcher at Google Cloud AT Research. Correspondence to: Zilong Wang cew.ang@ussd..dus>, edu>, Chen-Ye Yu Lee <chenyulee Scheeyulle@@googl..com google.com>

---

Published as a conference paper at ICLR 2024
Input Prompt
(a) Generic Reasoning
[Original Table]
Multjole Reasoning Steps Reisoning fails to solve the complex table. Question suko cycllsts out Harmer not top
There 2 cyclists from Spain. They are Alejandro and Halmar. Spain. X
Rank
Cyclist
1
Alejandro (ESP)
Davide (TTA)
3
Poolo (TT))
Haimar (ESP)
(b) Program-aided Reasoning
[Question]
Generated Programs Program-aided Reasoning falls to salve the complex table.
Which country had the
9gz: SETECT Coustry FROM Lable KHERE Rankc=3 GROUP
most cyclists finish with
SQL can't oxacufi since "Country" is "Neme"
top
BY Country ORDER BY
LLM
COUNT (*) DESC LINIT
X
(c) Chain-of-Table (ours)
Iteratively repeat Step 1, 2 3 with the intomeddiate tuble the oporation histagy
Sample operation based Table, Question, Operation History
Step Generate arguments for the sampled operation
Complete Operation History
Represent Tabular Reasoning Chain
Iter
Ladd, collCountry)
Header "Country"
Laadd_coll
1_select_row(1 2.
Added Col Colls "EPPTA.....
Iter 3:
group by(Country
Iter 4:
f soort by(Count)
OP Pool
Input Prompt (next Iteration)
Step 3:
Final Query Prompt
[Intermediate Table] [tank]]
Cyc
Ales
Country
Transform table to: store the tabular reasoning process
which country had Country the most cyclists finish
Count
[Operation History)
csl)
with the top 3?
ESP
[Quastion] Which country... ... mh top
Italy V
Illustration of the comparison between (a) generic reasoning, (b) program-aided reason- ing, and (c) the proposed CHAIN-OF-TAPEE and name are in the same cell, (a) is unable to provide the correct answer through multi-step rea- soning due to the complexity; (b) generates and executes programs (e.g. SQL queries) to deliver the answer, but it also falls short accurately parsing the name and nationality in the table. In contrast, (c) CHAIN-OF- -AALLE iteratively samples chain of operations that effectively transform the complex table into a version specifically tailored to the question. With the assistance of CHAIN- OF-TABLE, the LLM can arrive at the correct answer.
Figure 1:
Given a complex table where cyclist's 's nationality
2023; Cheng et al., 2022; et al., 2023). However, these approaches (Hsieh et al., 2023) often rep- resent reasoning steps in free-form text or code, which are not ideally suited for addressing scenarios involving complex tables, as shown in Figure 1(a) and Figure ((b).
On the other hand, inference on tables typically involves series of intermediate reasoning steps and each them aligns with specific tabular operations. We propose CHAIN-OF-AABLLE where we con- duct step-by-step reasoning step-by-step tabular operations to form chain of tables. The tables in the chain are the transformed tables by the tabular operations, representing the intermediate rea- soning results. This procedure resembles the thought of reasoning in Chain-of-Thought (Wei al., 2022). Specifically, we define a set of table operations, such as adding columns, selecting rows, grouping, and more, which are commonly-used 1995; Shi et al., 2020; Katsogiannis- Aatoogiianni-Mimmaraaai & Koutrika, 2023). We then prompt LLMs to con- duct step-by-step reasoning. In each step, the LLM dynamically generates an operation as the next step along with its required arguments, and then we execute the operation on the table programmat- ically. This operation can either enrich the table by adding detailed intermediate results or condense it by removing irrelevant information. Intuitively, visualizing the intermediate results is essential for reaching correct predictions. We feed the transformed table back for the next step. This iterative process continues until an ending state is achieved. We argue that the tables obtained during the rea- soning steps are better structured representations the intermediate thoughts than free-form text. Finally, the CHAIN-OF- -F-TABLE reasoning results in tables from which it is easier for LLMs derive a final answer to the question.
SQL and DataFrame development (Pönighaus,
We validate CHAIN-OF-TADE with three tabular benchmarks to evaluate based reasoning: WikiTQ (Pasupat & Liang, 2015). TabFact (Chen et al., 2019), and FeTaQA (Nan et al., 2022). We conduct experiments using PaI.M 2 (Anil et al., 2023) and GPT-3.5 (Brown et al., 2020; OpenAl,
2

---

Published as a conference paper at ICLR 2024
2023) to demonstrate that our proposed method CHAIN-OF-TABLE is able to generalize to various LLM options. We summarize our contribution as follows:
We extend the concept of Chain-of-Tie -hooughht to the tabular setting, where transform the input table to store intermediate results. This multi-step tabular reasoning approach with table evolution leads to more accurate table understanding.
Extensive experiments on table-based fact verification and question answering show that CHAIN- OF-TABLE archives state-of-the-art performance in WikiTQ, TabFact, and FeTaQA datasets.
2 RELATED WORK
Fine-tuning Language Model for Table Understanding Tables are effective in organizing, stor- ing, and analyzing information. Efforts have been made to fine-tune language models (LMs) to tackle table understanding tasks. Following the successful mask language modeling (MLM) pro- posed in BERT (Devlin et al., 2019), TaPas (Herzig al., 2020) adopts this approach and asks the model to reconstruct certain cells in the table during pre-training. Pasta (Gu et al., 2022) and TUTA (Wang et al., 2021) further propose to mask the entire columns or segments in the table. On the other hand, TAPEX (Liu et al., 2021) pre-trains an encoder-decoder model with a large synthetic SQL dataset so that it can perform as SQL executor to better understand the tabular structure. Eisenschlos et al. (2020) and Jiang et al. (2022) also leverage synthesized SQL with additional con- sideration the alignment between SQL and natural language questions by pre-training the model with both natural and synthetic data.
Prompting Language Model for Table Understanding LLMs can learn from a few samples prompts through in-context learning. This strategy is widely used to give models additional instructions to better solve downstream tasks. Chain-of-Thouuhh (CoT) (Wei et al., 2022) proposes to generate reasoning steps before answering instead of directly generating an end-to-ent answer. Following COT, Least-to-Most (Zhou et al., 2022) and DecomP (Khot et al., 2022) propose to break down the question into subproblems in the reasoning chain. During reasoning, the latter steps aware of the previous ones. Such iterative chains with task decomposition further improve the results on complex problems by leveraging the intermediate results from solving subproblems. Jin & Lu (2023) enhances CoT through a table-fillie filling procedure, with a primary focus on text-based tasks where the input and output are in textual format. However, the line of works following CoT is not specifically designed for tabular data. As reported in Chen (2023), large language models with these generic reasoning methods can achieve decent results, but there are still gaps between these methods and those specialized for table scenarios (Cheng et al., 2022; Ye et al., 2023). We propose CHAIN- OA-TABLE to fill the gap by directly incorporating intermediate tables from tabular operations as a proxy of intermediate thoughts.
To better solve table-based tasks with LLMs, researchers go beyond general text and resort to using external tools. Chen et al. (2022); Gao et al. (2023) propose solving reasoning tasks by generating Python programs, which are then executed using the Python interpreter. This approach greatly improves the performance of arithmetic reasoning. In the scenario of table understanding. Text-to- SQL with L.Ms (Rajkumar et al., 2022) is a straightforward application of this idea. To further push the limits of programs, Binder (Cheng et al., 2022) generates SQL or Python programs and extends their capabilities by calling LLMs as APIs in the programs. LEVER (Ni et al., 2023) also proposes solving the table-based tasks with programs but with the additional step of verifying the generated programs with their execution results. However, the assistant programs in these program- aided methods still fall short in solving difficult cases that involve complex tables. These limitations are primarily due to the constraints of the single-pass generation process, where the L.MMs lack the capability to modify the table in response to a specific question, requiring them to perform reasoning over static table. Our method, on the contrary, is multi-step reasoning framework that conducts tabular reasoning step by step. transforms the tables tailored to the given question.
To the best of our knowledge, Dater (Ye et al., 2023) is the only model that modifies the tabular context while solving table-based tasks. However, the table decomposition in Dater is motivated by the idea that tables could be too large for LLMs conduct reasoning. It is, therefore, more similar to an LLM-aided data pre-proc pre-processing than to a part of the reasoning chain since the tabular operations limited to column and row selections, and fixed for all tables and questions. In contrast, our
3

---

Published as a conference paper at ICLR 2024
CHAIN-OF-TABLE generalizes larger set of generic table operations and dynamically generates reasoning chains in an adaptive way based on the inputs, leveraging the planning ability (Valmeekam et al., 2022; Hao et al., 2023) of LLMs.
CHAIN-OF-TABLE REASONING
Problem Formulation. In table-based reasoning, each entry can be represented as triplet (T, Q, A), where T stands for the table, Q represents question or statement related to the ta- ble, and A is the expected answer. Particularly, in the table-based question answering task, Q and A are the question and expected answer in natural language form; in the table-based fact verification task, Q is a statement about the table contents and A E (True, False] is a Boolean value that indicates the statement's correctness. The objective to predict the answer given the question Q and the table T. To facilitate table-based reasoning within the same paradigm employed for generic reasoning, we convert all data values, including tables, into textual representations (see Appendix D for the tabular format encoding method).
3.1 OVERVIEW
CHAIN-OF-TABLE enables LLMs to dynamically plan a chain of operations over a table T in re- sponse given question Q. It utilizes atomic tool-based operations construct the table chain. These operations include adding columms, selecting rows columns, grouping, and sorting, which are common in SQL and DataFrame development (see Appendix A for more details).
Previously, Dater (Ye et al., 2023) employs a dedicated yet fixed procedure for decomposing ta- bles and questions, which limits compatibility with new operations. Also, Binder (Cheng al., 2022), while potentially compatible with new operations, is restricted to those that work with code interpreters such as SQL or Python. In contrast, our framework is extendable and can incorporate operations from wide range of tools thanks to the flexible in-context learning capability to sample and execute effective operations.
As illustrated in Algorithm 1, at each iteration, prompt the LLM sample one of the pre-defined atomic operations denoted as f using the corresponding question Q, the latest table state T, and the operation chain chain (Line 4). Then, query the LLM to generate the required arguments ar args for (Line 5) and execute it to transform the table T (Line 6). We keep track the operation performed the table in the operation chain chain (Line 7). The process finishes when the ending tag [E] is generated (Line 8). Finally, we feed the latest table into the LLM to predict the answer (Line 9). This series of operations serves the reasoning steps leading LLMs understand the input table and better generate the final answer.
Algorithm 1: CHAIN-OF-TABLL Prompting
Data: (T, Q) table-question pair.
Result: Å is the predicted unswer the question.
Function Chain of Ta-bll (T: Q):
chain + [([B]
Initialize the operation chain charnn with [B]] d, where [B] is beginning tag, and øi means requires arguments
repeat
f DynamicPlan (T,aaiicclaan,,,,,aeee Qchhain)))
Generate operation f based on the table, the question. and D the current operation chain > Generate the arguments args for the next operation Perform the next operation on table to obtain updated Keep Iteratively update operations the ending tag [E] generated b Query the LLM with the resulting table to get the linal answer Á
args + GemerateArgs(T.QQ.7
T - f(T f(T args)
chain chain. (f, (frggs))
operation chain charin
[E]
A Query(TT.)
return A
3.2 DYNAMIC PLANNING
CHAIN-OF- TABLE instructs the LLM to dynamically plan the next operation by in-context learning. As shown in Figure 2(a), Dynamic cP involves three components: the most recent intermediate table T (Figure 2(a)())), the history of the previous operations chain chain (Figure 2(a)(i))) and
4

---

Published as a conference paper at ICLR 2024
Input Prompt
Input Prompt
Intermediite
Operation
Infermedlete Table (1)
Arguments: Table
headers case
Oyclii
AMa (PSP
Ae
JESP
ESP
Plant
Cyclist
Operation History (chamr)
Current Operstion (1) Arguments (orgs)
Selected Operation
Solected Argument
col(Country)
besed on Choals
group
by
bused E,
Country
select
args:
group by
(!)) Question (Q))
(i) Question (Q)
country
Which had
tyyclett
mose cyclists fresh with
(a) DynamicPlan (7. Q, chain)
(b) GenerateArgs (7. Q.
Figure 2: Illustration of Dynami cPaan (T, Q,,,, chain) and GenerateArgs Q2) in the proposed CHAIN-OF-TANE TABLE, where T is a intermediate table; Q is the question; chain is a list of operations already performed DynamicPlan samples the next operation from the operation pool, according to (T, chain, Q)- Right: GenerateArg gs takes the selected operation f as input and generates its arguments based on (T, f, Q). The operations, along with their arguments, act as a proxy of the tabular reasoning process effectively tackle table understanding tasks.
the table; f is the operation selected by DynamicPlan Left:
the question Q (Figure (ia)) We guide the LLM to select the subsequent operation f from the operation pool given (T, chain, Q). The LLM is then able to dynamically plan the next operation and build a tabular reasoning chain step by step. See Appendix E.1 for detailed prompts.
3.3 ARGUMENT GENERATION
The next step. GenerateArgs involves generating arguments for the selected table operation sampled by DynamicPly as depicted Figure GenerateArs gg involves three key com- ponents: the most recent intermediate table T (Figure 2(b)(i)), the selected operation f along with its arguments args (Figure 2(b)(ii), and the question (Figure ((b)(i We employ simple regular expressions to account for varying number of arguments required by different operations (see Ap- pendix E.2 for more details). Finally, we apply programming languages to execute the operation and create the corresponding intermediate tables.
3.4 FINAL QUERY
We transform the table through dynamic planning (Section 3.2) and argument generation (Section 3.3). During this process, we create chain of operations that acts a proxy for the tabular reason- ing steps. These operations generate intermediate tables that store and present the results of each step to the LLM. Consequently, the output table from this chain of operations contains comprehen- sive information about the intermediate phases of tabular reasoning. We then employ this output table in formulating the final query. As illustrated Figure (bottom right), we input both the output table and the question into the LLM, which provides the final answer to the question (see Line 9 in Algorithm 1).
EXPERIMENTS
We evaluate the proposed CHAIN-OF- TABLE on three public table understanding benchmarks: Wik- TTQ (Pasupat & Liang, 2015), FeTaQA (Nan et al., 2022). and TabFact (Chen et al., 2019). WikiTQ and FeTaQQ are datasets focused on table-based question answering. They require complex tab- ular reasoning over the provided table to answer questions. WikiTQ typically requires short text span answers, whereas FeTaQA demands longer, free-form responses. TabFact, on the other hand, table-based binary fact verification benchmark. The task is to ascertain the truthfulness of given statement based on the table. For WikiTQ evaluation, we use the official denotation aceu- racy (Pasupat & Liang, 2015), and for TabFact, employ the binary classification accuracy. Given the nature of FeTaQA, which involves comparing predictions with longer target texts, we utilize BLEU (Papineni et al., 2002), ROUGE-1, ROUGE-2, and ROUGE-L (Lin, 2004) for assessment. In
5

---

Published as a conference paper at ICLR 2024
Table 1: Table understanding results WikiTQ and TabFact with PaI.M and GPT 3.5. (underline denotes the second- becond-best performance; bold denotes the best performance; the improvement is mea- sured against the second-best performing method.)
PaLM
GPT 3.5
Prompting
TabFact
WikiTQ
TabFact
WikiTQ
Generie Reasoning
End-to-End QA
77.92
60.59
70.45
51.84
Few-Shot QA
78.06
60.33
71.54
52.56
Chain-of Thouu-t (Wei et al., 2022)
79.05
60.43
65.37
53.48
Program-aided Reasoning
Text-to-SQL (Rajkumar et al., 2022)
68.37
52,42
64.71
52.90
Binder (Cheng al., 2022)
76.98
54.88
79.17
56.74
Dater (Ye et al., 2023)
84.63
61.48
78.01
52.81
CHAIN-OF- TABLE (ours)
86.61 (+1.98)
67.31 (+5.83)
80.20 (+1.03)
59.94 (+3.20)
our experiments, we use PaLM 2-S', GPT 3.5 porate few-shot demo samples from the training into the prompts to perform in-context learning. Examples of these prompts can be found in Appendix E. Details regarding the LLM inference pa- rameters and the number of demonstration samples used are provided in Appendix C.
as the backbone LLMs. We incor-
4.1 BASELINES
The baseline methods are categorized into two groups: (a) generic reasoning, which includes End- to-End QA, Few-Shot QA, Shain-of-Thoughh (Wei et al., 2022); and (b) program-aided reasoning, which includes Text-to-SQL. (Rajkumar et al., 2022), Binder (Cheng al., 2022), Dater (Ye al., 2023)). Detailed descriptions of these baseline methods are provided below.
Generic Reasoning End-to-End QA guides the LLM to directly produce the answer when pro- vided with table and question as input prompts. Few-Shot QA operates similarly, but it includes few-shot examples of (Table, Question, Answer) triplets in the prompt, detailed in Brown et al. (2020). select these examples from the training set, and the model also outputs the answer rectly. Chain-of-Thougg (Wei et al., 2022) prompts the LLM to articulate its reasoning process in text format before delivering the question. See Appendix for the prompts of baselines.
Program-aided Reasoning Text-to-SQL. (Rajkumar al., 2022) utilizes in-context samples to guide LLMs in generating SQL queries for answering questions. This approach follows the concepts introduced by Chen al. (2022); Gao (2023). Binder (Cheng et al., 2022) integrates language model API with programming languages such SQL Python. This integration prompts the LLM to produce executable programs that perform table reasoning tasks on the given table and question. Dater (Ye et al., 2023) employs few-shot samples for efficient deconstruction of table contexts and questions, enhancing end-to-end table reasoning with decomposed sub-tables and sub-questions.
4.2 RESULTS
We compare CHAIN-OF-TABLL with generic reasoning methods and program-aided reasoning methods on three datasets: WikiTQ, TabFact, and FeTaQA. The results on WikiTQ and TabFact are presented in Table 1. We have additional results on FeTaQA in Appendix B. We follow the previous works and report the performance using the official evaluation pipeline
Chttps://clett google. watggee.ccm//eee www/coot//eeeeee tex-ai htt///dcss/geeeattiveee a1 /tt.a///ggneraaiieaaaaaaaaa httpp:/ Phttp://openai. con/api/
Dater Ye et al. (2023) with OpenAl Codex LLM achieves 65.9% and 85.6% accuracy on WikiTQ and TabFact, respectively. It also achieves 27.96 in BLEU, 0.62 in ROUGE-1, 0.40 in ROUGE-2, and 0.52 in ROUGE-L on FeTaQQ. However, hecause Codex is no longer publicly available, do not compare CHAIN- OF-TABLE with Dater with Codex.

---

Published as a conference paper at ICLR 2024
Chain-ofFThouhh (Wei et al. 2022)
Dater (Ye et al. 2023)
Chain-of-Tabee (ours)
80
(-3.7) (-5.3) 69.3 67.7
69.5
70
(-7.2)
62.3 61.9
(-3.6) 61.0
Accuracy (%%
60
(-7.5)
(-7.9)
(-2.1)
53.5
(-10.8|52.2
50
49.3
(-11.6)
40.0
40
30
20
3
4
Length of Operation Chain
Figure 3: Performance of Chain-of-Thought, Dater, and the proposed SHAIN-OF-TABLLE Wik- iTQ for questions that require an operation chain of varying lengths. Our proposed atomic op- erations allow our proposed method CHAIN-OF-TABLE dynamically transform the input table through multiple reasoning iterations. This significantly improves performance over generic and program-aided reasoning counterparts.
Table 2: Distribution of the number of samples v.s. the required length of operation chain in CHAIN- OF- TABLE with PaLM 2 WikiTQ and TabFact datasets. We observe that the majority of samples need 2 to 4 operations to generate the final output.
Length of operation chain
Dataset
2
3
4
5
WikiTQ TabFact
95
1308
1481
1084
341
547
732
517
223
Table shows that CHAIN-OF- TabLe significantly outperforms all generic reasoning methods and program-aided reasoning methods TabFact and WikiTQ across PaLM 2 and GPT 3.5. This is attributed to the dynamically sampled operations and the informative intermediate tables in CHAIN- OF-TABLE. CHAIN-OP-TABL iteratively generates operations that as proxies for tabular reason- ing steps. These operations produce and present tailored intermediate tables to the LLM, conveying essential intermediate thoughts (see the example in Figure 4). With the support of CHAIN-OF- TABLE, the LLM can reliably reach the correct answer.
From the results, we observe a performance decrease on WikiTQ due the complexity of tabular structure when vanilla Ohain--f-Thhuuhhe is introduced to End-to-End QA using PaLM 2. In contrast, our proposed CHAIN-OF- TABLE consistently enhances End-to-End QA performance by 8.69% on TabFact and 6.72% WikiTQ with PaLM 2.
4.3 PERFORMANCE ANALYSIS UNDER DIFFERENT OPERATION CHAIN LENGTHS
In CHAIN-OF-TABLE the selection of each operation is dynamically determined based on the dif- ficulty and complexity of the questions and their corresponding tables. Therefore, conduct detailed study the performance under different numbers of operations by categorizing the test samples according their operation lengths. We report the distribution of the number of samples v.s. the required length of operation chain in Table 2. This analysis focuses on samples that require operations in the reasoning process. We use the results with PaLM as an example. Our observa- tions reveal that the majority of samples require to 4 operations generate the final output.
For each chain length, we further compare CUAIN-OF- TABLE with Chain-of-Thoughh and Dater, as representative generic and program-aided reasoning methods, respectively. We illustrate this using results from PaLM on WikiTQ. We plot the accuracy of all methods using bar charts Figure 3, highlighting the gap between the compared methods and method. Notably. CHAIN-OF- T consistently surpasses both baseline methods across all operation chain lengths, with a significant margin up 11.6% compared with Chain-of-Thouuhh and up 7.9% compared with Dater.

---

Published as a conference paper at ICLR 2024
Table 3: Performance of Binder, Dater, and the proposed CHAIN-OF-TABLE small (<2000 kens), medium (2000 to 4000 tokens), large (>4000 tokens) tables from WikiTQ. We observe that the performance decreases with larger input tables while CHAIN-OF-TABLE diminishes gracefully, achieving significant improvements over competing methods. (underline denotes the second-best performance; bold denotes the best performance; the improvement measured against the second- best performing method.)
Table Size
Prompting
Small (<2k) Medium (2kx-.4) Large (>:4k)
Binder (Cheng et al., 2022)
56.54
26.13 13
6,41
Dater (Ye al., 2023)
62.50
42.34
34.62
CHAIN-OF- TABLEE (ours)
68.13 (+5.63)
52.25
(+9.91)
44.87 (+10.25)
Table 4: Number of samples generated for single question in Binder, Dater, and the proposed CHAIN-OF- OF-TABLE on the WikiTQ dataset. Notably, CAAAIN-OF-ABBLLE generates the fewest sam- ples among the baselines- - 50% less than Binder and 75% less than Dater. For a detailed description of the steps involved in Binder and Dater, please refer to the corresponding papers.
Total # of generated samples
# of generated samples in each steps
Prompting
Binder (Cheng et al., 2022)
50
Generate Neural-SQL: 50
Decompose Table: 40; Generate Cloze: 20; Generate SQL: 20; Query: 20
Dater (Ye al., 2023)
100
DynamicPlan: < GenerateArgs: $19;
CHAIN-OF- TABLE (ours)
<5
Query: 1
Generally, the performance of these methods decreases as the number of tabular operations required in the tabular reasoning chain increases due higher difficulty and complexity of questions and ta- bles. Nevertheless, our proposed CHAIN-OF DAAAIN-F--TABEE declines gracefully compared to other baseline methods. For example, CHAIN-OP-TABL TABLE exhibits only minimal decrease in performance when the number of operations increases from four to five.
4.4 PERFORMANCE ANALYSIS UNDER DIFFERENT TABLE SIZES
Large tables present significant challenges LLMs since LLMs often struggle to interpret and in- tegrate contexts in long input prompts (Liu et al., 2023a; Ye et al., 2023). To assess the performance on tables of various sizes, we categorize the input tables from WikiTQ into 3 groups based on token count: small (<2000 tokens), medium (2000 to 4000 tokens) and large (>4000 tokens). We then compare CHAIN-OF- TFBLLE with Dater (Ye al., 2023) and Binder (Cheng et al., 2022), the two latest and strongest baselines, representative methods. Detailed results are presented in Table 3.
As anticipated, the performance decreases with larger input tables, as models are required to pro- cess and reason through longer contexts. Nevertheless, the performance of the proposed CHAIN- OF-TABLE diminishes gracefully, achieving a significant 10+% improvement over the second best competing method when dealing with large tables. This demonstrates the efficacy of the reasoning chain in handling long tabular inputs.
4.5 EFFICIENCY ANALYSIS OF CHAIN-OF-TABLE
We analyze the efficiency of CHAIN-OF-TABLE by evaluating the number of required generated samples. We compare CHAIN-OF-TABLE with Binder (Cheng et al., 2022) and Dater (Ye et al., 2023), the two latest and most competitive baseline method. The analysis results on WikiTQ are presented in Table 4. Binder generates Neural-SQLL queries, requiring 50 samples for self-consistentt results. Dater involves multiple delicate yet fixed steps, such as decomposing the tables and gener- ating cloze queries for the questions. In each step, Dater also employs self-consistency to improve accuracy of the L.LM outputs, leading high number of required generated samples. For de-
8

---

Published as a conference paper at ICLR 2024
Chain-of-Tabee (ours)
Input Prompt
Iteration 1:
Iteration 2:
Iteration 3:
Iteration 4:
[Original
Egroup
Pial
args: : Ireadien:: Your Actor
args:
args
Headeve
Order Deou
[END]
Afreoss
Tranaforo the table with
Transform the table with
Tharsform the sable with
Final Query
-ry
Intermodiute Table 17
Ontermociute Toble 2)
[intermediute
Cownt
Thergforz
[Quvestion) Wht actor hos the most Feacp image awards??
Morgon Freemon
Figure 4: Illustration of the tabular reasoning process in CHAIN-OF -F-BLE. This iterative process involves dynamically planning an operation chain and accurately storing intermediate results in the transformed tables. These intermediate tables serve as tabular thought process that can guide the LLM to land to the correct answer more reliably.
tailed description of these frameworks, please refer to the corresponding papers, Ye et al. (2023) and Cheng et al. (2022).
Unlike these previous methods, our proposed CHAIN-OF- TABLE employs a greedy search strategy in its tabular reasoning process, instead of relying on self-consistency sampling for boosting perfor- mance. This approach results in reduced query count for our method, despite CHAIN-OF- BI, adopting an iterative reasoning process. To be more specific, we observe that the number of queries needed by CHAIN-OF- MAALN-OF-TABLE is the lowest among the most recent baselines 50% less than Binder and 75% less than Dater. We attribute the query efficiency our method to the proposed dynamic operation execution through the tabular reasoning, The model is able to find effective reasoning process that reaches the final output quicker and more reliably.
4.6 CASE STUDY
In Figure 4, we illustrate the tabular reasoning process by CHAIN-OF-TARLE, The question is based on complex table and requires multiple reasoning steps to 1) identify the relevant columns, 2) conduct aggregation, and 3) reorder the aggregated intermediate information. Our proposed CIAINN- TA-BEBL involves dynamically planning an operation chain and accurately storing intermediate results in the transformed tables. These intermediate tables serve tabular thought process that can guide the LLM to land to the correct answer more reliably.
5 CONCLUSION
Our proposed CHAIN-OF- TABLE enhances the reasoning capability of LLMs by leveraging the tabu- lar structure express intermediate thoughts for table-based reasoning. It instructs L.LMs dynam- ically plan an operation chain according to the input table and its associated question. This evolving table design sheds new light on the understanding of prompting LLMs for table understanding.
REPRODUCIBILITY STATEMENT
We include the prompt examples of DynamicP QT, chain) in Appendix E.I. the demo examples of GenerateArge (T' Q, f) in Appendix E.2, the prompt examples of Query (T,Q) Q) in Appendix E.3. We the generic reasoning methods Ennd-toEEnd QA, FewShot QA, Chain- of-Thought) using the prompts reported in Appendix F. We run Text-to-SQL and Binder us- ing the official open-sourced code and prompts in /ttt:::/ggthub com HK/NLP P/innder.. We run Dater using the official open-sourced code and prompts in https: / igi thub. com/ Al AiiabbaResearda ChteearrhDDMM---CCC DAMOO-CMVOOOLL We revise the code to use publicly available GPT 3.5 and PaLM 2 (Section 4) as the LLM backbone instead of the OpenAl Codex due to its inaccessibility.
REFERENCES
Ewa Andrejezuk, Julian Eisenschlos, Francesco Piccinno, Syrine Krichene, and Yasemin Altun. Table-to-text generation and pre-training with TabT5. In Findings of the Association for Compu-
9

---

Published as a conference paper at ICLR 2024
national Linguisties: EMNLP 2022, pp. 6758-6766, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.186533//2222.nnndnngss www.:iiiinnng-emll55...3.333 1
Rohan Anil, Andrew Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 2023. 2
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 1,
2, 6
Michael J. Cafarella, Alon Halevy, Daisy Zhe Wang. Eugene Wu, and Yang Zhang. Webtables: Exploring the power of tables on the web. Proc. VLDB Endow. 1(1):538-544 (4988-54,,00 aug 2008. ISSN 21 2150-8097. doi: 00.177781453855 1453916.
Wenhu Chen. Large language models are few(l)-shot table reasoners. In Findings of the Association for Computational Linguisties: EACL 2023, pp. 1120-1130, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 10. (0.18633/v1/203. http://2iiiniinggeaaaa..888 1, 17
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-based fact verification. In International Conference on Learning Representations, 2019. 1, 2,
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.1255 2022. 1, 3,3.6
Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, al. Binding language models in symbolic languages. In International Conference on Learning Representations, 2022. 2, 3, 4, 6, 8, 9, 16
Jacob Devlin, Ming-
Beig-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, 2019.
Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming- Wei Chang, Dipanjan Das, and William Co- hen. Handling divergent reference texts when evaluating table-to-text generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4884 4895, 2019. 14
Julian Eisenschlos, Syrine Krichene, and Thomas Müller. Understanding tables with intermediate pre-training. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 281-296, Online, November 2020. Association for Computational Linguistics. doi: 10. 18653/vll 0020.iindnns-emmlp.22. 1,3
Luyu Gao, Aman Madaan, Shuyan Zhou, Uni Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAI:: Program-aided language models. In International Conference on Machine Learning, pp. 10764-10990 10799. PMLR, 2023. 6
Zihui Gu, Ju Fan, Nan Tang, Preslav Nakov, Xiaoman Zhao, and Xiaoyong Du. PASTA: Table- operations aware fact verification via sentence-table cloze pre-training. In Proceedings of the 2022 Conference Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 10.18653/v1/022 2tw.:0222.emmmmmmm333111111 1,3
Empirical Methods in Natural Language Processing, pp. 4971-4983 Abu
Shibo Hao, Yi Gu, Haodi Ma. Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305. 14992, 2023. 4
10

---

Published as a conference paper at ICLR 2024
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisen- schlos. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4320-4333, Online, July 2020. Association for Computational Linguistics. doi: 10.
1,3
Cheng- Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen- Yu Lee, and Tomas Pfister, Distilling stp-bb--tepp outperforming larger language models with less training data and smaller model sizes. In Findings of the Association for Computational Linguistics: ACI. 2023. Association for Computational Linguisties, 2023." 2
Shima Imani, Liang Du, and Harsh Shrivastava. MathPrompter: Mathematical reasoning using large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pp. 37-42, Toronto, Canada. July 2023, Association for Computational Linguistics. doi: 10.18653/v1/202..a -nnduuusrrr....y 13
Zhengbao Jiang, Yi Mao, Pengcheng He, Graham Neubig, and Weizhu Chen. OmniTab: Pretraining with natural and synthetic data for few-shot mable-based question answering. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguis- fics: Human I Language Technologies, pp. 932-942, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 10111865311/2222 http://2222naaamaa..ieeee
1, 3,
Nengzheng Jin, Joanna Siebert, Dongfang Li, and Qingcai Chen. A survey on table question an- swering: recent advances. In China Conference on Knowledge Graph and Semantic Computing, pp. 174-186. Springer, 2022.
Ziqi Jin and Wei Lu. Tab-cot: Zero-shot tabular chain of thought. arXiv preprint arXiv:2305. 2023. 3
George Gatsogiannis-Meimaraaa and Georgia Koutrika. A survey on deep learning approaches for text-to-sql. The VLDB Journal, pp. 1-32, 2023. 2
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. In Interna- tional Conference on Learning Representations, 2022.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, 2022.
Chin Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. 5, 14
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172. 2023a. 8
Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. TAPEX: Table pre-training via learning a neural sql executor. In International Conference on Learning Representations, 2021. 3,3, 16
Qian Liu, Fan Zhou, Zhengbao Jiang, Longxu Dou, and Min Lin. From zero to hero: Examining the power of symbolic tasks in instruction tuning. arXiv preprint arXiv:2304.079995 2023b. 16
Joshua Maynez, Priyanka Agrawal, and Sebastian Gehrmann. Benchmarking large language model capabilities for conditional generation. In Proceedings of the 61 Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers), pp. 9194 0194-9213, 2023. 14
Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryšciński, Hailey Schoelkopf, Riley Kong, Xiangru Tang, Mutethia Mutuma, Ben Rosand, Is- abel Trindade. Renusree Bandaru, Jacob Cunningham, Caiming Xiong, Dragomir Radev, and Dragomir Radev. FeTaQA: Free-form table question answering. Transactions of the Association for Computational Linguistics, 10:35-49, 2022. doi: 10. 1622taccl__ a 00446. 2,5
11

---

Published as a conference paper at ICLR 2024
Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. Lever: Learning to verify language-to-cou generation with execution. In International Conference Machine Learning, pp. 26128. PMLR, 2023. 3
OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Associa- tion for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguisties. doi: 10.3 10.3115/107303310733333
Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Anmual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference Natural Language Processing (Volume I: Long Papers), pp. 1470-1480, Beijing, China, July 2015. Association for Computational Linguistics. doi: 10.3115//1/15--11422 5,5
Richard Pönighaus. 'favourite sqll-sllllsss sgl-statements an empirical analysis of sql-usage in commercial applications. In International Conference on Information Systems and Management of Data, pp- 75-91. Springer, 1995. 2
Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. Evaluating the text-to-sql capabilities of large language models. arXiv preprint arXiv:2204. 00498, 2022. 3, 6
Tianze Shi, Chen Zhao, Jordan Boyd-Graber, Hal Daumé III, and Lillian Lee. On the potential of lcxico-logical alignments for semantic parsing to sql queries. Findings of the Association for Computational Linguistics: EMNLP 2020, 2020.
Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subharao Kambhampati. Large lan- guage models still can't I plan (a benchmark for llms planning and reasoning about change). In NeurlPS 2022 Foundation Models for Decision Making Workshop, 2022.
Zhiruo Wang, Haoyu Dong, Ran Jia, Jia Li, Zhiyi Fu, Shi Han, and Dongmei Zhang. TUTA: Tree- based transformers for generally structured table pre-training. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 1780-1790, 2021. 1,3
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi. Quoe V Le, Denny Zhou, al. Cfain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 0422824-248833 2022. 1, 2, 3,
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv 2rXiv:2305.10001 2023.
Yunhu Ye, Binyuan Hui, Min Yang. Binhua Li, Fei Huang, and Yongbin Li. Large language models are versatile decomposers: Decompose evidence and questions for table-based reasoning. arXiv preprint arXiv:2301. 2023. 3, 4, 6, 9, 13, 14, 16
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuur- mans, Claire Cui, Olivier Bousquet, Quoc V Le, et al. Least-to-most prompting enables complex reasoning large language models. In International Conference on Learning Representations. 2022. 1,
12

---

Published as a conference paper at ICLR 2024
APPENDIX
A ATOMIC OPERATIONS IN CHAIN-OF-TABLEE
A.1 INTRODUCTION
In this study, we adopt set of five table operations, which are commonly-used in SQL and DataFrame development, as an example. We note that our framework can trivially accommodate additional operations, which we leave for future work.
f_ __add_columm () adds a new column to the table to store intermediate reasoning or computa- tional results.
seeeeettroowe selects subset of rows that are relevant to the question. Tables may contain irrelevant information for the given question (Ye et al., 2023). This operation helps locate the necessary context.
f_sselec__eeeeeeeeee co lumn() () selects a subset of columns. A column usually corresponds tribute in the table. This operation allows the model to locate the necessary attributes to answer the question.
at-
f -_group_by by () groups the rows by the contents of specific column and provides the count of each enumeration value in that column. Many table-based questions statements involve counting, but L.MSs are not proficient at this task (Imani et al., 2023).
f f_sort_by () sorts the rows based on the contents of a specific column. When dealing with questions statements involving comparison extremes, LLMs can utilize this operation to rearrange the rows. The relationship can be readily inferred from the order of the sorted rows.
A.2 ABLATION STUDY
To demonstrate the effectiveness of proposed atomic operations, perform ablation study by creating five leave-one-out variants of our method, each of which removes one of the pre-defined operations from the pre-defined operation pool. For example, w/o f add_ co lumn () means f_ f_add_ co cumm () is removed from the operation pool. As a result, the LLM is only able to plan from the remaining four operations (f L___seeect__e cottttoouuu commn f f_lf___________e select__ row, f __group__i byy, and f aessrtrbbyyy to construct operation chains. We report the results of the ablation study in Table 5.
Table 5: Ablation study of the atomic operations used in CHAIN-OF- with PaLM 2 on Wik- iTQ and TabFact datasets. observe that row selection and group-by operations have the biggest impact on the final table understanding performance.
TabFact
WikiTQ
Prompting
Accuracy
Accuracy
CHAIN-OFE MAAN-OF-TABLL TA-LEBLE w/o
86.61
67.31
[_add_collume
85.23 (-1.38)
65.88 (-1.43)
w/o
f_select_colume
82.61 (4.00)
65.68 (-1633)
w/o asselec_roww(
82.21 (4440)
65.066 (-2.25)
w/o f_group_by (y)
84.78 (.183)
61.88 (-5.4))
w/o
f f_sort_by()
86.21 (-0.40)
65.85 (-1.46
As shown in Table 5, all five operations contribute to the final state-of-the-art performance of CHAIN-OP-TABLE, removing any operation results in a decrease in performance. In particular, we observe that seelect_roww
and f_selectt ceselecc_cllumme collum () contribute the most on TabFact, while f_group_be by () contributes the most WikiTQ. This suggests that different tasks require different operations to help the LLM determine the correct answer. Therefore, leveraging the LLM design custom operation chains through dynamic planning naturally fits different tasks, resulting in superior performance of our method.
13

---

Published as a conference paper at ICLR 2024
B EXPERIMENTS OF CHAIN-OF-TABLE ON FETAQA
Table 6 shows that CHAIN-OF-TABLE also improves the performance free-form question answer- ing on FeTaQQ across all metrics, whereas Dater (Ye al., 2023) fails to improve the ROUGE scores compared with End-to-End QA. We also observe the marginal improvement of CHAIN--FFTABLL compared with the baseline methods. We attribute this to the nature of the n-gram text similarity metrics of ROUGE-1/2/L (Lin, 2004). As discussed in Maynez et al. (2023); Dhingra et al. (2019), these metrics are known be insensitive capturing improvements when using in-context learning since the model is unable to learn the expected style of the long form text just from an instruction or few examples. We sample several cases from FeTaQQ as shown in Figure 5 where the ROUGE met- rics assign low scores; however, upon review, observe that the generated answers were correct.
Table 6: Table understanding results on the FeTaQQ benchmark using PaLM 2 with the best results in bold and improvements over Dater (Ye et al., 2023) reported. (underline denotes the second-best performance; bold denotes the best performance; the improvement is measured against the second- best performing method.)
FeTaQA
Prompting
BLEU
ROUGE-1
ROUGE-2
ROUGE-L
End-to- te-t---nd QA
28.37
0.63
0.41
0.53
Dater (Ye et al., 2023)
29.47
0.63
0.41
0.53
CHAIN-OF- TABLE (ours)
32.61 (+3.14)
0.66 (+0.03)
0.44 (+0.03)
0.56 (+0.03)
Example from FeTaQA
Who the last finishers in the 2000 Summer Olympics Mens 108 metre freestyle?
Question:
Answer:
Russia's Denis Pimankov (49.36) and Australia's Chris Fydler (49.44) rounded out the finale.
Prediction: The last two finishers the 2008 Summer Olympics Mens 100 metre freestyle were Chris Fydler and Denis Pimankov. Results: ROUGE-1-8.33; ROUGE-2=0.122 ROUGE-L=8.11
Explanation: The generated response correctly answers question the sentence styles are different. From the metrics, we see the ROUGE scores below the average.
Figure 5: Result example of CHAIN-OF-TABLE on FeTaQA using the ROUGE scores as metrics, where the ROUGE metrics assign very low scores but the generated answers were correct.
14

---

Published as a conference paper at ICLR 2024
C INFERENCE PARAMETERS AND NUMBER OF DEMO SAMPLES OF CHAIN-OF-TABLA
We report the parameters and demo sample numbers used in CHAIN-OF-TABLEF in Table 7, and 9. Overall, we annotate 29 samples and use them across different datasets. There are a large overlapping between the usage on different functions. For example, we use the same demo sample to introduce how to use __add_co lumn in the function Dynami cP across different datasets. We guarantee that all demo samples are from the training set so they are unseen during testing. We argue that this further demonstrates our framework does not rely on a specific set of demos and can be well generalized new datasets with the same prompts.
Table LLM parameters and number of demo samples in CHAIN-OF- TABLE on WikiTQ
WikiTQ
Function
temperature
top_p
decode_steps
n_ n_samples
n_ n_des
DynamicPlan()
1.0
200
f_add_column! ()
0.0
1.0
200
f f_select_ r_w ()
1.0
1.0
200
8
f_select__ollue
1,0
1.0
200
8
f_group_byc f_sort_by() ()
1.0
0.0
1.0
200
0,0
1.0
200
Table 8: LLM parameters and number of demo samples in CHAIN-OF-TABLE on TabFact
TabFact
Function
temperature
top_p
decode_steps samples
n_demos
DynamicPlan()
0.0
1.0
200
f_add_column( ()
0.0
1.0
200
f_select_row
0.5
1.0
200
8
f_select_ peellct_ccluunç
1.0
f_group_by f_sort_by query
0.0
1.0
200
0.0
1.0
200
0.0
1.0
200
Table 9: LLM parameters and number of demo samples in CHAIN-OF-TABLE on FeTaQA
FeTaQA
Function
temperature
top_p
decode_sted
samples
n_ demes
DynamicPlan() f Ladd_collumn(
0.0
1.0
200
0.0
1.0
200
f f_select_row(
1.0
f_select (eellct_coluun()
1.0
1.0
200
8
f f_group_by()
0.0
1.0
200
f_sort_by()
0.0
1.0
200
query
1.0
15

---

Published as a conference paper at ICLR 2024
D TABULAR FORMAT ENCODING COMPARISON
In alignment with prior studies Liu et al. (2023b; 2021); Jiang et al. (2022) and the baseline methods Cheng et al. (2022); Ye et al. (2023), adopt PIPE encoding in CHAIN-OFTTABLE (as shown in Appendix E). This decouples the performance gains the proposed tabular CoT with atomic operations from the influence of various table formatting choices.
To further understand the impact of different encoding methods on table understanding performance, conduct additional experiments using 3 additional table representations: HTML, TSV, and Mark- down. For these experiments, we use End-to-End QA on WikiTQ with PaLM 2 as a running exam- ple. The results shown in Table 10. These findings show that different tabular format encoding methods lead to different outcomes. Notably, the PIPE format adopted in our study yields the highest performance among the four encoding methods tested.
Table 10: Tabular format encoding comparison on WikiTQ with PaLM
Tabular Format Encoding
Prompting
PIPE
HTML
TSV
Markdown
End-to-End QA
60.6
56.1
58.1
58.0
E PROMPTS IN CHAIN-OF-TABLE
E. DynamicPlan
We illustrate the prompting method used by DynamicPlan Q, is the latest intermediate table and Q is its corresponding question; chain is the list of operations performed on the table.
Q,, chain) in Figure 6 where T
With Dynami cala,, the LLM can generate the rest of the operation chain for the current sample (Figure 6(c)). We denote the generated operations as f (args,))) + the last operation of the imput open-ended operation chain. Although a complete chain is generated, we only consider the first generated operation, fi+1, and ignore the rest of the generation including the arguments and remaining operations. fi+1 is generated based on the latest intermediate table from the previous operations, while the generation subsequent operations are not based on the most up-to-date intermediate table so there could be mistakes in the generated contents. Therefore, we believe f,+1 is the most reliable generation among all operations the generated chain. See Figure for more detailed prompts.
- [E] given that f, is
E.2 GenerateArgs
We illustrate the demonstration examples and prompts used by GenerateArgs (T, f) in Fig-
7 where is the latest intermediate table and Q its corresponding question; f is the selected tabular operations. The detailed prompts for each operation and the regular expressions for extract- ing the generated arguments are follows.
f add_ cold_coluue lumn: See Figure 10.
f_ select_row See Figure 12.
f_select_co lumn: See Figure 11.
f f_groupbby by: See Figure 13.
_y:rt_by: See Figure 14.
E.3 Query
We illustrate the prompts used by Query (T, Q) Figure 8 where T is the resulting table from CHAIN-OF-TAALLE and Q is the question. See Figure 15 for more detailed prompts.
16

---

Published as a conference paper at ICLR 2024
F IMPLEMENTATION DETAILS OF BASELINE METHODS
We run Text-to-SQL. and Binder using the official open-sourced code and prompts https: // g thub. com com/HKun HHKUNLP/Binded
We Dater using the official open-sourced code and prompts in https:: / / thub. com/ / Allilaalll babaResearch) DAMO- ConvAI. We revise the code to use publicly available GPT 3.5 and PaLM 2 (Section 4) as the LLM backbone instead of the OpenAl Codex due to its inaccessibility. We report the detailed prompts used in other baseline methods as follows.
End-to-End QA: See Figure 16.
Few-Shot QA: See Figure 17.
Chain-of- Thought: The demonstration samples of Chain-of- Thouugh for WikiTQ and TabFact are from Chen (2023) (https: /gg thub. www/com///////e how///whhcchheeeTaaleoooo.o See Figure 18.
Prompt Template of DynamicPlan(T. Q, chain)
(4) Atomic Operation Deme
Introduce how atomic operations
[SRRTALT (ARER]
Operation: www..e/.coommnfooann...
Papulation)
Queation: (QUO2TACM))
Explanation: The question asks We only need column Country and
Population to answer the question we use select column to selecc them.
Explenation: CEEECCUCION 20 TOW.TT000EEEAAAA00000000000 OCERATION)
Complete operation chain with ending
II: Operation
Question: was the year [_selett rovtrow row 2)
part a-laguu??
IOFERATTOR CTATE, TNSTRICTTO))
Operation Chain: [_dd__clluaa _oluun(YYar)
1BERIAL
setect column(Year, League) - coluttoooum [E]
Queation: (QUHSTICK)
Incomplete operation chain
trigsering
the LLM complete the choin with condidate operations
Part III: Enpot Sample
Question:
with
ESERTALLEO JATEST (7))
Operation Chain: [_add column(Country)
Questioo: (OUTSTTTON.C))
Candidabes:: (ROOGIDER MERO OPSEATIONS1
// Extroct the first generated operation [sselct, row next based
Operation Chain: IINOOMELENS OPERATION CHATN [chainl
operation
up-to-dete
select row (ro rew 2... row 10 [8]
Illustration of Dynami cP (T, Q, chain). Left: Overall prompt template and expected generation, including (a) demonstration of how atomic operations work, (b) demonstration of how to generate a complete operation chain to answer given question, and (c) prompt for actual input table and its question, and its expected generation from the LLM (highlighted in green). Right: Examples and brief explanations of each part the prompt and generation.
Figure 6:
generation pre-defined regular expressions orguments.
Prompt Template of GenerateArgs(T, Q, f)
column(j:
[OPERATION INSTRUCTICN
answer tsct_ sodt_ccouun/oooutrr.... The value: ESP, RUS,
1SRTTALIPED (ARGR(E))
When [- seleec_rrowwe)
Question: (OURSTION
The answer ig: sesecct, rawtrow row 2, 3...)
Explanation:
When selett [=[Seleet_cluumn
[EXPLAMAT 2081
The answer ia: /OPERATTON AND ARG(OKNTTS)
The answer [eeeecc Bobeo__oourm//oontyy Rank)
When f= sort bytk:
answer byyCont)
from-large-to-mmlll
When f= fof. f.r.group.bee by(:
The answer 1.grupp. w.t.roup.by.Counne byContyry)
Illustration of GenerateArgs (T' (I,, After a specific operation f is sampled by the LLM as the next operation, we ask the LLM to generate the required arguments by calling GenerateArgs. Then we parse the generation results of the LLM according the pre-defined templates to extract the arguments.
Figure 7:
17

---

Published as a conference paper at ICLR 2024
Prompt Template of Query(T, Q)
// Directly generate the onswer with the resulting table from Cholit-of-Table
(GENERAL _ AUCTIOCT
(SERIAL:BED RISULTING
Question: (OUE3:1ON
Anewer: (answer)
Figure 8: Illustration of Query ,)) The resulting table from operation chain serves as proxy for the intermediate thoughts of reasoning, allowing to directly generate the answer without providing the reasoning chain in textual format.
Prompt
If the table only needs few rows these rows for it. For example,
the question, f_select_row() select
/*
col Home team Home Team Score Away Team Away Team Score | Venue Crowd
row 1 st kilda 13.12 (90) melbourne 13.11 (89) moorabbin oval 18836
row south melbourne 9.12 (66) footscray 11.13 (79) lake oval 9154
row 3 richmond 20.17 (137) fitzray | 13.22 (169) meg 27651
*/
Question Whase home tean score is higher, richmond or st kilda?
Function: sselect_row(ret 1, row
Explanation: The question asks about the home team score of richmond and st kilda. We need to know the the information of richmond and st kilda in row and row We select row and row
If the table only needs few calumns answer question, f_select_column( to select these columns for it. For example,
If the question asks about items with the same value the number of these items, we use f_group_by() to group the items. For example,
If the question asks about the order of items in column, use f_sort_by() to sort the items. for example,
Here examples using the operations to the question. /+
Division League Regular Season Playoffs Open Cup
row 1 2601/01/02 2 USL A-League 4th, Western Quarterfinals Did not qualify
row 2 2002/08/06
USL A-League 2nd, Pacific 1st Round
not qualify USL First Division 5th Quarterfinals 4th Round
row 5 2005/03/24
*/
Question: what was Function Chain: cod__column(Yerr) -> r_select_rowrrow 1, row 2) ->
year this
part of the a-league?
f_select_column(ver League) F_sort_by(Year
/*
col Rank Cyclist Team Time
UCI ProTour; Points
Country
Alejandro Valverde (ESP) Caisse d'Épargne
10"
48
ESP
row 2
Alexandr Kolobnev (RUS) Team CSC Saxo Bank
s.t. | 30
RUS
Davide Rebellin (ITA) Gerolsteiner s.t.
ITA
row Paclo Bettini (ITA) Quick Step s.t. 20
ITA
row
Franco Pellizotti (ITA) Liquigas s.t. 15 ITA
row Denís Menchov (RUS) Rabobank s.t. 11 RUS
row 7 7
Samuell Sánchez (ESP) Euskaltel-Euskadi s.t.
ESP
Stéphane Goubert (FRA) Ag2r-La Mondiale
row 9 9
Haimar Zubeldia (ESP) Luskaltel-Euskade
2" 3
ESP
David Mancoutié (FRA) Cofidis
*/
Question: which country had the most cyclists finish within the top 18? The next operation must of F_select_row() f_select_columm f_group_by( or f_sort_by () Function Chain: (_add_collum((ountr)))
Completion
L.select_row(rogn 1, row 10) -> f_select_coll (oabbctt_cllmm((Cutrrr)) -> f_group_by (Counttr) -> <END>
Figure 9;
chain) Prompt used for WikiTQ
18

---

Published as a conference paper at ICLR 2024
To answer the question, can first use f_add_column() to add more columns to the table.
The added columns should have these data types:
1. Numerical: the numerical strings that can be used in sort, sum
Datetype: the strings that describe date, such year, month, day
3. String: other strings
/+
col week when | kickoff Opponent | Results; Final score Results; Team record
row 1 Saturday, April 13 7:00 at Rhein Fire W 27-21 1-8
row 2 : 2 Saturday, April 20 7:08 p.m. London Monarchs | 37-3 | 2-8
Sunday, April 6:00 p.m.
Barcelona Dragons | 33-29
*/
highest attendance? The existing columns are: "Week", "when", "Kickoff", "Opponent", "Results; Final score", "Results; Team record", "Game site", "Attendance". Explanation: the question asks about the date of the competition with highest row is about one competition. We extract the value from column "Attendance" and create a fifferent column "Attendance number" for each The datatype Numerical, Therefore, the answer 1s: f_add_ co (Attennaneee number). The value: 32092 34186 17503
Question: what
competition
Each
/+
col Rank Lane Player | Time
Tereshkova (KAZ) 51.86
row 2 : 6 Manjeet Kaur (IND) $2.17
Asami
(JPN)
*/
Question: tell
number athletes
japan.
The existing columns
Rank, Lane, Player, Time.
Explanation: the question asks about the number of athletes from japan. Each row is about athlete. need know the country each athlete. extract the value from column "Player" and create different column "Country of athletes" for each row. The datatype is string. Therefore, the answer is: f_add_colummee colddcoolunncnuttttcc of athletes). The value: KAZ
IND
JPN
Figure 10: Demos used for GenerateArgs(T,e) add_collunn) We use the regular expres- sion: f f_add_ co lumn (.*)). The value: (.*) to extract the arguments from the generated text.
Use F_select_column' to filter out useless columns in the table according to information in the statement and the table.
/*
stable_caption": "south wales derby", "columns": ["competition" "total matches", "cardiff win", "draw", "swansea win"],
["competition", "league", "fa cup", "league cup"), ["total matches", "55" "2", "5"], ["cardiff win", "19", "g", "2"1, ["draw", "16", "27" "g"], ["swansea win", "28", "2", "3"1
*/
statement there
cardiff wins that have draw greater than 27.
similar words link to columns cardiff wins cardiff win draw -> draw
column
Link columns
27 => draw
semantic sentence Link to columns
None
The answer is f_select_columm(ccrrrt win, draw])
Figure 11: Demos used for GenerateArgs selecc_collueee We the regular expression: f f_select_colum ())+]] to extract the arguments from the generated text.
19

---

Published as a conference paper at ICLR 2024
Using [_select_row() to select relevant rows in the given table that support or oppose the statement. Please use f_select_row((x) select all in the table.
/+
table caption 1972 vfl season.
home
score away
away score venue crowd
row 1 kilda 13.12 (90) | melbourne 13.11 (89) moorabbin oval 18836
row 2 south melbourne | 9.12 (66) footscray 11.13 (79) lake oval
9154
row richmond 28.17 (137) fitzroy 13.22 (160)
27651
row :gellong 17.10 (112) collingwaod | 17.9 (111) kardinia park 23108
north melbourne 8.12 (60) carlton 23.11 (149) arden street oval 11271
row 6 hawthorn 15.16 (196) essendon 12.15 (87) vfl park 36749
*/
statement :
what is the away team with the highest score?
explain the statement want to ask the away team of highest away team score. the hi highhst is 23.11 (149). it
away tean
the 5.50 need row 5.
The answer is r_select_row[rrow 5])
Figure 12:
Demos used for GenerateArgs (T, f f0f__seelll _es_eeeee_erreeee row). We use the regular pression: F_select_row([( to extract the arguments from the generated text.
To answer the question, we can first use f_group_by() to group the values in column.
/*
col Rank
Lane | Athlete Time Country
row 1
Nanjeet Kaur (IND) 52.17 IND
row 2
Olga Tereshkova (KAZ) 1 51.86 KAZ
Pinki Prananik (IND) 53. IND
row 4 : 4
Tang Xiaoyin (CHN) 53.66 CHN
Narina Maslyonko (KAZ)
KAZ
*/
The existing columns number
Question: tell
from japan.
Rank, Lane, Athlete, Time, Country.
Explanation: The question asks about the number of athletes from India. Each row is about athlete. We group column "Country" to group the athletes from the country. Therefore, the answer is: f_group_by
Figure 13: Demos used for GenerateArgs ff leff_rrou_bby/ We the regular expres- sion: f f_group_by (.*)) to extract the arguments from the generated text.
To answer the question, can first use f_sort_by() to sort the values in column to get
order of the items. The order can be "large small" or "small to large",
The colunn to sort should have these data types:
1. Numerical: the numerical strings that can be used in sort
DateType: the strings that describe date, such year, month, day
3. String: other strings
/+
col Position I Club Played
Points
Wins
| Draws
Losses
|
Goals for | Goals against
row
Malaga CF | 79
22
13
7
72
row 10 : 10 CP Merida 42 59
15
14
13
48
41
3 Numancia
73
68
*/
Question: what club placed
last position?
The existing columns are: Position, Club, Played, Points, Wins, Draws, Losses, Goals for, Goals against
Explanation: the question club. We need to know the order of position from last to front. There is a column for position and column name Position. datatype Numerical. Therefore, the answer is: __sort_yy(Posttion) the order is "large to small".
about the club in the last position, Each about
Figure 14: Demos used for GenerateArgs Q,, f f_sort_by). We the regular expression: f_sort_by ), the order is "(.*)", to extract the arguments from the generated text.
20

---

Published as a conference paper at ICLR 2024
Prompt
Here is the table this question. Please understand the table and the question:
/*
City Passengers Number Ranking Airline
row 1
Uni ted States, Los Angeles
14749 |2 Alaska Airlines
row 2
Uni ted States, Houston 5465 8 United Express
Canada, Calgary 3761
Air Transat, WestJet
row 4
4
Canada, Saskatoon 2282
4
Canada, Vancouver
Air Transat
row 6
United States, Phoenix
1829 1 Airways
row Canada, Toronto 1282
Transat, CanJet
row
Canada, Edmantan 110
row 9 United States, Oakland
187 |
Question: how many more passengers flew to los angeles than to saskatoon from manzanillo airport in
The anwser is: 12467
Here is the table answer this question. Please understand the table and the question:
/+
Country
row 1
1
ESP
row 2
RUS
ITA
row 4
4
ITA
row 6
RUS
row 8 FRA
row 9 9
|
ESP
FRA
*/
Group the rows according column "Country":
/*
Group
Country Count
ITA 3
ESP 3
RUS
2
FRA 2
Question: which country had the most cyclists in top 19? The answer
Completion
Italy.
Figure 15: Prompt Example used for Query (T Q)
21

---

Published as a conference paper at ICLR 2024
Prompt
Here is the table answer this question. Answer the question.
/*
col
League FA
League Cup
Trophy
Total
row 1
Scot Bennett
5
0
0
0
5
Danny Coles
row 3
Liam Sercombe
1
0
0
6
row Alan Gow 4
8
John O'Flynn
11
6
12
row 6
Guillem Bauza
2
6
Jimmy Keohane
row 8 Patt Baldwin
0
Jamie Cureton
row 10 Arron Davies
3
8
6
row : Jake Gosling
1
6
OWN GOALS
row 13 Total
8
8
*/
Question: does pat or john have the highest total? answer
Completion
John.
Figure 16: Prompt of End-to-end QA used for WikiTQ.
22

---

Published as a conference paper at ICLR 2024
Prompt
Here is the table to answer this question. Answer the question.
/*
col Rank
Cyclist Team Time
UCI ProTour; Points
row 1 111
Alejandro Valverde (ESP) Catsse d'Epargne 5h 29' 10"
40
Alexandr Kolobnev (RUS) Team Saxo Bank s.t. |
row 3
3
Davide Rebellin (ITA) GeroLsteiner s.t.
25
row
Paclo Bettini (ITA) Quick Step s.t. 28
row
Franco Pellizotti (ITA) Liquigas s.t. 15
row 6
Den's Menchov (RUS) Rabobank s.t.
row
Samuel Sánchez (ESP) Euskaltel-Euskadt
s.t.
row 8 8
stéphane Goubert (FRA)
Ag2r-La Mondiale 2" Zubeldía (ESP) Euskaltel-
row 10 10
David Moncoutié (FRA) Cofidis 2"
+/
Question: which country had the most cyclists finish within the 10? The answer 1s: Italy.
Here is the table to answer this question. Please provide your explanation first, then answer question
short phrase starting by 'therefore, answer is:
/*
col Rank Cyclist | Team Time UCI ProTour; Points
row
Alejandro Valverde (ESP) Caisse d'Epargne 5h 29' 10" 40
row 2
Alexandr Kolobnev (RUS) Team CSC Saxo Bank s.t. | 30
Davide Rebellin (ITA) Gerolsteiieer s.t.
row 4 4
Paolo Bettini (ITA) Quick Step s.t. | 20
row
Franco Pellizotti (ITA) Liquigas | s.t.
row 6
Denis Henchov (RUS) Rabobank s.t. 11
row 7
Samuel Sánchez (ESP) Euskaltel-Euskadi s.t. 7
row
Stéphane Goubert (FRA) Ag2r-La Mondiale
2"
row 9 9
Haimar Zubeldia (ESP) Euskaltel-Euskade
2"
David Moncoutié (FRA) Cofidis 2"
+/
Question: how many players got less than points? The answer is: 4.
Here is the table answer this question. Answer the question.
/*
col
League FA
League Cup
Trophy
Total
row 1 Scot Bennett
|
0
0
0
row Danny Coles
0
row Liam Sercombe
1
0
1
row Alan Gow 4
0
row John O'Flynn
11
6
12
row 6 Guillem Bauza
2
6
Jimmy Keohane
row 8 Pat Baldwin
0
row 9 Jamie Cureton
row Arron Davies
0
row Jake Gos Gosling
1
0
OWN GOALS
row 13 : Total
8
0
*/
Question: does pat or john have the highest total? The answer is:
Completion
John.
Figure 17: Prompt of Few-shot QA used for WikiTQ
23

---

Published as a conference paper at ICLR 2024
Prompt
Here is the table
this question. Please provide your explanation first, then answer the question in short phrase starting by 'therefore, the answer is:'
/*
col Rank
Cyclist Team Time
UCI ProTour; Points
:
Alejandro Valverde (ESP) Caisse d'Epargne
29'
49
row
Alexandr Ko Kolobnev (RUS) Team C5C Saxo Bank 1 s.t. | 30
row 3 3
Davide Rebellin (ITA) Gerolsteiner s.t.
25
Paolo Bettini (ITA) Quick Step
28
row 5 5
Franco Pellizotti (ITA) Liquigas | s.t.
Denis Nenchov (RUS) Rabobank s.t.
row 7 7
Samuel Sánchez (ESP) Euskaltel-
s.t.
row
Stéphane Goubert (FRA) Ag2r-La Mondiale
row Haimar Zubeldia (ESP) Euskaltel-EEske
2"
row 19 David ié (FRA) Cofidis
*/
Question: which country had the most cyclists finish within the top 10? Explanation: occurs answer is: Italy.
times in the table, more than any others. Therefore, the
Here is the table answer this question. Please provide your explanation first, then answer the question in short phrase starting by 'therefore, the answer is:"
/+
col Rank Cyclist | Team Time UCI ProTour; Points
Alejandro Valverde (ESP) Caisse d'Epargne
29'
49
row 2 2
Alexandr Kolobnev (RUS) Team CsC Saxo Bank s.t. | 30
row 3
Davide Rebellin (ITA) Gerolsteiner s.t. 25
row
Paolo Bettini (ITA) Quick Step s.t. 28
row 5 5
Franco Pellizotti (ITA) Liquigas | s.t.
Denis Menchov (RUS) Rabobank s.t.
row 7 7
Samuel Sánchez (ESP) Euskaltel- Euskadi
s.t.
stéphane Goubert (FRA) Ag2r-La Mondiale
row 9
Haimar Zubeldia (ESP) EuskaltellEEusadh
2"
row 10
David Moncoutié (FRA) Cofidis |
*/
Question: how many players got less than 10 points?
Explanation: Samuel Sánchez, Stéphane Goubert, received less than 19 points. Therefore, the answer is: 4.
Zubeldia and David Moncoutié
Here is the table to answer this question. Please provide your explanation first, then answer the question in short phrase starting by 'therefore, the answer is:"
/*
col Name League FA Cup
League Cup
JP Trophy Total
Bennett |
row Danny Coles
3
0
0
3
row Liam Sercombe
1
row 4 Alan Gow 4
0
row John O'Flynn
11 0
6
12
row Guillem Bauza
row 7 Jimmy Keohane
3 0
0
6
Baldwin
|
row Jamie Cureton
20
8
29
row Arron Davies
row Jake Gosling
1
Total row : OWN GOALS 6
*/
Question: does pat john have highest total? Explanation:
Completion
John 0' O'Flynn has the highest total of goals. Pat Baldwin has the lowest total of 1 goal. Therefore, the answer is: John.
Figure 18: Prompt of Chain-of- Chain-of-Thought used for WikiTQ
24


Reviews:
[{"review_id": "7z70rzmbzR", "reviewer": "Reviewer_4HBR", "summary": "Chain-of-thought (CoT) prompting enables complex reasoning capabilities through intermediate reasoning steps. Still, these approaches incorporate the reasoning chain in a textual context and can't handle tabular data. The paper proposes a Chain-of-Table framework, where tabular data is explicitly used in the reasoning chain as a proxy for intermediate thoughts. The framework uses in-context examples to iteratively generate operations and update the table to represent a complex reasoning chain.", "questions": "Should the entire table fit inside the context length of the models? What happens if the tables are large? I would encourage the authors to evaluate their approach on larger tables.\n\nIn addition, a recent work that tries to modify CoT to leverage tabular structure [1] isn't discussed. I encourage the authors to distinguish with [1], further strengthening the submission. \n\nMost existing LLMs often ignore the semantics of table structure, and it is beneficial to encode the structure in some form. Have the authors considered ways to incorporate the structure and semantics of the tables? As of now, the proposed techniques ignore them sans the operators defined and treat the table as yet another set of tokens.\n\n[1] Jin Ziqi and Wei Lu. 2023. Tab-CoT: Zero-shot Tabular Chain of Thought. In Findings of the Association for Computational Linguistics: ACL 2023", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Recently, there has been a lot of interest in developing methods to improve the performance of LLMs on tabular data.\n- The paper addresses one of the key stumbling blocks in improving the performance of LLMs on reasoning over tables.\n- The paper is well-written and easy to follow.\n- Empirical results show that the proposed approach outperforms the baselines on real-world datasets.", "weaknesses": "One of the issues with the paper is how they evaluate, especially the choice of benchmarks. I encourage the author to evaluate their approach over a diverse class of tasks, especially tasks like Table summarization, Column type annotation, Row augmentation, etc. In addition, I also encourage the authors to include open models too.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xyegwnMopa", "reviewer": "Reviewer_eSFE", "summary": "The paper introduces a novel framework called \"chain-of-table\" for reasoning on tabular data. This framework extends \"chain-of-thought\" reasoning to tables. In the chain-of-table approach, a dynamic planning process is employed, which utilizes LLMs to choose predefined operations. These operations are executed on the table at each step of the reasoning process.\n\nThe proposed chain-of-table framework demonstrates superior performance compared to both single-turn baseline methods for tabular reasoning and traditional textual chain-of-thought reasoning. This improved performance is validated on two table QA datasets, WikiTQ and FeTaQA, as well as TabFact, a dataset designed for table fact verification.", "questions": "1. In each dataset, could you clarify how many reasoning steps the chain-of-table method requires? Additionally, it would be helpful to understand the total number of input and output tokens required in comparison to the baselines.\n2. For Figure 2, could you specify the number of examples utilized in each of its parts?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed chain-of-table is simple and effective. This highlights the value of decomposing reasoning in tabular tasks, as opposed to employing single-step table reasoning methods.\n2. The good performance underscores its effectiveness compared to baseline methods across multiple tabular reasoning datasets.", "weaknesses": "1. The proposed method is an extension of chain-of-thought to tabular data. Each reasoning step is constrained by predefined operations on the table. However, it raises questions about the adaptability of the chain-of-table framework to incorporate new operations or external knowledge, such as contextual information related to the table.\n2. Chain-of-table requires a table, which could be large, in each reasoning step. This may significantly increase the computational cost. (See my questions below.)\n3. While it is not necessarily a weakness, it would be beneficial to evaluate the proposed method with an open-sourced model (e.g. Llama-2) to understand whether the framework can be easily adopted by other models. One concern is that chain-of-table relies on LLMs' ability to comprehend the defined operations and reasoning chains, and it is uncertain whether other LLMs can seamlessly adapt to these requirements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dTZc7RsELr", "reviewer": "Reviewer_3YkD", "summary": "The paper aims to effectively leverage tabular data in the reasoning chain. To achieve this, the paper proposes the chain-of-table that conducts step-by-step reasoning as step-by-step tabular operations to form a chain of tables. Empirical results on three benchmarks verify the effectiveness of the proposed method.", "questions": "Please refer to the above weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The investigated problem of leveraging and understanding structural data like tables is important and practical while existing LLMs can not solve it well.\n\nThe proposed method does not require training (or fine-tuning) of the existing LLMs.\n\nThe design of atomic operations is novel and also reasonable.\n\nThe overall reasoning procedure of chain-of-table is step-by-step, explainable, and effective.\n\nThe paper is solid from a technical perspective, and extensive experiments are conducted.\n\nThe presentation and drawn figures are generally clear and easy to understand.\n\nSeveral case studies are also elaborated on in the Appendix.", "weaknesses": "The proposed method only achieves marginal improvements in some cases, e.g., TabFact and ROUGE-1/2/L datasets. I would suggest the paper discuss the potential reasons.\n\nThe observations in Figure 4 are quite interesting. It seems that a longer chain does not consistently bring more accurate results. What are the underlying reasons for this?\n\nDater (Ye et al., 2023) should be the most important baseline for comparison. I would suggest the paper make a further comparison with in-depth analysis from the perspective of methodology, e.g., a comparison between one-step and multi-step reasoning on tabular data.\n\nBesides, how efficient is chain-of-tables when dealing with large-scale data? It seems that the running-time efficiency is known from the current draft.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t9AbVC8TSd", "reviewer": "Reviewer_xr4a", "summary": "This paper proposes chain-of-tables prompting for solving table-based reasoning tasks. Specifically, instead of generating reasoning chains in natural language or code, the paper generates reasoning chains as a list of atomic operations on the table, and use the intermediate updated table to represent the reasoning process. LLMs are used to generate the operation chains, and each operation is done by executing it in some programming languages. Experiments show that the proposed method outperforms existing baselines on three benchmarks.", "questions": "1. The added API calls in Binder also add additional columns to the table, how is the proposed method different from the operation used in Binder?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is new in that it adapts the previous chain-of-thought prompting to the table-related tasks, and it improves the understanding of intermediate reasoning results.\n2. Experiments on three benchmarks show the advantages of proposed method over a broad range of baselines including end-to-end methods and other program-aided methods.", "weaknesses": "1. The proposed method requires significantly more number of queries to LLMs compared to baselines such as chain-of-thought and Binder, since the method needs to query LLMs twice (one for operation generation and one for arguments generation) for each operation. However, no comparison is provided for performance under the same number of queries. For example, what is the performance of baselines when they adopt the self-consistency idea (which is shown to be beneficial in the Binder paper)? Does the proposed method still have advantage when using the same number of queries?\n2. The pre-defined five atomic operations seem to significantly limit the available operations. For example, aggregations such as SUM, MAX, MIN, etc. can be easily done in SQL. How are these aggregations done in the proposed method? No explanation is provided for why these five operations are used in the paper and why they cover the complete set of operations on table.\n3. The presentation of argument generation process is not clear. Based on the prompts in the Appendix, it seems add_column operation directly uses LLMs to generate the new column, whereas other four operations only prompt LLMs to generate arguments that will be fed to the programming language.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)