CRT-QA: A Dataset of Complex Reasoning Question Answering over Tabular Data
Authors: Zhehao Zhang
Xitao Li
Yan Gao
Jian-Guang Lou
Venue: EMNLP 2023 Main (2023)
Keywords: Table QA
Table analysis
Large language model reasoning
Large language model with tool-use
TL;DR: We construct a dataset of complex reasoning question answering over tabular data to evaluate LLMs and propose a effective method.

Abstract: Large language models (LLMs) show powerful reasoning abilities on various text-based tasks. However, their reasoning capability on structured data such as tables has not been systematically explored.  In this work, we first establish a comprehensive taxonomy of reasoning and operation types for tabular data analysis. Then, we construct a complex reasoning QA dataset over tabular data, named CRT-QA dataset (Complex Reasoning QA over Tabular data), with the following unique features: (1) it is the first Table QA dataset with multi-step operation and informal reasoning; (2) it contains fine-grained annotations on questions' directness, composition types of sub-questions, and human reasoning paths which can be used to conduct a thorough investigation on LLMs' reasoning ability; (3) it contains a collection of unanswerable and indeterminate questions that commonly arise in real-world situations. We further introduce an efficient and effective tool-augmented method, named ARC (Auto-exemplar-guided Reasoning with Code), to use external tools such as Pandas to solve table reasoning tasks without handcrafted demonstrations. The experiment results show that CRT-QA presents a strong challenge for baseline methods and ARC achieves the best result.

CRT-QA: A Dataset of Complex Reasoning Question Answering over Tabular Data
Zhehao Zhang'; Xitao Li Yan Gao', Jian-Guang Lou" 'Darmouth College, 2Xi'an an Jiaotong University
Microsoft Research Asia
zhehao. zhang. trouattmouth, edu, wui tenye@stu. jtt.. edu.eduucom cn {yan.gao, jlou]emicrosoof. com com
Abstract
Question
Competitions
Does being in the bundeslige increase
Large language models (LLMs) show power- ful reasoning abilities on various text-based tasks. However, their reasoning capability on structured data such tables has not been sys- lematically explored. In this work, we first establish a comprehensive taxonomy of reason- ing and operation types for tabular data analy- sis. Then, we construct a complex reasoning QA dataset tabular data, named CRT-QA (Complex Reasoning QA over Tabular data), with the following unique features: (1) it is the first Table QA dataset with multi-step op- eration and informal reasoning; (2) it contains fine-gre fine-grained annotations on questions' direct- composition types of sub-questions, and human reasoning paths which can be used to conduct a thorough investigation on LLMs reasoning ability; (3) it contains collection unanswerable and indeterminate questions that commonly arise in real-world situations. We further introduce an efficient and effective tool-augmented method, named ARC (Auto- ceempla-gguieed Reasoning with Code), to external tools such as Pandas to solve table reasoning tasks without handcrafted demon- strations. The experiment results show that CRT-QA presents strong challenge for base- line methods and ARC achieves the best re- sult. The dataset and code are available at https://g/ /ggi thub. -ow.//h--SSTUU/CTTOO..
97.98
bundesilga
of competing in Europe?
bundesliga played lorac
98-99
Answer
11-12
reasoning path
Fine-grained arnotation
Diractness: Implicit @
Step 1:
Find the row/columns of "bundesliga" (Indexing)
Composition Type: Bridging
Step 2: Group the rows/columns based on "Ieague" (Grouping)
Sub QE: what is "Bundesliga"?
Stop 3: Ground colurn "Earapaan Compection" (Grounsing)
what indicative y (3: the count corpeting in European
Step Conduct "COUNT" each value of "European competitions (Augregut
corpeticions higher in
Stop Reasoning "Increase the chances". (Arithectic)
Bundesliga?
Figure I:
This example demonstrates the format of CRT- QA dataset: tables, question-ansse -answers, and fine-grained annotations.
Yao al., 2023a) focus on LLMs' reasoning abili- ties on text-based NLP tasks. However, the capabil- ity LLMs on table reasoning tasks has not been systematically investigated (Chen, 2023a). Eval- uating LLMs' reasoning ability over tabular data and improving their performance can produce sig- nificant impact efficient data analysis, decision- making, and on in real-life applications.
Current Table question answering (Table QA) datasets are primarily concerned with obtaining factoids to answer simple queries and lack in-depth analysis. Although recent works (Chen et al., 202 start to investigate multi-hop "reasoning" questions over tables, they do not have a clear defi- nition of reasoning types and the "reasoning" they investigate (e.g., operations like filtering) does not align with current research LLMs' reasoning ability. Besides, the current Table QA datasets only contain explicitly questions. However, in real-life scenarios, users frequently ask implicit even am- biguous questions over tables.
1 Introduction
Large language models (LLMs) (Brown et al., 2020; Chowdhery et al., 2022; Chung et al., 2022: Touvron et al., 2023; OpenAl, 2023a,b) have re- cently shown emergent abilities, such as the capac- ity for "reasoning". when they are sufficient in size (Wei et al., 2022). A large number of works (Zhang al., 2022a; Wei et al., 2023; Kojima al., 2023;
"Work done during Zhchao and Xitao's 's internship at Microsoft Rescarch Asia.
To fill these gaps and conduct an in-depth anal-

---

ysis of LLMs' reasoning abilities over tabular data, we first establish fine-grained laxonomy of commonly-used reasoning and operations types for table analysis. Different from previous works (Chen et al., 2021b), we separate the steps that can be casily executed using single Pandas or SQL query from reasoning and categorize them oper- ations. Following recent studies on the reasoning capacity of L.LMs(Wei et al., 2023), we focus on informal reasoning which utilizes intuition, experi- ence, and common sense to deduce outcomes.
cant challenge for baseline methods, the current most powerful model, GPT-4, achieves an accuracy of 56.32% through few-shot in-context learning. Our proposed ARC achieves the best result, outper- forming various prompting and tool-use haselines.
Related Works
2.1 TableQA Datasets
Table QA is the task of answering queries concer- ing tabular data. A large number of datasets have been proposed for this task. Datasets such as WTQ (Pasupat and Liang. 2015). WikiSQL (Zhong et al., 2017), SQA (lyyer et al., 2017) and Spider (Yu et al., 2018) contain tables for QA or text-to-SQL tasks. Recently, numerous works construct datasets that require multi-hop reasoning tables: OT- TQA (Chen et al., 2021a), HybridQA (Chen et al., 2021b), TabFact (Chen et al., 2020b), LogicNLG (Chen al., 2020a). AIT-QA (Katsis et al., 2021), MultiModalQA (Talmor et al., 2021), FeTaQA (Nan et al., 2021). However, they are focused on iterated factoid retrieval (Ho et al., 2022) where the definition of reasoning does not align with the reasoning ability of LLMs. Datasets like FinQA (Chen et al., 2022b), TAT-QA (Zhu et al., 2021), MultiHiertt (Zhao al., 2022) and TABMWP (Lu et al., 2023b) focus on numerical reasoning over tabular data. Yin et al., 2022 propose ARCADE, a benchmark of 1,082 code generation using the pandas for tabular data analysis. However, they do not introduce commonsense in the datasets and their labels are not natural languages.
Then, we construct CRT-QA dataset (Complex Reasoning QA over Tabular data) over Wikipedia tables. Answer-based evaluation proves inadequate for assessing LLMs' reasoning ability, as does not fully capture the complexity of their cognitive processes. Nonetheless, devising a robust method for evaluating such reasoning capabilities remains formidable challenge within the field. When deal- ing with complex table analysis queries, humans typically begin by reformulating the questions (pos- sibly implicitly) into more explicit ones, followed by decomposing them into sub-questions, and ulti- mately conducting atomic reasoning. Inspired by this process, we propose fine-grained annotations on the directness of questions, composition types of sub- -ube-uutssnns, and human reasoning paths. To explore the ambiguous questions mentioned earlier, we incorporate a subset of unanswerable and inde- terminate queries. During question collection, we propose a comaan-n--the-loo question generation pipeline that utilizes L.M to generate questions ne- cessitating complex, multi-step reasoning. Our pro- posed pipeline can efficiently produce high-qualig queries while mitigating issues such as biases, in- sufficient complexity, and lack of diversity.
2.2 Language Models for Reasoning
LLMs' reasoning abilities Numerous works (Fu et al., 2023b; Wang et al., 2023b; Zelikman et al.,
We evaluate LLMs (e.g.. GPT-4) with different prompting methods CRT-QA. Inspired by the finding that LLMs can often generate correct rea- soning plans but fail on execution, we propose an efficient and effective method, named ARC (Auto- exemplar-guided Reasoning with Code), to allevi- ate such limitation. Instead of expensive human ef- fort for code design, ARC first uses an instructional prompt to generate exemplar code on the dev set queries and serve as an in-context demonstration for test questions. After executing the generated code with an external Python interpreter, we then inject the output into the prompt and L.LM gen- crates the final answer by reflection. Experiment results demonstrate that CRT-QA poses a signifi-
2022; Creswell al., 2022; Yao et al., 2023a) fo- cus on increasing LLM's arithmetic (Lewkowycz et al., 2022; Chen et al., 2022a; Zhou et al., 2022: Taylor et al., 2022), commonsense (Liu et al., 2022; Madaan et al., 2022) and symbolic reasoning (Zhou et al., 2023). Notably, simply adding "Let's 's think step by step" before each answer using chain-of- thought (CoT) (Wei et al., 2023) prompting which contains number of intermediate steps can better elicit LLM's reasoning ability.
LLM with tools External tools such as web browsers, search engines, Python interpreters, and models of other modalities have been incorpo- rated to complete complex tasks (Nakano et al.,

---

Sub type
Example
Percentage
Which county has had the fastest rate of population growth between 1960 and 2040, in terms of percentage change per decade?
Indexing
84.97%
Did any drivers who retired due to un accident complete more laps than those who retired for other reasons?
Filter
30.20%
Operation
Grouping
there a particular player that Tom has faced more frequently than others?
38.39%
Was there any significant increase decrease in the number of points wins for Juan Garriga over the years?
Sorting
5.64%
Aggregating
How many matches in the 1974-75 FA Cup tournament had scoreless draw?
58.12%
What is the average time difference between a manager's dismissal and the subsequent appointment of their replacement?
Arithmetic
29.26%
Does Juan Garriga have higher success when competing with the Yamaha compared the JJ Cobas Cagiva teams?
Grounding
17.99%
Details: The term of success the question is mapped to win column)
What proportion of the Malaysia Airlines group companies are involved in the airline industry? Details tairline industry belongs principal activities (a column)
Auto-
1.07%
Reasoning categorization
Temporal Rea- soning
How did the Tampa Bay Buecaneers perform during the first half of the 1983
3.89%
season compared to the second half of the season?
Geographical/-
Does Andrea Petkovic have a higher winning percentage in finals matches held in Europe or outside of Europe?
2.55%
Spatial Reasoning
Reasoning with Quantifiers
Are there any shows that have been airing consistently throughout all networks the Canadian Network Television Schedule in 1998-99?
24.16%
Was there consistent difference in the duration of operation between satel-
Others
50.20%
lites launched earlier in the program and the later ones?
Table Our proposed tuxonomy for operation and reasoning types Table QA, accompanied by examples and their proportion in CRT-QA. We emphasize keywords for their respective categories.
2022: Shuster et al., 2022; Cheng et al., 2023; Cobbe et al., 2021; Paranjape et al., 2023; Shen et al., 2023; Lu et al., 2023a). Toolformer (Schick et al., 2023) uses self-supervision to teach LLMs to use multiple tools. However, it needs to fine- tune LLM's parameters, which makes it imprac- tical to apply it to close-sourced LLMs like GPT- 4. Yao et al., 2023b propose ReAct, a prompt- based paradigm to integrate reasoning and act- ing for LLMs. However, ReAct requires multi- ple API callings and hand-crafted exemplars of (Thought, Act, Obs) triplets, which has high call- ing cost and not flexible to transfer to other tasks.
The format of answers a is free-form natural lan- guage. Our dataset focuses on the questions that require multiple steps of operation (0), 02, 02, Onl and reasoning (r1, (n).
Previous TableQA datasets (Chen et al., 2021b) definition of "reasoning" primarily encompasses basic operations like filtering, which does not align with the more comprehensive understanding of rea- soning in current LLM research, which involves higher-order cognitive tasks such as logical, nu- merical, and commonsense reasoning. As re- sult, we separate these steps from reasoning types and define them as operations. Following recent works on LLMs' reasoning ability (Wei et al., 2023; Cobbe et al., 2021), we examine informal reason- ing. which relies intuition, experience, and com- mon sense to draw conclusions and solve problems. Inspired by benchmarks such Big-bench (Sri- vastava al., 2022), we propose a taxonomy on fine-grained reasoning types commonly used in ta- ble analysis. The operation and reasoning types are illustrated in Table
3 CRT-QA Dataset
In this section, we describe the task formulation, and process for collecting tables, questions, an- swers, and detailed annotations for CRT-QA.
3.1 Desiderata
The table-based QA task defined the problem of generating an answer a to a question q based on a table with metadata m using a model M which can be formulated as a MMT,, m, q). In our dataset,

---

A. The overall framework of our proposed quesioo--eeneratoon pipeline.
B. The questions generated using the inibal prompt may meet requirements (e.g. too simple). With human leedback, LLMs can generate questions (features are highlightd)).
Tide: al--raland sarror
huriin championship
Feedback Categories
Player
County
Ave
Human Feedback Instructional prompes that contain low-level lexical features
Incresse complesity
Points
Injoct reasoning types
podraig
offaly
8.0
Midiaaee out-of-scope
Which county the highest points?
bily
kikenny
7.5
Increase diversity
christy
lukenny
7.0
Gonerate complex questions that require math solve them.
Role-play data requirements
LUM
Humzn Evaluzzion
Question Collection
What the probability a player from particular county having cally higher than the average?
You dara analyst and are good at coming up with
Candidate Questions
insightfull questions. Question requirements
1. What average points counties! X Too simlle
2. How many players has higher average score per game
than the overall average score!
Our proposed pipeline of humann-n trmman-n--the-loo toop question generation using LLMs. we first design a role-playing prompt as data analyst with desired questions' requirements for initial question generation. Then, human annotators collect the questions that meet the requirements and provide feedback for question improvement. Our proposal can efficiently collect high-quality and diverse table-bused questions. B: A example of interaction between LLMs and human feedback. As we can see, specific feedhack (e.g., complex and mark) can greatly improve the quality of our generated questions.
Figure 2: A:
3.2 Dataset Collection
ity, ChatGPT usually generates simple questions that do not contain multi-hop reasoning; (ii). lack of diversity: When we ask ChatGPT to generate multiple questions, we find that many queries have similar formats. For example, the majority of them start with "Is there': (iii). unanswerable ques- tions: ChatGPT may generate questions that can not be answered only given the table. We collect some unanswerable and indeterminate questions and conduct an in-depth analysis in Section 6. The next paragraph described the approach we use to mitigate the above issues.
We select open-domain tables from the TabFact (Chen et al., 2020b) datasets, where the tables are from Wikipedia'. 1 Then, inspired by recent works on LLM's s ability to aid human annotations (Bartolo et al., 2022: Törnberg, 2023), design pipeline efficiently generate multi-step complex reason- ing questions by incorporating LLMs and human feedback. After obtaining the questions, we con- duct fine-graing -ne-grrined annotations on their directness, decomposition types, and human reasoning paths.
3.2.1 Human-in-the-loop question generation using LLMs
As shown in Figure 2, the pipeline has two main steps: initially generating queries using LLMs, fol- lowed by human selection and feedback to enhance them in accordance with human preferences.
Human selection and feedback Human feed- back is essential for LLMs because it helps them align with human preferences and values. Inspired by recent works on model refinement (Ouyang et al., 2022; Huang al., 2022; Shinn et al., 2023), we let human annotators select the questions that meet our requirements and then provide LLMM with feedback to improve the quality of the questions. For feedback design, we use several lexical features such as use math and more complex to resolve the problems mentioned above and reduce potential biases. Empirically, we find that ChatGPT can bet- ter improve their generated questions by providing them with specific lexical features than high-level instructions. Details on the feedback design can be found in Appendix A.
Initial question generation Inspired by the ef- fectiveness of LLMs' role-playing capability (Park al., 2023; Wang et al., 2023a; Fu et al., 2023a; Liu et al., 2023), we use LLM (i.e.. ChatGPT) ) as the question generator, which largely reduces the cost of data annotations. Specifically, we design an instructional prompt containing question require- ments to generate question candidates. However, there are three problems when use such prompts for ChatGPT: (i). lack of complexity: Although we provide corresponding instructions on complex-
lttpss/wwww.wkiipccii..ggg

---

3.2.2 Fine-grained annotations
Property
Value
Among the reasoning datasets, most of them only contain label-related annotations without human reasoning paths or fine-grained reasoning types. However, we argue that only goal-oriented anno- tations are insufficient to analyze the reasoning ability of LLMs. To fill in this gap, after annotating the answer, we further annotate whether ques- tion is implicit or explicit and how sub-questions are composed. We also annotate the main steps of table operations and reasoning. After that, we use a template-fie -fllling method to efficiently anno- tate human reasoning paths to solve the questions. The details on template design and the complete annotation interface can be found in Appendix E and F.
Unique Tubles
423
Total Questions
1000
Answerable Questions
744
Unanswerable Questions
256
Question Length (Avg/Median))
141.2/ 144.5
Answer Longth (Avg/Median)
5.5 3.0
Annotation Length (Avg/Median)"
54.3 /4.0
Rows per Tahle (Avg/Median)
12.6/ 10.0
Num of reusoning (Avg/Modian))
3.2/3.3
Num of operation (Avg/Median)
3.1/
Length of reasoning path (Arg/Median)
3.0
Complexity (Agreement)"
4.1 (88%)
anter-annotator A Agreement
93.7%
Table 2: Core Statistics of CRT-QA. Lengths the number of characters. Both and complexity (assessed by humans) demonstratc the significant challenge posed by our dataset.
that specifies the objectives of the chosen type. For example, Aggregation involved in solving the query, the annotator should select which type of this aggregation (e.g., sum) and its objectives (e.g., column names). The template can be found in Ta- ble 8 in the Appendix. This method can efficiently annotate the key reasoning steps of humans.
Directness Inspired by StrategyQA (Geva et al., 2021), we first introduce implicit questions over tabular data. Following Geva al., 2021, we use the following rule-of-thumb determine whether a question is implicit or explicit: the question is ex- plicit it can be written using words from the ques- tion, their inflections, and function words, while implicit questions require new content words to describe the reasoning process.
3.3 Dataset Analysis and Statistics
Key statisties Table 2 shows the key statistics of CRT-QA. Following Nan et al., 2021, we ask hu- man annotators to rate the complexity of 100 sam- ples on scale of 1 to 5 and report the average rat- ing and agreement. We compute the inter-annotator agrecment of 3 annotators 100 samples². The large proportion of agreement (92.7%) indicates the annotation quality of our dataset.
Decomposition types As the queries in our dataset contain multi-step reasoning, we further composed
annotate how these sub-questions together. Following Min et al., 2019, categorize the question decomposition into the following types?: bridging needs to find the first-hop evi- dence in order to find the second-hop evidence; in- tersection requires finding an entity that meets two independent requirements; comparison requires comparing the property of two different entities. Our annotation be used to analyze LLMs' ques- tion decomposition abilities.
Question and topic types We show the distribu- tion of topics and question types within CRT-QA through visualizations. Due to page limitations, we have included them and their details in Appendix C and D. These visualizations show that our dataset encompasses a diverse array of topics and includes a variety of question types, ensuring comprehen- sive coverage and versatility in the QA domain.
Human reasoning path To better evaluate LLMs' reasoning ability, we further annotate hu- man reasoning paths for solving these queries. However, it is impractical for annotators to write their detailed reasoning paths due to the great vol- ume of data. Hence, we design a template-fille paradigm to let annotators fill the objects of rea- soning or operation. We first let annotators select the type of reasoning or operation for each step in order (selections are listed in Table 1). Then, for each step, they are asked fill in a template
Comparison to existing datasets Compared with previous datasets CRT-QA dataset exhibits several distinctive features: (1) CRT-QA is the first Table QA dataset that contains both multi-step op- erations and informal reasoning. (2) CRT-QA is the first Table QA dataset that contains fine mne-grained annotations on questions' directness, composition types, and human reasoning paths. (3) CRT-QA
Examples of these 3 decomposition types are in Appendix B
TTabl 11 in the Appendix comprehensively compare them.

---

Dev set data
Zero-shot Python code generation Based on the table, answer the following question ONLY using Pychon code
such as code generation for complex table analysis. Inspired by recent works on auto-demonstration generation (Zhang et al., 2022b), we first randomly sample a data instance from the development set and input LLMSs with an instructional prompt for code generation. The prompt we use is a simple instruction to generate Python code and print inter- mediate or final results. The ablation of different selections will be discussed in Appendix H.
(use comments reasoning process)
Question: ...
Player
Paints
Python code:
pádraig
8.0
Layerto
quee
christy
print(...)
Test set data
Involve code and its output for final answer gonoration
Titdl: 2007 08 new jersey devils sesson
exempar
Question: many times did the New
Code
team
score
jersey Devils win or by goal margin during the 2007.
stlanta
2.3
In-context code generation For every data ex- ample in the test set, we use the exemplar generated from the dev set to conduct in-context learning for code generation. As Pandas is the most commonly- used library in Python for tabular data analysis which may frequently occur in LLMs' pretraining data, the generated codes are proficient in the use of Pandas to process tables.
Code Ourput
08 season?
buffalo
1-2
Python
Answer:
Figure 3: Our proposed ARC for complex table reasoning tusks. We first sample a data instance from the dev set and in- put it with an instructional prompt to L.LMs for zero-shot code generation. We then use the generated code the in- nn-context exemplar generate code for test data. After that, execute the code and inject the output to the prompt, and iteratively input LLM to get the final answer. The prompts and dev-set demonstration can be found in Appendix. ARC can mitigate the shortcoming of LLMs for aperation/reasoniig execution and climinate the effort of handcrafted code demonstrations.
Code execution with external tools We fur- ther use the generated code for execution using Python interpreter with a Pandas installed envi- ronment. We then obtain the output of the program as the intermediate or final results for the query.
has a sub-set of unanswerable and indeterminate questions, which are frequently occurred due to the complexity of real-life scenarios.
Iterative LLM calling with code output How- ever, for queries that require in-depth common- sense reasoning, only the code sometimes can not directly solve them. As a result, inspired by Re- Act (Yao et al., 2023b), we also integrate Acting and Reasoning by injecting code output into the prompt design for final step reasoning. By prompt- ing LLMs with code output, LLMs can generate more accurate final answers by avoiding step exe- cution errors.
4 Method
Although LLMs show powerful reasoning abilities on various tasks (Qin et al., 2023), they have limi- tations on various fields (Lewkowycz et al., 2022; Ziems et al., 2023). From our pivot experiments of prompting baselines, we find that LLMs can often generate correct reasoning plans but are unable to appropriately execute them. However, such steps (e.g., arithmetic, counting) can be perfectly per- formed by external tools such as SQL or Pandas. Inspired by recent works on tool-augmented LLMs (Gao et al., 2023; Yao et al.. 2023b; Shen et al., 2023; Lu et al., 2023a; Paranjape et al., 2023), we propose an efficient and effective approach, named ARC (Auto-exemplar-guiin guided Reasoning with Code), use external tools such as Pandas solve table reasoning tasks without handcrafted demonstrations. Figure 3 illustrates the pipeline of our proposed ARC and the detailed prompt design for each step can be found in Appendix H.
Experiments
5.1 Experiment Settings
For all the experiments, we use the powerful Chat- GPT, GPT-3,5-turbe and GPT-4 as the LLMs to investigate their reasoning ability on tabular data. Following Chen, 2023b, we use Markdown as the format of tables and use Exact Match (EM) as the main metric for evaluations. For each experiment, run three times with different random seeds and report the average EM score.
5.2 Baselines
Auto-exemplar generation Although manually- annotated demonstration shows significant effec- tiveness in in-context learning for L.Mss, nontrivial hand draddddf of effective exemplars makes it not flexible enough to be applied to sophisticated tasks
To evaluate LLMs' reasoning ability on the com- plex TableQA task, we select the following base- lines:4 Few-shot/Zero-shot prompting (Brown
*Implementation details can he found in Appendix G

---

Operation Types
Reasoning Types
Method
Overall
Index
Sort
Group
Filter
GRO
CAT
TEM
AGG
ARI
SPA
QUA
OTH
Prompting w/ ChaiGPT
shro-shot
47.20
50.00
46.34
37.15
46.34
44.44
53.09
36.39
37.68
42.11
62.22
52.41
46.11
Zero-shot-CoT
40.22
40.00
48.07
32.38
28.37
44.44
44.44
27.92
33.61
21.05
50.56
37.70
37.23
Few-shot (2-shot)
47.48
37.57
55.44
50.00
42.31
40.74
56.79
34.73
37.54
42.11
51.80
65.56
45.92
Few-shot- CoT (2-shot)
46.45
46.67
53.10
38.80
44.21
37.04
37.04
36.39
40.48
24.56
62.41
50.96
45.47
Tool Use w/ ChaIGPT
PAI.
46.51
42.96
51.27
36.39
42.41
11.11
1
42.59
35.54
41.87
17.54
63.38
45.37
44.11
ReAct
47.69
58.97
51.58
38.21
43.57
22.22
44.44
42.08
22.22
46.77
60.00
46.77
45.24
ARC (Ours)
50.62
43.33
58.20
40.42
47.15
37.04
37.74
44.19
45.46
40.35
64.83
51.15
49.41
Prompting w/ GPT-3,5-turbu
Zero
43.83
45.00
50.18
34.02
39.01
44.44
55.56
31.93
31.51
10.53
63.78
48.97
42.11
Zero-shot-Con
43.84
55.00
52.58
33.20
39.01
33.33
37.04
34.80
33.19
26.32
52.78
44.37
41,58
Pew-shot (2-shol)
49.69
47.50
62.40
46.10
55.56
59.26
38.62
43.70
43.11
42.11
66.67
54.48
49.05
Few-shot-Coo (2-shot)
47.20
45.00
56.49
37.30
45.39
44.44
33.33
37.86
39.92
53.63
62.22
51.26
46.33
Tool Use w/ GPT-3.3 turbo
PAL
53.19
50.00
61.59
43.03
50.35
11.11
51.85
46.07
54.81
31.58
67.22
51.38
52.17
ReAct
42.71
35.00
45.96
34.42
38.30
22.22
40.74
33.07
36.55
21.05
52.77
42.76
40.22
ARC (Ours)
55.28
52.50
64.21
51.77
66.67
60.71
46.56
46.56
52.12
16.67
63.53
54.38
53.26
Prompting w/ GPT-4
Zero-shot
46.99
51.28
56.47
35.27
50.46
11.11
48.15
34.92
39.74
36.84
60.23
55,16
46.21
Zero-shot-CoT
42.00
43.59
50.00
37.29
39.45
22.22
37.03
32.14
36.32
31.58
53.80
45.08
41.01
Few-shot (2-shot)
57.75
43.59
64.39
55.05
66.67
55.56
44.25
44.25
51.71
36.84
78.95
63.31
56.32
Few-shot-CoT (2-shot)
59.29
41.03
67.63
50.00
56.88
66.67
51.85
49.01
54.70
52.63
77.19
61.15
58.69
Tool Use w/ GPT-4
PAI.
61.20
48.72
65.47
52.97
56.88
33.33
40.74
54.56
62.39
36.84
74.85
60.67
59.83
ReAct
61.88
65.00
56.94
45.00
56.94
75.00
44.44
46.95
54.69
57.14
76.86
61.34
58.69
ARC (Ours)
62.14
64.10
64.75
51.06
54.13
55.56
55.56
52.09
59.40
52.63
72.94
65.16
60.11
Table 3: Evaluation results of various haselines and our method on our proposed CRT-QA: GRO: Grounding; CAT: Auto- categorization; TEM: Temporal reasoning; AGG: aggregating; ARI: Arithmetic; SPA: Spatial/ Geographical reasoning: QUA: Reasoning with quantifiers; OTH: Other commonsense reasoning. 'he mean p-values for the paired f-test between ARC and other performing baselines is 0.041, indicating significant differences. Among all the methods except Zero-shot and Zero-shot-Con CoT, ARC is the Only method that requires no handcrafted exemplar.
al., 2020): simply prompts LLMs with few- shot examples or instructions. Few-shot/Zero- shot CoT (Wei et al., 2023; Kojima et al., 2023): inputs LLMs few-shot exemplars with manually- crafted reasoning path or Let's think step by step. PAL (Gao et al., 2023): uses few-shot examples of only Python code to encourage LLMs to gen- erate correct code for problem-solving. ReAct (Yao al., 2023b) utilizes in-context examples of (Thought, Act, Obs) tuples to combine logical path and sask-ip actions. ARC utilizes a zero-shot- generated code exemplar to perform in-context code generation and incorporate code output for final answer generation.
of 60. 11, indicating the difficulty of our dataset. (2). among all the baselines, our proposed ARC achieves the best average EM scores with an aver- age improvement of 1.846 across all models with- out using any handcrafted exemplar, indicating the effectiveness of our proposal. For ChatGPT base- lines. (3). we find that Zero-shot-CoT performs worse than the vanilla Zero-shot approach. By checking the reasoning paths elicited by Let's ''s think step-by-step, we find that the reason may arise from the phenomenon that the reasoning paths are un- ruly and even generate codes that the model does not have the ability to solve. As a result, Ler's think step-by-step is not a one-fits-for-all solution. (4), although Few-shot-CoiT can not outperform Zero-shot for ChatGPT. As the model evolves (i.c., from ChatGPT Turbo GPT-4), Few-shot-CoTT can have better performances than Zero-shot pre- dictions, indicating that the model increases its
5.3 Experiment Results
Table shows different methods' EM scores on CRT-QA dataset. We can see that (1). overall, the most efficacious approach achieves a maximum

---

CoT reasoning ability. (5), Among the 3 tool-use baselines, ReAct can not have comparative perfor- mances with the other two methods with GPT-3.5- turbo and GPT-4. By investigating the reasoning path, we find that ReAct often finishes without any answer. Alternatively, ReAct often conducts a sub- stantial number of iterations, resulting in not only increased costs but also an extremely long reason- ing pathway that becomes out of control.
the question "If the score increase by years, ...". but the table content cannot support the implied assumption. For indeterminate questions, anno- tators can yield different answers due to different metrics, algorithms, and criteria. These questions can be answered from both subjective and objective perspectives. A "best guess" can be made using subjective reasoning, while these questions can also be objectively asked for user's further clarification in certain scenarios.
From fine-grained reasoning types shown in Ta- ble 3, we observe that all prompting- based methods are bad at aggregation and arithmetic compared with other reasoning types. Noticeably, our pro- posed ARC and PAL can greatly improve LLMs' ability on these two reasoning types. Besides, we observe that among all the reasoning types, LLMs perform the best in reasoning with quantifiers. Due page constraints, comprchensive ablation study on the number of exemplars, error analysis, and case study are in Appendix H, J, and
A nswerabilliyy Study We evaluate the language model's ability to determine whether to answer question under three approaches. As baseline ap- proach, Random approach randomly predicts the responses under the prior probability distribution. Binary Classification presents a binary classifi- cation problem, wherein the model must output with either "unanswerable" or "answerable". Ques- tion Answering approach produces the correct an- swer if the question is answerable and respond with "unanswerable" otherwise. We conduct these experiments on the whole dataset, including unan- swerable/indeterminate and "normal" questions.
6 Unanswerable and Indeterminate Question
Most Table QA datasets are designed for answer- ing the questions with golden labeling (Pasupat and Liang, 2015; Chen et al., 2020b, 2021b,a), but real users possibly ask questions that inherently dif- ficult to answer due to the complexity of the real world. Motivated by this, we incorporate a sub-set unanswerable and indeterminate queries where some questions go beyond common external knowl- edge, while others are inherently problematic. We categorize these questions into four categories and conduct the answerability of LLMs based on them.
Acc
P
Fl
Random
0.596
0.731
0.727
0.729
Binary Classification
0.680
0.908
0.637
0.749
Question Answering
0.779
0.928
0.763
0.838
Table 5: Results for identifying answerability. We report common metrics for binary classification, i.e., Ace (Accuracy). P( Precision), R(Recall) and Fl score.
Based on the results presented in Table 5, Binary Classification shows improvements over Random, indicating its effectiveness in identifying questions' answerability. Question Answering proves be the most effective approach for identifying answer- ability, probably because generating answers is casier than determining whether a question can be answered. It mimics some pre-training tasks like reading comprehension. This study benefits a broader understanding of how language models can tackle unanswerable and indeterminate questions and provides directions enhance performance.
Type
Delinition
Percentage
Lacking essential information based given table.
Out scope
73.2%
The assumption in the question is invalid hased the table.
Hallucination
10.4%
Problematic
Question itself contains logical errors.
4,8%
Subjective
The answer varies from annolaters due to different metrics, algoritmms criteria.
6.4%
Others
Other types of questions that aot labeled
5.2%
Table Category and ratio of indeterminate and unan- swerable questions in CRT-QA dataset.
Conclusion
As shown in Table 4, out-of-scope, hallucination, and problematic questions are unanswerable. The main reason is the absence of essential information or logical flaws within the question itself. For ex- ample, there is an implicit assumption underlying
In this work, systematically evaluate LLMs' rea- soning ability on tabular data, we first establish a comprehensive taxonomy on operation and rea- soning types for table analysis. Then, we propose CRT-QA, a dataset of complex reasoning QA over

---

tables. We propose ARC which effectively utilizes table analysis tools to solve table reasoning tasks without manually-annotater exemplars. Extensive experiments show CRT-QA poses a significant chal- lenge for LLMs and our proposed ARC achieves the best EM scores. Besides the main experiments, we also conduct thorough ablation studies, error analyses, answerability study, and case study for further analysis.
Max Bartolo, Tristan Thrush, Sebastian Riedel, Pontus Stenetorp. Robin Jia, and Douwe Kiela. 2022. Mod- els in the loop: Aiding crowdworkers with generative annotation assistants.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemnrr, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Limitations
(1) CRT-QA is a test-only dataset, which means no gradient updates are performed. While striv- ing for problem complexity, we face challenges in balancing the quantity of our dataset. This is primarily due to the intricate nature of our annota- tion process, which demands more time for answer generation and fine-grained process labeling. (2) Similar to previous works discussed in Table I1, we only foucs on single-table question answering. However, queries across multi-tables are also com- mon in real-life table analysis scenarios. (3) We don't research the boundary of external knowledge. The appearance of unanswerable and indetermi- nate questions is associated with our data genera- tion goal, which is to generate complex and diverse questions. Specifically, the indeterminate questions are contrasted with implicit questions, while inde- terminate questions stand out beyond implicit ques- tions. We leave this study as future work. (4) In our study, utilize a combination of exact match and human evaluation reasonable because, during the question generation process, we only select the questions that can be answered within several words without ambiguity. Although this is not comprehensive for free-form answer-generation tasks, alternative metrics such as F1, ROUGE-L, and BLEU-1 also possess inherent limitations. Evaluation of the free-form answer- generation task seems promising. Moreover, our fine-grained annotations provide feasible path to help answer the question. Although the path is not unique. Currently, there is no effective method to evaluate the reasoning path. This aspect will be left for future research and development.
Wenhu Chen. 2023a. Large language models are few( shot table reasoners.
Wenhu Chen. 2023b. Large language models are few( ))- shot table reasoners. In Findings of the Associ- ation for Computational Linguistics: EACL 2023, pages 1 1120-1130, Dubrovnik, Croatia. Association for Computational Linguistics.
Wenhu Chen, Ming- Wei Chang, Eva Schlinger. William Wang, and William W. Cohen. 2021a. Open question answering over tables and text.
Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020a. Logical natural lan- guage generation from open-domain tables.
Wenhu Chen, Xueguang Ma, Xinyi Wang. and William W. Cohen. 2022a. Program of thoughts prompting: Disentangling computation from reason- ing for numerical reasoning tasks.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang. Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020b. Tabfact: A large-scale dataset for basle fact verification.
evaluation metric. It is
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong. Hong Wang, and William Wang. 2021b. Hybridqa: A dataset of multi-hop question answering over tabular and textual data.
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, lana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2022b. Finqa: A dataset of numerical reasoning over financial data.
Zhoujun Cheng. Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong. Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2023. Binding language models in symbolic languages.
References
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek
Sumit Asthana and Aaron Halfaker. 2018. With few eyes, all hoaxes are deep. Proc. ACM Hamm-Compp Interact, 2(CSCW).

---

Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivami Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz. Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Bck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu. Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022. Large language models can self-improve.
Mohit lyyer, Wen-tau Yih, and Ming- Wei Chang. 2017. Search-based neural structured learning for sequen- tial question ans answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 1821- 1831, Vancouver, Canada. Association for Computa- tional Linguisties.
Yannis Katsis, Saneem Chemmengath, Vishwajeet Ku- mar, Samarth Bharadwaj. Mustafa Canim, Michael Glass, Alfio Gliozzo, Feifei Pan, Jaydeep Sen, Karthik Sankaranarayanan, and Soumen Chakrabarti. 2021. Ait-qa: Question answering dataset over com- plex tables in the airline industry.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2023. Large lan- guage models are zero-shot reasoners.
Hyung Won Chung. Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghami, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai. Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang. Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le. and Jason Wei. 2022. Scaling instruction finetuned language models.
Aitor Lewkowyez. Anders Andreassen. David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wo, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra, 2022. Solving quantita- tive reasoming problems with language models.
mnstructionfiiettueed
Jiacheng Liu, Skyler Hallinan, Ximing Lu, Pengfei He, Sean Welleck, Hannaneh Hajishirzi, and Yejin Choi. 2022. Rainier: Reinforced knowledge introspector for commonsense question answering.
Karl Cobbe, Vineet Kosaraju. Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word prob- lems.
Ruibo Liu, Ruixin Yang. Chenyan Jia, Ge Zhang, Denny Zhou, Andrew M. Dai, Diyi Yang. and Soroush Vosoughi. 2023. Training socially aligned language models in simulated human society.
Antonia Creswell, Murray Shanahan, and Trina Higgins. 2022. inleeeion-inference:: Exploiting large language models for interpretable logical reasoning.
Pan Lu, Baolin Peng, Hao Cheng. Michel Galley, Kai- Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023a. Chameleon: Plug-and-pla compositional reasoning with large language models.
Yao Fu, Hao Peng. Tushar Khot, and Mirella Lapata. 2023a. Improving language model negotiation with self-play and in-context learnngg from ai feedback.
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song- Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. 2023b. Dynamic prompt learn- ing via policy gradient for semi-structured mathemat- ical reasoning.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2023b. Complexity-bart based prompting for multi-step reasoning.
Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang. and Graham Neubig. 2022. Language models of code are few-shot commonsense learners.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. 2023. Pal: Program-aided language models.
Sewon Min, Victor Zhong, Luke Zettlemoyer, and Han- naneh Hajishirzi. 2019. Multi- reading compre- hension through question decomposition and rescor- ing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6097-6109, Florence, Italy. Association for Compu- tational Linguistics.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle a laptop? a question answering benchmark with implicit reasoning strategies.
Matthew Ho, Aditya Sharma, Justin Chang, Michael Saxon, Sharon Levy, Yujie Lu, and William Yang Wang. 2022. Wikiwhy: Answering and explaining cause-and-effect questions.
Rejichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders,

---

Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022. Webgpt: Browser- assisted
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal rein- forcement learning.
answering with human feedback.
Linyong Nan, Chiachun Hsieh, Ziming Mao. Xi Victoria Lin, Neha Verma, Rui Zhang. Wojciech Kryšcińsski, Nick Schoelkopf, Riley Kong, Xiangru Tang. Murori Mutuma, Ben Rosand, Isabel Trindade. Renusree Bandaru, Jacob Cunningham, Caiming Xiong. and Dragomir Radev. 2021. Fetaqa: Free-form table ques- tion answering.
Kurt Shuster, Mojtaha Komeili, Leonard Adolphs, Stephen Roller, Arthur Szlam, and Jason Weston. 2022. Language models that seek for knowledge: Modular search generation for dialogue and prompt completion.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz. Akshat Agarwal, Alethea Power. Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Par- rish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, An- drew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang. Angelica Chen, Anh Vuong, Animesh Gupta. Anna Gottardi, Antonio Norelli. Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabas- sum, Arul Menezes, Arun Kirubarajan, Asher Mul- lokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret 7oph, Bartlomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ek- mekei, Bill Yuchen Lin, Blake Howald, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Ar- gueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng. Chitta Baral, Chiyu Wu, Chris Gallison-Burch. Chris Waites, Christian Voigt, Christopher D. Manning. Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raf- fel, Courtney Asheraft, Cristina Garbacea, Damien Sileo, Dan Garrette. Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González. Danielle Perszyk, Danny Hernandez. Danqi Chen, Daphne Ippolito. Dar Gilboa, David Dohan, David Drakard, David Ju- rgens, Dehajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko. Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dim- itri Coelho Mollo. Diyi Yang, Dong-Ho Ho Lee, Ekate- rina Shutova, Ekin Dogus Cubuk. Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, El- lie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem. Emie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Bunice En- gefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martinez--Plumee Francesca Happé, Francois Chollet, Frieda Rong. Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Ger- mán Kruszewski, Giambattista Parascandolo, Gior- gio Mariani, Gloria Wang. Gonzalo Jaimovitch- López, Gregor Betz, Guy Gur-Ari, Hana Galijase- vic, Hannah Kim, Hannah Rashkin, Hannanch Ha- jishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang,
OpenAl. 2023a. Chatgpt.
OpenAI. 2023b. Gpt-4 technical report.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. 2023. Art: Automatic multi- step reasoning and tool-use for large language mod- cls.
Ankur P. Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. Totto: controlled table-to-text generation dataset.
Joon Sung Park, Joseph C. O'Brien, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative agents: Interactive sim- ulacra of human behavior.
Panupong Pasupat and Percy Liang. 2015. Composi- tional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pages 1470- 1480, Beijing. China. Association for Computational Linguistics.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language process- ing task solver?
Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugging- gpt: Solving ai tasks with chatgpt and friends in hugging face.

---

Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jae- hoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Ko- coń, Jana Thompson, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Ja- son Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse En- gel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song. Jil- lian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gim- pel, Kevin Omondi. Kory Mathewson, Kristen Chi- afullo. Ksenia Shkaruta, Kumar Shridhar, Kyle Mc- Donell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin. Lidia Contreras- Ochando, Louis-Phillippe Morency, Luca Moschella, Lucas Lam. Lucy Noble, Ludwig Schmidt, Luheng He. Luis Oliveros Colón, Luke Metz, Lütfi Kerem Senel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazcika. Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Pouthast, Matthew L. Leavitt, Matthias Hagen, Mátýás Schu- bert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Co- hen, Michael Gu, Michael Ivanitskiy, Michael Star- ritt. Michael Strube, Michat Swedrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mo Tiwari, Mo- hit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Ni- tish Shirish Keskar, Niveditha S. Iyer, Noah Con- stant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy. Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormo- labashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Fickesslyy, Phu Mon Hutt Pinyu Hwang, Piotr Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramón Risco Delgado, Raphaël Millière, Rhythm Garg. Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le- Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Rus- lan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Sto- vall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sehastian Bischoff, Sehas- tian Gehrmann, Sebastian Schuster, Sepideh Sadeghi,
Shadi Hamdan, Sharon Zhou, Shashank Srivastava. Sherry Shi. Shikhar Singh, Shima Asaadi, Shixi- ang Shane Gu, Shubh Pachchigar, Shubham Tosh- niwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas De- haene, Stefan Divic, Stefano Ermon, Stella Bider- man, Stephanie Lin, Stephen Prasad, Steven T. Pi- antadosi, Stuart M. Shieber, Summer Misherghi. Svet- lana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan. Tianle Wang. Tiberius Nkinyili, Timo Schick, Timofei Kornev, Timothy Telleen-Lawton, Titus Tunduny, Tobias Gerstenberg. Trenton Chang, Trishala Neeraj. Tushar Khot, Tyler Shultz, Uri Sha- ham, Vedant Misra, Vera Demberg. Victoria Nyamai. Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fe- dus, William Saunders, William Zhang. Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song. Yasaman Bahri, Yejin Choi, Yichi Yang. Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang. Zirui Wang, and Ziyi Wu. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.
Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Han- naneh Hajishirzi, and Jonathan Berant. 2021. Mul- timodalga: Complex question answering over text, tables and images.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshom, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
Petter Tömberg. 2023. Chatgpt-4 outperforms experts and crowd workers in annotating political twitter mes- sages with zero-shot learning.
Guanzhi Wang, Yugi Xie, Yunfan Jiang. Ajay Man- dlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and An- ima Anandkumar, 2023a. Voyager: An open open-ended embodied agent with large language models.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery. and Denny Zhou. 2023b. Self-consistency improves chain of thought reasoning in language models.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama,

---

Maarten Bosma, Denny Zhou, Donald Metzler, Ed Chi, Tatsunori Hashimoto, Oriol Vinyals. Percy Liang, Jeff Dean, and William Fedus. 2022. Emer- gent abilities of large language models.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe Le, and Ed Chi. 2023. Least-to-most prompting enables complex rea- soning in large language models.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought promptiin elic- its reasoning large language models.
Fan Zhou, Haoyu Dong. Qian Liu, Zhoujun Cheng, Shi Han, and Dongmei Zhang. 2022. Reflection of thought: Inversely eliciting numerical reasoning in language models via solving linear systems.
Zhilin Yang. Peng Qi. Saizheng Zhang, Yoshua Ben- gio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop hop question answer- ing.
Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang. Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. Tat-qa: question answering benchmark on hybrid tabular and textual content in finance.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023a. Tree of thoughts: Deliberate problem solving with large language models.
Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. Can large lan- guage models transform computational social sci- ence?
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023b. React: Synergizing reasoning and acting language models.
Prompt Design
Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Alex Polozov, and Charles Sutton. 2022. Natural lan- guage to code generation in interactive data science notebooks.
Syper-parrametr
Value
Temperature
0.7
max max_len (CoT)
1024
max max_len len (Code)
1024
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang. Zifan Li, James Ma, Trene Li, Qingn- ing Yao, Shanelle Roman, Zilin Zhang, and Dragomir lamaee- dataset for complex and cross-domain semantic pars- ing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods Natural Lan- guage Processing, pages 3911-3921, Brussels, Bel- gium. Association for Computational Linguistics.
max (Few-shot/Zero-shot)
16
top_p
1.0
Radev. 2018. Spider: A large-scabe
best_on
1
Table Elyper-parameter setting for LLMs.
Selection from dev set
Average EM score
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. 2022. Star: Bootstrapping reasoning with reasoning.
Selection
49.41
Selection 2
49.89
Selection
48.96
Zhexin Zhang, Jian Guan, Guowei Xu, Yixiang Tian, and Minlie Huang. 2022a. Automatic comment gen- eration for Chinese student narrative essays. Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing: System Demon- strations, pages 214-2212 223,, Abu Dhabi, UAE Associa- tion for Computational Linguistics.
Table 7: Different dev set selections' performance for ARC.
A.1 Human feedback for question generation
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022b. Automatic chain of thought prompt- ing in large language models.
The followings are examples of feedback for LLMs to generate desired questions.
Generate another 10 more complex questions.
Yilun Zhao, Yunxiang Li, Chenying Li, Rui Zhang, 2022. Multihiertt: Numerical reasoning over multi hierarchical tabular and textual data.
Generate another 10 questions with different question types.
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning.
Generate another more complex questions that require math to solve them.

---

Generate another 10 more complex questions that require common sense for column 1. Other choices may also improve the quality of LLMs' generated questions.
Sports
Biography
A.2 Prompt for Baselines
All prompt designs for the main experiment and experiment in Section 6 can be found in Table and Table 9 respectively.
Misial ars
B Question Decomposition Types
Media
Rerseone
Following Min et al., 2019, we study the follow- ing three different types of question decomposition types:
Figure 4: Topic distribution CRT-QA. Categories in the figure originate from the mid-level WikiProjects directory.
Bridging: requires finding first-hop evidence before moving on to the second-hop evidence. Example question: "What was the average number of years between TV station's s affil- iation with the e! Canadian TV system and their eventual disaffiliation?".
Intersection requires finding an entity that meets two independent conditions. Example question: "Are there any counties within the Mid-Indiana Football Conference that contain more than one school?".
what
the
WAG
thore
Comparison requires comparing the features of two distinct entities. Example question: "How often does Tim Lajeik win fights in the first round compared to subsequent rounds?".
Figure 5: Types of questions in CRT- QA. Only high- frequency words are labeled, and empty blocks indicate that the frequencies the suffixes are too rare to shown individually
C Data Topic Distribution
Wikimedia Foundation's topic categorization model (Asthana and Halfaker, 2018) visualize the topic distribu- tion of our dataset. Figure shows that our data are mostly related to sports, biography, regions, and media. Overall, CRT-QA dataset covers a fairly wide range of topic domains.
Following Parikh et al., 2020,
F Data Annotation Details
We enroll 2 undergraduate students and Ph.D. student majoring in computer science for data an- notations. All them have at least one year data analysis experience.
Question Type Distribution
Following (Yang et al., 2018) 8), by taking the three neighboring tokens along with the central question determine the question types.
G Experiment Implementation Details
word (CQW), A visual representation of the distribution is shown in Figure 5, which illustrates the syntactic diversity questions in our proposed CRT-QA.
The models we use for experiments are text-chat-davinci--oe
GPT-3.5-tume 5-turbo, and GPT-4 through Microsoft Azure API. For tool-use baselines, empirically, find that the LLM-generated code may contain some syntax errors which make it impossible to run and generate output. For these cases, we let LLM re re-generate code a maximum of five times. Once it
E Data Annotation Interface
Figure 8 shows the detailed interface for data anno- tation.

---

Pronupt Design for Baselines
pero-Anot
Table
Read the table below regarding "yugeslavia national football results"
date
city
opponent
results
type of gare
spril may
beigrade
france
1:0
belgrade
england
1:1
friendly
June 16 septenber 4
norway
norway
1966
mascow russia
ussr
8:0
friendly
sestenber
luxenbourg
luxenbourg
5:2
1966
october
parts france
france
18:1
1965 weq
6 novenber 7
helgrade
norway
1:1
1966 wee
Question
Yugoslavia national football
play any ganes against outside Europe in table?
with only 'Yes' or 'No' that is nost accurate and nothing else.
Answer
Table
Read the table below regarding "yugoslavia national football tean results" results
date
city
upoonent
april 18
belgrade belgrade
france
1:8
1966 wee
nay
england
friendly
2 june 16
oslo norway
norway
8:3
1966 wee
september
russia
8:0
friendly
septenber 19
luxenbourg
luxenbourg
5::2
1966 wee
october
paris france
france
00:1
1966
1 novenbar helgrade
norway
weq
Quastion
Did the Yugoslavia
only 'Yes' national foothall
tean play ganes against teams outside of Europe in the table? Answer
accurate nothing else.
Answer
Let's think step-by-step
I-shot Table
Read the table balow regarding *198 ireland hurl championship" the following questioon.
[rank] player
county
tally
total
matchee
average
pádraig horan
offaly
5 17
billy fitzpatrick
ki1kenny
2 24
7.5
tony 'sullivan
3
4
nolloy
galney
3
28
10
christy heffernan
kilkenny
4.5
5 5
pat horgan
cork
8- 18
4.5
Question
How many players in the 1982 all-Irelame senior hurling championship had higher the overall average score per gane the conpetition?
than regarding "yugoslavia national foutball results" city
Answer
4
date
apponent results
type of game
april 18
belgrace belgrade
France
1:8
1966 wee
england
1:1
friendly
June 16
aslo norway
norway
8:3
1966 wee
september 4
nescon russia
ussr
1::0
friendly
september
luxenbourg
luxenbourg
1966
octaber paris france
france
8:1
1966 wee
novenber belgrade
1:1
1966
Quastion
Did Yugoslavia national Tootball
tean play any ganes agalinst teans outside Europe in the table? Answer accurate and nothing else.
with only 'Yes" 'No" that is nost
Answer
Table
Read the table below regarding "1982 all ireland senior hurling championship" to answer the following questions. average
player
county
tally
total
matches
pádraig horan
offaly
billy fitzstricc
kilkerny
2 24
7.5
tony 'sullivan
cork
8 28
7
nolloy
galway
4
christy hefferman
kilkerny
-
18
4.5
horgan
cork
4.55
Question
How many players in the 1982 all-Ireland senior hurling championship had higher average score per gane than the everall
the conpetition?
Answer Table
Read the table ballw regarding "g.d. astoril prata" ansuer the following questionn. away
season conpetition
opponent
2013
uefa europa league
hapael gan
2813
14
vefa eurepa league
play off
pasching sevilla
2
2613
14 uefa
leaque
1
2013
uefa europa league
group slovan liberec
11-1
[
4
2013 14 uefa curopa league
correlation between group freiburg
11-1
Question:
performance gates and away gates durring 2013-14
UEFA Europa League conpetition?
Answer:
Continued page

---

generates runnable code, we execute it and get the output. If the LLM can not generate runnable code five times, we keep the code the prompt and set the output to "None". The hyperparameters we use can be found in Table 6.
by the metric. To ease the trouble of format error, augment the EM with human evaluation. (5) Refuse answer (4%). The answerable query is re- garded as an unanswerable question. The model's s completion contain some expressions that refused to answer like "I am unable to write Python code for this question the data does not provide in- formation ...". We find that the ARC method of the program enhances numerical processing while weakening the semantic recognition of the text. So the method makes mistakes for the match of strings such as "n/a" and "n a".
Ablation Study
For our proposed ARC, we select different amples from the dev set conduct zero-shot code generation as exemplars for the test set. Table shows that the performance difference among different selections is within EM score demon- strating the robustness of our proposal.
We also design four sets of contrast experiments for the ablation study as Figure 6 shows. We find the table reasoning ahility differs from the models. GPT4 is best and turbo performs on par with Chat- GPT. For the in-context learning, GPT4 benefits 1ot from the increase in the number of demonstra- tions, but the increase is not significant for other models. We study the impact of up to 2-shot be- cause structured tables consuming lots of tokens can easily break the input limitation.
As expected, different decomposition types vary in difficulty with bridging being the most challeng- ing and comparison being the easiest. Besides, LLMs obtain similar performances on implicit and explicit questions in our Table QA dataset.
I
Case Study
We show how method ACR uses external tools solve table reasoning tasks in Figure 7. The comments in ACR show the reasoning sketch and guide the generation code. Using external tools enhances numerical computation compared with plain text reasoning. In contrast, CoT fails even with the right reasoning path.
J Error Analyses
To analyze how the error was caused, we randomly choose samples and go depth into error analysis based on the performance ARC.
We find five types of errors: (1) Code generation етror (20%). The code is not executable and the output is none or an illegal type. (2) Gross error of reasoning (32%). The reasoning path deviates from the requirements of the user query. (3) Condition missing error (18%). The code framework has no problem in general, but some subtle conditions or operations are missed. (4) Format error (26%). The model return with an answer but can't be judged

---

70%
#280 # shot #2 shat
60%
Accuracy
5005
Co #Col shot # 2 shot
409
Bridging Intersaction
Comparsion
ximplich
Explisit
ChaiGPT
ChaiGPT
Turbo
GPT4
ChatGPT
Turbo
GPT4
ChaiGPT
(4)
(b)
(c)
(d)
Figure 6: Ablation experiments in four groups: (a) number of demonstrations, (b) number of demonstrations for COT, (c) decomposition types, (4) directness.
CoT
Table: 2007 08 new jersey devils season
Prediction: firstfind the column of score and then use our knowledge hockey to understandwhat means to win or lose by one goal margin. If team wins or loses by one goal margin, thedifference in the score 1.After that, we count the number of games whose difference in the score isl.As result, the answer G. [False]
date
visitor
score
docomber
atanta
2
booton
Label: 7
wasbington
decomber
ARC
december 10
new
2-3
Prediction: Python code (only code with comments stop when code ends):
december
new Jencyy
3-3
december
phoenis
philadelphca
# get only rows where the margin of victory is or -1 one goal_ggmes .gmmes ddf[abs (fffhhoesfhhoe_sscree] score'] df''visitor score') }]] 1]
december
Jensyy
december
new
3-
# count the number of rows with 1-goal margin of victory
december 23
Dew jersey
1-0
december 28
buffalo
1-2
one goal games =len(one Loatlonneoooll___e loallooooaassssl_ games) # print the result
Jersey
2
print(("Number 1-goal margin games: \",
How many times did the New J ersey Devils win or lose by one goal margin during the 2007-08 season?
Question:
num one _oe__oalll___ggggge games)
Code output:
Number of 1-goal margin games:
[True]
Figure 7: Case study CoT and ARC. We find the CoT generate correct reasoning plans but fail reasoning execution. On the contrary, ARC can obtain the right answer by using code.
Multi-hop
Multi-hop
Numerical
Common
Query Directness
Annotations
Unanswerahle
Dataset
Domain
Reasoning
Operations Reasoning
Implicit
Explicit Decomposition Reasoning
Queries
WTQ (2015)
open
TubFacn(2020b)
open
HybridQA (2021b)
/
X
V
X
open
X
OTTQA (2021a)
/
X
X
V
X
X
open
X
FinQA (20226)
/
X
d
X
X
finance
X
TAT-QA (2021)
finance
/
J
airline
MukiHiertt (2022))
X
V
finance
FeuQA (2021)
/
X
X
V
X
X
open
X
TABMWP (2023b)
/
X
J
X
X
open
CRT-QA (ours)
/
/
/
open
Table A comparison CRT-QA and other Table QA datasets. CRT-QA is TableQA dataset that contains implicit questions, detailed annotations of human reasoning paths, and question decomposition types. t: MultiHiertt only contains math expression as the reasoning path.

---

Procupt Design for Baselines
Table
Bead the tabla bolow regarding "yugoslavta national
football results
tean rasults"
date
opporent
type of
spril 18
belgrade
france
1:0
1966 wcq
may
belgrude
england
friendly
2 june 16
oslo norway
norway
8:3
1966 wee
russia
friendly
septenber 19
luxenbourg
luxenbourg
5:2
1966 woq
octuber paris france
france
0:1
1966
novenber belgrado
norway
1966 woq
Question
Did the Yugoslavia national Football I
tean play any games against teans outside of Europe in the table? Answer nothing else.
only 'Yes'
Answer
CaT
2-shot
Read table below regarding
ireland tally
Ling charpionship" matches
Following questions.
|rank| player
county
total
average
8
pädraig horan
effaly
5
31
1
billy Fitzpstrick
kilkenny
24
36
7.5
3
tony 'sullivan
cork
28
7
3 4
molloy
galney
3-
28
18
christy har fernan
kilkenny
pat horgon
cork
8-
18
4.5
Question
How players the 1982 all-Ireland senior hurling charpionship had higher average score gane than compatition?
averall average score per gane
Explanation
Ke first find the column of "average" and conpute the average of all the players, which is (8 7.5 + player whose average larger then
4.5)/6 6.917. Then count
result,
answer is 4.
Table
Read the table below regarding 'g.d. estoril praia"
the fallowing questions.
season
competition
round opponent
hone
uaay
europa league
hapael ranat gan
2013 14 uefa europa Ieague
play off
pasching
2 2 2
europa leaque
group
sevilla
3 2013 14 uefa suropa Ieague
group h
slovan libersc
1
2013 14 uefa
league group
freiburg
Question:
Was there correlation between GO Estoril Praia's 5 per formance In home gares and away gares during the 2013-14 UBFA Eurapa League competition?
Explanation
column "hore" and "eway" corpare outcone "hone" "anay" gares. find there is no correlation between "hone" and "away" ganes. result, the answer is No.
first
Table
the table regarding "yugoslavia date
football oppanent results
city
type af gave
spril 18
belgrade
france
1966 woq
may 9 segpeebeer nescon russis
belgrade
england
1:1
friendly
norwey
nоrway
1966 weq
user
1::0
friendly
segtenber luxenbourg
luxenbourg
1966
octaber paris france 16 novenber belgrade
france
0::1
1966 weq
1:1
1966
Question
the Yugoslavia national foothall tean play any ganes against autside Europe Answer
with only 'Yes" or 'No" that is nost accurate and nothing else.
Explanation
Instruction
Let's use Python to solve Table-based question answering. Here is an exapple how to do Tittle: "1982
Table
ireland sentor hurling charpionshii"
|rank| player
county tally
total natches
average
pádraig
offaly
1
billy fitzpatrick
kilkenny
2- 24
38
7.5
tony 'sullivan
cork
3 4
nolloy
galney
3--11
28
18
christy heffernan
kilkemny
5 5 pat horgan
cork
8-
18
4.5
Question
How players the 1982 all-Treland sentor hurling charptonship had higher average score gane than averall average score per gone the corpetition? import pandas as pd
Code
get the overall everage score per gare averall_avg ofl averagee' nean()
conpetition
filter the detafrane only include players with higher average score per gane than the overall average
hi iighr_avg_dd df[df[ "averagge" ] averall_avey
the number players with higher averoge score per gane uverall average nun, higher_ave _av_p_aya_sssssss_ = len(higher_avve print the result arine(
"riinn""ubberr "Nunber of players with higher average score per gome than the overall average: ", mun min_higher..wg_playee "How about this questien?
Instruction
Table
Titcle: "yugoslavia national football city
tean results"
date
oppanent
results
type of gane
april 18
belgrade
france
1:8
1966 woq
nay belgrade
england
friendly
june 18
aslo norway
norway
6:3
1966 woq
Continund on next page

---

Promppt Design for Daselines
3 september 4
moscon russia
ussr I 8::
friendly
september luxenbourg
luxerbourg 5:2
1966 wce
octaber 9 paris frence
france
8:1
1966 wee
novenbar bel gradd
norway
1966 weq
Question
0id the Yugoslavia national football team play any ganes against teans outside of Europe in the table? Answer
with only 'Yes' 'No' that is nost accurate and tiininn else.
Code
RoACE
Table
Tittle: "1982 all ireland senior hurling charptonship"
Irank| player
county
tally
total
average
padraig horan
offaly kilkerny
5- 17
2
billy fitzpatrick
2- 24
38
7.5
tony 'sullivan
cork
7
3 p J nolloy
galnay
3 3-
|
4
christy haffernan
kilkenny
3--9
18
4.55
1.5
horgan
cork
4.5
Question
How nany players in the 1982 ell-Irelard senior hurling chanpionship had higher average score per gane than the everall average
gane the conpatition?
Thought 1
I need to first get the everall averege score per gane of the convetition and then count the number of players with higher average score game than the overall average. overall_avg dfl average" neann)
Action
princ( The overall average score per gane of the corpetition is overall_avg) The everall average score per gane the competition 6. 9.966666666667
Observation
Thought 2
Then I should filter the datafrane to only include players with a highar average score per gane than the overall which 9.918869866666 and count the length of the filtered datafrane. aigher_avg..e of df[df[' everegeee > overell.evg]
Action 2
nun_highes ham_ttgher_agg.llae pag..playersss lnn@nigher_aagg...
print( (Number of players with higher average score per game than the overall average: ", mun.highhrr_wggpplaaa. Number of players with higher average score gane than the everall average: 4 Number players
Observation 2
Thought Action 3
higher average score gane than overall average
answer
Finish. The answer 15
Table
Tittle: "yugoslavia national footbull tean results"
date
city
opponent
results
type of gare
april 18
helgrade belgrade
france
1:0
1966 wce
may
england
1:1
friendly
June 16
oslo norway
norway
8:3
1966 weq
septenber 4
nescow russia
ussr
8::0
friendly
restember luxenbourg
luxenbourg
5:22
1966 wee
october
paris france
france
8:1
1966 weq
6 novenber 7
belgrade
norway
1:1
1966 wee
Question
Did Yugoslavia national football tean play any games against teans outside Europe in the table?
with only 'Yes' or 'No' that is nost accurate and nothing else.
ARC
Instruction
Based the table, answer following question ONLY using Python code. comments reasoning process) little: "1982 all ireland senior hurling chanpionshia" Irank] player
Table
county telly
total
natches
average
0
pádraig haran
offaly
5
17
8
billy fitzpatrick
kilkenny
2
7.5
I 2 3 tony o 'sullivan
cork
8 28
28
7
molloy
galway
I 4 1 christy heffernan
kilkerny
13- 3-
18
4.5
5 1
horgan
cork
8- 18
18
1 4.5
Question
How nany players in the 1982 all-Treland senior hurling championship had higher average score per gane than the overall average score game of the compatition?
Code Table
Zero-shot Generate through LLMs
Tittle: "yugoslavia national football tean results"
date
results
type
e april 18
helgrade
france
1:a
1966 wec
nay
belgrade
england
1:1
friendly
2 june 16
aslo norway
norway
8:3
1966 wce
september noscon
8:0
friendly
[ septenbar 19
luxenbourg
luvanbourg 5:2
1966 weq
october
paris france
france
8:1
1966
1 6 noverber 7
belgrade
norway
| 1:1
1966 weq
Question
Did the Yugoslavia national foothall tean play ganes against teans autside of Eurape in the table? Answer
only "Yes"
that is nost accurate nathing else.
Code
In-context Generate through LLMs
Code Output
Output executed by external Python Interpreter
Ansver
Table 8: Prompt design for all baseline methods the main experiment

---

Prompt Design Asswerabiiity
description
tasks, not all questions need to answered. should whether to answer according to table and comonsense knowledge, that is to judge if the following question is answerable or follows. unanswerable. The definitions of unanswerablee quastions Definition
Definition of unanswerable
Type of unanswerable
questions
Out scope
Lacking essential information
given table.
Hellucination
The assunption in the question is invelid besed on the table.
Problenatic
question itself contains logical enror.
3 I Subjective
The answer vartes due to differant netrics, algorithns, and criteria.
Others
other types questions that not be labeled.
4-shot
Read the table below regarding *1982 ireland senier hurling championship" to judge if the following
question i5 answerable unanswerable.
|rank] player
county
tally
natches
average
[-:
horan
offaly
17
2
hilly fitzpatrick
kilkenny
2- 24
38
7.5
2
3 tony "sulliven
cork
1e- 28
7
3
J nolloy
galway
13-11
4
5 christy heffernan
kilkenny
13-
18
4.5
5
horzan
cork
Question
How nany players in the 1982 all-rraland sentor hurling charpi onship had a higher average score per game the overall average answerable. (4.)
gare corpetition?
Answer:
Read the teble belom regarding "g.d. estoril praie" to judge if the following question answerable unanswerable.
season
conpetition
round
opponent
hone
oway
2813
14 uefa auropa league 3q
hapoel ronat gan
R 8
2013
14 uefa league
off
pasching
2
0
2
2 2013 14 uefa europa league group
sevilla
1
[3] 2813 14 uefa europa league group
slovan 1iberec
11-2
2013
europa league group
freiburg
Question
Mas there correlation between GO Estoril Praie's 's
formonannce in hore gones and anay ganes during the
2013-14 UEFA Furopa League corpatition?
Answer
answerable. (No.)
Read the table below regarding 11941 in brazilian football" Judge if the Following question is answerable or unanswerable.
position
tean
points
played
drawn
lost
against
difference
|-:
corinchians
35
28
3 |
17
44
são paulo
32
23
2
3 portuguesa
palestru itália 5p
38
28
5
19
25
3
20
28
6
7
46
1-3
santos
28
60
5
5
sãa paulo railway
18
28
4
9
53
hespanhee
portuguesa santista
15
20
43
8
ypiranga
52
1
Juventus
28
49
17
18
11 conercial
78
3
16
76
52
Question Answer
teams highar points nora 1ikely to teans with points?
unanswerable.
table below regarding geo online collection" to judge following question is answerable or unanswerable.
(Due its excessive size. table is disregarded cannot be displayed) Does the longth of the Japanese titles differ significantly between pre-2985 and post-2805 releases? unanswerable. the table
Question Answer
Interface
regarding "(titley" to judge the following question answerable or unanswerable.
[table]
Question: Cuuestion)
Answer with only "answerable" of "unanswerable" and no other outputs. (If the question is ansverable, give answers, otherkise answer with only "unanswerable".)
Answer:
Table 9: Prompt design for binary classification and answering task is enclosed in parentheses and highlighted in green.
tasks. The prompt for the question-

---

Sub type
Definition and Template
D: Mapping between the values in a specific column and the corresponding rows in a table
Indexing
T: Find the row/columns of [ENTITY]
D: Retrieve data from a tahle hased specific conditions.
Filter
T: Filter the rows/columns based on the [ENTITY]
Group data bused one or more columns/rows. Filter the rows/columns based on [ENTITY]
Grouping
D: Order data in specific way.
Sorting
Sort the rows/columns based on [ENTITY]
D: Determining whether given statement logically follows from of premises or background knowledge.
Grounding
T: Group the rows/columns based on [ENTITY]
Auto-
Cutegorizing classifying information into prodelined cutegories groups based content. T: The term of [ENTITY i] in the question mapped to [ENTITY2]
categorization
Temporal
D: Make presumptions about humans' knowledge times, durations, time intervals. Based the [ENTITY time, temporal indicator (ENTTTY2]
Reasoning
Goographical/ Spatial Reason-
D: Reasoning ubout Geographical/Spattal knowledge.
T: Conduct geographical/spiee reasoning [ENTITY]
ing
D: Combining multiple values into single value to summarize data and make it easier to understand.
Aggregatiigg
T. Conduct the aggregating operation of [ENTITYI] I] on the value of [ENTITY2].
D: Basic mathematical operations.
Arithmetic
T. Conduct the arithmetic operation of [RNTITY[] on the value of [FNTITY2].
Reasoning with Quantifiers
The process making logical and mathematical inferences from statements contain quantifiers. Conduct the reasoning with quantifiers of [ENTITY]] the domain of discourse of [ENTITYY]
Table 10: The definitions (D) and templates (T) reasoning and operations types in our proposed taxonomy.

---

Question:
Are there anomalous observations in terms of popular vote percentage for candidates in the grassroots party?
Answer:
Type the answor
Bridging: requires finding the hrrs--hpp evidence to second-hog hop one.
Intersection: requires find entity that satisfles two independent conditions.
Comparison: requires com comparing the property of two different entities.
How do sub questions compose?
Brdging
Intersection
Camparison
Other
Explicit: The question decomposition be writzen with vecabulary limited words from questions, their inflections, and function words, Implicit: otherwise.
the question implicit or explicit? +
impler
Explich
Operation Types
Indexing:
Definition: mapping between the values specific column and the corresponding rows in table. Examples: For the data the second row and third column,
b. Filter:
Definition: retrieve data from table based specific conditions.
Examples: find the events happened in 2013,
Grouping:
Definition: group data based on more columns/rows.
Examples: For male athletes,
Sorting;
Definition: order data in specific way.
Examples: select the player who have the third highest scores,
Reasoning Types
Aggregating:
Definition: combining multiple values into single value to summarize data and make it easier to understand.
Examples: e.g., sum(), average(), min(, max0.
b. Arithmetic:
Definition: basic mathematical operations
Examples: e.g, ,,5,,
Grounding
Definition: Determining whether given statement logically follows from premises background knowledge.
Examples: Turoverr exceeds make profit
iuto-categorizatio::
Definition: Categorizing or classifying information into predefined categories groups based on content context.
Examples: yellow, blue, green colors
Temporal Reasoning:
Definition: Make presumptions about humans' knowledge of times, durations and time intervals.
Examples: In the next two years,...
Geographicall/ Spatial reasoning:
Examples: e.g, For palyers from Asia,
Reasoning with Quantifiers"
Definition: The process of making logical and mathematical inferences from statements that contain quant
Examples: Are all dogs mammals? (universal quantifier) There exists a dog that friendly. (existential quantifier) Other commonsense reasoning: social reasoning.

---

Step 1
Indkeing
Filer
Grouping
Sorting
Grounding
Reasoning
Geographto rSpatial Ressoning
Commoneenue
Aggregating
Anthmmtio
Reasuniin Quantiferr
Submit
operation Indexing, template: row//columns (ENTITY)
[ENTITY
Step
Indecing
Grouplng
Sorting
Grounding
Tamporal
Geugnach Reasoning
Other Commonsenoe Reasoning
Aggregating
Arthmmeio
Reasoning with Quantherrs
Submi
operation Grouping, template: Group rows/columns [ENTITY) TENTTTM
Step 3
Indeding
Fittr
O Grouping
Sorting
Groundigg
categorization
Reasoning
GeograpNzzl /Spttal Ressoning
Commonsense Reasoning
Aggregating
Arthmetio
Ressoring with Quartifrrs
S.bmit
the reasoning of Aggregating, the template: Conduct the aggregating
operation [ENTITYA] [ENTITY2].
SJM
JENTTT]
Step
Indesing
Filter
Grouping
Sotting
Groundin ng
ontegorization
Reasoning
Geograpiical Ressoning
Commonsense freasoring
Arthmetic
Ressoning with Quanniiera
Submit
the reasoning of Reasoning with Quantifiers, the template: Conduct the reasoning
with quantifiers [ENTITYA] domain (ENTITY2).
JENTTY]
Submit
Figure Our detailed data annotation interface.


Reviews:
[{"review_id": "F21JGxcrFU", "reviewer": "Reviewer_r1r7", "summary": "", "questions": "- From Figure 3, we see that ARC is based on Python Pandas code generation. Will SQL be a better choice for this table question answering?\n- Could you provide more baseline results and analysis on CRT-QA? e.g., open-source code generation models (starcoder, santacoder,…) or finetuned language models.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful and positive feedback! We are encouraged that you found our proposed CRT-QA **pioneering and potential impact** on **future AI research in data science**. We are also glad that you highlighted our TQA dataset’s **uniqueness in multi-step reasoning and reasoning chain annotations**, which differentiate ours from others in the TQA landscape. \n\n- W1: Lack of traditional fine-tuned models’ performance.\n    - As our work focuses on investigating LLMs’ reasoning abilities over tabular data, we do not incorporate finetune-based models in our experiments. In response to your comments, we further conduct **additional experiments** with **two types of fine-tuned models** on CRT-QA. First, we test our dataset on TaPex [1], one of the **SOAT tabular pre-trained models** based on BART. Then, we test a SOTA **code-completion model**, Starcoder [2]. We conduct zero-shot prompting and PAL for TaPex and code-completion models respectively.\n    \n    | Model (Method) | Index | Sort | Group | Filter | GRO | CAT | TEM | AGG | ARI | SPA | QUA | OTH | Overall |\n    | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n    | Tapex (zero-shot) | 35.74 | 22.5 | 45.26 | 32.38 | 27.66 | 0.00 | 25.93 | 28.68 | 27.31 | 26.32 | 54.44 | 37.70 | **35.05** |\n    | Starcoder (PAL) | 26.15 | 25.00 | 26.95 | 12.70 | 26.95 | 22.22 | 25.93 | 14.91 | 18.49 | 5.26 | 40.56 | 26.67 | **23.64** |\n    \n    From the table, we find that both Tapex and Starcoder can not reach the performance of ChatGPT. Besides, Tapex performs much better than Starcoder. One of the reasons may be that Starcoder was not trained specifically on tabular data.\n    \n- W2: Only the EM score for evaluation may not be comprehensive enough.\n    - As described in the section of **Limitation (lines 552-567)**, we acknowledge that the EM score has its own limitations. For example, EM fails in many cases like “15” and “fifteen”. As a result, we **manually calibrate (see lines 553-554)** such cases. In the section of Limitation, we also further discuss the limitations and challenges of current metrics on free-form answer-generation tasks. Generally, the EM score is suitable for our task because, during the data construction process, we **only** select the questions that can be answered **within several words without ambiguity**, which can be easily evaluated using EM for most cases.\n- Q1: Why use Python Pandas instead of SQL  for code generation? Will SQL be a better choice?\n    - Our ARC framework can **easily incorporate** **any other programming language** such as SQL. In our experiments, we opted for Python Pandas because it is much more **flexible and versatile** than SQL. For example, when processing comparison tasks, SQL typically outputs a concrete query result or numerical value instead of a boolean value (a judgment of true or false).\n\n        | Name | Age | Gender |\n        | --- | --- | --- |\n        | John | 25 | Male |\n        | Sarah | 30 | Female |\n        | Alex | 25 | Male |\n\n        For example, if we have a table with columns Name, Age, and Gender, and we want to know which gender has a higher average age, we can use the following SQL query: `SELECT Gender, AVG(Age) as AvgAge FROM table_name GROUP BY Gender`. This will return a result set with two rows of data: one row displaying the average age of males and another row displaying the average age of females. However, further logical processing is needed to determine the final result. Although our proposed ARC can handle it in the last step of Iterative LLM calling with code output, **other code-based baselines** such as PAL **can not deal with such cases**.\n\n- Q2: Can you provide more baseline results and analysis on CRT-QA?\n    - Please refer to the response to W1.\n\n[1] Liu, Qian, et al. \"TAPEX: Table Pre-training via Learning a Neural SQL Executor.\" *International Conference on Learning Representations*. 2021.\n\n[2] Li, Raymond, et al. \"StarCoder: may the source be with you!.\" arXiv e-prints (2023): arXiv-2305.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper's significance in the TQA domain is evident due to the following:\n\n- The CRT-QA is a pioneering dataset tailored for intricate reasoning over tabular data, highlighting the existing challenges for state-of-the-art pretrained language models.\n- CRT-QA can motivate future AI research in data science. It distinguishes itself from other datasets, as many lack multi-step reasoning annotations. CRT-QA incorporates a reasoning chain, delineating the process to derive answers from the table – a valuable feature in the current TQA landscape.\n- The paper showcases clarity in its writing and is methodically structured, facilitating a coherent understanding.", "reasons_to_reject": "While the paper is commendable, there are some aspects that require further consideration:\n\n- A significant proportion of the baselines leverage the ChatGPT series models. To provide a holistic understanding, the inclusion of results from other architectures, such as finetuned language models and end-to-end models, would be beneficial.\n- Exact Match (EM) might not provide a comprehensive evaluation of a model's performance on CRT-QA. The authors could enrich their analysis by incorporating additional evaluation metrics.", "questions_to_authors": "- From Figure 3, we see that ARC is based on Python Pandas code generation. Will SQL be a better choice for this table question answering?\n- Could you provide more baseline results and analysis on CRT-QA? e.g., open-source code generation models (starcoder, santacoder,…) or finetuned language models.", "paper_topic_and_main_contributions": "The paper introduces the CRT-QA dataset, curated for facilitating complex reasoning over tabular data. This contribution is a commendable addition to the table question answering (TQA) research domain. Furthermore, the authors provide a rigorous baseline evaluation on this novel dataset, demonstrating the challenges posed by the dataset even to contemporary large language models.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PZghNeOga4", "reviewer": "Reviewer_ejgp", "summary": "", "questions": "Ignore if space constrained (won't influence rating), prioritize weaknesses:\n\n- A. How are the reasoning path annotations used in ARC prompts, if at all? Although these reasoning paths are useful, I'm not sure how and where you are using them. Perhaps, only in building CoT prompts for non-ARC baselines?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review of our paper. The **novel and useful features** of **implicit questions, reasoning paths, and unanswerable questions** indeed set our dataset apart from the existing ones, and we are glad to see that you find these contributions **significant**.\n\n- W1: It's unclear what's the (human) UB or solvability of this dataset.\n    - The **inter-annotator agreement** reported in **Table 2** can reflect the human upper bound. The over **93% of the inter-annotator agreement** indicates our dataset’s considerable solvability for humans. During the data construction process, we have **separated the unanswerable and Indeterminate questions** which can alleviate the ambiguity to ensure the solvability for \"normal\" questions.\n        \n- W2: no fine-tuned model baseline for this new dataset.\n    - As our work focuses on investigating LLMs’ reasoning abilities over tabular data, we do not incorporate finetune-based models in our experiments. In response to your comments, we further conduct **additional experiments** with **two types of fine-tuned models** on CRT-QA. First, we test our dataset on TaPex [1], one of the **SOAT tabular pre-trained models** based on BART. Then, we test a SOTA **code-completion model**, Starcoder [2]. We conduct zero-shot prompting and PAL for TaPex and code-completion models respectively.\n    \n    | Model (Method) | Index | Sort | Group | Filter | GRO | CAT | TEM | AGG | ARI | SPA | QUA | OTH | Overall |\n    | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n    | Tapex (zero-shot) | 35.74 | 22.5 | 45.26 | 32.38 | 27.66 | 0.00 | 25.93 | 28.68 | 27.31 | 26.32 | 54.44 | 37.70 | **35.05** |\n    | Starcoder (PAL) | 26.15 | 25.00 | 26.95 | 12.70 | 26.95 | 22.22 | 25.93 | 14.91 | 18.49 | 5.26 | 40.56 | 26.67 | **23.64** |\n    \n    From the table, we find that both Tapex and Starcoder can not reach the performance of ChatGPT. Besides, Tapex performs much better than Starcoder. One of the reasons may be that Starcoder was not trained specifically on tabular data.\n    \n- Q1: How are the reasoning paths used?\n    - As Review r1r7 mentioned, \"our TQA dataset’s uniqueness in multi-step reasoning and reasoning chain annotations\". Evaluating LLMs’ reasoning path is an **important but challenging** task and the current approaches are **most goal-oriented**, which is **far from comprehensive**. Our annotation of fine-grained reasoning paths is geared towards providing a reference to evaluate LLM’s reasoning process (**process-oriented**). For example, we use human-annotated reasoning paths to evaluate LLM’s reasoning process in vanilla prompting baselines **(lines 343-348)** and find that LLMs can often generate correct reasoning plans but are **unable to correctly execute them**, inspiring us to **incorporate external tools** to augment LLMs.\n- Typos Grammar Style And Presentation Improvements:\n    - T1: What does \"(for past datasets) definition of reasoning does not align with the reasoning ability of LLMs” mean?\n        - It has the exact same meaning as lines 183-185. As you suggested, we will bring it introduction or related work in the camera-ready version.\n\n[1] Liu, Qian, et al. \"TAPEX: Table Pre-training via Learning a Neural SQL Executor.\" *International Conference on Learning Representations*. 2021.\n\n[2] Li, Raymond, et al. \"StarCoder: may the source be with you!.\" arXiv e-prints (2023): arXiv-2305.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This new dataset has multiple novel and useful features over the existing TableQA datasets -- implicit questions, reasoning paths, and unanswerable questions. Each of these is an important contribution and can be beneficial to people working in this TableQA field. \n\n- Although it's not a modeling paper, this paper also introduces an interesting approach that solving the task, which can be applied easily to other TableQA datasets as well. The authors also compare their modeling approach with several other few-shot approaches across different reasoning types, providing a good analysis. (Although not necessary, it would be useful to compare ARC's performance with other approaches on existing TableQA datasets, perhaps in the appendix)\n\n- The paper is well written.", "reasons_to_reject": "__It's unclear what's the (human) UB or solvability of this dataset.__\n\nFirst of all, it's unclear how a ground-truth answer is generated. Perhaps, reasoning path annotation already involves writing the final answer, but I'm not sure. But importantly, the information about assessing the quality of the dataset solvability or (human) upper bound is missing. This is especially important as the questions are partly autogenerated. On a related note, it'd also be useful to discuss how the difficulty of this dataset compares with other datasets for the current best models.\n\n__There is no fine-tuned model baseline for this new dataset.__\n\nAlthough I understand people are mainly interested in zero/few-shot settings these days, assessing the solvability of tasks with finetuning can also be useful for people who are interested in it.\n\nPlease correct me if I'm missing something here. I can increase the score post-rebuttal.", "questions_to_authors": "Ignore if space constrained (won't influence rating), prioritize weaknesses:\n\n- A. How are the reasoning path annotations used in ARC prompts, if at all? Although these reasoning paths are useful, I'm not sure how and where you are using them. Perhaps, only in building CoT prompts for non-ARC baselines?", "paper_topic_and_main_contributions": "This work proposes a new TableQA dataset called CRT-QA. Unlike past datasets,\n\n1. that mainly have explicit reasoning questions (having operations like select, filter, etc are easily identifiable from question words), this work introduces questions that are more implicit in nature (requiring informal/commonsense reasoning).\n2. this dataset has unanswerable or interminate questions to assess models' answerability prediction.\n3. this dataset has a fine-grained annotation of reasoning paths, subquestions, and answers required to answer complex questions.\n\nTo build this dataset, they prompt LLMs to generate questions from tables, and then have humans give feedback to improve the question. They have humans annotate the reasoning paths using templates with slots.\n\nThis work also introduces a model, ARC, that essentially works by iterative execution of pandas code. They bootstrap demonstrations based on the dev set instead of hand-crafting them. Then they compare it with many other few-shot LLM strategies and show it works the best (~60%), which still doesn't solve the dataset.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "In LLM with tools section of related work: Decomposed Prompting (https://arxiv.org/abs/2210.02406) which also uses tools (or modules) for complex tasks."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "It was not clear to me what you meant by \"(for past datasets) definition of reasoning does not align with the reasoning ability of LLMs\" in related work and also in the intro. I only understood it in 183-185. This is a good point, consider bringing it introduction or related work to better distinguish your work early enough.\n\nThere a capitalization mistakes at 2-3 places (e.g., L564). Consider proofreading again.\n\nL060: on -> of"}}, {"review_id": "uJDbeIQeLm", "reviewer": "Reviewer_dBCv", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your meaningful feedback and comments on our work. We're glad you recognized the **uniqueness of our complex reasoning** TQA dataset and the value of the additional meta-data we provided. Your observation about our method's advantages in terms of **performance, interpretability, and the reduction of human-crafted few-shot examples** is on point. We appreciate your thorough review and are open to any additional suggestions or insights you may have.\n\n1. W1: Test on a table-oriented model\n    \n    As our work focuses on investigating LLMs’ reasoning abilities over tabular data, we do not incorporate finetune-based models in our experiments. In response to your comments, we further conduct **additional experiments** with **two types** of fine-tuned models on CRT-QA. First, we test our dataset on **TaPex** [1], one of the **SOAT tabular pre-trained models** based on BART. Then, we test a **SOTA code-completion model, Starcoder** [2]. We conduct zero-shot prompting and PAL for TaPex and code-completion models respectively.\n\n    \n    | Model (Method) | Index | Sort | Group | Filter | GRO | CAT | TEM | AGG | ARI | SPA | QUA | OTH | Overall |\n    | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n    | Tapex (zero-shot) | 35.74 | 22.5 | 45.26 | 32.38 | 27.66 | 0.00 | 25.93 | 28.68 | 27.31 | 26.32 | 54.44 | 37.70 | 35.05 |\n    | Starcoder (PAL) | 26.15 | 25.00 | 26.95 | 12.70 | 26.95 | 22.22 | 25.93 | 14.91 | 18.49 | 5.26 | 40.56 | 26.67 | 23.64 |\n\n    From the table, we find that both Tapex and Starcoder can not reach the performance of ChatGPT. Besides, Tapex performs much better than Starcoder. One of the reasons may be that Starcoder was not trained specifically on tabular data.\n    \n2. W2: To prove the challenge of CRT-QA, the author might need to show that the same LLM can easily achieve better results on the previous dataset.\n    - In response to your comments, we have conducted **additional experiments** with GPT-4 on **two widely used TQA datasets**, WikiTableQuestions (**WTQ**) [3] and **HiTab** [4]. WTQ is one of the earliest and one of the most widely-used TQA datasets which contains 4344 tabular questions. We use the same prompt design as Chen et al. [5] for zero-shot and CoT experiments. The more recent HiTab contains 1584 tabular QA in the test set and all of their tables are complex hierarchical tables with multi-level headers. The experiment results and previous SOTA performance are listed as follows:\n    \n        |  | GPT-4 Zero-shot | GPT-4 CoT | Reported SOTA |\n        | --- | --- | --- | --- |\n        | WikiTableQuestions | 71.39 | 73.76 | 52.7 [6] |\n        | HitTab  | 87.84 | 91.36 | 40.7 [4] |\n        | Averaged EM difference between CRT-QA | 33.41 | 26.24 | - |\n    \n        From the table, we find that GPT-4 can achieve **much better results** on previous TQA datasets than CRT-QA **even without external tools**, indicating the challenge of our proposed dataset. Besides, GPT-4 can easily surpass previous SOTA model on these datasets.\n    \n3. W3: Details about Figure 5.\n    \n    The experiment is tested on the whole dataset, including unanswerable/indeterminate and \"normal\" questions. We conducted tests on ChatGPT using a few-shot prompt and a temperature setting of 0 in order to investigate the language model's capabilities preliminarily to determine whether to choose to answer. Detailed prompt design can be found in Table 7 in the Appendix. The confusion matrix of binary classification is `TN:202 FP:48 FN:270 TP:474`, while `TN:206 FP:44 FN:176 TP:568` for question answering.\n    \n    ChatGPT sometimes considers a “normal” question to be unanswerable/indeterminate both in binary classification and question answering. We will provide more details about this experiment in camera-ready version. \n    \n\n[1] Liu, Qian, et al. \"TAPEX: Table Pre-training via Learning a Neural SQL Executor.\" *International Conference on Learning Representations*. 2021.\n\n[2] Li, Raymond, et al. \"StarCoder: may the source be with you!.\" arXiv e-prints (2023): arXiv-2305.\n\n[3] Panupong Pasupat and Percy Liang. 2015. Compositional Semantic Parsing on Semi-Structured Tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1470–1480, Beijing, China. Association for Computational Linguistics.\n\n[4] Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmei Zhang. 2022. HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1094–1110, Dublin, Ireland. Association for Computational Linguistics.\n\n[5] Wenhu Chen. 2023. Large Language Models are few(1)-shot Table Reasoners. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1120–1130, Dubrovnik, Croatia. Association for Computational Linguistics.\n\n[6] Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir R Radev, Richard Socher, and Caiming Xiong. 2021. Grappa: Grammar-augmented pre-training for table semantic parsing. In ICLR.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The author introduces a more complex reasoning QA dataset over tabular data, which is created by human-machine-in-the-loop method. Additional meta-data is given in the dataset which can be beneficial for the model diagnosis. \n2. They proposed a method that can achieve better performance than zero-shot models or few-shot learning models. Their methods have advantage in terms of interpretability and also get rid of human crafted few-shot examples. \n", "reasons_to_reject": "Overall, this is a good resource for the community and the proposed method is intuitive and achieves good performance. But I think there are few things can be done to make the paper better. \n\n1. The dataset is a test-only dataset, since most of the models are not trained on table corpus, it might be reasonable that such language models can not do well on table dataset. The author did not test a table-oriented model such as tableBERT on the proposed dataset, which can be interesting to see. \n2. To show that CRT-QA is more complex than the previous benchmark, the author might need to show that the same language model can easily achieve better results on previous dataset. \n3. It is unclear that Table 5 is the result from which language model. And also, is this result just evaluated based on the unanswerable/indeterminate subset? If so, why not do the evaluation on the mix of unanswerable/indeterminate and normal questions? Will the model say a normal question is unanswerable/indeterminate? \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a new Question Answering dataset over Tabular data which requires complex reasoning, CRT-QA. This dataset includes fine-grained information such as type of operation and type of reasoning, which can be useful for diagnosis models. They show that CRT-QA is a difficult task for the large language model. \nThey propose a method that utilizes a language model to translate a question into code, which can then use an external python library tool to answer the question.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Naming: “ARC” has been taken as the name for the dataset AI2 Reasoning Challenge[1], to avoid duplication, maybe the author can have a different name for the proposed model.\n\n[1]Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge"}}]

Author rebuttal:

Meta-review: This paper presents a noteworthy addition to the field of table question answering (TQA) research. The authors present a new dataset and offer a comprehensive baseline evaluation of this innovative dataset, illustrating the substantial challenges it presents even to modern large language models. It stands out from other datasets, as many lack annotations for multi-step reasoning. CRT-QA incorporates a reasoning chain, delineating the step-by-step process required to derive answers from the table, which is a valuable feature in the current TQA landscape.

While not primarily a modeling paper, this work also introduces an intriguing approach to solving the task, which can be readily applied to other TableQA datasets. Additionally, the authors compare their modeling approach with several other few-shot approaches, encompassing various reasoning types, and providing a thorough analysis.

The authors have effectively addressed the reviewers' concerns by promptly conducting additional experiments to furnish more evidence, thereby offering a more holistic understanding of their proposed model.
Decision: Accept-Main