API-Assisted Code Generation for Question Answering on Varied Table Structures
Authors: Yihan Cao
Shuyi Chen
Ryan Liu
Zhiruo Wang
Daniel Fried
Venue: EMNLP 2023 Main (2023)
Keywords: table question answering
code generation

Abstract: A persistent challenge to table question answering (TableQA) by generating executable programs has been adapting to varied table structures, typically requiring domain-specific logical forms. In response, this paper introduces a unified TableQA framework that: (1) provides a unified representation for structured tables as multi-index Pandas data frames, (2) uses Python as a powerful querying language, and (3) uses few-shot prompting to translate NL questions into Python programs, which are executable on Pandas data frames. Furthermore, to answer complex relational questions with extended program functionality and external knowledge, our framework allows customized APIs that Python programs can call. We experiment with four TableQA datasets that involve tables of different structures --- relational, multi-table, and hierarchical matrix shapes --- and achieve prominent improvements over past state-of-the-art systems. In ablation studies, we (1) show benefits from our multi-index representation and APIs over baselines that use only an LLM, and (2) demonstrate that our approach is modular and can incorporate additional APIs.

API-Assisted Code Generation for Question Answering on Varied Table Structures
Yihan Cao"
Ryan Liu" Zhiruo Wang Daniel Fried
Shuyi Chen'
Carnegie Mellon University
(yihanc, shuyic, ryanlii zh mhoi, (eroooddddreddedadreeee cmu
Abstract
Who more likely have cancer, the elder or the young?
A persistent challenge to table question answer- ing (TableQA) by generating executable pro- grams has been adapting varied table struc- tures, typically requiring Nomain--spciifee logi- cal forms. In response, this paper introduces unified TableQA framework that: (1) provides a unified representation for structured tables as multi-index Pandas data frames, (2) uses Python as powerful querying language, and (3) uses feo- prompting to translate NL questions into Python programs, which are ex- ecutable on Pandas data frames, Furthermore, to answer complex relational questions with extended program functionality and external knowledge, our framework allows customized APIs that Python programs can call. We ex- periment with four TableQA datasets that in- volve tables of different structures relational, multi-table, hierarchical matrix shapes and achieve prominent improvements over past state- satte-of-the-are art systems. In ablation studies, we (I) show benefits from multi-index repre- sentation and APIs over baselines that use only an LLM, and (2) demonstrate that our approach is modular and incorporate additional APIs.
Illness
data.loc
total
percent
total
percent
All patients
10,000
2.5%
200
0.3%
Elders
2,400
3.5%
126
0.9%
larger
Young
2,600
1.5%
74
0.2%
Code LM
API function
compare_largen
[ latalles((lllness,, 'Cancer', 'percent'), ('All patients', 'Elders')] I],
data.loc((lllness', 'Cancer', 'percent'), patients', 'Young'))
I. ['Elders, "Young']
Python program
Figure I: Our approach answers questions about com- plex tables by representing the tables as Pandas multi- index data frame, and using a code generation LM translate the question into a Python program that uses assistant API functions to operate on the data frame.
tures and required tailoring logical forms to each specific type of table structure (Wang et al., 2015; Guo et al., 2019). These methods struggle to work on tables with structures outside of their original domain. For example, the neural-symbolic ma- chine (NSM) approach designed for relational ta- bles (Liang al., 2017) only achieves 29.2% accu- racy on the hierarchical matrix table dataset HiTab (Cheng al., 2022) due to the ineffectiveness of its logical forms on hicrarchical tables.
1 Introduction
Tables are an important and widely used format for storing and retrieving information. However, since they are often constructed to present information in visually effective way, they consequently oc- in diverse formats (Chen and Cafarella, 2013; Nishida al., 2017: Wang al., 2021b). Thus, to effectively answer questions about all tabular in- formation, we must consider information stored in relational (Pasupat and Liang, 2015), matrix, and hierarchically indexed tables (Cheng et al., 2022), and also address scenarios where multiple tables are presented conjointly (Yu et al., 2018).
Our work focuses on developing unified frame- work solve TableQA tasks across diverse table structures using Python an intermediate lan- guage. In contrast with previous approaches that use custom logical forms to query table data (Guo et al., 2019; Cheng et al., 2022), we propose to query entries and perform operations within the
In the past, works have focused achieving strong results datasets with particular table struc- "Equal contribution.

---

widely used Python Pandas library. This allows us leverage the strong few-shot Python generation capabilities of large code generation models, whilst saving costs by using only a few training exam- ples. An overview of the framework is presented in Figure 1, with more details in Figure 3. Our framework consists of three main parts:
We find that our framework surpasses the state-of- the-art few-shot baselines on the HiTab, AIT, and Spider datasets, achieving absolute improvements of 24.5%, 26.2%, and 2.3%, respectively, while retaining non-trivial performance on WikiTQ. Fur- thermore, we perform an ablation study the API functions we introduced and find that they bring sig- nificant improvements across datasets and models. Our framework also allows the use of existing API operations within Pandas in modular way. For example, we show that combining Pandas' SQL interface API with our multi-indes representation and APIs improves performance by 3.7-7.6% on relational datasets.
First, we transform tables with varied structures into unified multi-index data frame representation adopted from the Python Pandas library ($3.1). The multi-index objects can effectively retain the struc- tural information in a wide range of table formats. From the implicit first step illustration in Figure 1 and more detailed one Figure 2c, we convert tables from a hierarchical format to the multi-index representation, which will enable generated code to successfully query elements in the table.
2 Problem Setting and Datasets
In this section, we provide a formal description of the TableQA tasks ($2.1) and describe the four datasets used in our experiments. containing tables of varied structures ($2.2).
Second, we translate TableQA questions into Python programs an executable intermediate lan- guage by prompting code generation models ($3.2). Specifically, we use few-shot paradigm where we provide the multi-index headers of the table, a few rows of the table in textual form, the question, and exemplar program annotations. As shown in Fig- ure 1, the table and question are input to the code generation model along with the few-shot prompt, and the generated Python code uses the Pandas data. loc function query the appropriate cells.
2.1 Problem Statement
Each example in a TableQA task contains table and question q about the table, and aims to gener- ate the correct answer a to the question. While both q and a are textual sequences, the table can vary in structure - from relational tables (Figure 2a, Figure 2b) to hierarchical matrix tables (Figure 2c). The common symbolic approach to TableQA generates an intermediate program from model M using the table and question, p M t). The program is then executed on / to yield the answer a' exec(p, t). In traditional methods, tables are serialized text sequences by cell string concate- nation, and programs are logical forms specifically designed for certain domains. To preserve and leverage more structural information, we represent tables as Pandas multi-index objects, tm. and gen- erate Python programs accordingly.
Third, introduce assistant API functions to extend the capabilities of our framework beyond Python Pandas and achieve broader coverage over various TableQA tasks. These functions enable a model-generated program to query external knowl- edge and perform various additional operations on the multi-index representation. In this paper, we demonstrate the usage of two simple types of API functions, Operation APIs and QA API ($3.3), and show through ablations that they increase the per- formance of our framework across various datasets, For example in Figure 1, the model outputs code using one of our API functions, compare_largen to solve the question.
2.2
QA over Varied Table Structures
We use four diverse TableQA datasets evaluate our approach: WikiTQ (Pasupat and Liang, 2015), HiTab (Cheng al., 2022), AIT-QA (Katsis et al., 2022), and Spider (Yu et al., 2018). As listed Table 1, these datasets cover a broad spectrum of table structures and topical domains, thereby pre- senting various challenges associated with different table structures.
We evaluate our method across relational, hier- archical, and matrix tables, as well as multiple ta- ble paradigms using the WikiiTableQuestionss (Wik- iTQ) (Pasupat and Liang, 2015), HiTab (Cheng al.. 2022), AIT-QA (Katsis al., 2022), and Spider (Yu al.. 2018) datasets ($4). For code generation models, we use more capable propri- etary model, CODEX (Chen et al., 2021), and an open-source model, STARCODER (Li et al., 2023).
WikiTableeuestionss (WikiTQ) WikiTQ (Pasu- pat and Liang, 2015) contains relational tables with

---

Size
Table Info
Dataset
dev
test
structure
source
WikiTQ (Pasupat and Liang, 2015)
2,381
4,344
relational, single
Wikipedia
HiTab (Cheng et al., 2022)
1,671
1,584
hierarchical matrix
Stat. reports, Wikipedia
AIT-QA (Katsis al., 2022)
515
hierarchical
Airline documents
Spider (Yu et al., 2018)
1,034
relational, multiple
College data, WikiSQL
Spider has an undisclosed online test set. We treat the entire AIT-QA dataset as a single test set in our evaluations.
Table I: Statistics and properties of the datasets we use to evaluate our approach, encompassing varied table structures and sources.
relatively simple structures. All tables are collected from Wikipedia articles. The questions usually in- volve basic arithmetic operations, such as sum and max, and sometimes require compositional reason- ing table entries. Since our method does not require any training, we use the standard validation set and test set with 2,381 and 4,344 samples.
(Katsis et al., 2022) contains tables and questions that particularly focus on the airline in- dustry domain. It also contains tables with com- plex structures such hierarchical headers bi- dimensional matrix shapes. Both the complex struc- ture and domain-speete speccifc terminology are challeng- ing for TableQA tasks (Zhu et al., 2021; Katsis et al., 2022). AIT-QA contains 515 questions that are distributed across 116 tables. We use all 515 questions as the test set for evaluation purposes.
AIT-QA
HiTab (Cheng et al., 2022) involves hierarchical matrix tables collected from statistical reports and Wikipedia articles, covering diverse domains such economies, education, health, science, and more. HiTab challenges existing methods with its broad set of operations, complex table structures, and diverse domain coverage. It contains 3,597 hierar- chical tables with 10,672 questions. We evaluate on their test set containing 1,584 samples.
Spider (Yu et al., 2018) is multi-domain text- to-SQL benchmark with tables in a database for- mat. It contains 200 databases across 138 differ- ent domains. All "databases" in Spider consist of multiple relational tables connected by primary foreign keys, which poses unique challenge for methods the multi-table structure. Evaluations on the hidden test set are supported by its online benchmark which only accepts SQL programs. We instead evaluate on the validation set containing 1,034 questions.
Fable taazador
Your
City
Country
Natien
18p0
Athere
Greee
9900
Paca
Trnas
24
Tille depertgnnn
3008
B-sjig Londaz
Ciise
204
UK
.011
relational table.
(b) multi-table relation.
Method
Cold
Illness
We first describe our approach for representing var- iously structured tables with a unified multi-index format ($3.1). We then introduce our method to answer TablcQA questions by generating Python programs ($3.2) from a code LLM. Lastly, we de- tail the use of custom helper API functions ($3.3) that are learned to be used by the LLM in context ($3.4). Figure gives overview of our method.
percent
percent
pilients
10,000
Fiders
3.5%
126
Young
2,800
15%
0.2%
Original Table
Table
Top Rosts
Left Roct
(1llrase, cold, parcensi, (iltnase, cancae, tocatl,
Illneac
oOM
Yoongg
paliento, Siderats passante, Youngr
3.1 Unifying Representation of Diverse Table Structures with Multi-Index Parsing
praxi
Parsed Tree Structure
Multi-inex Object
(c) hierarchical matrix table parsed into multi-index object: step (1) transforms table headers into tree representation. and step creates Pandas multi-index object.
To represent diverse table structures in unified way that is naturally compatible with Python pro- grams, we propose to transform any table with vary- ing structures, denoted as f, into a Pandas multi-
Figure 2: Examples of diversely structured tables.

---

Whols more licyy nowc cinerr the oider young?
may assisaa
Tide Number and pereentage people who intertewed who have or had illnesses.
ilosse
Cold
treal
Casser
PYSTET
Pytton Interpreter
Code
Fiders
7,00
3.7%
176
Young
2.00
17%
barrle
2ample
Cote
Extrittn
Python
Seowee with Pyt
Few shot examples
Figure 3: Our framework consists of three stages: (1) Multi-index parsing accepts one or more tables with diverse table structures, and transforms them into a single hierarchical table, represented using a Pandas multi-index representation m (a process described in $3.1). (2) We then provide natural language question, in-context exemplar annotations, and multi-index representation m as a prompt for pretrained code models like CODEX or STARCODER for Python code generation, as described $3.2 and $3.3. The prompt specifies examples of using our QA API and operation APIs. (3) Finally, execute the Python code, which could contain QA or operation APIs, on the Pandas data frame, outputting the final results. Our API definitions are given in Figure and Figure 4.
index representation," m. We denote this process as multi-indes parsing, which first converts raw table bi-dimensional
Algorithm 1 Preorder tree traversal for multi- indexing.
structure b (T, L, V). then parses it into multi-index representation m. The two steps are illustrated Figure 2c.
result list()
tempList + list()
procedure PREORDER(P: coordinate tree node, result, lup)
First, we first transform a table t of any struc- ture into a bi-dimensional tree representation b = (T, I, V) as defined by Wang et al. (2021b). It yields two coordinate trees: (i) the left tree L by indexing the values V from the left header, and (ii) the top tree T by indexing data cells V from the top header. Specifically in the example Figure 2c, T represents the deep blue area (i.e., two rows at the top): L represents the first column to the left; and V covers all the data cells from 10, 000 from the lef- to 0.2% to the bottom right.
if p is not tree root then tempList <- tempList + (p.value,)
end if
if p.index is None then results results + tempList end if for child in p.children do PRE- ORDeR(child, result, templist) end for return result end procedure
Next, we traverse the left and top trees and ex- tract header path tuples from both header trees to define the row indexes and column indexes in the multi-index representation m, which can be used to reconstruct the original table using Pandas seman- tics. Specifically, we use a preorder traversal (as illustrated in Algorithm 1) on both the top tree T and the left tree L, and enumerate all paths starting from the root node to each leaf header node. For every such root-to-leaf path, we aggregate all cell strings along the path into tuple. For each header tree, we collect all such tuples and correspondingly define the row/column index list in the multi-index representation m. For example, for the top header in Figure 2c, Illness - Cold >> total is one
possible top header path, then we add (Illness, Cold, total) to the column index list in m.
Note that this multi-index parsing process can be readily applied to tables of diverse structures. with a unified representation efficiently preserving all structural information. In the special case of simple- strpleestructured tables such as flat relational ta- bles, both the top and left headers are represented by single-layer trees and undergo similar parsing process as hierarchical structures. When dealing with questions that involve multiple tables, we con- vert all tables into multi-index objects, feed them to the model, and let the model choose which parts it will use to answer the question. For more detailed
"htrss//aandasppydatt... http:///hw..yyy..ooooccccc/oe//a Multilndex.hml

---

illustrations of this parsing process for tables of different structures, please refer $D.
def compore larger(values: list(fioat], args: list|str]) -> str: ""Return the argument associated with the larger value," return sesasvaaues.ndeexmaxxxeeeee
3.2 Generating Python Programs
def compare_smaae (value: list/float], args: list(str]) ] str: "*
With table environments represented as multi-index representation m, we generate Python programs to execute on m to yield the answer prediction a'. Our method is general and not tied to any particular code generation model, as long as the model has sufficient natural language understanding and code generation capability. As such, we validate our approach using two code language models (code LMs) for Python program generation: the propri- etary CODEX model (Chen et al., 2021) and the open-source STARCODER model (Li et al., 2023). CODEX is code LM based GPT-3 (Chen al., 2021). We access it through OpenAl API calls.? We experiment with the strongest ver- sion DAVINCI-002 with 175 parameters. STAR- CODER is an open soun-source 15B code LM, which at the time of this writing achieved top performance among open-source models on many code gener- ation benchmarks such as HumanEval (Li et al., 2023). Both models are trained mixture of natural language and code sequences. We gener- ate code from both models using few-shot prompt- ing (Brown et al., 2020), with prompts specifying examples of the mapping from natural language questions and tables to Python Pandas code. More details follow in $3.4.
"""Retum argument associated return htgp///lleeeiinddxdiivv/vulssssl
Figure Illustration of the two comparison operation APIs.
the age of Person A?", than qualitative questions like "which is higher, A or B?". We design two comparison APIs, as shown in Figure 4, to address this issue. Each API takes a list of values and list of indices and returns the largest or the smallest index according the values. An example show- ing how the comparison APIs are used is shown in Figure 1. We find in $5 that the additional opera- tions enabled by these APIs substantially improve performance.
QA API Sometimes TableQA questions require external knowledge which is not included in the given table. The QA API, proposed by Cheng al. (2023), is designed access external knowledge when solving table QA tasks, by making calls to an LLM.
def table_ qatbaalleteeeooooo question: values: list(flcatt, arge: list[st]]) *"*Glven table, write code to answer the question""
mapped qa, column = question, Inswwringg(["ss () a city in Canada?" for in table(cities]] values, angs)
3.3 Incorporating Assistant APIs
selected columns - mapped_qa_ (olupe_mappee_ee oa_column--True)
We describe two types of API functions, Ques- tion Answering (QA) APIs and Operation APIs, to expand the ability of code LMs to incorporate external knowledge and extra table operations.
def question_ www.eeennnggueettiooooooo list|sti]::
"""Return model answer."" ...
Q [f"Answer question with True or False. [q]" for q question]
(call generative caaptteeeeeeeeeeeeeeeee model__pple) qn return
Operation APIS Operation APIs are designed enhance the functionality of LM-generated pro- grams. Since we use Python as the query language and represent tables in Pandas, we are able to incor- porate simple Python APIs into the solution. For demonstration, we employ two simple comparison APIs in this paper. We emphasize that our frame- work is not tied to any specific operation API, and therefore offers flexibility in designing customized APIs for different datasets or question types.
Figure 5: Illustration of the question answering API.
As illustrated in Figure during the table ques- tion answering stage, we prompt the LLM to write questions itself. These generated questions will direct the code parsing program to call the QA API function. The QA API takes in this question and a set of table indices as parameters. For each value in the questioned column, we insert the value into the generated question and prompt the model again to give an answer to this question. Then, we select the rows with true values the returned column. We include real case from HiTab (Cheng et al., 2022) in Figure 11 which requires geographical knowledge to accurately answer the question.
The design of the comparison APIs inspired by the empirical observation that CODEX is better at handling numerical questions, such as "what is
"One can call the CODEX model via API without knowing the internal process or model parameters.

---

3.4 Learning APIs In Context
4.3 Experiment Results
Building upon the in-context learning ability of large language models (Brown et al., 2020), we use examples of API usage as few-shot context for LM generation of programs. Our LM prompt is designed to be mixture of instructions and solu- tion examples. The instructions specify API usage guidelines and scenarios in which should be used. The solution examples incorporate a balanced mix of examples, some of which utilize the assistant API to perform tasks, while others tackle the ques- tion with Pandas semantics directly. With this di- versity of input examples, we aim to encourage the usage of the assistant APIs in the code gener- ated by the LMs, while avoiding over-reliance on some APIs. Refer şB for more complete prompt templates we used for different datasets.
Table shows our main experiment results across the datasets described in $2.2, encompassing varied table structures, using the CODEX (code-davinci- 002) code LM. We compare full method (Ours) with an ablated version (CODEX) that only uses few-shot prompting of the code LM, without includ- ing our multi-index structure and API functions (see $5.2 for details and more ablations). We also give the performance of state-of-the-art baselines for each dataset from past work.
Dataset
HiTab
Spider
AIT-QA
WikiTQ
MAPO 40.7
DIN-SQL.
RCI
BINDER
Baseline
61.5
51.8
54.8
Codex
59.6
61.2
77.8
41.7
w/ API (Ours)
69.3
63.8
78.0
42.4
4 Experiments
this section, we first introduce the experiment (54.1) and evaluation settings ($4.2), then present our results and analysis ($4.3). We also test the robustness of our method with different code mod- cls ($4.4) and intermediate query language forms ($4.5).
Table 2: Execution accuracy on four TableQA dataset with CODEX generated programs, comparison to state-of-the-art baselines: the MAPO (Liang al., 2018) baseline from HiTab (Cheng et al., 2022), DIN-SQL (Pourreza and Rafiei, 2023) Spider, RCI (Glass et al., 202 I) for AIT-QA, and BINDER (Cheng et al., 2023).²
Our approach, which uses unified representa- tion for tables and question semantics, typically improves over past work despite their use of rep- resentations tailored to individual datasets. In par- ticular, our approach obtains an improvement of 28.6% over the past state-of-the-an training-basee -aased method (Liang et al., 2018), and a 9.7% improve- ment on plain few-shot prompted Codex models for the HiTab dataset.
4.1 Implementation Details
We use the STARCODER 16B and code-davinci- 002 (CODeX) models to generate Python programs. We set the sampling temperature to 0.7 for both models. For every question, generate five model predictions and select the candidate with the high- est log probability. We include five shots in the prompt for WikiTQ, HiTab, and Spider datasets, and cight shots for the AIT dataset.
One exception the WikiTQ dataset, where our method underperforms the BINDER approach (Cheng et al., 2023). We attribute this to the supe- rior alignment of BINDER's 's API with the question characteristics of the WikiTQ dataset, which in- cludes many questions that require external knowl- edge.
4.2 Evaluation Metrics
We evaluate performance using the Execution Ac- curacy (EA) metric following Pourreza and Rafiei (2023) and Cheng al. (2023). EA measures the percentage of questions for which our method pro- duces programs that yield correct answers. We did not include exact matching (EM) as the measure to determine if the predicted query is equivalent to the gold query, since it can lead to false nega- tive evaluations when the semantic parser generates novel syntax structures (Pourreza and Rafiei, 2023; Yu al., 2018). Moreover, Python is our main query and APT function language and most datasets do not provide golden queries in Python, EM is not a practical measure in our case.
Meanwhile, we find strong performance from us- ing Python as a QA representation of the code LM alone: in particular, on the HiTab dataset, we see that the Codex model already achieves an 18.9% improvement compared to previous training- based methods (Liang al., 2018).
'For a fair comparison with our approach, we use 5-shot prompting for BINDER rather than the 12-shot experiments in Binder et al. (2022), which decreases performance compared to the results in their paper.

---

4.4 Robustness to Code Generation Model
IiihiQ SQL Prompr:
You are helpful assstant in writing code to answer questions. Write sqi query behind "say" to amserr the given question correctly.
We next analyze method performance when using the recently released open-source code generation model STARCODER- (Li et al.. 2023). The findings are compiled and presented in Table 3.
All column types are sot by default. Please first convert needed columns to correct needeed.
outputs are dataframes,
exaples:
[FEW SHOTS]
question
the given table below.
While the base model performance with STAR- CODER is lower than CODEX, still observe an improvement from our generated API across all settings, with substantial improvement on the AIT-QA dataset, presumably because a large por- tion of the questions in AIT-QA can be more easily solved using our API functions. This indicates that our approach is compatible with a range of code generation LMs, and performance may scale with future improvements in open-source code LMs.
CREATE TABLE datal,
Year text.
Attendance
EXAMPLEE ROWE] what was year where this team was part of ielengue?
Figure 6: An example prompt for model to generate SQL queries on the WikiTQ dataset.
Query Language
Code Model
Python
w/ API
SQL
w/ API
STARCODER
15.6
15.6
22.4
23.2
CODEX
41.7
42.4
45,4
46.7
Dataset
HiTab
AIT-QA
WikiTQ
STARCODER
29.0
4.0
22.4
Table 4: Execution accuracy WikiTQ when generat-
ing Python and SQL programs.
w/ API
30.1
26.1
23.2
Table 3: Execution accuracy on four TableQA datasets with STARCODER generated programs.
reason for the need to generate very long Python functions is to accommodate different data input types. Nevertheless, our API functions when applied on top of either query language in nearly all settings.
improvements from
4.5 Robustness to Intermediate Forms
One feature of our Python and Pandas-based method for TableQA is that it is modular, and can existing API methods from other Python li- braries when appropriate. In this section, we per- form a case study where we use the SQL querying APT of the pandasql extension library for Pandas. We experiment with the WikiTQ dataset since all tables within are relational types, the operations on which can be readily covered by the SQL grammar. Similarly to the in-context learning configuration introduced in $3.2, the model to generate SQL queries. An example prompt shown in Figure 6.
Ablation Studies
We conduct ablation analyses on the HiTab dataset to examine the improvements brought by assistant API functions ($5.1) and multi-index parsing ($5.2). We refer multi-index parsing w/ MI, and de- note the opposing table flattening approach as w/o MI in the result tables.
API Functions
Multi-Index w/o MI
adapt our input to prompt
QA
Operation
w/ MI
X
X
59.6
56.0
The generated SQL. queries are executed on Pan- das data frames and return Python objects, hence, we can still retain our same API functions and multi-index data frame representation. Please find more details about the prompt and code in SC.
57.4
58.7
59.1
69.4
Table 5: Comparing CODEX execution accuracy on HiTab by removing multi-index parsing helper API functions. MI stands for Multi-Indes parsing. For exper- iments without MI, we employ the flattening approach (Figure 7).
The experimental results with and without the SQL API are presented in Table 4. We see that SQL outperforms Python on the WikiTQ dataset, which we suspect is because this relational dataset affords short SQL queries, which are easier to gen- crate than long Python functions for code LMs trained on sufficient amounts of SQL. The primary
5.1 API Functions
We first study the improvement brought by API functions of different types. As described $3.3, we provide the model with (i) the QA API, and (ii)
"httss:/pyyi.orrg//ae http:///pypppor///oooc////mmm///

---

Cancer
Ilinees
Intel
total
persent
Hattonig
patlents
Finess
Cold total
Cold, pereent
2.5%
Caneer total
Caucer perceat
MI potlents
10,000
2,5%
200
0.3%
EXers
7,400
3,5%
126
0.9%
Eiders
7,400
Yourgg
2,600
Youngg
2,000
1.5%
74
0.2%
Hierarchizal table
Flattened Pandas dataframe
Figure 7: Representing structured table by flattening hierarchical headers.
operation APIs. comparison to the base setting in which neither type of API is utilized, we add one type of API at time to examine their gains.
within specific table structures and across table types, SQL queries perform well in solving ques- tions with relational tables (Pasupal and Liang, 2015; Yu et al., 2018), but their grammar is often unable to cover the entire spectrum of questions (Guo al., 2019). Domain-specific logical forms are often favored when handling more complicated table structures (Wang et al., 2021a: Katsis et al., 2022), queries on text (Gupta et al.. 2019), and grounded environments such as visual question an- swering (Andreas et al., 201 ba; Johnson et al., 2017). However, these specifically designed log- ical forms do not support the operations needed outside their intended domain (Cheng et al., 2022). In contrast previous methods which focus on adapting their solutions to certain table structures, we transform table structures using the multi-index format and utilize the natural flexibility within the Python language. Furthermore, we also introduce the usage of custom API functions as a modular approach to extend the framework.
As shown in Table 5, both types of APIs are helpful. Comparing the two API types, the opera- tion API brings more marginal improvement than that of the QA API. In particular, adding the QA API increases performance by 2.7%, while further adding the operation APIs improves by another 10.7%, compared to the full system without APIs.
5.2 Multi-Index Transformation
Next, we study the contribution of adopting multi- index hierarchical table representation.
In comparison to multi-index parsing base- line, experiment with alternative table header flattening approach that has been adopted for ta- ble processing in many works (Parikh al., 2020; Liu et al., 2022; Wang et al., 2022). More specif- ically, we flatten the top header hierarchy into a single-row relational header by concatenating all cell strings from the root to each leaf header node. Note that this process ignores the header informa- tion in dimensions other than the top and treats them normal table cells. This flattened represen- tation then integrated into our framework using Python queries, code models, and API calls to fa- cilitate the ablation study. An illustration of this flattening process is shown in Figure 7.
Program generation and assistant functions Several works have leveraged program generation to answer questions or carry out planning tasks (Singh et al., 2022; Liang et al., 2022) within other environments such those involving visual modali- ties (Subramanian et al., 2023; Surís et al., 2023; Gupta and Kembhavi, 2023). Past work has also proposed connecting natural and programming lan- guages by adding textual QA abilities programs (Cheng et al., 2023) or adding programmatic opera- tions to NL (Gao et al., 2022; Chen et al., 2022). comparison, we focus more on extending the lim- ited built-in operations in programming languages (e.g., Python), and incorporate helper API func- tions enable broader capabilities for TableQA tasks. This is similar to the goal of extending func- tions into given programming language (Herzig et al., 2020; Guo et al., 2019), however, our APIs also benefit from being casily implemented as Python functions themselves.
As shown in Table 5, the execution accuracy decreases by 3.6% with the multi-index approach compared to the table flattening approach when no API is included. However, as involve QA and operation APIs, the multi-index approach demon- strates a 10.3% improvement over that without multi-indexing. This indicates that the proposed multi-index representation is essential in preserving the rich structural information of hierarchical ta- bles, and can be leveraged by our API operations.
6 Related Work
QA over Varied Table Structures Existing methods for table QA face many challenges both

---

7 Conclusion
References
We introduced a framework for solving general TableQA tasks. This framework is built upon three core components: a unifying multi-index represen- tation, Python programs as the query language, and code generation by prompting a large pretrained code model. Using CODEX the code model, demonstrate improvements over the state-of- the-art performance for multiple datasets, each with its unique table structures and challenges. Through ablations, we show that our proposed multi-index and API functions are both critical to the success of the framework, with the largest improvements datasets involving hicrarchical tables. We also observe improvements in performance for open- source model, STARCODER, demonstrating the ef- fectiveness of our approach with different code models.
Jacob Andreas. Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016a. Learning to compose neural net- works for question answering. In NAACL.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016b. Neural module networks. In CVPR, pages 49.48.
Arne Binder, Leonhard Hennig, and Bhuvanesh Verma. 2022. Full-text argumentation mining scientific publications. Proceedings of the first Workshop on Information Extraction from Scientific Publications. pages 54-66, Online. Association for Computational Linguisties.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu. Clemens Winter, Christopher Hesse, Mark Chen, Firic Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems. volume 33, pages 1877-1901.
Limitations
We note that though our general framework con- sistently improves performance across datasets, we identify through ablations components in our framework that, individually applied, adversely af- feet performance on particular datasets. However, these same components are able to improve perfor- mance in combination with other features of the framework, as well as across other datasets. Thus, we consider these capabilities to be step towards solving general TableQA tasks.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374
One limitation of our current framework is its manual definition of API functions, which may not extend to other datasets with disparate table struc- tures, or involve richer grounded settings. Thus, a future direction for our work is to automate the API generation for different QA datasets and ques- tions. We envision an interactive framework that, given feedback on its performance on dataset, constructs new API functions to solve question types with lower performance.
Wenhu Chen, Xueguang Ma. Xinyi Wang, and William W Cohen. 2022. Program thoughts prompting: Disentangling computation from reason- ing for numerical reasoning tasks. arXiv preprint arXiv:2211.12588
Zhe Chen and Michael Cafarella. 2013. Automatic web spreadsheet data extraction. In Proceedings of the 3rd International Workshop on Semantic Search over the Web, pages 1-8.
Zhoujun Cheng. Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han. MianGuang Lou, and Dongmei Zhang. 2022. HiTab: A hierarchical table dataset for question answering and natural language generation. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers). pages 1094-1 I 10, Dublin, Ireland. Association for Computational Linguistics.
Zhoujun Cheng. Tianbao Xie. Peng Shi, Chengzu Li. Rahul Nadkarni, Yushi Hu. Caiming Xiong. Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2023. Binding language models symbolic languages. The Eleventh Inter- national Conference on Learning Representations.

---

Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Gra- ham Neubig. 2022. Pal: Program-aided language models. arXiv preprint arXiv:2211.1043:
Thomas Wang. Olivier Dehaene, Mishig Davaadorj. Joel Lamy- Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel 7ebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Ahulkhanov, Marco 7occa, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya. Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan rihert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Dan- ish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. 2023, Starcoder: may the source be with you!
Michael Glass, Mustafa Canim, Alfio Gliozzo, Sa- neem Chemmengath, Vishwajeet Kumar, Rishav Chakravarti, Avi Sil, Feifei Pan, Samarth Bharadwaj, and Nicolas Rodolfo Fauceglia. 2021. Capturing row and column semantics in transformer based question answering over tables. In Proceedings of the 2021 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, pages 1212-1222 Online. Association for Computational Linguisties.
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao. Jian- Guang Lou, Ting Liu, and Dongmei Zhang. 2019. To- wards complex text-to-SQL in cross-domain database with intermediate representation. In Proceedings of the 57th Annual Meeting of the A Association for Com- putational Linguistics, pages 4524 4535, Florence. Italy, Association for Computational Linguistics.
Chen Liang. Jonathan Berant, Quoc Le, Kenneth D. For- bus, and Ni Lao. 2017. Neural symbolic machines: Learning semantic parsers Freebase with weak supervision. Proceedings of the 55th Annual Meer- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 23-33, Vancouver, Canada. Association for Computational Linguistics.
Nitish Gupta, Kevin Lin, Dan Roth. Sameer Singh, and Matt Gardner. 2019. Neural module networks for soning over text. arXiv preprint (xXiv:9912.455
Tanmay Gupta and Aniruddha Kembhavi. 2023. Vi- sual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 14953-14962.
Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoe Le, and Ni Lao. 2018. Memory augmented pol- icy optimization for program syuthesis and semantic parsing. In Proceedings of the 32nd International Conference on Neural Information Processing Sys- lems, NIPS' 18, page 10015-10027, Red Hook, NY, USA. Curran Associates Inc.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 4320 4333, Online. Association for Computa- tional Linguisties.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. 2022. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:209.007555
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C. Lawrence Zit- nick, and Ross Girshick. 2017. Inferring and execut- ing programs for visual reasoning. In Proceedings of the IEFE International Conference on Computer Vision (ICCV).
Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. 2022. TAPEX: Table pre-training via learning a neural SQL executor. In International Conference on Learning Representations.
Kyosuke Nishida, Kugatsu Sadamitsu, Ryuichiro Hi- gashinaka, and Yoshihiro Matsuo. 2017. Understand- ing the semantic structures of tables with hybrid deep neural network architecture. In Proceedings of the Thirty-Firs First AAAI Conference on Artificial Intelli- gence, page 168-174.
Yannis Katsis, Saneem Chemmengath, Vishwajeet Ku- mar, Samarth Bharadwaj. Mustafa Canim, Michael Glass, Alfio Gliozzo, Feifei Pan, Jaydeep Sen, Karthik Sankaranarayanan, and Soumen Chakrabarti. 2022. AIT-QA: Question answering dataset over complex tables in the airline industry. In Proceed- ings of the 2022 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies: Industry Track, pages 305-314 14, Hybrid: Seattle, Washington + Online. Association for Computational Linguistics.
Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Man- aal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipan- jan Das. 2020. ToTTo: A controlled table-to-text generation dataset. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 1173-1186. Online. As- sociation for Computational Linguistics.
Raymond Li, Loubna Ben Allal, Yangtian 7i, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li. Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo
Panupong Pasupat and Percy Liang. 2015. Composi- tional semantic parsing on semi-structured tables. In

---

Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference Natural Language Processing (Volume I: Long Papers), pages 1470- 1480, Beijing, China. Association for Computational Linguistics.
Conference on Empirical Methods in Natural Lan- guage Processing, pages 3911-3921, Brussels, Bel- gium. Association for Computational Linguistics.
Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. Tat-qa: A question answering benchmark on hybrid of tabular and textual content in finance. arXiv preprint 10X.v:2005.07624.
Mohammadreza Pourreza and Davood Rafiei. 2023. Din-sql: Decomposed in-context learning of text-to-sql with self-correction. arXiv preprint arXiv:2304.10015.
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. 2022. Progprompt: Generating situated robot task plans using large language models. arXiv preprint S.Ki:2200.11102..
Sanjay Subramanian, Medhini Narasimhan, Kushal Khangaonkar, Kevin Yang. Arsha Nagrani, Cordelia Schmid, Andy Zeng. Trevor Darrell, and Dan Klein, 2023. Modular visual question answering via code generation.
Dídac Surís, Sachit Menon, and Carl Vondrick. 2023. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint ar 2xX1v:2303.08/28
Fei Wang, Kexuan Sun, Muhao Chen, Jay Pujara, and Pedro Szekely. 202 la. Retrieving complex tables with multi-granulle graph representation learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, SIGIR '21, page 1472-1482, New York, NY, USA. Association for Computing Machin- ery.
Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building a semantic parser overnight, In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Juint Conference on Natural Language Processing (Volume I: Long Papers), pages 1332-1342, Beijing, China. Association for Computational Linguistics.
Zhiruo Wang. Haoyu Dong, Ran Jia, Jia Li, Zhiyi Fu, Shi Han, and Dongmei Zhang. 2021b. Tuta: Tree- based transformers for generally structured table pre- training. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1780-1790.
Zhiruo Wang, Zhengbao Jiang, Eric Nyberg, and Gra- ham Neubig. 2022. Table retrieval may not necessi- tate table-specific model design. In Proceedings of the Workshop on Structured and Unstructured Knowl- edge Integration (SUKI), pages 36-46. Seattle, USA. Association for Computational Linguistics.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga. Dongxu Wang, Zifan Li, James Ma, Irene Li. Qingn- ing Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic pars- ing and text-to-SQL task, In Proceedings of the 2018

---

A Few-shot Results
We further conduct experiments on WikiTableQues- tions and HīTab datasets to study the influence of different shots in the prompt. Results are shown in Table 6.
Dataset
5 shots
12 shots
HITAB
70.0 42.4
69.3
WIKITQ
42.8
Table 6: Execution accuracy on HITAB and WIKITQ with different number prompts.
B Prompt Templates
Below we show the templates that we used to prompt code generation models to give answers.
HiTabb and A/T-Q44 Pronpt
Cenerate Pytoon code paen the note and queetion stoun
(Pele SHOTS)
Tall thte Number and perpentage of people Interteannt who howe lllngs.
top, rose (Jitees,, coss, most, [liness cast percertt (Rimme,, cancer, totell
(llrese career, porsenti
left roct- (IAl patiens, Bidens, potients Young))
+pd ramadaate, matra, induonfall, neok, columrontus_rrte rast)
lklyy
Figure prompt example for hierarchical table question answering problems.
C Incorporating SQL into Python Pandas
The prompt template we use for SQL querying in shown in Figure 9a and Figure 9b. The generated SQL query wrapped in pandasqi queries and exccuted using Python SQLite tools. The return value for these queries is pandas data frames, upon which our self-designed APIs will take effect. This way of combining SQL and Python makes it possi- ble to directly upply Python API functions to SQL-returned results.
D Examples for Multi-Index Parsing
See examples of multi-index parsing in both hierar- chical table and flat tables in Figure 10.

---

candoors White
an rppu by ofiecd servant meded soluma
lable
firesded
ensen eemoos comectyy me
Tournne
SQI Pronget
aonerr the onler cornatty. caftult
White
Plocee vide argars bones mogple
redddd
cecrect Topa
CREATE Ta croeue TILES
Posee TCJ hove onth poee bellee
Ariee sco tax))
I. wnat
thoue Aogae?
CRAATE THI STAALE TRUS: IAALL ALSOEVADD
what usa vou of ttl Aaga??
(a) An prompt example for the Spider dataset.
(h) An prompt example for the WikiTQ dataset.
Figure 9
Talk
Noke
DOWT
All MBKe
11201
603
15.8
Peaeree your
Telved
4180
153
147H
19030
436
3219
250
20225
623
3008
Teal
Poced
Deiginal NererckialTele
Rimantied Toe froe BiTak
Multinder Objec
(a) A hierarchical table example adapted from HiTab
Tolble dapuetmant
leddeg
kedge
Real
Tulio undrustor
15 peinery
Agg
Paraed Spider
Mudiciade: Ofjjecs
(b) A multi-tablc example adapted from Spider
Figure 10: Illustration of multi-index parsing with two datasets. We first transform the original table(s) into parsed hierarchical tree(s), similar to HiTab (Cheng et al., 2022). Then we parse the hierarchical tree(s) using depth-first traversal to form the multi-index lists. In Figure 10a, the top root is Source and mechanism, whose children define different kinds of source and mechanism of the participants, while the left root is All full-time, whose children consist of different kinds of full-time jobs. With the final output of the multi-index representation, all cells can be uniquely identified, and thus can reconstruct the table using a simple Pandas code. We also present an example of flat tables in Figure 10b, demonstrating the general applicability of method.
Cold
Cancer
Province
total
percent
total
percent
Canada
10,000
2.5%
200
0.3%
Newfoundland and Labrador
7,400
3.5%
126
0.9%
Prince Edward Island
2,600
1.5%
74
0.2%
Winnipeg
1,600
5.5%
92
0.4%
Rest of Manitoba
2,600
2.5%
14
0.2%
Figure 11: The table is the distribution of new immigrants in different regions/provinees of Canada adapted from HiTab. One question related to this table "how much percentage point has Manitoba rose when it comes to immigrants intended landing destination". However, the information that Winnipeg a city Manitoba is not included in the table itself. In such case, no method we tested could give the correct answer, which should contain a numerator that adds up "Winnipeg" and "Rest of Manitoba". In contrast, by calling QA API (MMnn toba?", left root, 1) Codex, which returns answer yes both "Winnipeg" and "Rest of Manitoba", code models could return the accurate answer to this question.


Reviews:
[{"review_id": "5Brwl2jgay", "reviewer": "Reviewer_rUJ6", "summary": "", "questions": "If a hierarchical table has multiple sections, e.g., rows 1-2 are about gender, rows 3-10 are about different age groups, rows 11-20 are about funding types, how can this table be transformed to multi-index representation? One multi-index representation object is ok or multiple objects are needed?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, and for recognizing the novelty and effectiveness of our work! We are encouraged that you find novelty in our multiindex representation as a unified hierarchical table representation to solve the important representation issue in using LLMs to solve QA tasks, in particular for complicated and diverse tables. \n- **R.1 WikiTQ Improvement**\n\nWhile this improvement still demonstrates the generalizability of our method, e.g., to extend to arbitrary table structures (including relational tables), we believe that one major reason for this small gain is the simplicity of the questions and tables in WikiTQ, where complex operations supported by advanced APIs are not necessary for most examples. As we note in Table 1, WikiTQ only contains single table structures from a single data source, while all other datasets where we demonstrate more significant gains contain questions over either hierarchical or multiple tables. \n\nAdditionally, we would like to note that the focus of Table 1 is to demonstrate the wide applicability of our method across all datasets, with the comparison against the SOTA serving to help inform the reader. We also refrained from over-tuning our method towards specific datasets (e.g., micro-adjusting hyperparameters like number of shots or adding customized APIs, as well as prompt engineering) to evaluate the generalizability of our method. \n\n- **Q.1 Multiindex representation for multi-section tables**\n\nYes, a multi-section hierarchical table can be represented with a single multi-index object. We can simply split the multiple sections and organize them with respective head nodes, serving as children nodes for the left root in one bi-dimensiontal tree.\n\nAs per the example you give, the rows of the original table are (we assume there is only one column “Amount” with two sub columns “total” and percent for simplicity):\n|Category |\n|---|\n|gender  —> 1st section, Gender|\n|g_1 |\n|g_2 | \n|age —> 2nd section, Age | \n|a_1 |\n|a_2 |\n … \n|a_N| \n|funding type —> 3rd section, Funding|  \n|f_1 |\n|f_2 |\n … \n|f_M| \n\nwhere gender, age, and funding type are three child nodes of left_root, Category. So such a table can be transformed into a single multi-index representation using our bi-dimentional tree representation and Algorithm 1 as:\n```\ntop_root = [(Amount, total), (Amount, percent)]\nleft_root =  [(Category, gender, g_1), (Category, gender, g_2), (Category, age, a_1), (Category, age, a_2), …, (Category, age, a_N), (Category, funding type, f_1), (Category, funding type, f_2), …, (Category, funding type, f_M)]\n```\nSubsequently it can be transformed into a Python dataframe with:\n`df = pd.DataFrame(data_matrix, index= top_root, columns= left_root)`\n\nIn addition, we also refer to a similar example and more details description of multi-index transformation process in Figure 10 (a) in Appendix C of our paper (more details about the example table can be found in Figure 1 in the HiTab paper [1]). The two examples show the compatibility of our method with multiple hierarchical sections included within one table. In addition, we note our proposed method is not only compatible with multiple hierarchical sections (as described above), but also multiple levels within each hierarchical section by expanding the depth of tree representation and using Algorithm 1 in our paper.\n\n[1] Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmei Zhang. 2022. HiTab: A hierarchical table dataset for question answering and natural language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1094–1110, Dublin, Ireland. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes a novel way to generate executable programs for varied table structures. For examples, hierarchical tables are commonly-used but largely been neglected by previous research works.\n2. Multi-index representation is a novel way. \n3. Experiment results are significant on several important and related datasets.\n4. Good writing and easy to follow.", "reasons_to_reject": "The experiment results on WikiTQ is not significant. ", "questions_to_authors": "If a hierarchical table has multiple sections, e.g., rows 1-2 are about gender, rows 3-10 are about different age groups, rows 11-20 are about funding types, how can this table be transformed to multi-index representation? One multi-index representation object is ok or multiple objects are needed?", "paper_topic_and_main_contributions": "This paper proposes to generate executable programs for varied table structures, which is interesting. It transforms the tables to multi-index data frames in Panda Python library, then generated python programs are executable on them. It also introduces a set of special API functions to achieve broader coverage of queries.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No missing reference.", "Typos_Grammar_Style_and_Presentation_Improvements": "-"}, "extra_text": {}}, {"review_id": "j3RCOemXvU", "reviewer": "Reviewer_VuHH", "summary": "", "questions": "Now you just provide a small number of API functions, and you use a few-shot to prompt. But if we want to extend this method's capability, it is likely to add more API functions, which need more few-shot examples to prompt.  Have you thought of this question and how do you think to solve this problem?\n\nAlthough your method achieves start-of-the-art performance on HiTAB, Spider, and AIT-QA datasets, its performance on the WikiTQ dataset is not so good and you attribute this to question characteristics of the WikiTQ dataset, which includes many questions that require external knowledge. Have you thought about how to improve its performance, such as using some APIs that can be queried on the Web? And more, have you thought about how to make it answer questions that require specialized domain knowledge?\n\nNow you define some operation APIs and QA APIs in advance. Do you think it is a good idea to train a model which can make necessary APIs automatically because there may be cases that it needs a new API that you haven't defined in advance?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and for your thoughtful feedback! We are glad for your positive comments on the novelty of our approach to addressing general table QA tasks via the multi-index approach, and your comments on the thoroughness of our experiments and ablation studies.\n\n- **R1. Extensive Comparison with Baselines**\n\nWe agree that sufficient baseline comparisons help solidify our findings. We would like to clarify that the baselines listed in Table 2 are not the only baselines we considered, but the *best* baseline methods according to individual datasets, as described in the caption of Table 2. We include them as representatives of all the competitive methods evaluated on each dataset. The fact that our method outperforms these previous SoTA baselines sufficiently shows that our method also outperforms other baseline methods. \n\nAdditionally, the row “Codex” in Table 2 denotes another baseline we built to quantify the performance improvement due to the proposed multi-index structure and API functions. The Codex baseline incorporates few-shot prompting with Python code generation using flattened tables as a vanilla method to handle complex table structures (a process described in 5.2), and is also why it already outperforms some of the state-of-the-art baselines.\n\nIn light of your feedback, we will add more explanations about the baselines in the experiment section and Table 2&3 in the revised paper version. \n\n- **Q1. Adapting to More APIs**\n\nIn addition to extending the prompt to include examples of all newly introduced APIs, one could also add selection/retrieving modules to filter the most relevant ones, and only include an efficient and acceptable number of API examples to our framework. Nonetheless, that would demand designing a whole additional set of approaches which is currently beyond the scope of this work. \n\nOverall, we believe our paper provides a structured framework with unified table representation and API capability, with Operation and QA APIs as two working examples.\n\n- **Q2. Improving WikiTQ Performance**\n\nThanks for pointing out possible ways to improve the performance over WikiTQ. While we refrained from over-tuning our method towards specific datasets (e.g., micro-adjusting hyperparameters like number of shots or adding customized APIs, as well as prompt engineering) to evaluate the generalizability of our method, we think the following methods could be compatible with our framework to improve the performance on certain datasets:\n1. Additional effective APIs to query external knowledge could help. For example, using our framework, one can incorporate APIs designed specifically to search for relative knowledge from web, which can potentially improve the performance. \n2. Add chain-of-thought when utilizing the API to better help the model understand why the API is used under this situation. Large language models can be confused of when and why to use the APIs even they are given with few-shot examples. To help them better understand the usage of the API methods, we can potentially add chain-of-thought explanation to few-shot examples.\n3. More specialized data query APIs for domain-specific knowledge (e.g., internal database) could help. \n\n- **Q3. LLMs Create APIs Automatically**\n\nYes, we have indeed considered and experimented in this direction, in light of other promising findings [1,2]. However, at the time of experimenting, neither GPT3.5 nor Codex could generate viable API functions in our setting, presumably due to limited capability or the broad variety of questions. This is understandable, as generating APIs is much more complicated than using them. We have also considered automatically generating APIs in chain-of-thought style, but found it requires substantial effort and is more suitable for a future research project. \n\n[1] Cai, Tianle, et al. \"Large language models as tool makers.\" arXiv preprint arXiv:2305.17126 (2023).\n\n[2] Qian, Cheng, et al. \"CREATOR: Disentangling Abstract and Concrete Reasonings of Large Language Models through Tool Creation.\" arXiv preprint arXiv:2305.14318 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a novel and innovative approach to address the general table QA tasks. It is quite novel and impressive to convert tables into multi-index representation, while in traditional methods, tables are serialized to text sequences. Moreover, the paper provides a thorough experimental evaluation, doing ablation studies and other comprehensive evaluations to make the conclusion more convincing and reliable. \nAnd the presented approach achieves state-of-the-art performance on benchmark datasets, surpassing existing methods. Furthermore, the paper is well-written and effectively communicates the proposed approach, making it accessible and understandable to the NLP community.\n", "reasons_to_reject": "The authors should include a more extensive comparison with relevant baselines to showcase the superiority of their method convincingly. Now you just display one baseline for each dataset to compare. The paper may benefit from a more comprehensive review of related literature, including recent and relevant works in table QA tasks.\n", "questions_to_authors": "Now you just provide a small number of API functions, and you use a few-shot to prompt. But if we want to extend this method's capability, it is likely to add more API functions, which need more few-shot examples to prompt.  Have you thought of this question and how do you think to solve this problem?\n\nAlthough your method achieves start-of-the-art performance on HiTAB, Spider, and AIT-QA datasets, its performance on the WikiTQ dataset is not so good and you attribute this to question characteristics of the WikiTQ dataset, which includes many questions that require external knowledge. Have you thought about how to improve its performance, such as using some APIs that can be queried on the Web? And more, have you thought about how to make it answer questions that require specialized domain knowledge?\n\nNow you define some operation APIs and QA APIs in advance. Do you think it is a good idea to train a model which can make necessary APIs automatically because there may be cases that it needs a new API that you haven't defined in advance?", "paper_topic_and_main_contributions": "The authors propose a novel approach to solve general table QA tasks, rather than limiting it to some specific table formats. They introduce a framework to solve this problem, which uses a unifying multi-index representation, assistant API functions, and Python program generation by prompting a large pre-trained code model.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No missing references", "Typos_Grammar_Style_and_Presentation_Improvements": "I didn't find any typographical or grammatical errors and I think this paper is well-organized and clearly written.\n"}, "extra_text": {}}, {"review_id": "IaGTd3ayE3", "reviewer": "Reviewer_87ms", "summary": "", "questions": "1. Please elaborate on the considerations of baseline approaches. \n2. Please clarify the gain from multi-index transformation and API. It would be better if we can have the same ablation on Spider, AIT-QA, and WikiTQ (optional).", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer 87ms for thoughtful feedback. Your positive comments on our approach to deal with question answering over hierarchical tables are truly encouraging. For your concerns and questions about our work, please find responses below.\n\nWe would like to emphasize that one contribution of our work is the use of Pandas DataFrames as a unified representation for tables of varied structures, in conjunction with prompting LLMs. We will clarify this framing in the paper, which we believe should help to address two of your points: (1) For your point that SQL is a strong querying language for multi-table datasets – we absolutely agree! Our Pandas representation is also compatible with SQL; as our experiments in Sec 4.5 show – finding improvements of 6.8% with StarCoder and 3.7% with Codex on WikiTQ by using SQL instead of Python as querying language. We will clarify our positioning, and make it clear that we do not claim that Python is the optimal query language, only that we advocate for Pandas as a unified representation. (2) We show that LLMs, when combined with our Pandas representation, achieve strong, often state-of-the-art performance across datasets. While our additional multi-index and API contributions do produce the largest improvements on HiTab, our Pandas+LLM baseline (without multi-index and APIs), referred to as “Codex” in the results tables, is also a contribution of our work. We will make these contributions clearer in the paper. \n\nWe will also refine our statements about arbitrarily-structured tables – while our representation handles both the range of table structures seen in our 4 datasets in the paper, and can also handle other sorts of tables (see our response to rUJ6 about multi-section tables), we agree that “arbitrary” was strong wording, and are willing to change to language like “unified representation for varied table structures”, or other suggestions you might have, in future versions of the paper.\n\n- **Q1 & R1. Considerations of baseline selection**\n\nThanks for pointing this out. As mentioned in the caption of Table 2, for each dataset, we selected baselines based on the state-of-the-art methods from the literature at the time of submission. For the case of DIN-SQL – their paper presents both few-shot performance results, and results with the incorporation of an additional self-correction generation mechanism, which prompts one model several times iteratively to finally complete one generation pass. As our method focuses on table representation and prompt design, for fair comparison we compare to their few-shot performance. Incorporating a self-correction mechanism like DIN-SQL’s on top of our table representation and prompt design strategy is a promising direction for future work.\n\nAdditionally, we present few-shot performance on Codex, especially since datasets such as HiTab and AiT-QA remain relatively underexplored. We recognize that there are other approaches proposed to improve the few-shot performance, however, they are concurrent works during our submission period. \n\n- **R1. few-shot results with different shots not shown**\n\nWe provide further experiment results comparing 5-shot and 12-shot prompt as follows:\n| Dataset | 5 shots | 12 shots |\n|---|---|---|\n| HiTab |69.3 | 70.0 |\n| WikiTQ |42.4 | 42.8 |\n\nFrom the results, we can see that with adding the number of shots in the prompt, the performance didn’t improve a lot. Also, 5-shots is more compatible with open-source code generation models which have a limited context window.\n\n- **R2. Effectiveness of Multi-Indexing (MI)**\n\nFirst, multi-indexing is easier to use than implementing four different data processing and table querying pipelines for different datasets. In addition, MI brings better performance on all types of structures. This improvement appears the most significant on HiTab (which contain hierarchical tables), but doesn't impair its efficacy on any other datasets. Intuitively, compatible data processing and table querying schemes provide the best result, whereas Table 5 suggests that it is best to use MI together with our Python-based QA schema. Conversely, simply unifying data processing with little consideration of the query step (only using MI) may lead to degraded performance.\n\n- **Q2. Gain from MI and API on Spider, AIT-QA and WikiTQ**\n\nFor Spider and WikiTQ (relational) dataset, using MI is basically the same as using the normal representation of a table. Therefore, the performance improvement on Spider and WikiTQ mainly comes from the API. We only performed ablation studies on HiTab since it can properly reveal the benefits of MI and API. We will add ablation results on other datasets in the revised paper version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A simple and generic framework for TableQA. This framework is especially helpful for tables that cannot be queried by SQL and is also good at leveraging the strong Python code generation capability of advanced LLM and code generation models. \n\n2. The paper is well-written and easy to follow.", "reasons_to_reject": "My primary concern about this paper is the experiment part. \n\n1. Baseline approaches selection. This paper compares 4 baseline approaches in the experiment, but it does not explain the reasons for choosing them or the baseline settings. For example, DIN-SQL using Codex achieves far better performance (69.9%) than the one reported in Table 2 (61.2%). Also, the proposed approach can align with BINDER in WikiTQ, i.e., using 12-shots, but this paper chose to use 5-shot and did not show the performance by the number of few-shot examples. The reasons for such settings are essential for readers to understand the performance and superiority of the proposed approach. \n\n2. The performance gain of using multi-index transformation and API is not impressive except for HiTab. Specifically, compared with Codex, w/ MI and API only provides 2.6%, 0.2%, and 0.7% absolute improvement on Spider, AIT-QA, and WikiTQ, respectively. Also, we can observe from Table 5 that only using MI itself even hurts the performance.", "questions_to_authors": "1. Please elaborate on the considerations of baseline approaches. \n2. Please clarify the gain from multi-index transformation and API. It would be better if we can have the same ablation on Spider, AIT-QA, and WikiTQ (optional).", "paper_topic_and_main_contributions": "This paper targets question-answering on structured tables, TableQA for short. Owing to the diverse types of tables, such as relational tables, hierarchical tables, and databases, previous work had proposed various approaches and used different logical forms for a specific kind of table. This paper instead proposes a unified framework for TableQA. This unification occurs in 3 levels. First, various types of tables are represented as a multi-index pandas DataFrame. Second, Python serves as a generic intermediate representation of natural language. Third, powerful code generation models are leveraged to generate Python code in a few-shot manner. In addition, this framework allows customs API functions to extend the Pandas functionality and to incorporate external knowledge. Experimental results on few-shot HiTab, Spider, and WikiTQ demonstrate the performance of the framework to some extent.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The claim of supporting arbitrarily structured tables is not safe in my opinion. For example, the proposed algorithm may not be easy to support nested tables, which are frequently observed in Wiki tables and Excel sheets. I suggest the authors refine the scope of supported tables. \n\n2. The claim of experimental results in the Abstract and Introduction is not safe and somewhat misleading. Various approaches have been proposed to boost the few-shot performance of LLM on TableQA, but most of them are not considered, compared, and discussed in the paper. I suggest the authors make a more rigorous claim on the position and contribution. \n\n3. In general, I suggest the authors should better position the paper. Python is a feasible intermediate representation for multi-table query, but obviously not the best one, because SQL has been optimized for a long time for relational database management systems and LLM has demonstrated awesome SQL generation capability in both zero-shot and few-shot settings. The proposed paper has far more advantages on irregular table structures, like hierarchical tables and nested tables, which to the best of my knowledge misses a generic and well-optimized intermediate representation. In short, a “unified” framework is a good point but does not fully reflect the advantages of the framework."}}]

Author rebuttal:

Meta-review: This paper targets question-answering on structured tables, TableQA for short. Previous work has proposed various approaches and used different logical forms for a specific table due to the diverse types of tables, such as relational tables, hierarchical tables, and databases. This paper instead presents a unified framework for TableQA. This unification occurs in 3 levels:
 1. Various types of tables are represented as a multi-index pandas `DataFrame`.
 2. Python serves as a generic intermediate representation of natural language.
3. Powerful code generation models are leveraged to generate Python code in a few-shot manner.

In addition, this framework allows customs API functions to extend the Pandas functionality and incorporate external knowledge. Experimental results on few-shot HiTab, Spider, and WikiTQ demonstrate the framework's performance to some extent.

**Positive:** The reviewers have liked the simplicity and novelty of the work.

**Negatives:** More extensive comparison with the relevant baselines and elaborating the rationale behind the choice of the baselines (VuHH, 87ms): the authors have agreed. They promised to elaborate on the existing baselines and add more. Similarly, it has raised concerns about the limited choice of baselines, which seem arbitrary. Given the number of questions I see in the reviews, there is a need to improve the writing of the paper. I hope the authors revise their work for future revisions. 


**Given the "soundness" scores, I support accepting this paper. I am also leaning toward suggesting it for the "main track."**
Decision: Accept-Main