Venue: NAACL 2024

# FREB-TQA: A Fine-Grained Robustness Evaluation Benchmark

for Table Question Answering

Wei Zhou\({}^{1,3}\)   Mohsen Mesgar\({}^{1}\)  Heike Adel\({}^{2}\)    Annemarie Friedrich\({}^{3}\)

\({}^{1}\)Bosch Center for Artificial Intelligence, Renningen, Germany

\({}^{2}\)Hochschule der Medien, Stuttgart, Germany  \({}^{3}\)University of Augsburg, Germany

{wei.zhou|mohsen.mesgar}@de.bosch.com annemarie.friedrich@informatik.uni-augsburg.de

###### Abstract

Table Question Answering (TQA) aims at composing an answer to a question based on tabular data. While prior research has shown that TQA models lack robustness, understanding the underlying cause and nature of this issue remains predominantly unclear, posing a significant obstacle to the development of robust TQA systems. In this paper, we formalize three major desiderata for a fine-grained evaluation of robustness of TQA systems. They should (i) answer questions regardless of alterations in table structure, (ii) base their responses on the content of relevant cells rather than on biases, and (iii) demonstrate robust numerical reasoning capabilities. To investigate these aspects, we create and publish a novel TQA evaluation benchmark in English. Our extensive experimental analysis reveals that none of the examined state-of-the-art TQA systems consistently excels in these three aspects. Our benchmark is a crucial instrument for monitoring the behavior of TQA systems and paves the way for the development of robust TQA systems. We release our benchmark publicly.1

Footnote 1: [https://github.com/boschresearch/FREB-TQA](https://github.com/boschresearch/FREB-TQA)

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 2479–2497 June 16-21, 2024 02024 Association for Computational Linguistics

## 1 Introduction

Table Question Answering (TQA) deals with answering natural language questions related to information organized in a table. TQA systems serve as a fundamental component for interacting with relational databases Zhong et al. (2017); Yu et al. (2018) through natural language and for processing information across diverse domains, e.g., science Desai et al. (2021) and finance Zhu et al. (2021).

Processing tabular knowledge poses notable challenges. While tables are structured, there are no standard table layouts for representing a particular type of data. Table cells may contain various data types, including text and numbers. Moreover, tables can have nested structures. Thus, TQA systems should be able to robustly integrate textual commonsense understanding with numerical reasoning applied to structured data.

Recent benchmarks for TBA systems Zhao et al. (2023); Yang et al. (2022) reveal that current TQA systems generate inconsistent responses by performing coarse-grained changes in tables and questions. However, these benchmarks contain a mix of questions requiring different levels of retrieval and aggregation capabilities.

As a result, these benchmarks fail to differentiate between various aspects of robustness, such as errors stemming from incorrect cell retrieval versus those arising from failures in value aggregation steps. Pinpointing the exact failures of TQA systems, however, is a necessary diagnostic step towards enhancing their robustness.

In this paper, we lay the groundwork for a more fine-grained systematic evaluation of TQA systems.

Figure 1: Our benchmark addresses three aspects of robustness shown in the yellow boxes. Answers are bold in tables. Original tables (left) are what exist in a TQA dataset and changed tables (right) show tables after perturbations. We demonstrate three perturbations in this figure (top to bottom): table transposing, removing relevant cells and modifying values to change answers.



To foster the development of robust TQA systems, we address three2 important desiderata, as illustrated in Figure 1. With these aspects, we intend to separate several steps required when answering questions based on tabular data, i.e., identifying the relevant table cells and aggregating over or comparing them. In particular, we propose to evaluate TQA systems according to three aspects: (1) **Retrieval robustness in case of table structure changes**: Systems should retrieve the correct relevant cells regardless of table structure changes; (2) **Attention to relevant cells**: The system should use the relevant cell values from retrieval instead of exploiting shortcuts, such as model-internal knowledge or positional biases, when composing answers; and (3) **Aggregation/comparison robustness in case of value changes**: The system should aggregate the values of relevant cells correctly regardless of cell value changes. Aggregation can involve different types of reasoning. In this study, we focus on numerical operations, such as counting and comparisons, as they are the most common reasoning types in TQA Zhu et al. (2021).

Footnote 2: We acknowledge that there are additional aspects that future work should address.

To evaluate TQA systems with regard to these aspects, we create **FREB-TQA**, a new benchmark based on four well-studied TQA datasets using seven novel automatic perturbations, one perturbation from previous work Zhao et al. (2023), and extensive manual annotations.

In our extensive experiments, we study pipeline and end-to-end state-of-the-art TQA systems Liu et al. (2021); Jiang et al. (2022); Herzig et al. (2020); Cheng et al. (2022), as well as large language models LLMs Youvron et al. (2023) using our benchmark. The fine-grained results uncover shortcomings of these systems that have not been previously explored. For example, we show that model performance diminishes substantially when column order changes, or when cells containing the answer are positioned at the bottom of the table. This finding indicates a strong positional bias in TQA systems, motivating more informative table encodings.

Our main contributions are: **(1)** We propose a novel benchmark of 8,590 selected and partially manually annotated questions and tables, resulting in a total of 75,205 instances using seven perturbation methods; **(2)** using several inter-annotator agreement studies, we demonstrate the solidness of our benchmark; and **(3)** using our benchmark, we experiment extensively with state-of-the-art TQA systems, discovering that for almost all robustness tests, system performances drop, demonstrating the difficulty of our benchmark for TQA systems.

## 2 Related Work

We provide an overview of previous research on TQA systems and their evaluation for robustness.

TQA systems.End-to-end TQA systems Liu et al. (2021); Jiang et al. (2022) compose an answer given a question and a serialized version of the table without intermediate steps or using additional tools. Pipeline systems convert questions into a command language, e.g., SQL Cheng et al. (2022); Ni et al. (2023), filtered tables Glass et al. (2021); Lei et al. (2022); Ye et al. (2023); Herzig et al. (2020), and then generate answers either via executing the commands Ni et al. (2023); Zhang et al. (2023) or via using a trained neural network Lei et al. (2023). Our benchmark reveals the key strengths and weaknesses of these types of systems.

Robustness evaluation.Several studies have shown that recently proposed TQA systems suffer from robustness issues Yang et al. (2022); Zhao et al. (2023); Lin et al. (2023). Zhao et al. (2023) provide a robustness evaluation dataset for TQA, which includes header perturbations, content perturbations, and question perturbations. Their work is closely related to ours. However, they do not further disentangle the various aspects of robustness, thus not providing detailed insights into why systems are not robust. As we show in our experiments, the fine-grained aspects formulated in FREB-TQA contribute to a deeper understanding of TBA systems.

To the best of our knowledge, no previous work on TQA provides analyses of models for our proposed aspects. However, there are several studies looking into those aspects for other tasks. In the context of tabular natural language inference, Gupta et al. (2022a,b) study to what extent models pay attention to relevant cells. The robustness of large language models in case of numerical value changes has primarily been studied in the context of solving math world problems Stolfo et al. (2022) and tabular natural language inference Akhtar et al. (2023) 



## 3 Our Benchmark: FREB-TQA

FREQ-TQA is a Fine-grained Robustness Evaluation Benchmark for Table Question Answering. From four TQA datasets (Table 1), we first classify questions with regard to whether questions merely require cell value retrieval or further reasoning. We then generate perturbations for evaluating each aspect of robustness by making use of seven perturbation methods and collecting human annotations.

### Source Datasets

For building FREB-TQA, we leverage the development sets of four well-studied TBA datasets: WikiTableQuestions (**WTQ**, Pasupat and Liang, 2015), **WikiSQL**Zhong et al. (2017), Sequential Question Answering (**SQA**, Iyyer et al., 2017), and Tabular And Textual dataset for Question Answering (**TAT**, Zhu et al., 2021). The first three datasets feature tables from Wikipedia, TAT addresses the financial domain. See Appendix A.1 for detailed statistics. For all source datasets, we eliminate questions that relate to the table structure, such as "What is the name of the actor in the first row?" To identify such questions, we use a word list provided in Appendix A.3. Around 10% of the questions in WTQ and 3% of TAT questions are filtered out by this criterion.

### Extraction and Reasoning Questions

To decouple the robustness aspects in TQA, we group questions from the source datasets into extraction questions (EQs) and reasoning questions (RQs) by applying heuristics and classification models. The answer to an EQ can be retrieved from a single cell of the table. The answer to an RQ additionally requires aggregating over several cell values. In Figure 1, the question "How many floors does the Fosh Tower have?" is an EQ since it only requires retrieving a cell value. The question "Who received the least amount of votes?" is an RQ as it requires comparing values of several cells.

Question type classification for WTQ.WTQ provides a diverse set of questions. We group them into EQs and RQs using two different methods. Each method is tuned to identify either EQs or RQs with high precision, as our final benchmark will only contain the set of questions identified as EQ or RQ by the respective method.3

Footnote 3: Around 20% of questions are selected by neither of the models and thus eliminated from our benchmark. We also found that the final set of EQ and RQ cases do not overlap.

Given a table, a question, and an answer, the **lexical rule-based** method conducts a string match between the answer and the table's cell values. If there is no match, the corresponding question is labeled as RQ. Otherwise, we detect if the question contains any comparative or superlative words using POS tags.4 Since these words are signals for aggregation over table cells, we mark the question as RQ. Otherwise, the question's type is set to EQ. We also prompt **LLaMA2**-13b (Touvron et al., 2023) to label the question type (The prompt is provided in Appendix A.4). Finally, we combine the lexical rule-based and LLaMA2 models. If the lexical rule-based model labels a question as EQ, we obtain the LLaMA's prediction in addition. If they agree, the question type is set to EQ. Otherwise, it is set to RQ. To estimate the quality of these methods, we manually annotate questions sampled randomly from WTQ (100 EQs and 100 RQs). Details are given in Appendix A.2.

Footnote 4: [https://www.nltk.org](https://www.nltk.org)

Table 2 shows the results achieved by these models. As we aim for a high-precision question-type classification, we apply the combined model for identifying EQs and the lexical rule-based method for identifying RQs on all questions in WTQ.

**Question type classification for WikiSQL.** Questions in WikiSQL do not require complex reasoning Zhao et al. (2023); Lin et al. (2023). In WikiSQL, questions are labeled with regard to the operations required to answer them. We select the questions that can be answered without performing

\begin{table}
\begin{tabular}{l r r r} \hline \hline
**Source Dataset** & **\# EQs** & **\# RQs** & **\# ORI** \\ \hline WTQ Pasupat and Liang (2015) & 205 & 1562 & 2831 \\ WikiSQL Zhong et al. (2017) & 6013 & 0 & 8418 \\ SQA Iyyer et al. (2017) & 157 & 0 & 2265 \\ TAT Zhu et al. (2021) & 114 & 539 & 1668 \\ \hline \hline \end{tabular}
\end{table}
Table 1: The first two columns show the total number of questions selected from the dev part of each source dataset for extraction questions (EQs) and reasoning questions (RQs). The last column shows the original number of questions in each source dev set.

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Model** & **EQ** & **RQ** \\ \hline LLaMA2 & 74.27 & 57.02 \\ Rule-based & 75.23 & **83.42** \\ Combined & **93.81** & 60.17 \\ \hline \hline \end{tabular}
\end{table}
Table 2: The precision of examined models for question type classification on 200 questions from WTQ.



aggregation operations and label them as EQs.

**Question type classification for SQA.** The SQA dataset consists of dialogues in the form of questions and answers related to the information in a table. The answers to all questions except for the first question in a dialog rely on the dialog history. Thus, we select only the first question from each sequence, which usually asks for retrieving information from the table (Iyyer et al., 2017). Hence, we mark them as EQs.

**Question type classification for TAT.** TAT consists of questions about tables and text snippets and requires numerical reasoning. The dataset also features annotations about how an answer is derived, whether the answer is based on the table or the text, and which operations are required to answer a question. We leverage these annotations to identify EQs and RQs. More specifically, we first select questions that can be answered only based on tables. Next, if a question needs derivation or comparison, we classify them as RQ. Otherwise, we classify them as EQ.

### Perturbations for Testing Retrieval Robustness against Table Structure Changes

A robust TQA system should answer an EQ by retrieving the answer from a table, regardless of table structure changes. We perturb the structure of tables associated with extraction questions. We replicate one perturbation type from previous work and introduce two new perturbation types for measuring this robustness aspect.

Shuffle all rows (columns)Following Zhao et al. (2023), we randomly shuffle all rows (columns) in a table. This perturbation allows us to study if TQA systems are robust against changes in re-arranging all rows (columns). However, it does not reveal which biases may impact the robustness of TQA systems. Thus, we introduce the following two new perturbation methods for this aspect.

Shift target rows (columns).For each EQ, the _target_ row (column) in a table contains the cell that corresponds to the answer. This perturbation type shifts target rows (columns) either to the top, to the middle, or to the bottom part of a table. We identify the target cell by applying exact match between the answer and table cell value. For shifting target rows, we partition a table into three equal-length parts, referred to as top, middle, and bottom. For shifting target columns, we partition a table into two equal-length parts: front and back. We use more partitions for rows because on average tables include more rows than columns (Table 3). For our benchmark, we remove the target rows (columns) from the table and re-insert them at a random position in each partition. This perturbation method allows us to study whether TBA systems exhibit any positional biases.

Transpose.This perturbation type transposes the table, i.e., it rotates the table by 90 degrees and turns rows into columns and columns into rows.

This perturbation allows us to study if TQA systems have a bias towards particular table layouts.

### Perturbations for Testing Attention to Relevant Cells

TQA systems may answer a question by exploiting shortcuts (model-internal knowledge or positional biases) without paying attention to relevant cells (cells that are important to compose an answer). We propose three new perturbation methods for reasoning questions (RQs) to study this aspect of TQA robustness.

Remove relevant cells.We perturb a table by removing relevant cells from tables. The relevant cell annotations associated with RQs in our benchmark have been created by Zhu et al. (2021) and Ye et al. (2023) for TAT and WTQ, respectively. We test the validity of the latter since the annotations are gathered from LLMs (see Appendix A.2). This perturbation lets us investigate to what extent TQA systems bypass relevant cells to derive their answers. We observe that 70% of relevant cells contain non-numerical values for WTQ. For TAT, all relevant cells contain numerical values.

Remove table.TQA systems may bypass the whole table and use their internal knowledge to answer a question. To test for this behavior, for any RQs, we replace the table with a dummy table, consisting of one cell with a "None" value.

Shift relevant rows.This perturbation evaluates to what extent TQA systems bypass table cell values and rely on the position of relevant cells. For instance, to answer the question "Who received the least amount of votes?" in Figure 1, a TQA system may exploit a shortcut between the last row and the question since in most cases, rows in tables from TQA datasets are sorted. Because of the correlation between cell values and positions, this type of 

shortcut mostly occurs for RQs that require comparing cell values. In this perturbation type, for each RQ, we remove relevant rows and re-insert them at a random position of the table.

### Perturbations for Testing Robustness when Aggregating/Comparing Values

TQA systems should compose a correct answer to an RQ by aggregating over or comparing values in the provided table independent of value changes. To evaluate this capability, for RQs, we manually modify cell values that should be aggregated to answer questions. We ask three human annotators to first identify the table cells that numerical aggregations or comparisons operate on (step C in Figure 2). We refer to the part of the table relevant to answering the question as the _shortened table_. We use the shortened table versions in our experiments to minimize the effect of table length in this test. Then, annotators are asked to change one or two cell values per question, once resulting in an answer change and once not resulting in an answer change (step E in Figure 2). We define two types of perturbations.

**Modify values to change answers.** This perturbation allows us to study to what extent TQA systems perform correct aggregations and adapt answers to the value changes accordingly. An example of numeric value changes is shown in Figure 1. The votes for _Olsson_ is changed from _4 to 361_, resulting in a change of the answer (from _Olsson_ to _Leslie_). Figure 2 shows an example of string value changes. The first entry _Go West Young Man_ is removed, changing the answer from _2 to 1_.

**Modify values without changing answers.** This perturbation aims to study if systems articulate correct answers because of their biases to certain values. For instance, in Figure 1, systems might be capable of comparing _15_ and \(4\). However, if we change the votes for _Leslie_ from _15_ to _1500_, without changing the answer, systems might face difficulties as _1500_ might not fall into the cell values distribution during pre-training. Figure 2 shows an example of string value changes: _West_ is changed to _East_ without changing the answer (2).

We filter out instances that annotators find unanswerable. In the case of TAT, all modified values are numeric. In the case of WTQ, 50% of the modified cell values are numeric, the others are string-based.

To assess the quality of the output of these perturbations, for each perturbation type, we randomly sample 50 instances created using this type from each dataset, resulting in 200 instances in total. Then, we ask two annotators who were not involved in the perturbation creation to provide the answers given the changed tables and questions. We compute the exact match accuracy of answers provided by the two annotators, which amount to 92.5% and 93.5%. We find that most wrong cases are related to questions asking for percentage changes: here, annotators sometimes neglected to add the minus symbol to negative percentage changes.

## 4 Experimental Settings

We use our benchmark to evaluate the robustness of state-of-the-art TQA systems with regard to our proposed aspects. Table 3 shows the main statistics of our benchmark grouped by the robustness aspects for which TQA systems are evaluated.

Figure 2: An example of aggregation/comparison robustness in case of string change. C illustrates shortening an original table to table cells on which numerical aggregations or comparisons operate. E illustrates modifications on a shortened table, leading either to a change in answer or not. The orange parts mark changed strings.

 



### Examined TQA Systems

We analyze the robustness of three types of systems: end-to-end systems that are fine-tuned for the TQA task; pipeline systems that generate relevant cells or SQL queries which are then executed on a table, and off-the-shelf LLMs. In particular, we compare the following TQA systems. **TAPEX**Liu et al. (2021) is an end-to-end TQA system based on BART Lewis et al. (2019). **OmniTab**Jiang et al. (2022b) further fine-tunes TAPEX on both more natural and synthetic data. **TaPas**Herzig et al. (2020) first predicts relevant cells and an aggregation function, backboned by BERT Devlin et al. (2019). Then, it articulates answers based on outputs from the previous step with a numeric tool. We categorize it as a pipeline model as answers are not directly generated. **Binder**Cheng et al. (2022) is a pipeline model consisting of a parsing and an executing step. First, intermediate representations (e.g., SQL queries) are generated by GPT-3.5, and then the queries are executed by a program interpreter. **GPT-3.5** is an LLM and the backbone of various TQA models Ye et al. (2023); Zhang et al. (2023). In the prompt to GPT-3.5, we use a three-shot demonstration to obtain answers (see Appendix A.7). LLaMA Touvron et al. (2023) is an open-source LLM. We fine-tune its 7b chat version with LoRA Hu et al. (2021). We describe details of fine-tuning in Appendix A.5.

### Evaluation Metrics

To compare results on our benchmark, we use the following metrics.

**Exact match accuracy (Em)** checks if the predicted answers and the ground truth are the same. It is a widely used metric for evaluating TQA systems Pasupat and Liang (2015); Yang et al. (2022); Jiang et al. (2022b).

**Exact match difference (Emd, Zhao et al., 2023) measures system performance change before and after perturbations (negative values indicate performances drop). Both Em and Emd focus on overall system performance. Variation percentage (VP, Yang et al., 2022) measures to what extent predictions change before and after performing perturbations from an instance-level perspective. It is defined as follows:

\[\text{VP}=\frac{\text{C2W}+\text{W2C}}{\text{N}} \tag{1}\]

where C2W counts the number of instances whose predictions change from correct to wrong and W2C is the number of instances whose predictions change from wrong to correct. N is the total number of instances. For perturbations involving randomness, we report the mean and standard deviation of scores over five runs with different random seeds.

## 5 Experimental Results

In this section, we discuss the performance of TQA systems on our benchmark, and provide a detailed analysis of these models for each robustness aspect.

### Retrieval robustness in case of table structure changes

To study this aspect of robustness, we use the first part of our benchmark (see Section 3.3) which consists of shuffling rows (columns), shifting target rows (columns), and transposing tables for EQs. To rule out the effect of different maximum input lengths of the models in this test, we use only instances that are within the maximum input length (512) of TaPas, the model accepting the smallest input. This means we make use of 91% of the EQs.

Figure 3 and Figure 4 show Emd and VP scores for row and column shuffling, respectively, averaged across all source datasets. We report the detailed results for each dataset in Appendix A.6. In addition, the first column of Table 4 provides the model performance in terms of Em on the original data (without perturbations).

Figure 3 shows that the performance of almost all systems drops when evaluating them on row (column) shuffling, which is consistent with prior results Zhao et al. (2023). However, our fine-grained benchmark enables us to draw more conclusions beyond this general observation. First, systems are more vulnerable to column shuffling than to row shuffling. This is apparent for LLMs (GPT-3.5 and LLaMA2). When comparing model types, LLMs are most affected by row and column shuffling, followed by pipeline systems. The end-to-end TQA systems (TAPEX and OmniTab) are more robust in this regard. Second, regarding row perturbations, systems are highly impacted by the position of the target row. The more it is moved down the table (TM and TB), the more the performance drops. Notably, moving the target row to the top (TT) even leads to performance improvements for some of the systems, confirming that the systems have encoded some positional biases. It further reveals that systems are likely to fail on tables with many rows, where the target row can be

located anywhere. Similar observations are found by Lin et al. (2023). However, we show that even within the maximum input window of a model, the positional bias exists during cell retrieval.

Third, Table 4 shows that systems fail on transposed tables as well (see columns Emd and VP). Only GPT-3.5 suffers less. We assume that this model has seen a larger variety of tables during its pre-training. Since our analysis of table transposing is new and especially the fine-tuned end-to-end TQA systems perform considerably worse on that perturbed data, we look into this aspect in more detail and investigate their ability to adapt to new table structures (here: transposed tables). For this, we further fine-tune them on transposed table data. The last column of Table 4 shows the results. By fine-tuning on transposed data, the performance of these models improves. However, the gap to GPT-3.5 is still large, showing TQA systems might need architectural changes or pre-training datasets that feature more diverse table structures.

### Attention to relevant cells

To investigate if systems use relevant cell values to compose answers or rather use shortcuts, such as implicit knowledge or biases, we use the second part of our benchmark (see Section 3.4). To the best of our knowledge, this aspect has never been tested in prior work on TBA robustness. We split our analysis into two parts. First, we investigate system behavior when removing relevant information (relevant cells or the whole table). Second, we examine to what extent systems exploit linkages between questions and relevant cell positions rather than paying attention to cell content.

Table 5 shows the Em results for the first part, analysing the behavior of systems when removing relevant information. The column ORI shows the original performance of systems, the column RRel shows the performance when removing relevant cells and the column RT shows results when removing the whole table and replacing it with a

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
* \\ GPT-3.5 & 71.77 & **-17.32** & **26.15** &
- \\ LLaMA2 & 61.10 & -30.58 & 38.42 &
- \\ \hline \hline \end{tabular}
\end{table}
Table 4: Exact match difference (Emd) and variation percentage (VP) for **table transpose**. Emd-Ft stands for Emd after fine-tuned with transposed tables. Bold values suggest best performances.

Figure 4: Variation percentage (VP) on retrieval robustness against table structure changes perturbations **for extraction questions**, averaged across four datasets and seeds. R, C and stand for row and column. SA, TT,TM, TB and TF stand for shuffle all, target top, target middle, target bottom/back, target front, respectively.

Figure 3: Exact match difference (Emd) on retrieval robustness against table structure changes perturbations **for extraction questions**, averaged across four datasets and seeds. R, C and stand for row and column. SA, TT,TM, TB and TF stand for shuffle all, target top, target middle, target bottom/back, target front, respectively.

 

dummy table. The results indicate that TaPas pays most attention to the cell values, i.e., its Em score is the lowest among the systems when removing the relevant information, which is the desired behavior.

In general, the pipeline models are more robust against changes in the position of the relevant cells than end-to-end models. This intuitively makes sense as they consist of an executing component that performs a function or query directly on the structured table. For a deeper investigation, we analyze instances which the end-to-end systems TAPEX and OmniTab still predict correctly after removing relevant information in tables for TAT. All of them feature the same answer "2019", indicating that answer values are not well distributed in the TQA benchmarks and systems learn this bias during fine-tuning.

For investigating the second question, i.e., to what extent systems exploit positional biases when answering questions, we analyze the variation percentage (VP) with and without shuffling relevant rows or columns. To account for the effect that shuffling data leads to challenges on its own (the first robustness aspect we analyzed before), we compare VP for comparison questions (involving cues, such as "least") with VP for non-comparison questions. Table 6 shows the results. For all systems, the gaps are larger than zero, indicating that they all use question-related shortcuts. Among them, TaPas features the largest gap, suggesting that among the tested systems, it exploits the most question-related shortcuts.

### Aggregation/comparison robustness in case of value changes

We investigate the numerical reasoning abilities of TBA systems, i.e., their robustness against value changes. As many tables contain numeric data and questions might require the aggregation of several numeric values, this aspect is of utmost importance for real-world applications. While numerical reasoning abilities of models have been analyzed for other domains or tasks Stolfo et al. (2022); Akhtar et al. (2023), a targeted analysis for TQA is missing in prior work. We analyze this aspect using the third part of our benchmark (Section 3.5). Table 7 shows the Em results for tables with original values (ORI) and shortened tables (ST) that contain only the part of the cell necessary for composing the answer. Additionally, the VP between the results before modifying the cell values and those after changing values is provided, once for the case where answers change (AC) and once where they do not (NC). For shortened tables, performance increases compared to original tables for all systems, again indicating that systems might perform a decent job on tables with a few rows but strongly fail on tables with many rows. When changing values, the pipeline systems TaPas and Binder are among the most robust for both AC and NC settings. We account this to the fact that they execute a predicted function or query on the table and, thus, directly involve the cell values when deriving the final answer. End-to-end systems (TAPEX and OmniTab) show small VP on TAT. However, they do not outperform LLMs, e.g., GPT-3.5.

## 6 Discussion

In our experiments, we use our new FREB-TGA benchmark to analyze end-to-end, pipeline, and LLM-based TQA systems. Our experimental re

\begin{table}
\begin{tabular}{l c r r} \hline \hline
**Model** & **Compare** & **Non_compare** & **Gap** \\ \hline TAPEX & 10.24 & 9.70 & 0.54 \\ OmniTab & 7.37 & 6.02 & 1.35 \\ TaPas & 10.98 & **4.04** & 6.94 \\ Binder & **9.47** & 9.07 & 0.40 \\ GPT-3.5 & 16.62 & 16.40 & **0.22** \\ LLaMA2 & 17.11 & 12.89 & 4.22 \\ \hline \hline \end{tabular}
\end{table}
Table 6: VP of instances (not) requiring value comparisons on WTQ. **Compare** refers to instances requiring comparison of cells to be solved (e.g., questions contain “highest”, “lowest”) and **Non_compare** refers to instances do not require comparisons. **Gap** refers to the gap between the two settings.

\begin{table}
\begin{tabular}{l l l l l l l} \hline \hline  & \multicolumn{3}{c}{**WTQ**} & \multicolumn{3}{c}{**TAT**} \\ \cline{2-7} Model & **ORI** & **RT** & **RRel** & **ORI** & **RT** & **RRel** \\ \hline TAPEX* & 52.26 & 8.84 & 9.57 & 9.65 & 3.53 & 5.38 \\ OmniTab* & **58.12** & 7.76 & 12.55 & 10.02 & 3.90 & 6.12 \\ TaPass** & 47.02 & **2.80** & **8.21** & - & - & - \\ Binder** & 45.67 & 6.86 & 11.62 & **37.65** & **0** & **1.01** \\ GPT-3.5 & 42.06 & 6.95 & 16.52 & 36.17 & **0** & 1.73 \\ LLAMA2 & 35.92 & 7.44 & 11.91 & - & - & - \\ \hline \hline \end{tabular}
\end{table}
Table 5: The exact match (Em) of examined TBA systems regarding attention to relevant cells. ORI shows results on original tables. RT and RRel show the results when the removing table perturbation and removing relevant cells are used, respectively. We do not report results for TaPas and LLaMA2 on perturbations for the TAT subset as their ORI accuracy scores are too low (<4%) to draw conclusions from them.

 

sults show that all examined systems suffer from substantial issues when it comes to robustness. However, different system types show different patterns. **End-to-end TQA systems, for instance, seem to be more robust against changes in the row/column arrangement. However, they are more likely to fail on numerical reasoning questions.** This might be because the datasets these models are pre-trained on do not feature complex questions requiring numerical reasoning.

**LLMs are more affected by row or column perturbations but much more robust against transposing tables.** Their performance in performing aggregations or comparisons is highly dependent on the length of the serialized tables, i.e., they perform much better on tables that can be serialized to fewer tokens.

**Finally, the pipeline models in our study are more robust against changes in relevant cells, including value changes.** This is likely due to their symbolic execution component, which executes a predicted function or query on the given structured table. However, the prediction of the function or query itself still suffers from various perturbations of table data, including changes in table structure. Yet, another benefit of pipeline TBA systems is that the intermediate representation, i.e., the predicted function or query, makes the model explainable to a certain extent. We hence argue that **more research in the pipeline-based paradigm is a promising step towards more robust TQA.**

## 7 Conclusion and Outlook

In this paper, we have proposed to evaluate three aspects of robustness of TQA systems: retrieval robustness in case of table structure changes, attention to relevant cells, and aggregation/comparison robustness in case of value changes. We have presented a novel benchmark for a fine-grained analysis of those aspects. The main building blocks of our benchmark are targeted table perturbation methods and high-quality human annotations. Finally, we have evaluated a range of architecturally varied state-of-the-art TQA systems, as well as off-the-shelf LLMs. Our study has shown that while none of the systems was consistently robust, their weaknesses and strengths differ from each other. Systems trained in an end-to-end fashion are able to deal with changes in row/column arrangement but LLMs perform better when it comes to numeric operations. The answers of pipeline-based models, which use symbolic methods for part of their computations, are more faithful towards the table they are supposed to use when composing their output.

Our new benchmark constitutes an important first step for evaluating TQA systems in a fine-grained way, thereby directing research efforts towards building more robust TQA systems. As pipeline systems offer at least some explainability, we argue that research should concentrate on these types of systems. In particular, future work could explore the possibility of utilizing LLMs in a pipeline manner to build more robust TQA systems. Further, aggregating or extracting information from long tables is an important next step given the shortcomings of current systems in this regard which our analysis revealed.

We thank the anonymous reviewers for their helpful feedback. We also thank Tianxing Liu, Xinyue Shi and Yidan Chen for their annotations.

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{4}{c}{**WTQ**} & \multicolumn{3}{c}{**TAT**} \\ \cline{2-9}  & **ORI** & **ST** & **AC-VP** & **NC-VP** & **ORI** & **ST** & **AC-VP** & **NC-VP** \\ \hline TAPEX & 47.55 & 48.03 & 30.00 & 5.65 & 9.68 & 10.17 & 8.38 & 2.23 \\ OmniTab & **52.79** & **54.69** & 26.46 & 4.76 & 10.06 & 12.96 & **6.33** & **2.05** \\ TaPas & 41.09 & 48.57 & 16.39 & **3.81** & - & - & - & - \\ Binder & 42.99 & 50.88 & **14.56** & 7.96 & **37.78** & **41.19** & 9.50 & 8.19 \\ GPT-3.5 & 38.91 & 51.70 & 23.20 & 8.78 & 36.30 & 46.37 & 26.82 & 14.71 \\ LLAMA2 & 32.59 & 33.88 & 32.45 & 6.31 & - & - & - & - \\ \hline \hline \end{tabular}
\end{table}
Table 7: System evaluation on aggregation/comparison robustness in case of value changes. ORI stands for original performance without perturbations. ST stands for passing short tables on which numerical aggregations operate. We report Em for the ORI and ST settings. AC-VP and NC-VP stand for variation percentage for the answers change and not change settings. We do not report results for TaPas and LLAMA2 on the TAT subset, as the performances are too low (<4%) to derive reasonable analysis from. Bold values suggest best performances.

 



### Limitations

This work focuses on building a benchmark for analyzing the robustness of TQA systems in three fine-grained aspects. In terms of the reasoning type, we mainly discuss numerical reasoning. However, other types of reasoning, e.g., commonsense reasoning or temporal reasoning can also occur during the aggregation phase of TQA. Future studies can explore these aspects and extend our benchmark. We build our benchmark with English TBA datasets, this could also be extended by incorporating TQA datasets in other languages. Additionally, though we report statistics about percentages of numeric and non-numeric values when removing relevant cells, our benchmark does not distinguish these two value types explicitly. Future work could extend our benchmark on this aspect to explore how non-numeric value changes affect model performance.

## Ethical Considerations

The development sets of the source datasets we use: WTQ (Pasupat and Liang, 2015), WikiSQL (Zhong et al., 2017), SQA (Iyyer et al., 2017) and TAT (Zhu et al., 2021) are publicly available under the licenses of CC-BY-SA-4.05, BSD-3 Clause6, MIT7 and MIT, respectively. These licenses all permit us to compose, modify, publish, and distribute additional annotations upon the original dataset. Experiments in this paper are run on a single NVIDIA Tesla V100-32G GPU. Benchmark and code will be released along with the paper. For annotation tasks where humans are involved, we recruit 5 undergraduate students (3 females and 2 males) studying Linguistics in China. All 5 annotators voluntarily participate in the annotation tasks. Three out of four of our annotation tasks requires less than 2 hours to finish. The other one took two weeks and we suggest annotators to spend less than three hours per day to ensure enough rest.

Footnote 5: [https://creativecommons.org/licenses/by-sa/4.0/](https://creativecommons.org/licenses/by-sa/4.0/)

Footnote 6: [https://opensource.org/licenses/bsd-3clause/](https://opensource.org/licenses/bsd-3clause/)

Footnote 7: [https://opensource.org/licenses/mit/](https://opensource.org/licenses/mit/)

## References

* M. Akhtar, A. Shankarampeta, V. Gupta, A. Patil, O. Cocarascu, and E. Simperl (2023)Exploring the numerical reasoning capabilities of language models: a comprehensive analysis on tabular data. ArXivabs/2311.02216. Cited by: SS1.
* Z. Cheng, T. Xie, P. Shi, C. Li, R. K. Nadkarni, Y. Hu, C. Xiong, D. R. Radev, M. Ostendorf, L. Zettlemoyer, N. A. Smith, and T. Yu (2022)Binding language models in symbolic languages. ArXivabs/2210.02875. External Links: Link Cited by: SS1.
* H. Desai, P. Kayal, and M. K. Singh (2021)TableX: a benchmark dataset for structure and content information extraction from scientific tables. In IEEE International Conference on Document Analysis and Recognition, Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)Bert: pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics, Cited by: SS1.
* J. L. Fleiss (1971)Measuring nominal scale agreement among many raters.. Psychological Bulletin76, pp. 378-382. Cited by: SS1.
* M. R. Glass, M. Canim, A. Gliozzo, S. A. Chemmengath, R. Chakravarti, A. Sil, F. Pan, S. Bharadwaj, and N. Rodolfo Fauceglia (2021)Capturing row and column semantics in transformer based question answering over tables. In North American Chapter of the Association for Computational Linguistics, Cited by: SS1.
* V. Gupta, R. A. Bhat, A. Ghosal, M. Shrivastava, M. Singh, and V. Srikumar (2022a)Is my model using the right evidence? systematic probes for examining evidence-based tabular reasoning. Transactions of the Association for Computational Linguistics10, pp. 659-679. Cited by: SS1.
* V. Gupta, S. Zhang, A. Vempala, Y. He, T. Choji, and V. Srikumar (2022b)Right for the right reason: evidence extraction for trustworthy tabular reasoning. In Annual Meeting of the Association for Computational Linguistics, Cited by: SS1.
* J. Herzig, P. Krzysztof Nowak, T. Muller, F. Piccinno, and J. Martin Eisenschlos (2020)Tapas: weakly supervised table parsing via pre-training. In Annual Meeting of the Association for Computational Linguistics, Cited by: SS1.
* E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2021)Lora: low-rank adaptation of large language models. External Links: Link Cited by: SS1.
* M. Iyyer, W. Liu, and M. Chang (2017)Search-based neural structured learning for sequential question answering. In Annual Meeting of the Association for Computational Linguistics, Cited by: SS1.
* Z. Jiang, Y. Mao, P. He, G. Neubig, and W. Chen (2022a)OminiTab: pretraining with natural and synthetic data for few-shot table-based question answering. In Proceedings of the 2022 Conference of the North American Chapter of

the Association for Computational Linguistics: Human Language Technologies_, pages 932-942, Seattle, United States. Association for Computational Linguistics.
* Jiang et al. (2022b) Zhengbao Jiang, Yi Mao, Pengcheng He, Graham Neubig, and Weizhu Chen. 2022b. Omnitab: Pretraining with natural and synthetic data for few-shot table-based question answering. In _North American Chapter of the Association for Computational Linguistics_.
* Landis and Koch (1977) J Richard Landis and Gary G Koch. 1977. An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. _Biometrics_, pages 363-374.
* Lei et al. (2023) Fangyu Lei, Xiang Lorraine Li, Yifan Wei, Shizhu He, Yiming Huang, Jun Zhao, and Kang Liu. 2023. S3hqa: A three-stage approach for multi-hop textable hybrid question answering. In _Annual Meeting of the Association for Computational Linguistics_.
* Lei et al. (2022) Weixian Lei, Difei Gao, Yuxuan Wang, Dongxing Mao, Zihan Liang, Lingmin Ran, and Mike Zheng Shou. 2022. AssistSR: Task-oriented video segment retrieval for personal AI assistant. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 319-338, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdel rahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In _Annual Meeting of the Association for Computational Linguistics_.
* Lin et al. (2023) Weizhe Lin, Rexhina Blloshmi, Bill Byrne, Adria de Gispert, and Gonzalo Iglesias. 2023. An inner table retriever for robust table question answering. In _Annual Meeting of the Association for Computational Linguistics_.
* Liu et al. (2021) Qian Liu, Bei Chen, Jiaqi Guo, Zeqi Lin, and Jian-Guang Lou. 2021. Tapex: Table pre-training via learning a neural sql executor. _ArXiv_, abs/2107.07653.
* Ni et al. (2023) Ansong Ni, Srini Iyer, Dragomir R. Radev, Ves Stoyanov, Wen tau Yih, Sida I. Wang, and Xi Victoria Lin. 2023. Lever: Learning to verify language-to-code generation with execution. _ArXiv_, abs/2302.08468.
* Pasupat and Liang (2015) Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In _Annual Meeting of the Association for Computational Linguistics_.
* Stolfo et al. (2022) Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Scholkopf, and Mrinmaya Sachan. 2022. A causal framework to quantify the robustness of mathematical reasoning with language models. In _Annual Meeting of the Association for Computational Linguistics_.
* Touvron et al. (2022) Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasming Babaei, Nikolay Bashkykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Bleicher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyn Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Innan, Marcin Kardas, Viktor Kerkaez, Madin Khabsa, Isabel M. Kolumann, A. V. Korenaev, Punit Singh Kouva, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihalov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Pavin Xu, Zhengxu Yan, Diyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. _ArXiv_, abs/2307.09288.
* Yang et al. (2022) Jingfeng Yang, Aditya Gupta, Shyam Upadhyay, Luheng He, Rahul Goel, and Shachi Paul. 2022. Tableformer: Robust transformer modeling for tablet-text encoding. In _Annual Meeting of the Association for Computational Linguistics_.
* Ye et al. (2023) Yunhu Ye, Bingyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. 2023. Large language models are versatile decomposers: Decomposing evidence and questions for table-based reasoning. _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_.
* Yu et al. (2018) Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 3911-3921, Brussels, Belgium. Association for Computational Linguistics.
* Zhang et al. (2023) Yunjia Zhang, Jordan Henkel, Avrilia Floratou, Joyce Cahoon, Shaleen Deep, and Jurgensh M. Patel. 2023. Reactable: Enhancing react for table question answering. _ArXiv_, abs/2310.00815.
* Zhao et al. (2023) Yilun Zhao, Chen Zhao, Linyoung Nan, Zhenting Qi, Wenhlin Zhang, Xiangru Tang, Boyu Mi, and Dragomir R. Radev. 2023. Robut: A systematic study of table qa robustness against human-annotated adversarial perturbations. In _Annual Meeting of the Association for Computational Linguistics_.
* Zhong et al. (2017) Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. _ArXiv_, abs/1709.00103 

Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat seng Chua. 2021. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. _ArXiv_, abs/2105.07624.

 



## Appendix A Appendix

### Source Dataset Overview

#### To distinguish between extraction and reasoning questions.

They are told to classify whether a question is of type extraction or reasoning after giving the definitions and examples of each question type. The instructions are listed as follow:

* Thank you for participating in this annotation task! In this task, you will be given a pair of question, answer and table and to determine if a question is extraction-based [1] or reasoning-based [0], or neither of them [-1]. We define extraction-based as: answers of the questions can be retrieved from table without numerical reasoning abilities. For instance, for the question: what is the sale number of 2018, given the rows showing the year and the column showing the sale number, the answer can be obtained by looking at certain cells without needing operations on numerical levels. We define reasoning-based as: answers of the questions should be firstly retrieved from table and then numerical reasoning abilities are needed to obtain the answer. For instance, for the question: how many countries have received 4 goals. Entries of countries receiving 4 goals should be retrieved, then, counting is needed to obtain the answer. Common numerical operations are: counting, comparing (date or number), summing, averaging, subtracting, etc. If you find the question unanswerable given the table information, please put a [-1] in the annotation field.

Agreement on the binary question type classification of the 200 questions amounts to a \(\kappa\)-score [7] of 0.85, which shows good agreement among the annotators [6].

To test the validity of relevant cells annotations.We randomly sample 100 annotations for WTQ and ask the same 3 annotators to decide if removing the relevant cells prevents one from answering the questions or not. The instruction is listed as follows:

* Thank you to participate in this annotation task! In this task, you will be given a pair of question, answer, table and relevant cells from the table to determine if removing the relevant cells prevent one from answering the question (and deriving the same correct answers) or not. If you find removing the relevant cells prevent one from answering the question, please put one from answering the question, please put [1] in the annotation field. Otherwise, please put [0]. If you find a question unanswerable or strange, please put [-1] in the annotation field.

The Fleiss' \(\kappa\)-score is 0.53, which shows moderate agreement.

### Eliminating Questions Related to Table Structures

We collect common positional prepositions (suggesting positions) and ordinal numbers (suggesting order) in English. The list we use is as follows: ['first', 'second','third', 'last', 'top', 'bottom', 'before', 'previous', 'latter', 'after', 'next', 'below', 'above']. We eliminate questions containing words in the list.

### LLMs prompt

Figure 5: LLaMA2 Prompt for classifying extraction and reasoning questions.





### Training/Prompting Details

Similar as in Zhao et al. (2023), we fine-tuned the Large version of pre-trained end-to-end TQA systems for 20 epochs and use the random-split-1-train if multiple train-dev splits are provided. For TAT, the training epochs is increased to 50 epochs. We then split the training set further into training and development sets with the ratio of 80:20. The batch size we use are 32 and gradient accumulation is 4. The best model is selected based on validation loss. As for fine-tuning LLaMA2, we use the 7B chat-hf version. We use the code provided in LLaMA-Factory8 and keeps the default settings of all parameters in the code. In terms of GPT-3.5, we use three-shots demonstrations. The demonstrations can be found in Appendix A.7. As for Binder, since the systems used in the original paper (Codex) is no longer provided by OpenAI, we use GPT-3.5 as the backbone. The performance might degrade due to the switch, as what is observed in Zhang et al. (2023).

Footnote 8: [https://github.com/hiyouga/LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory)

### Results for Robustness Against Table Structural Change for Different Datasources

The following four tables show the exact match difference and variation percentage for each subset on retrieval robustness against table structure changes perturbations.

### LLM prompts for GPT3.5

Figure 6, Figure 7, Figure 8 and Figure 9 shows the prompts we used for prompting GPT-3.5 for WTQ, WikiSQL, SQA and TAT, respectively.





\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c c} \hline \hline
**Pv** & \multicolumn{8}{c}{Exact match difference} & \multicolumn{9}{c}{Variation percentage} \\ \cline{2-13}  & TAPEX & OmatiTho & TaPlus & GPf7.35 & LLAMA2 & Binder & TAPEX & OmariTho & TaPlus & GPf7.35 & LLAMA2 & Binder \\ \hline NP & 67.39 & 70.29 & 68.44 & 57.26 & 63.89 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ \hline R.S.A. & \(-12.81\pm 1.06\) & \(-1.45\pm 0.7\) & \(-2.90\pm 1.18\) & \(-1.45\pm 0.78\) & \(-2.17\pm 0.98\) & \(-1.39\pm 0.99\) & \(5.56\pm 1.18\) & \(4.38\pm 1.37\) & \(2.75\pm 1.69\) & \(5.78\pm 1.40\) & \(5.78\pm 1.40\) & \(5.84\pm 1.14\) \\ R.T.T. & \(+0.5\,1.12\pm 0.90\) & \(-1.42\pm 0.90\) & \(-1.45\pm 0.93\) & \(-0.72\pm 0.97\) & \(-1.71\pm 1.24\) & \(-1.43\pm 0.57\) & \(3.86\pm 1.31\) & \(3.86\pm 0.83\) & \(3.38\pm 1.03\) & \(3.38\pm 1.03\) & \(4.37\pm 1.24\) & \(1.94\pm 1.24\) \\ R.T.M. & \(-3.48\pm 0.9\) & \(-0.82\pm 0.94\) & \(-5.07\pm 0.95\) & \(-1.52\pm 0.88\) & \(-2.71\pm 1.53\) & \(-4.84\pm 1.01\) & \(3.86\pm 0.82\) & \(3.56\pm 0.84\) & \(3.56\pm 0.34\) & \(3.97\pm 1.22\) & \(10.22\pm 0.93\) & \(9.46\pm 1.27\) \\ R.T.R. & \(-16.6\pm 0.73\) & \(-1.45\pm 0.93\) & \(-4.83\pm 0.34\) & \(-5.80\pm 0.43\) & \(-4.90\pm 1.57\) & \(-6.12\pm 0.67\) & \(3.42\pm 1.20\) & \(4.34\pm 0.97\) & \(6.02\pm 0.35\) & \(11.59\pm 1.66\) & \(13.68\pm 2.54\) & \(10.87\pm 1.21\) \\ \hline C.SA & \(-31.41\pm 1.02\) & \(-1.09\pm 1.07\) & \(-2.14\pm 0.86\) & \(-2.51\pm 1.34\) & \(-2.51\pm 1.47\) & \(-1.76\pm 1.79\) & \(-3.79\pm 1.03\) & \(1.22\pm 1.71\) & \(-2.17\pm 1.76\) & \(11.42\pm 1.37\) & \(16.21\pm 1.37\) & \(16.21\pm 1.37\) & \(16.21\pm 1.37\) \\ C.TP & \(-21.71\pm 1.02\) & \(-21.71\pm 0.96\) & \(-20.21\pm 0.31\) & \(-6.14\pm 1.54\) & \(-7.57\pm 1.19\) & \(-3.52\pm 1.42\) & \(-1.71\pm 1.41\) & \(7.27\pm 0.51\) & \(5.11\pm 1.74\) & \(1.49\pm 1.02\) & \(11.41\pm 1.29\) & \(12.14\pm 1.36\) & \(2.69\pm 1.51\) \\ C.TB & \(-29.1\pm 1.57\) & \(-31.44\pm 0.68\) & \(-12.12\pm 0.98\) & \(-5.81\pm 1.17\) & \(-4.38\pm 1.13\) & \(-35.8\pm 1.11\) & \(10.14\pm 1.76\) & \(7.49\pm 0.08\) & \(3.69\pm 0.08\) & \(11.61\pm 2.78\) & \(12.89\pm 2.25\) & \(5.57\pm 1.12\) \\ \hline Tr & -51.45 & -58.70 & 6.04 & 24.00 & -34.78 & -55.68 & 57.25 & 60.14 & 64.49 & 52.56 & 39.13 & 68.24 \\ \hline \end{tabular}
\end{table}
Table 11: SWIQ: system performance on extraction data set. Ps stands for perturbations. NP stands for no perturbation. R, C and Tr stand for row, column and transpose perturbations. SA, TT, TF, TM and TB stand for shuffle all, target top, target front, target middle, target bottom/back respectively. We shade the best performances (minimal absolute values) with blue.

\begin{table}
\begin{tabular}{l l c c c c c c c c c c c c c c} \hline \hline
**Pv** & \multicolumn{8}{c}{Exact match difference} & \multicolumn{9}{c}{Variations percentage} \\ \cline{2-13}  & TAPEX & OmatiTho & TaPlus & GPf7.35 & LLAMA2 & Binder & TAPEX & OmariTho & TaPlus & GPf7.35 & LLAMA2 & Binder \\ \hline NP & 89.27 & 91.33 & 92.99 & 63.06 & 56.45 & 66.70 & 0 & 0 & 0 & 0 & 0 & 0 \\ \hline R.S.A. & \(-0.08\pm 0.4\) & \(0.84\pm 0.06\) & \(0.88\pm 0.04\) & \(-0.27\pm 0.13\) & \(-1.82\pm 0.22\) & \(-1.14\pm 0.11\) & \(1.08\pm 0.06\) & \(0.62\pm 0.02\) & \(0.63\pm 0.08\) & \(10.37\pm 1.11\) & \(11.25\pm 0.88\) & \(3.21\pm 0.63\) \\ R.T.T. & \(-0.08\pm 0.01\) & \(0.40\pm 0.02\) & \(-0.03\pm 0.03\) & \(-0.21\pm 0.07\) & \(-1.01\pm 0.24\) & \(-0.03\pm 0.03\) & \(-0.07\pm 0.04\) & \(-0.02\pm 0.02\) & \(0.64\pm 0.02\) & \(11.05\pm 0.56\) & \(12.71\pm 0.94\) \\ R.T.M. & \(-0.08\pm 0.01\) & \(0.01\pm 0.01\) & \(-0.001\pm 0.02\) & \(-1.01\pm 0.11\) & \(-1.24\pm 0.14\) & \(-1.25\pm 0.65\) & \(0.57\pm 0.36\) & \(0.38\pm 0.04\) & \(0.41\pm 0.02\) & \(11.56\pm 0.11\) & \(11.41\pm 0.24\) & \(12.84\pm 0.47\) \\ R.T.B. & \(-0.12\pm 0.02\) & \(-0.08\pm 0.02\) & \(-0.03\pm 0.03\) & \(-0.22\pm 0.13\) & \(-1.88\pm 0.09\) & \(-1.21\pm 0.04\) & \(-1.05\pm 0.05\) & \(-0.03\pm 0.02\) & \(0.61\pm 0.02\) & \(11.01\pm 0.56\) & \(12.71\pm 0.94\) \\ R.T.B. & \(-0.12\pm 0.02\) & \(-0.08\pm 0.02\) & \(-0.03\pm 0.03\) & \(-0.02\pm 0.13\) & \(-1.01\pm 0.15\) & \(-1.24\pm 0.04\) & \(-1.25\pm 0.05\) & \(0.57\pm 0.36\) & \(0.38\pm 0.04\) & \(0.41\pm 0.02\) & \(11.01\pm 0.56\) & \(11.71\pm 0.94\) & \(12.60\pm 0.05\) \\ C.SA & \(

You are given a question and a table. Please answer the question with regards to the table and return the answer in the list format. Please return ONLY the answer as output in a list. In the table, \(|\) sperates columns and n seperates rows. Below are the three examples:

Question: in how many games did the winning team score more than 4 points? Table:

Home Team \(|\) Score \(|\) Away Team \(|\) Date \(|\) Agg Aberdeen \(|\)7-1 \(|\) Hamilton Academical \(|\) 11-10-1978 \(|\) 8-1 Airdriconians \(|\) 1-2 \(|\) Arbroath \(|\) 10-10-1978 \(|\) 2-3 Ayr United \(|\) 1-1 \(|\) Falkirk \(|\) 11-10-1978 \(|\) 3-1 Clydebank \(|\) 1-1 \(|\) Hibernian \(|\) 11-10-1978 \(|\) 1-2 Morton \(|\) 5-2 \(|\) Klimmrock \(|\) 11-10-1978 \(|\) 5-4 Montrose \(|\) 5-1 \(|\) Raith Rovers \(|\) 11-10-1978 \(|\) 5-4 Motherwell \(|\) 1-4 \(|\) Celtic \(|\) 11-10-1978 \(|\) 2-4 St. Mirren \(|\) 0-0 \(|\) Rangers \(|\) 11-10-1978 \(|\) 2-3 Answer: 3 Question: at the women's 200 meter individual medley sm10 event at the 2012 summer paralympics, how long did it take aurelie rivalor to finish? Table:

Rank \(|\) Lane \(|\) Name \(|\) Nationality \(|\) Time \(|\) Notes \(|\) 4 \(|\) Sophie Pascoe \(|\) New Zealand \(|\) 2:25.65 \(|\) WR \(|\) 5 \(|\) Summer Ashley Mortimer \(|\) Canada \(|\) 2:32.08 \(|\) 3 \(|\) Zhang Meng \(|\) China \(|\) 2:33.95 \(|\) AS \(|\) 4 \(|\) 6 \(|\) Katherine Downie \(|\) Australia \(|\) 2:34.64 \(|\) 5 \(|\) 2 \(|\) Nina Ryabova \(|\) Russia \(|\) 2:35.65 \(|\) 6 \(|\) 8 \(|\) Aurelie Rivard \(|\) Canada \(|\) 2:37.70 \(|\) 7 \(|\) Harriet Lee \(|\) Great Britain \(|\) 2:39.42 \(|\) 8 \(|\) 1 \(|\) Gemma Almond \(|\) Great Britain \(|\) 2:42.16 \(|\) Answer: 2:37.70

Question: what is the difference in attendance in tie no 1 and 4? Table:

Tie no \(|\) Home team \(|\) Score \(|\) Away team \(|\) Attendance \(|\) IFK Vasteras (D2) \(|\) 3-2 \(|\) IK Slebier (D2) \(|\) 875 \(|\) Kamfors IF (N) \(|\) 2-4 (aecu) \(|\) BK Kernty (D3) \(|\) 2,808 \(|\) Reynersholds IK (D2) \(|\) 5-2 \(|\) Akridbadges FF (D2) \(|\) 1,504 \(|\) Wifsta/Ostrands IF (N) \(|\) 4-3 \(|\) Ludvka FI (D2) \(|\) 974 \(|\) RAi IF (D3) \(|\) 3-2 \(|\) Sandvikens IF (D3) \(|\) 2,116 \(|\) Sandvikens AIIK (D2) \(|\) 2-3 (aecu) \(|\) Tidaholms GIF (D2) \(|\) 822 \(|\) Karlatads BIK (D2) \(|\) 2-5 \(|\) IF Friska Viljor (N) \(|\) 1,550 \(|\) 8 \(|\) Sandviks IK (N) \(|\) 2-3 \(|\) Suralhammars IF (D2) \(|\) Answer: 99 Now please answer this question: Question: {question} Table:

[table]Answer:

Figure 6: Prompt for GPT-3.5 for WTQ.

 

You are given a question and a table. Please answer the question with regards to the table and return the answer in the list format. Please return ONLY the answer as output in a list. In the table, \(|\) operates columns and 'in seperates rows. Below are the three examples:

Question: How many number does Fordham school have? Table: Player \(|\) No. \(|\) Nationality \(|\) Position \(|\) Years in Toronto \(|\) School/Club Team Patrick O'Bryant \(|\) 13 \(|\) United States \(|\) Center \(|\) 2009-10 \(|\) Bradley Jermaine O'Neal \(|\) 6 \(|\) United States \(|\) Forward-Center \(|\) 2008-09 \(|\) Eau Claire High School Dan O'Sullivan \(|\) 45 \(|\) United States \(|\) Center \(|\) 1995-96 \(|\) Fordham Charles Oakley \(|\) 34 \(|\) United States \(|\) Forward \(|\) 1998-2001 \(|\) Virginia Union Hakeem Olajuwon \(|\) 34 \(|\) Nigeria / United States \(|\) Center \(|\) 2001-02 \(|\) Houston Answer: 45 Question: How many schools are in Bloomington, IN? Table: School \(|\) Location \(|\) Founded \(|\) Affiliation \(|\) Enrollment \(|\) Team Nickname \(|\) Primary conference Indiana University \(|\) Bloomington, IN \(|\) 1820 \(|\) Public \(|\) 40354 \(|\) Hoosiers \(|\) Big Ten Conference ( D-I ) Iowa State University \(|\) Ames, IA \(|\) 1858 \(|\) Public \(|\) 27945 \(|\) Cyclones \(|\) Big 12 Conference ( D-I ) Lindenwood University \(|\) St. Charles, MO \(|\) 1827 \(|\) Private/Presbyterian \(|\) 11421 \(|\) Lions \(|\) MIAA ( D-II ) Ohio University \(|\) Athens, OH \(|\) 1804 \(|\) Public \(|\) 20437 \(|\) Bobcats \(|\) Mid-American ( D-I ) Robert Morris University \(|\) Chicago, IL \(|\) 1913 \(|\) Private/Non-Sectarian \(|\) 7277 \(|\) Eagles \(|\) Chicagoland ( NAIA ) Answer: 1 Question: How many votes did Devil in a Hood receive in total? Table: Song \(|\) Mobiles \(|\) Northern Ireland \(|\) Northern England \(|\) Scotland \(|\) Southern England \(|\) Wales \(|\) Total "Groovy Chick" \(|\) 10 \(|\) 3 \(|\) 2 \(|\) 3 \(|\) 2 \(|\) 3 \(|\) 23 \(|\) Clear the Air" \(|\) 5 \(|\) 5 \(|\) 10 \(|\) 8 \(|\) 3 \(|\) 4 \(|\) 35 "Devil in a Hood" \(|\) 4 \(|\) 1 \(|\) 3 \(|\) 4 \(|\) 1 \(|\) 17 "In My Life" \(|\) 2 \(|\) 6 \(|\) 8 \(|\) 5 \(|\) 5 \(|\) 10 \(|\) 36 "How Does It Feel" \(|\) 8 \(|\) 8 \(|\) 4 \(|\) 10 \(|\) 8 \(|\) 5 \(|\) 43 "The Girl" \(|\) 1 \(|\) 2 \(|\) 1 \(|\) 1 \(|\) 6 \(|\) 2 13 "About You" \(|\) 3 \(|\) 4 \(|\) 6 \(|\) 6 \(|\) 1 \(|\) 6 \(|\) 26 "Answer: 17 Now please answer this question: Question:{question} Table: {table} Answer:

Figure 7: Prompt for GPT-3.5 for WikiSQL.

 

You are given a question and a table. Please answer the question with regards to the table and return the answer in the list format. Please return ONLY the answer as output in a list. In the table, \(|\) generates columns and \(`un\) seperates rows. Below are the three examples:

Question: how many total deputies does Pottosi have? Table:

Department \(|\)Total Deputies \(|\) Uninonimal Deputies \(|\) Plurinominal Deputies \(|\) Special Indigenous'nor Campesino Deputies \(|\) Senators La Paz \(29\)\(|\)\(14\)\(|\)\(1\)\(|\) 4 Santa Cruz \(28\)\(|\)\(14\)\(|\) 3 \(|\) 1 \(|\) 4 Cochabamba \(|\)\(19\)\(|\)\(9\)\(|\) 1 \(|\) 4 Potosi \(|\)\(13\)\(|\) 7 \(|\) 6 \(|\) 0 \(|\) 4 Chuquisacan \(|\)\(10\)\(|\) 5 \(|\) 5 \(|\) 4 Oruro \(|\)\(9\)\(|\) 4 \(|\) 1 \(|\) 4 Tarija \(|\)\(9\)\(|\) 4 \(|\) 1 \(|\) 4 Beni \(|\)\(8\)\(|\) 4 \(|\) 3 \(|\) 1 \(|\) 4 Pando \(|\)\(5\)\(|\) 2 \(|\) 2 \(|\) 1 \(|\) 4 Total \(|\) 130 \(|\) 63 \(|\) 60 \(|\) 7 \(|\) 36 Answer: 13 Question: how many passengers arrived in 2011? Table:

Year \(|\) Domestic passengers \(|\) International passengers \(|\) Total passengers \(|\) Change

2006 \(|\) 764,831 \(|\) 83,115 \(|\) 847,946 \(+\)4.6%

2007 \(|\) 764,674 \(|\) 75,276 \(|\) 839,950 \(-\)0.9%

2008 \(|\) 709,792 \(|\) 2176 \(|\) 801,955 \(-\)4.5%

2009 \(|\) 655,347 \(|\) 82,424 \(|\) 687,958 \(-\)14.3%

2010 \(|\) 595,457 \(|\) 105,119 \(|\) 700,576 \(+\)1.7%

2011 \(|\) 850,305 \(|\) 123,607 \(|\) 973,912 \(+\)39.1%

2012 \(|\) 899 854 \(|\) 178,679 \(|\) 1,078,533 \(|\) +10.7%

2013 \(|\) 745,178 \(|\) 131,902 \(|\) 877,080 \(|\) -18.7%

Answer: 973,912

Question: how many silver medals did karine ruby win? Table:

Athlete \(|\) Nation \(|\) Olympics \(|\) Gold \(|\) Silver \(|\) Bronze \(|\) Total Philipp Schoch \(|\) Switzerland (SUI) \(|\) 2002-2006 \(|\) 2 \(|\) 0 \(|\) 0 \(|\) 2 Shaun White \(|\) United States (USA) \(|\) 2006-2014 \(|\) 2 \(|\) 0 \(|\) 0 \(|\) 2 Seth Wesccott \(|\) United States (USA) \(|\) 2006-2010 \(|\) 2 \(|\) 0 \(|\) 0 \(|\) 2 Karine Ruby \(|\) France (FRA) \(|\) 1998-2002 \(|\) 1 \(|\) 1 \(|\) 0 \(|\) 2 Hannah Teter \(|\) United States (USA) \(|\) 2006-2014 \(|\) 1 \(|\) 1 \(|\) 0 \(|\) 2 Ross Powers \(|\) United States (USA) \(|\) 1998-2002 \(|\) 1 \(|\) 0 \(|\) 1 \(|\) 2 Kelly Clark \(|\) United States (USA) \(|\) 2002-2014 \(|\) 1 \(|\) 0 \(|\) 2 \(|\) 3 Danny Kass \(|\) United States (USA) \(|\) 2002-2006 \(|\) 0 \(|\) 2 \(|\) 0 \(|\) 2 Answer: 1 Now please answer this question:

Question:{question} Table:

\(\{\)table\(\}\)Answer:

Figure 8: Prompt for GPT-3.5 for SQA.

 

You are given a question and a table. Please answer the question with regards to the table and return the answer in the list format. Please return ONLY the answer as output in a list. In the table, \(|\) operates columns and 'in seperates rows. Below are the three examples:

Question: What is the percentage change in net sales from Frozen Kefir between 2018 and 2019? Table:

\(|~{}\) 2019 \(|~{}\) 2018

In thousands \(|~{}\$~{}|\%~{}|\) \(|~{}\$~{}|\) \(|\%\)

Drinkable Kefir other than ProBugs \(|~{}\$~{}71,822 \(|~{}\)77% \(|~{}\$~{}\) 78,523 \(|~{}\) 76%

Cheese \(|~{}\) 11,459 \(|~{}\) 12% \(|~{}\) 11,486 \(|~{}\) 11%

Cream and other \(|~{}\) 4,228 \(|~{}\) 4% \(|~{}\) 5,276 \(|~{}\) 5%

ProBugs Kefir \(|~{}\) 2,780 \(|~{}\) 3% \(|~{}\) 2,795 \(|~{}\) 3%

Other dairy \(|~{}\) 1,756 \(|~{}\) 2% \(|~{}\) 3,836 \(|~{}\) 4%

Frozen Kefir (a) \(|~{}\) 1,617 \(|~{}\) 2% \(|~{}\) 1,434 \(|~{}\) 1%

Net Sales \(|~{}\$~{}93,662 \(|~{}\) 100% \(|~{}\$~{}\) 103,350 \(|~{}\) 100%

Answer: 12.76 Question: How much is the 2019 rate of inflation? Table :

\(|~{}\) 2019 \(\%\) \(|~{}\) 2018 \(\%\) \(|~{}\) 2017 \(\%\)

Weighted average actuarial assumptions used at 31 March! \(|~{}\mid~{}\mid\) Rate of inflation2 \(|~{}\) 2.9 \(|~{}\) 3.0

Rate of increase in salaries \(|~{}\) 2.7 \(|~{}\) 2.7 \(|~{}\) 2.6

Discount rate \(|~{}\) 2.3 \(|~{}\) 2.5 \(|~{}\) 2.6 Answer: 2.9 Question: What was the change in raw materials between 2018 and 2019? Table :

\(|~{}\)March 31, \(|~{}\)

\(|~{}\) 2019 \(|~{}\) 2018

Raw materials \(|~{}\$74,5|~{}\$26.0\)

Work in process \(|~{}\) 413.0 \(|~{}\) 311.8

Finished goods \(|~{}\) 224.2 \(|~{}\) 138.4 Total inventories \(|~{}\$711.7~{}\$476.2\) Answer: 48.5 Now please answer this question:

Question: {question} Table:

\(\{\)table\(\}\) Answer:

Figure 9: Prompt for GPT-3.5 for TAT.