Venue: NAACL 2024

# Rethinking Tabular Data Understanding with Large Language Models

 Tianyang Liu

UC San Diego

ti104@ucsd.edu &Fei Wang

USC

fwang598@usc.edu &Muhao Chen

UC Davis

muhchen@ucdavis.edu

###### Abstract

Large Language Models (LLMs) have shown to be capable of various tasks, yet their capability in interpreting and reasoning over tabular data remains an underexplored area. In this context, this study investigates from three core perspectives: the robustness of LLMs to structural perturbations in tables, the comparative analysis of textual and symbolic reasoning on tables, and the potential of boosting model performance through the aggregation of multiple reasoning pathways. We discover that structural variance of tables presenting the same content reveals a notable performance decline, particularly in symbolic reasoning tasks. This prompts the proposal of a method for table structure normalization. Moreover, textual reasoning slightly edges out symbolic reasoning, and a detailed error analysis reveals that each exhibits different strengths depending on the specific tasks. Notably, the aggregation of textual and symbolic reasoning pathways, bolstered by a mix self-consistency mechanism, resulted in achieving SOTA performance, with an accuracy of 73.6% on WikitTableQuestions, representing a substantial advancement over previous existing table processing paradigms of LLMs.1

Footnote 1: Our code is available at [https://github.com/Leolty/tablellm](https://github.com/Leolty/tablellm).

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 450–482


presents unique challenges to these models. Inherently designed to parse and process vast expanses of unstructured textual content, LLMs confront a paradigm shift when facing tabular data. Linearizing tables to suit the LLM paradigm can obscure the inherent structural and relational information, making tasks such as precise localization and complex statistical analyses. Additionally, the design variations in tables, whether 'column tables' with headers in the first row or 'row tables' with headers in the first column, further complicate the interpretation process. Beyond structural concerns, numerical reasoning and aggregation over tabular data present another layer of complexity. While LLMs excel at textual understanding, they occasionally stumble when confronted with tasks necessitating precise numerical computation within tables. Moreover, tables often present a dense amalgamation of textual or numerical data. The sheer volume and intricacy of this information can risk overshadowing crucial details, potentially impeding the LLM's decision-making abilities Shi et al. (2023).

With the emergence of instruction fine-tuning techniques Wei et al. (2022); Chung et al. (2022) and the application of Reinforcement Learning from Human Feedback (RLHF) Stiennon et al. (2022); Gao et al. (2022); Christiano et al. (2017), LLMs have witnessed significant enhancements in their alignment capabilities, paving the way for transitioning from few-shot to zero-shot learning settings Kojima et al. (2023). In light of these advancements, this paper delves deep into the the challenges and intricacies of tabular understanding and reasoning by LLMs, exemplified in Fig. 2. We organize our exploration around three **pivotal research questions**: (1) How well do LLMs perceive table structures and how can we ensure robustness against structural variations? (2) Comparing textual and symbolic reasoning for table data in LLMs, which prevails in effectiveness, and what advantages and challenges manifest in each strategy? (3) Will the aggregation of multiple reasoning pathways enhance the accuracy and reliability of tabular data interpretation by LLMs?

In pursuit of answering the aforementioned research questions, we conduct experiments on SOTA LLMs such as GPT-3.5 OpenAI (2023a). Our findings in SS4 underscore that while LLMs are adept at semantically interpreting tables, their capability to resist structural variance (SS4.1) and understandable structures (SS4.2) is suboptimal. Motivated by these findings, we propose a table structure normalization method to enhance LLMs' resilience against structural table variations in SS4.3. Intriguingly, SS5.1 reveals that textual reasoning surpassses symbolic reasoning in contexts with limited table content, defying conventional conceptions of symbolic reasoning's dominance in other domains Mi

Figure 2: Illustrative examples sampled from the WikiTableQuestions dataset, comparing textual reasoning (via direct prompting) and symbolic reasoning (via python shell interactions). _Top_: The table and its title. _Bottom Left_: In the first question example, textual reasoning yields an incorrect interpretation due to limitations in precision localization, while symbolic reasoning accurately locates the answer using Python code. _Bottom Right_: In the second question example, textual reasoning successfully identifies the answer, but symbolic reasoning incorrectly treats the special total row as the final answer.

 

alon et al., 2023). Both textual and symbolic reasoning strategies encompass different advantages and challenges, which is detailed in SS5.2. To harness the unique strengths of each, we implement mix self-consistency mechanism (SS6) that remarkably attains SOTA performance on Table QA, exemplifying the synergistic potential when both reasoning strategies are aggregated.

## 2 Related Work

**PLMs for Tabular Data Processing.** Tabular reasoning presents unique challenges due to the fusion of free-form natural language questions with structured or semi-structured tabular data, for which PLMs jointly trained on tables and text are developed in the past few years, including TaBERT Yin et al. (2020), Tapas Herzig et al. (2020), TAPEX Liu et al. (2022), ReasTAP Zhao et al. (2022), and PASTA Gu et al. (2022). The recent development of TableLlama Zhang et al. (2023), an open-source model excelling in a variety of table-based tasks, adds a new dimension to the field. Despite these advancements, recent studies have identified generalization issues under table perturbations Zhao et al. (2023); Chang et al. (2023), raising concerns regarding the robustness of PLMs. Specific efforts like LETA Zhao et al. (2023) and LATTICE Wang et al. (2022) have investigated and mitigated the vulnerabilities related to structural perturbations of tabular data, like row/column shuffling and table transpose, through various techniques, including data augmentation and order-invariant graph attention. However, these approaches require whitebox access to the models, limiting their applicability to SOTA LLMs with only blackbox accessibility, a limitation directly addressed in this work.

**Tabular Data Processing with LLMs.** Recent advancements in LLMs, notably within few-shot learning, have demonstrated their potential for tabular reasoning. Chen (2023) leveraged the Chain-of-Thought (CoT) technique Wei et al. (2023) to illustrate LLMs' effectiveness in this domain. Building upon CoT, Cheng et al. (2023) and Ye et al. (2023) introduced frameworks that incorporate symbolic reasoning for improved comprehension, with Ye et al. emphasizing their ability to adeptly decompose both evidence and questions. The advent of aligned models, such as ChatGPT, has enabled zero-shot table reasoning. However, these models often lack sensitivity to table structures, struggling with structural perturbations. StructGPT Jiang et al. (2023), while introducing a promising framework for LLMs to efficiently engage with structured data, has its effectiveness limited by not integrating symbolic reasoning, a critical aspect for enhancing the full capabilities of LLMs in tabular reasoning, which is the focal point of this study. Furthermore, while programming-based approaches can mitigate some challenges, they are limited in addressing free-form queries, creating a gap in the landscape. Innovations like AutoGPT Significant Gravitas (2023) have sought to address this, spawning the development of tabular agents like LangChain Chase (2022), SheetCopilot Li et al. (2023), and Datas Copilot Zhang et al. (2023). These agents offer solutions unattainable through conventional programming but still require rigorous evaluation in various scenarios. In our study, we delve into addressing these challenges for enhancing LLMs' reasoning capabilities within structural perturbations, hence providing insights that facilitate improved accuracy in the current context.

## 3 Preliminaries

This section succinctly introduces the foundational aspects of our study over structurally perturbed tabular data. SS3.1 formally defines the problem, delineating the critical notations and conceptual frameworks, and SS3.2 explicates our experimental setup details, elucidating dataset choice, model utilization, and evaluation strategy.

### Problem Definition

Question answering (QA) over tabular data, commonly known as the TableQA task, is an important challenge in NLP. In this study, we targets TableQA to explore and enhance the proficiency of LLMs, in reasoning over tabular data. Additionally, we probe the robustness and adaptability of these models by introducing structural perturbations to tables.

Let \({\cal T}\) represent a table consisting of \({\cal R}\) rows and \({\cal C}\) columns, and \(\tau\) represent its title/caption. Each cell in \({\cal T}\) is denoted by \({\cal T}_{i,j}\), where \(i\in[0,{\cal R}-1]\) and \(j\in[0,{\cal C}-1]\). \({\cal T}_{0,j}\) are headers. Given a question \({\cal Q}\) pertaining to the table, our task is to identify an answer \({\cal A}\). This answer is generally a collection of values, denoted as \(\{a_{1},a_{2},\ldots,a_{k}\}\), where \(k\in\mathbb{N^{+}}\).

Furthermore, to delve deeper into the structural comprehension of LLMs, we introduce structural 

perturbations, which include:2 Footnote 1: Columns shuffling was not employed as the typical number of columns is limited and this shuffling had minimal impact on accuracy Zhao et al. (2023).

Footnote 2: Column shuffling was not employed as the typical number of columns is limited and this shuffling had minimal impact on accuracy Zhao et al. (2023).

1. **Transposed Table (\({\mathcal{T}^{\top}}\)):** A table obtained by converting rows to columns and vice-versa, maintaining the row and column order: \[{\mathcal{T}_{i,j}^{\top}={\mathcal{T}_{j,i}}\quad\forall i\in[0,{\mathcal{R} }-1],j\in[0,{\mathcal{C}}-1].}\]
2. **Row Shuffled Table (\({\mathcal{T}_{\Pi}}\)):** A table obtained by randomly shuffling the rows (excluding the headers) with a random permutation function \(\pi\), while keeping the order of columns unchanged: \[{\mathcal{T}_{\Pi_{i,j}}={\mathcal{T}_{\pi(i),j}}\quad\forall i\in[1,{ \mathcal{R}}-1],j\in[0,{\mathcal{C}}-1]}\]
3. **Row Shuffled and Transposed Table (\({\mathcal{T}_{\Pi}^{\top}}\)):** A table obtained by first randomly shuffling the rows (excluding headers) and then applying transposition: \[{\mathcal{T}_{\Pi_{i,j}}^{\top}={\mathcal{T}_{j,\pi(i)}}\quad\forall i\in[1,{ \mathcal{R}}-1],j\in[0,{\mathcal{C}}-1]}\]

Defining our research problem more formally: our primary objective is to investigate the function, \(f\), that can appropriately answer the posed question using the provided table. Specifically, this function will take three arguments: the table variant \({\mathcal{T}^{\prime}\in\{\mathcal{T},{\mathcal{T}^{\top},{\mathcal{T}_{\Pi} },{\mathcal{T}_{\Pi}^{\top}}\}\}}\), its title \(\tau\), and the question \({\mathcal{Q}}\). It will output an answer \({\mathcal{A}}\). The entire problem can be formally framed as:

\[f({\mathcal{T}^{\prime},\tau,{\mathcal{Q}}})\to{\mathcal{A}},\quad\forall{ \mathcal{T}^{\prime}\in\{{\mathcal{T},{\mathcal{T}^{\top},{\mathcal{T}_{\Pi} },{\mathcal{T}_{\Pi}^{\top}}\}}\}}\]

### Experimental Setup

This section details the experimental setup adopted in our study, including the datasets employed, model selection, evaluation metrics, reasoning methods, and other details.

DatasetWe used the WikiTableQuestions WTQ; Pasupat and Liang (2015) dataset for our experiments. The test set comprises 421 tables. Each table provides up to two question-answer pairs; if a table has fewer than two, only one was chosen, totaling 837 unique data points. With our four table configurations (original and three perturbations), the overall evaluation data points amount to \(837\times 4=3,348\).

ModelsWe employ the GPT-3.5 OpenAI (2023a) series for our research. Given that tables usually have extensive data, depending on the prompt length, we dynamically use gpt-3.5-turbo-0613 and gpt-3.5-turbo-16k-0613, with a primary aim to optimize cost when querying the API.

Evaluation MetricsFollowing prior works Jiang et al. (2022); Ni et al. (2023); Cheng et al. (2023); Ye et al. (2023), we employ _Exact Match Accuracy_ as the evaluation metric to validate predictions against ground truths, embedding instructions in prompts for consistent and parseable outputs.

Reasoning MethodsOur evaluation hinges on two distinct zero-shot reasoning approaches:

* **Direct Prompting (DP)** is a textual reasoning method that prompts LLMs to answer questions in a zero-shot manner. Rather than directly providing the answer, LLMs are instructed to reason step-by-step before concluding. More details can be found in Appx. SSA.1,
* **Python Shell Agent (PyAgent)** is a symbolic reasoning approach where the model dynamically interacts with a Python shell. Specifically, LLMs use the Python Shell as an external tool to execute commands, process data, and scrutinize results, particularly within a pandas dataframe, limited to a maximum of five iterative steps. Detailed prompt is presented in Appx. SSA.2.

Other DetailsDepending on the scenario, we adjust the temperature setting. In cases not employing self-consistency, we set it to 0. For scenarios involving self-consistency, the temperature is set to 0.8. For further granularity, Appx. SSA offers an exhaustive list of the prompts implemented in our experiments. Importantly, it should be noted that all prompts are deployed in a zero-shot manner, without any demonstrations or examples.
 

(\({\cal T}_{\rm II}\)), and their combination (\({\cal T}^{\top}_{\Pi}\)). As demonstrated in Tab. 1, both reasoning methods, DP and PyAgent, exhibit significant performance declines, with more pronounced when transposition is applied. DP consistently outperforms PyAgent largely across perturbations, indicating that textual reasoning tends to be more resilient to these structural changes. This resilience can be attributed to LLMs' ability to grasp semantic connections and meanings irrespective of structural shifts. In contrast, symbolic reasoning, exemplified by PyAgent, is heavily reliant on table structure, making it more vulnerable, especially to transposition.

### Limitations of Table Transposition with LLMs

To better understand LLMs' capabilities with regard to table structures, we investigate their ability on detecting tables in need of transposition and performing table transposition.

**LLMs as Transposition Detectors.** Given a table \({\cal T}\), the goal is to detect whether a table should be transposed for better comprehension by LLMs. This is formulated as a binary classification task:

\[f({\cal T})\to 0,\quad f({\cal T}^{\top})\to 1,\]

Where 0 denotes 'no need of transposition' and 1 indicates 'transposition needed'. Tab. 2 shows the results using the prompt in Appx. SSA.4. GPT-3.5 correctly classified 93.35% of original tables \({\cal T}\) as not requiring transposition. However, its accuracy dramatically decreased to 32.54% on transposed tables \({\cal T}^{\top}\). Our observations highlight that LLMs suffer from structural bias in the interpretation of table orientations, predominantly leading to recommendations against transposition.

**LLMs as Table Transposers.** The objective is to switch between original and transposed table formats. Specifically, the goal is to directly yield \({\cal T}^{\top}\) given \({\cal T}\), and vice versa. Formally, the task is:

\[f({\cal T})\rightarrow{\cal T}^{\top},\quad f({\cal T}^{\top})\rightarrow{ \cal T}\]

We observed that GPT-3.5's proficiency in this task is limited, with an accuracy of 53.68% transposing row tables and 51.07% for the inverse operation, suggesting that LLMs can not transpose tables precisely. For a detailed error case study and further analysis, refer to the Appx. SSB.

### Table Structure Normalization

In addressing structural variations in tables, our goal is to ensure consistent interpretation and utility across diverse table structures. To normalize various table structures into well-ordered troubles prior to downstream tasks, we introduce Norm, which is a two-stage normalization strategy: the first stage detects column-tables and transposing them into row-tables, while the second stage sorts the row-tables for enhanced comprehensibility. Through this approach, Norm accommodates for structural perturbations without compromising the understanding of the standardized row-tables.

**Content-Aware Transposition Determination** In the straightforward methods mentioned in SS4.2, LLMs are affected by the loss of structure information of the table. Our approach aims to reduce this structural dependence by introducing a content-aware determination process, which leverages the semantic reasoning capabilities of LLMs, instead of perceiving the table's structure. Specifically, we analyze the inherent content within the first row (\({\cal T}_{0,*}\)) and the first column (\({\cal T}_{*,0}\)) of a given table

\begin{table}
\begin{tabular}{l l c} \hline \hline
**Perturbation** & **DP** & **PyAgent** \\ \hline Original (\({\cal T}\)) & 59.50 & 55.91 \\  +Shuffle (\({\cal T}_{\rm II}\)) & 52.21 & 47.91 \\  +Transpose (\({\cal T}^{\top}\)) & -12.25\% & -14.31\% \\  & 51.14 & 12.45 \\  & -14.05\% & -77.73\% \\  +Transpose\&Shuffle (\({\cal T}^{\top}_{\Pi}\)) & 37.51 & 8.96 \\  & -36.96\% & -83.97\% \\ \hline \hline \end{tabular}
\end{table}
Table 1: Accuracy of GPT-3.5 under different table perturbations using Direct Prompting (DP) and Python Shell Agent (PyAgent).

\begin{table}
\begin{tabular}{l l r} \hline \hline
**LLMs As** & **Task Description** & **Accuracy** \\ \hline Transposer & \(f({\cal T})\rightarrow{\cal T}^{\top}\) & 53.68 \\  & \(f({\cal T}^{\top})\rightarrow{\cal T}\) & 51.07 \\ \hline Detector & \(f({\cal T})\to 0\) & 93.35 \\  & \(f({\cal T}^{\top})\to 1\) & 32.54 \\ \hline Determinator & \(f({\cal T},{\cal T}_{0,*},{\cal T}_{a,0})\rightarrow{\cal T}_{0,*}\) & 97.39 \\  & \(f({\cal T}^{\top},{\cal T}_{0,*},{\cal T}_{e,0})\rightarrow{\cal T}_{*,0}\) & 94.77 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Evaluation results of GPT-3.5 on the 421 distinct tables of the WikiTTableQuestions (WTQ) dataset, covering three tasks: (1) _Transposer_, which involves switching between original (\({\cal T}\)) and transposed (\({\cal T}^{\top}\)) tables directly; (2) _Detector_, which identifies the need for table transposition (\(0\) for no transposition, \(1\) for transposition required); and (3) _Determinator_, which chooses probable table headings either from the first row (\({\cal T}0,*\)) or the first column (\({\cal T}*,0\)).

 

(\({\cal T}\)) to decide which is more semantically fitting to serve as the table's heading. This content-aware approach can be mathematically modeled as:

\[\begin{cases}f({\cal T},{\cal T}_{0,*},{\cal T}_{*,0})\rightarrow{ \cal T}_{0,*}\\ f({\cal T}^{\top},{\cal T}_{0,*},{\cal T}_{*,0})\rightarrow{\cal T}_{*,0}\end{cases}\]

Here, a selection of the first row suggests that the current table structure is preferred, whereas opting for the first column signifies a need for transposition. The prompt detailing this method is provided in Appx. SSA.5. Results in Tab. 2 highlight capability of GPT-3.5 in discerning table headings semantically, with accuracies of 97.39% and 94.77% respectively for original table and tranposed table.

**Row Reordering.** Upon transposition, our next objective is to ensure the logical coherence of the table data through reordering the rows. We instruct LLMs to suggest improved reordering strategies using the prompts as detailed in Appx. SSA.6. Due to the subjective nature involved in identifying the most suitable order of a tabular data, and given that there are no widely recognized standards for this process, the effectiveness of the proposed sorting strategy will be evaluated based its downstream impact on the results of table QA task. We notice that when the entire well-ordered table is exposed, GPT-3.5 occasionally suggests alternative sorting strategies, leading to unnecessary complexity. To counteract this tendency and ensure a better sorting proposal, we strategically present the model with only the first three and the last three rows of the table. This selective exposure typically allows the model to discern logical ordering patterns without being influenced by existing table configurations.

Tab. 3 underscores the efficacy of Norm when applied prior to the two reasoning methods - DP and PyAgent. Demonstrably, Norm robustly mitigates structural perturbations, optimizing table comprehensibility for LLMs. The results illustrate that applying Norm does not detrimentally affect the original results (\({\cal T}\)), and it effectively refines perturbated data, aligning the outcomes closely with the original results, and in some instances, even showing slight improvement. This suggests that Norm as a preprocessing step for preparing tabular data can enhance robust analysis by LLMs.

In addressing our initial research question, the analysis indicates that **LLMs' performance is sensitive to table structural variations**, with significant struggles observed in accurately interpreting the same tabular content under transposition and shuffling. While **textual reasoning demonstrates some resilience** to structural variations, **symbolic reasoning is significantly impacted**, particularly with transposed tables. **The Norm strategy effectively navigates these challenges** by eliminating dependency on table structures, providing consistent interpretation across diverse table structures without compromising the integrity or meaning of the original content.

## 5 Comparing Textual and Symbolic Reasoning

In this section, we delve into the comparison of textual and symbolic reasoning methods in LLMs for tabular data understanding (SS5.1), further conducting a detailed error analysis (SS5.2) to address the second research question (SS1). We evaluate the performance of each reasoning strategy using GPT-3.5, shedding light on their strengths and challenges. In SS4.3, we explored Norm to mitigate structural perturbations, enhancing generalized LLM performance and successfully restoring perturbed tables to accuracy levels similar to their original states. Therefore, subsequent analyses will exclusively consider the original tables (\({\cal T}\)).

### Results

Tab. 4 showcases the performance of GPT-3.5 when employed for both direct textual reasoning using DP and interactive symbolic reasoning using PyAgent. By instructing the model with the CoT Wei et al. (Wei et al., 2023) reasoning strategy to _think step by step, and then give the final answer_, as detailed in Appx. SSA.1, we can achieve an accuracy of 58.66%. This surpasses the StructGPT's Iterative Reading-then-Reasoning method, which concentrates reasoning tasks by continually collecting relevant evidence. For tables with limited tokens, symbolic reasoning via PyAgent offers an accuracy of 56.87%,

\begin{table}
\begin{tabular}{c c c|c c} \hline \hline
**Method** & \({\cal T}\) & \({\cal T}_{\Pi}\) & \({\cal T}^{\top}\) & \({\cal T}_{\Pi}^{\top}\) \\ \hline DP & 59.50 & 52.21 & 51.14 & 37.51 \\  +Norm & 58.66 & 58.66 & 58.30 & 57.71 \\  & -1.41\% & +12.35\% & +14.00\% & +53.85\% \\ \hline PyAgent & 55.91 & 47.91 & 12.43 & 8.96 \\  +Norm & 56.87 & 57.11 & **55.44** & **55.08** \\  & +1.72\% & +19.20\% & +346.02\% & +514.73\% \\ \hline \hline \end{tabular}
\end{table}
Table 3: Accuracy of GPT-3.5 under different table perturbations for Direct Prompting (DP) and Python Shell Agent (PyAgent) with Norm applied.

 

which is slightly behind the accuracy by DP in a single attempt. A distinct advantage of symbolic reasoning is its ability to only present parts of the table in the prompt. As our experiments revealed, after excluding the central rows and showcasing only the initial and final three rows, we manage to maintain an accuracy of 52.45% with a 4.42% drop compared to the full-table PyAgent results. This makes it possible to deal with larger tables with numerous rows using LLMs with limited context window. In the following sections, we will present a comprehensive analysis of the discrepancies and errors observed across these methods.

### Error Analysis

To elucidate the challenges and limitations of DP and PyAgent, this section presents an in-depth error analysis by sampling 50 erroneous outputs for each. Tab. 5 summarizes the predominant error types for DP and PyAgent methods. _Table interpretation errors_ significantly afflict the DP method, comprising 42% of its total errors, highlighting substantial challenges for LLMs in accurately interpreting table data. PyAgent primarily struggles with _coding errors_, constituting 38% of its total errors. These errors either originate from misunderstandings of table content, often overlooking subtle details, or manifest as inherent deficiencies in coding capabilities. These prevalent errors underscore the intrinsic challenges and limitations each method faces in the reasoning process. Detailed case studies on each error type are delineated in Appx. SSC.

In response to the second research question, the analysis indicates **DP marginally surpasses****PyAgent within single attempts**. Despite this, PyAgent can handle larger tables by processing partial table views. Notably, **DP encounters difficulties in accurate table interpretation**, while **PyAgent reveals instability in coding capabilities**.

## 6 Reasoning Aggregation

This section examines how combining multiple reasoning pathways can boost LLMs' accuracy in interpreting tabular data, which is in response to the third research question (SS1).

### Methods

**Self-Consistency.** Previous work has highlighted the advantages of generating multiple outputs from LLMs and adopting the most frequent answer, a mechanism known as self-consistency (SC; Wang et al.2023). Tab. 4 showcases the notable improvements realized through self-consistency (aggregating 10 outputs), with DP achieving an accuracy of 64.84% and PyAgent attaining 63.49%.

**Self-Evaluation.** Based on our error analysis in SS5.2, different reasoning methods excel at specific tasks. For instance, symbolic reasoning tends to outperform textual reasoning in counting and column localization tasks. To optimize the choice between these methods, we strategically use a prompt (referenced in Appx. SSA.7), which avoids directly validating answers against tables but guides the LLM to choose between the two reasoning approaches based on the question's nature and each answer's clarity. By weighing the problem against the known strengths and weaknesses of each reasoning strategy, this tactic mitigates potential bias

\begin{table}
\begin{tabular}{l c} \hline \hline
**Method** & **Accuracy** \\ \hline \multicolumn{3}{|r|}{_Few-shot Prompting Methods_} \\ Binder\({}^{\blacktriangle}\)[Cheng et al., 2023] & 63.61 \\ Binder\({}^{\blacklozenge}\)[Cheng et al., 2023] & 55.07 \\ DATER w/o SC\({}^{\bigstar}\)[Ye et al., 2023] & 61.75 \\ DATER w/ SC\({}^{\bigstar}\)[Ye et al., 2023] & 68.99 \\ \hline \multicolumn{3}{|r|}{_Zero-shot Prompting Methods_} \\ StructGPT\({}^{\blacktriangle}\)[Jiang et al., 2023a] & 51.77 \\ Norm+Dp\({}^{\blacktriangledown}\) & 58.66 \\ Norm+PyAgent\({}^{\blacktriangle}\) & 56.87 \\ Norm+PYAgent-Omitted\({}^{\blacktriangle}\) & 52.45 \\ Norm+DP@\&PryAgent w/ eval\({}^{\blacktriangle}\) & 64.22 \\ DP w/ sc\({}^{\blacktriangle}\) & 66.39 \\ +Norm\({}^{\blacktriangle}\) & 64.10 \\ +Norm w/o Resort\({}^{\blacktriangle}\) & 66.99 \\ PYAgent w/ sc\({}^{\blacktriangle}\) & 61.39 \\ +Norm\({}^{\blacktriangle}\) & 63.77 \\ +Norm w/o Resort\({}^{\blacktriangle}\) & 62.84 \\ DP\&PyAgent w/ Mix-Sc\({}^{\blacktriangle}\) & 73.06 \\ +Norm\({}^{\blacktriangle}\) & 72.40 \\ +Norm w/o Resort\({}^{\blacktriangle}\) & 73.65 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance comparison of various methods on a sampled subset (\(\mathcal{T}\)) of the WikiTableQuestions dataset. Methods marked with \({}^{\bigstar}\) are based on Codex, while those marked with \({}^{\blacktriangle}\) are based on GPT-3.5. SC stands for self-consistency, and the results for SC-based methods are obtained by averaging over 100 shuffles to handle cases of ties during majority voting. In the Misc SC method, answers from DP are prioritized over those from PyAgent due to DP’s superior performance. This prioritization is followed in all experiments involving SC. Norm w/o Resort refers to the Norm method without the reordering stage.

 

towards textual reasoning by LLMs and enhances answer accuracy. As evidenced by Tab. 4, using self-evaluation boosts accuracy to 64.99%. Impressively, this method, using only two reasoning paths, matches the performance of using 10 paths of DP or PyAgent independently.

Mix Self-ConsistencyAccording to SS5.1, symbolic and textual reasoning exhibit distinct focuses but deliver similar performance. Consequently, we introduce _Mix Self-Consistency_, a method that selects a predetermined number of outputs for each type of inference, aiming for self-consistency. This approach hinges on the idea that multiple outputs can reflect the confidence levels of LLMs in answer generation. In scenarios where LLMs are less proficient, they tend to produce a diverse set of answers. Conversely, for tasks that LLMs handle adeptly, consistent answers are often generated across multiple reasoning attempts, converging towards one answer. Such convergence allows for the aggregation of model outputs that align with areas where LLMs exhibit stronger reasoning capabilities, thereby substantially improving accuracy. The detailed mechanics of how this approach is implemented within the framework of _Mix Self-Consistency_, including the aggregation and interpretation of these outputs, are further elucidated in Appx. SSD.2.

Tab. 4 demonstrates that using mix self-consistency, which involves generating 5 outputs per inference type (totaling 10), enhances performance substantially, achieving an impressive accuracy of 72.40%, which achieves SOTA performance on the sampled WTQ data. The choice of generating 5 outputs per inference type (5+5) is a hyperparameter selection influenced by the dataset's distribution. We conducted an ablation study regarding this in Appx. SSD.1. The decision to use an equal split (5+5) is based on the observed comparable performance between the two reasoning strategies.

### Overall Evaluation

To evaluate our method thoroughly, we conduct a comprehensive pass of testing using the complete WTQ test set, integrating both Norm and Mix self-consistency mechanisms. Since re-sorting may change the answers of row index-related questions, we perform Norm without resorting in this evaluation. Originally, the Norm process included a re-sorting step to counteract the row-shuffling perturbation. However, as explored in the error analysis (SS5.2) with a detailed case study in Appx. SSC.6, re-sorting may inadvertently alter answers reliant on the initial sequence. Despite this limitation, it is noteworthy that re-sorting can be advantageous for questions not dependent on row indexes, particularly when dealing with tables that are initially unorganized or messy.

As illustrated in Tab. 6, our proposed method exhibits outstanding efficacy with an accuracy of 73.6%, significantly outperforming existing models to achieve SOTA performance on the complete WTQ test set. Importantly, our approach is conducted in a fully zero-shot manner.

To further analyze the impact of table size on method performance, we segmented the row num

\begin{table}
\begin{tabular}{l c c l p{184.9pt}} \hline \hline
* & 38\% & LLMs produce inaccurate code, typically due to issues with minor details. & Appx. §C.2.1, Appx. §C.2.2, Appx. §C.2.3 \\ \hline Misalignment Issue & 24\% & 28\% & Outputs are conceptually correct but the answers do not align with the instructions. & Appx. §C.3.1, Apx. §C.3.2 \\ \hline Logical Inconsistency & 20\% & 10\% & LLMs exhibit failures in reasoning, leading to tradictions or inconsistencies. & Appx. §C.4.1, Appx. §C.4.2 \\ \hline Execution Issue &
- & 12\% & Issues emerge related to the execution of Python code. & Appx. §C.5.1, Appx. §C.5.2 \\ \hline Resorting Issue & 10\% & 8\% & The resorting stage in Norm changes the answers of some sequence-dependent questions. & Appx. §C.6 \\ \hline \hline  & & & \\ \end{tabular}
\end{table}
Table 5: Categorization of error types for the DP and PyAgent methods. \({}^{\dagger}\)The _table interpretation errors_ is not explicitly used for PyAgent, as these errors are included under _coding errors_ to avoid overlap in categorization. The percentages for each reasoning method may not sum up to 100%, as the remaining percentage points are attributed to _other errors_, such as issues with dataset labeling, which are not categorized in our analysis.
 

bers into 10 ranges, each containing approximately 430 data points, and calculated the average accuracy within these intervals. Fig. 3 visualizes the average accuracy across these ranges for each method. It is evident that there is a shared trend of diminishing accuracy as the number of rows increases, suggesting that all methods are subject to decreased efficacy in the context of long tables.

The decline in performance with larger tables can be attributed to the complexity of handling long-context data and the abundance of potentially interfering information, often resulting in an increased error rate. The insights gained from this analysis point towards a need for the development of better symbolic methods for handling long tables, which might be capable of effectively narrowing down the scope of larger tables, either by selective attention to relevant segments or by intelligently summarizing the data, to mitigate the challenges posed by long-context information.

In response to the third research question, our findings reveal that **reasoning path aggregation significantly enhances LLMs' accuracy in table reasoning tasks**. Notably, **the _Mix Self-Consistency_ method achieves an accuracy of 73.6% on the WTQ dataset, surpassing the previous SOTA by a considerable margin**. The _Self-Evaluation_ strategy also contributes to this remarkable performance by adeptly selecting between reasoning approaches.

## 7 Conclusion

This study investigated the proficiency of LLMs in tabular reasoning. The findings suggest that LLMs are sensitive to the structural variance of tables, but the application of the proposed normalization strategy can stabilize table structures and improve resistance to structural perturbations. When comparing reasoning approaches, textual reasoning demonstrated a slight advantage over symbolic reasoning, with each strategy exhibiting unique strengths. Furthermore, integrating multiple reasoning strategies via mix self-consistency proved beneficial for overall interpretation accuracy, surpassing previous SOTA results on the WikiTableQuestions dataset. These observations contribute to the understanding of LLMs' capabilities in tabular reasoning and provide insights for further improvements.

\begin{table}
\begin{tabular}{l c} \hline \hline
**Method** & **Accuracy** \\ \hline \multicolumn{2}{|c|}{_Fine-tuning Based Models_} \\ TaPaS(Herzig et al., 2020) & 48.8 \\ T5-3B(Xie et al., 2022) & 49.3 \\ TaPAX(Liu et al., 2022) & 57.5 \\ ReasTAP(Zhao et al., 2022) & 58.7 \\ OmniTab(Jiang et al., 2022) & 63.3 \\ \hline \multicolumn{2}{|c|}{_LLM s Based Methods_} \\ StructGPT\({}^{\bigstar}\)(Jiang et al., 2023a) & 48.4 \\ Binder\({}^{\bigstar}\)(Cheng et al., 2023) & 55.5 \\ Binder\({}^{\blacktriangle}\)(Cheng et al., 2023) & 64.6 \\ Lever\({}^{\blacklozenge}\)(Ni et al., 2023) & 65.8 \\ Dater\({}^{\blacklozenge}\)(Ye et al., 2023) & 65.9 \\ \hline
**Ours\({}^{\bigstar}\)** & **73.6** \\ \hline \hline \end{tabular}
\end{table}
Table 6: Comparison of various methods on all test data of WTQ. \({}^{\bigstar}\) denotes methods based on the GPT-3.5 (OpenAI, 2023a); \({}^{\blacklozenge}\) denotes methods based on the Codex (OpenAI, 2022). Results are directly sourced from the referenced paper.

Figure 3: The impact of table size on the performance of DP, PyAgent, and Mix-SC (Combining 5 DP and 5 PyAgent) methods on the WikiTableQuestions test set. The horizontal axis represents the number of rows in the tables, divided into ranges, while the vertical axis denotes the average accuracy achieved by each method within the corresponding table size range.

 



### Limitation

While this study provides insights into tabular data reasoning with LLMs, it is pertinent to acknowledge its limitations. First, the exclusive utilization of GPT-3.5, due to the budgetary constraints, may limit the generalizability of our findings, as exploration with GPT-4 might offer enhanced outcomes. Second, all table data are sourced from Wikipedia, which may introduce potential data leakage or memorization issues, as certain answers might be implicitly available within the LLMs' training data, thus potentially biasing results. Lastly, several perturbation-sensitive table-based questions, especially regarding table perturbations like shuffling, may impact the precision of the reported accuracy, as demonstrated answers may change based on the structural modifications of the table.

## Acknowledgment

We appreciate the reviewers for their insightful comments and suggestions. Fei Wang is supported by the Annenberg Fellowship and the Amazon ML Fellowship. Muhao Chen is supported by the NSF Grant IIS 2105329, the NSF Grant ITE 2333736, the Faculty Startup Fund of UC Davis, a Cisco Research Award and two Amazon Research Awards.

## References

* B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei (2020)Language models are few-shot learners. In Advances in Neural Information Processing Systems _33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
* Chang et al. (2023) Shuachen Chang, Jun Wang, Mingwen Dong, Lin Pan, Henghui Zhu, Alexander Hanbo Li, Wuwei Lan, Sheng Zhang, Jiarong Jiang, Joseph Lilen, Steve Ash, William Yang Wang, Zhipuo Wang, Vittorio Castelli, Patrick Ng, and Bing Xiang. 2023Dr.spider: A diagnostic evaluation benchmark towards text-to-sql robustness. Cited by: SS1.
* Chase (2022)LangChain. External Links: Link Cited by: SS1.
* W. Chen (2023)Large language models are few(1)-shot table reasoners. External Links: Link Cited by: SS1.
* W. Chen, M. Chang, E. Schlinger, W. Y. Wang, and W. W. Cohen (2020)Open question answering over tables and text. In International Conference on Learning Representations, Cited by: SS1.
* Zhoujun Cheng, Tianbao Xie, Peng Shi, C. Li, R. Nadkarni, Y. Hu, C. Xiong, D. Radev, M. Ostendorf, L. Zettlemoyer, N. A. Smith, and T. Yu (2023)Binding language models in symbolic languages. ICLRabs/2210.02875. Cited by: SS1.
* A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. Won Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-Arri, P. Yin, T. Duke, A. Levskaya, S. Ghemawat, S. Devin, H. Michalewski, X. Garcia, V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito, D. Luan, H. Lim, B. Zoph, A. Spiridonov, R. Sepassi, D. Dohan, S. Agrawal, M. Omerick, A. M. Dai, T. Sankaranayana Pilali, M. Pillat, A. Lewkowycz, E. Moreira, R. Child, Q. Polozov, K. Lee, Z. Zhou, X. Wang, B. Saeta, M. Diaz, O. Firat, M. Catasta, J. Wei, K. Meier-Hellstern, D. Eck, J. Dean, S. Petrov, and N. Fiedel (2022)Palm: scaling language modeling with pathways. External Links: Link Cited by: SS1.
* P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei (2017)Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, Vol. 30, pp. . Cited by: SS1.
* H. Won Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. Shane Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdharey, A. Castro-Ros, M. Pellat, K. Robinson, D. Valter, S. Narang, G. Mishra, A. Yu, V. Zhao, Y. Huang, A. Dai, H. Yu, S. Petrov, E. H. Chi, J. Dean, J. Devlin, A. Roberts, D. Zhou, Q. V. Le, and J. Wei (2022)Scaling instruction-finetuned language models. External Links: Link Cited by: SS1.
* Leo Gao, John Schulman, and Jacob Hilton (2022)Scaling laws for reward model overoptimization. External Links: Link Cited by: SS1.
* Z. Gu, J. Fan, N. Tang, P. Nakov, Xiaooman Zhao, and Xiaoyong Du (2022)PASTA: label-operations aware fact verification via sentence-table clore pre-training. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, pp. 4971-4983. External Links: Link Cited by: SS1.



* [Hao et al.2023] Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. 2023. Toolkengptt: Augmenting frozen language models with massive tools via tool embeddings.
* [Herzig et al.2020] Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Muller, Francesco Piccinino, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics.
* Jiang et al.2023a] Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Wayne Xin Zhao, and Ji-Rong Wen. 2023a. StructGept: A general framework for large language model to reason on structured data.
* Jiang et al.2022] Zhengbao Jiang, Yi Mao, Pengcheng He, Graham Neubig, and Weizhu Chen. 2022. OmniTab: Pretraining with natural and synthetic data for few-shot table-based question answering. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 932-942, Seattle, United States. Association for Computational Linguistics.
* Jiang et al.2023b] Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023b. Active retrieval augmented generation.
* Kojima et al.2023] Takeshi Kojima, Shixiang Shane Gu, Mached Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. Large language models are zero-shot reasoners.
* Li et al.2023] Hongxin Li, Jingran Su, Yuntao Chen, Qing Li, and Zhaoxiang Zhang. 2023a. Sheetcopilot: Bringing software productivity to the next level through large language models.
* Li et al.2023] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Koccetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirjer, Joao Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muthanshan Obkulov, Zhriuo Wang, Rudra Murthy, Jason Stillerman, Siva Sanklap Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timp, Jennifer Ding, Claire Schlesinger, Hailey Schoekopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Munoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. 2023b. Starcoder: may the source be with you!
* Li et al.2023c] Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2023c. Making large language models better reasoners with step-aware verifier.
* Liu et al.2022] Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. 2022. Tapex: Table pre-training via learning a neural sql executor.
* Lozhkov et al.2022] Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joelamy-Poiferci, Noumanc Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Koccetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuqing Li, WenDing Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Ni Osae Oseae Dade, Wenhao Yu, Lucas Krauh, Naman Jain, Yixuan Su, Xunali He, Mannap Bey, Edoardo Abati, Yekun Chai, Niklas Muenninghoff, Xiangru Tang, Muthasham Olokulov, Christopher Akiki, Marcarone, Chengho Mou, Mayank Mishra, Alex Gu, Biyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Taljabakhsh, Yacine Jernite, Carlos Munoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. 2024. Starcoder 2 and the stack v2: The next generation.
* Malon et al.2023] Gregoire Malon, Roberto Dessi, Maria Lomeli, Christoferos Lalmpantis, Ram Pasnuru, Roberta Raileanu, Baptisse Roziere, Timo Schick, Jane Dwivedi-Yu, Asli Celikylmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. 2023. Augmented language models: a survey.
* Nakano et al.2022] Reichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022. Webgpt: Browser-assisted question-answering with human feedback.
* Ni et al.2023] Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen-tau Yih, Sida I Wang, and Xi Victoria Lin. 2023. Lever: Learning to verify language-to-code generation with execution. In _Proceedings of the 40th International Conference on Machine Learning (ICML'23)_.
* OpenAI 2022] OpenAI. 2022. Codex.
* OpenAI 2023a] OpenAI. 2023a. Chatgt.
* OpenAI 2023b] OpenAI. 2023c. Gpt-4 technical report.
* Paspupat and Liang 2015] Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables.

 

Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Scharli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context.

## References

* G. Gravitas (2023)Auto-gpt. External Links: Link Cited by: SS1.
* N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. Christiano (2022)Learning to summarize from human feedback. External Links: Link Cited by: SS4.
* H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Roziere, N. Goyal, E. Hambro, F. Azhar, et al. (2023)Llama: open and efficient foundation language models. ArXiv preprintabs/2302.13971. Cited by: SS1.
* H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton Ferrer, M. Chen, G. Cucurull, D. Esiboul, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Gowyal, A. Hartshorn, S. Hosseini, R. Hou, H. Innan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. Singh Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaloy, P. Mishra, Igor Mobbo, Y. Nie, A. Poulton, J. Reizenstein, R. Runga, K. Saladi, A. Schelten, R. Silva, E. Michael Smith, R. Subramanian, X. Ellen Tan, B. Tang, R. Taylor, A. Williams, J. Xiang Guan, P. Xu, Z. Yan, I. Zarorov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Sciolom (2023)LDLAMA 2: open foundation and fine-tuned chat models. External Links: Link Cited by: SS1.
* F. Wang, K. Sun, J. Pujara, P. Szekely, and M. Chen (2021)Table-based fact verification with salience-aware learning. In Findings of the Association for Computational Linguistics: EMNLP 2021, Punta Cana, Dominican Republic, pp. 4025-4036. External Links: Link Cited by: SS1.
* F. Wang, Z. Xu, P. Szekely, and M. Chen (2022)Robust (controlled) table-to-text generation with structure-aware equivariance learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Cited by: SS1.
* X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou (2023)Self-consistency improves chain of thought reasoning in language models. External Links: Link Cited by: SS1.
* J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le (2022)Finetuned language models are zero-shot learners. External Links: Link Cited by: SS1.
* J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou (2023)Chain-of-thought prompting elicits reasoning in large language models. External Links: Link Cited by: SS1.
* T. Xie, C. Henry Wu, P. Shi, R. Zhong, T. Scholak, M. Yasunaga, C. Wu, M. Zhong, P. Yin, S. I. Wang, V. Zhong, B. Wang, C. Li, C. Boyle, A. Ni, Z. Yao, D. Rabcey, C. Xiong, L. Kong, Rui Zhang, N. A. Smith, L. Zettlemoyer, and T. Yu (2022)Unifiedskg: unifying and multi-tasking structured knowledge grounding with text-to-text language models. EMNLP. External Links: Link Cited by: SS1.
* Y. Ye, B. Hui, M. Yang, B. Li, F. Huang, and Y. Li (2023)Large language models are versatile decomposers: decompose evidence and questions for table-based reasoning. External Links: Link Cited by: SS1.
* P. Yin, G. Neubig, W. Yih, and S. Riedel (2020)TaBERT: pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 8413-8426. External Links: Link, Document Cited by: SS1.
* S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. Victoria Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Korus, A. Sydfahr, T. Wang, and L. Zettlemoyer (2022)Opt: open pretrained transformer language models. External Links: Link Cited by: SS1.
* T. Zhang, X. Yue, Y. Li, and H. Sun (2023)Tabellama: towards open large generalist models for tables. External Links: Link Cited by: SS1.
* W. Zhang, Y. Shen, W. Lu, and Y. Zhuang (2023)Data-copilot: bridging billions of data and humans with autonomous workflow. External Links: Link Cited by: SS1.
* Y. Zhao, L. Nan, Z. Qi, R. Zhang, and D. Radev (2022)ReastTAP: injecting table reasoning skills during pre-training via synthetic reasoning examples. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, pp. 9006-9018. External Links: Link, Document Cited by: SS1.
* Y. Zhao, C. Zhao, L. Nan, Z. Qi, W. Zhang, X. Tang, B. Mi, and D. Radev (2023)Robut: a systematic study of table qa robustness against human-annotated adversarial perturbations. External Links: Link Cited by: SS1.
* D. Zhou, N. Scharli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, C. Cui, O. Bousquet, Q. Le, and E. Chi (2023)Least-to-most prompting enables complex reasoning in large language models 



**Appendices**

## Appendix A Prompts

### Prompt of Direct Prompting (DP).

You are an advanced AI capable of analyzing and understanding information within tables. Read the table below regarding "[TITLE]".

[TABLE] Based on the given table, answer the following question:

[QUESTION] Let's think step by step, and then give the final answer. Ensure the final answer format is only "Final Answer: AnswerName1, AnswerName2..." form, no other form. And ensure the final answer is a number or entity names, as short as possible, without any explanation.

### Prompt of Python Agent.

You are working with a pandas dataframe in Python. The name of the dataframe is "df". Your task is to use 'python_repl_ast' to answer the question posed to you.

[FIGURE:A3.F4] Figure 9 shows the results for the first run of the code.

[FIGURE:S0.F5] Figure 10 shows the results for the second run of the code.

[FIGURE:Sx

[TABLE] Please transpose this table. Maintain the format I give, with each row beginning with '|' and each cell separated by '|'. Do not change the content of any cell. Your response should solely consist of the transposed table, without any additional text.

### Prompt of LLMs as Table Transposition Detector

Please examine the provided table:

[TABLE] To enhance readability and facilitate efficient data analysis, it is often suggested that the table headings be horizontally located in the first/topmost row.

Please evaluate the table with this consideration in mind, and provide your response in the following format:

**Table Headings****: List the headings of the table, separated by commas.
**Table Evaluation****: Identify whether the headings listed are horizontally located in the first/topmost row. If not, describe the position.
**Transpose Recommended****: Indicate if transposing is recommended. Answer with only "YES" or "NO", without any additional explanation.

### Prompt of Content-Aware Transposition Determination

You are an advanced AI capable of analyzing and understanding information within tables. Read the table below regarding "[TITLE]".

[TABLE] Headings of a table are labels or titles given to rows or columns to provide a brief description of the data they contain. Based on the given table, the headings of the table are more likely to be:

(A) [FIRST_ROW] (B) [FIRST_COLLMW] (C) None of the above Directly give your choice. Ensure the format is only "Choice: (A)/(B)/(C)" form, no other form, without any explanation.

### Prompt of Resorting

You are an advanced AI capable of analyzing and understanding information within tables. Read the table below regarding "[TITLE]":

[TABLE] Note: Only selected rows from the beginning and end of the table are displayed for brevity. Intermediate rows are omitted and represented by "..." for clarity. The table column headings are provided below, separated by semicolons:

[HEADINGS] In order to optimize the interpretability and readability of the data, follow these guidelines to determine the most suitable sorting method: Sorting Guidelines:

1. Evaluate columns based on data types such as numerical, alphabetical, chronological, categorical, or other relevant sorting methods.
2. Identify any patterns or relationships in the data that would be highlighted by certain sorting methods.
3. Consider column position, as those on the left may sometimes have sorting priority.
4. If applicable, consider sorting by multiple columns in a prioritized sequence.

 

Provide your decision using one of the following statements:

- For sorting using a single column: "Sort by: [Name of Column]".

- For sorting using multiple columns: "Sort by: [Primary Column Name], [Secondary Column Name], ...'.

- If no specific sorting seems advantageous: "Sort by: N/A".

Your response should strictly follow the formats provided.

### Prompt of Self-Evaluation

Below is a markdown table regarding "[TITLE]":

[TABLE]

You're tasked with answering the following question:

[QUESTION]

You have 2 answers derived by two different methods. Answer A was derived by prompting the AI to think step-by-step. Answer B was derived by interacting with a Python Shell.

Answer A is [COLT_ANSWER]. Answer B is [AGENT_ANSWER]. Your task is to determine which is the correct answer. It is crucial that you strictly adhere to the following evaluation process:

1. **Preliminary Evaluations*: Begin by evaluating which of the two answers directly addresses the question in a straightforward and unambiguous manner. A direct answer provides a clear response that aligns closely with the query without introducing additional or extraneous details. If one of the answers is not a direct response to the question, simply disregard it.
2. **Nature of the Question**: If both answers appear to be direct answers, then evaluate the nature of the question. For tasks involving computation, counting, and column-locating, especially when for extensive table, the Python Shell (Answer B) might be more precise. However, always remain cautious if the Python Shell's output appears off (e.g., error messages, success notifications, etc.). Such outputs may not be trustworthy for a correct answer.
3. **Final Verdict**: Finally, after thorough evaluation and explanation, provide your verdict strictly following the given format: - Use "[LA]" if Answer A is correct. - Use "[IB]" if Answer B is correct. Note:

1. Each method has its own strengths and weaknesses. Evaluate them with an unbiased perspective. When in doubt, consider the nature of the question and lean towards the method that is most suited for such queries.
2. Ensure that your verdict is provided after evaluation, at the end.

 

Analysis for LLMs as Table Transposer

### Case Study

Fig. 4 illustrates a typical mistake made by LLMs when transposing tables, a problem that becomes more evident when a table has many identical or similar entries. Take, for example, the 'Nation Cup' column shown in the figure, which is filled with numerous ? symbols. LLMs, limited in processing structured data, often mishandle such tables, leading to misplacements or misalignments. This highlights the fundamental difficulties and limitations LLMs face in accurately transposing tables containing repetitive or similar data cells.

### Analysis

A further examination of the results, as shown in Fig. 5, illustrates that transposition accuracy for LLMs as direct table transposer is associated with the table's dimensions. The accuracy in row-to-column transposition (\({\cal T}\rightarrow{\cal T^{\top}}\)) is distinctly sensitive to the original table's row count, whereas column-to-row

Figure 4: An example error case of content misalignment occurring within cells when leveraging LLMs directly to transpose a table. **Blue Table (Top):** The original table subjected to a transposition operation. **Green Table (Bottom Left):** The ground truth table subsequent to transposition. **Purple Table (Button Right): GPT-3.5’s output of transposed table. Cells erroneously aligned or displaced are highlighted in red.

 

transposition (\({\cal T}^{\top}\to{\cal T}\)) accuracy is similarly related on the number of columns. This observation can be potentially attributed to the inherent characteristics and organizational structure of table data. In most of the row tables, cells within a given column often display homogeneous data types, such as numerical or temporal values. This homogeneity can pose significant challenges for LLMs, as the models might struggle to differentiate between semantically similar cells during the transposition process, thereby leading to potential misalignments and misplacements, particularly as the number of rows increases. Conversely, in those column tables, cells within a row may exhibit similar data types, introducing analogous challenges and potential errors during transposition.

Figure 5: Performance of GPT-3.5 as direct table transposer: from original to transposed tables (\({\cal T}\to{\cal T}^{\top}\)) and from transposed to original tables (\({\cal T}^{\top}\to{\cal T}\)), with different row and column counts.

 



## Appendix C Error Case Study for WTQ

### Table Misinterpretation

#### c.1.1 Counting Error

* Men's 1000 metres**

Fig. 6 highlights a typical error related to table comprehension, emphasizing a common problem in LLMs when dealing with tasks that involve statistical analysis. It points to a weakness in LLMs' ability to accurately process and respond to questions based on statistical information without the help of external tools.





#### c.1.2 Locating Error

Fig. 7 showcases a common error in table interpretation associated with LLMs. This error originates from the LLMs' linearization process, which impairs their ability to recognize table structures. Although the model efficiently identifies the highest value, \(132\), in the \(2\mathsf{nd}(\mathsf{m})\) column, it inaccurately associates this value with the \(1\mathsf{nd}(\mathsf{m})\) column, assuming it represents the same feature as in the \(2\mathsf{nd}(\mathsf{m})\) column. This leads to a misplacement of the value in the table's interpretation.

Figure 7: Example of a _table misinterpretation_ error by DP, where the model fails accurately locating the specific cell.





### Coding Error

#### c.2.1 Attribute Noise Error

List of The Citadel Bulldogs football seasons

Fig. 8 illustrates a minor issue stemming from a coding mistake. The table shown in the figure features a variety of data under the Team column. Alongside the expected Dallas Cowboy entries, there are cells with a slight variation: Dallas Cowboy\({}^{\dagger}\). The Python Shell Agent used failed to recognize these unusual variations. This is evident from the use of the df['Team'] = "Dallas Cowboy" command for calculating occurrences, leading to a discrepancy in the final count and resulting in inaccurate outcomes.

Figure 8: Example of a _Coding Issue_ error by PyAgent, where the abnormal variant data entries leads to inaccurate output.





#### c.2.2 Special Row Misinterpretation Error

Fig. 9 displays an error where the Python Shell Agent incorrectly interprets a special row, treating the Total row as a valid season entry. In this case, while calculating the seasons in which Nicolas won at least 10 races, the Agent did not properly exclude the cumulative Total row in its code for computation. Consequently, it erroneously reported one season as meeting the criteria.




Fig. 10 depicts an error where the Python Shell Agent incorrectly includes a special row in its calculations. Specifically, when counting the number of Linux distributions supporting the x86 architecture, the agent erroneously counts a nested heading row. As indicated in the figure, the row indexed at 9 is not a valid data entry but rather serves as a nested heading for the table. This row should have been excluded from the count, resulting in an inaccurate calculation (\(29\to 30\)) of distributions supporting the x86 architecture.

Figure 10: Example of another _coding error_ by PyAgent, where the PyAgent misinterpretes the special row which is a nested heading.

 



#### c.2.3 Incorrect Coding

Fig. 11 shows an issue that originates from a basic coding mistake. Although the agent correctly grasps the concept, aiming to filter for entries with a maximum velocity of at least 100 km/h, it falters in the implementation stage by using a contains("100 km/h") statement in the code. This error is akin to the _Logical Inconsistency_ error described in (DP), where a mismatch between the understanding of a concept and its practical execution becomes evident.

Figure 11: Example of a _coding error_ by PyAgent, where the coding is incorrect



### Misalignment Issue

#### 1.3.1 Answer Format Issue

Fig. 12 displays a scenario in which the final answer produced did not align with the specified prompt instructions. In this instance, the Python Shell Agent correctly executed a series of interactions and accurately identified the final answer as 77. However, the response given was Brooks Racing had the number 77.. Although the reasoning and the result are correct within their respective contexts, the format of the response impedes the parsing of the correct answer.

Figure 12: Example of a _misalignment issue_ by PyAgent, where the answer format does not follow the instruction in the prompt to give a parsable answer. Note that emojis presented in the figure are purely for visual aid and were not incorporated in actual experiments; the same applies to any figures below.





#### c.3.2 Answer Deviation Error

**2007-08 Washington Huskies men's basketball team**

Fig. 13 shows a DP example where the final answer does not directly address the posed question. Faced with the binary query Are there at least 13 names on the chart?, the anticipated response should be a simple yes or no. However, the LLM responds with 12. Although this answer aligns conceptually with the data in the table and the logic seems accurate, it does not conform to the direct and binary response format required by the question.




Fig. 14 illustrates a case with the Python Shell Agent where the final response fails to directly answer the posed question. The question Which team has the same number of playoffs appearances as the St. Louis Bombers? clearly requests the identification of a specific team. Yet, it is noted that the agent prematurely delivers an answer upon finding data related to the playoff appearances of the St. Louis Bombers. While the direction of the python shell agent's reasoning appears correct, the resultant answer ultimately falls short of resolving the question correctly.

Figure 14: Example of a _misalignment issue_ by PyAgent, where the final answer does not directly answer the question.

 



### Logical Inconsistency

#### c.4.1 Reasoning Conflict in DP

Fig. 15 presents an example of a _Logical inconsistency_ error occurring during the interpretation of tabulated data. The error in reasoning is occurred in determining whether George E. Leach or Kenneth F. Cramer was the Chief of the National Guard Bureau in 1934. The reasoning text accurately states that George E. Leach served from December 1, 1931, to November 30, 1935, and Kenneth F. Cramer served from September 30, 1947, to September 4, 1950. Despite this, the interpretation erroneously concludes that neither was in the role in 1934, leading to a contradiction between the information and the final answer.

Figure 15: Example of a _logical inconsistency_ by DP where a problem with logical reasoning leads to a reasoning conflict in the context.





#### c.4.2 Reasoning Mistakes in PyAgent

Fig. 16 depicts an instance of a _Logical Inconsistency Error_ during the Python Shell interaction. In this case, LLMs successfully the necessary steps to gather information about Tablet 9 and Tablet 10, a misinterpretation of the retrieved data results in flawed reasoning. This leads to an error in the conclusion drawn from the interaction.

Figure 16: Example of a _logical inconsistency_ error by PyAgent where the LLMs make mistakes in reasoning on the observations.





### Execution Issue

#### c.5.1 Interaction Bound or Looping Error

Comparison of Linux distributions

Fig. 17 demonstrates an instance of _Execution Issue_ error made by the Python Shell Agent. In the process of identifying the team that participated in Euro 2000 qualifying for the most consecutive years, the agent faces difficulties in the data processing phase. Initially, an error occurs due to the Date column not containing datetime objects. Then the agent successfully converts the entries into the appropriate format. However, the agent, in trying to compute the number of consecutive participation years for each team, gets stuck in a loop of continually refining its calculation method without arriving at a conclusive answer within the given interaction steps.

Figure 17: An example of an _execution issue_ by PyAgent, where the agent attempts to fix an coding error but falls into a loop.





#### c.5.2 Non-Observable Action Error

Fig. 18 illustrates a situation where PyAgent encounters a _Non-Observable Action Error_. Specifically, the actions given by the PyAgent, for example, consecutive_games_count = len(df_filtered), do not generate any observable output in the Python Shell. Consequently, the agent does not receive any valid observations to aid it in deriving the correct answer, thus, leading to an incorrect count of consecutive games played in the ARCO Arena from January 2nd to January 24th.

Figure 18: Example of a _Execution Issue_ error by PyAgent, where the agent inputs non-observable actions into Python Shell.





### Resorting Issue

Fig. 19 shows a case of data inconsistency due to the application of the resorting stage in the Norm procedures. The figure's upper table displays the original format, with Gillig as the manufacturer in the final row. However, after resorting as suggested by LLMs, the lower table in the figure lists New Flyer as the last row's manufacturer. This change, while seemingly minor in the broader context of table comprehension, significantly impacts responses to specific queries like "What is the name of the last manufacturer on the chart?"

Figure 19: Example of a _Normalization Issue_ error by DP, where the correct answer changes due to the resorting stage in Norm.





## Appendix D Analysis of Mix Self-Consistency

### Ablation Study of Output Selection

This section presents an ablation study conducted to elucidate the effect of various combinations of DP and PyAgent outputs on the performance of the _Mix Self-Consistency_ method. For this experiment, we systematically explored different combinations while keeping the total output count constant at ten. Each combination was tested 100 times through random shuffling. For each test, maximum, minimum, and average accuracies were recorded.

Fig. 20 shows the results of the ablation study. The 5+5 combination (5 DP + 5 PyAgent) consistently gives the highest minimum and average accuracies among all tested combinations, making it a robust and reliable choice for this task. The 4+6 combination (4 DP + 6 PyAgent) secured the highest maximum accuracy in our tests.

Through this ablation study, we aim to provide insights into how different output selections influence the effectiveness of the _Mix Self-Consistency_ method. Importantly, the choice of output combination should be considered as a hyperparameter that is intimately related to the distribution of the dataset being used. Given that different reasoning strategies exhibit unique strengths and weaknesses, it is crucial to tailor the output combination to align with the characteristics of the specific tasks and datasets in question, thereby maximizing the performance of the _Mix Self-Consistency_ method.

### Mechanics of Mix Self-Consistency in Output Selection

The effectiveness of the _Mix Self-Consistency_ method in achieving high accuracy largely stems from its ability to harness the strengths of different reasoning methods. Intuitively, the multiple outputs from certain reasoning method can be interpreted as the confidence score for the generated answers. In scenarios where a method excels, its outputs often tend to converge towards a common answer, signifying higher

Figure 20: Accuracy results for the _Mix Self-Consistency_ method applied to the sampled WTQ dataset, with varying combinations of DP and PyAgent outputs (depicted as DP vs. PyAgent on the x-axis). The combinations range from 10 DP vs. 0 PyAgent to 0 DP vs. 10 PyAgent. Each data point represents the maximum, minimum, and average accuracies obtained from 100 tests per combination, conducted using random sampling. Note that for the 10 DP vs. 0 PyAgent and 0 DP vs. 10 Py Agent combinations, there is no random sampling of paths. However, variance is observed due to the presence of multiple equally probable answer sets generated by the 10 paths, leading to different possible selections of answers even without sampling, thereby introducing randomness into the results.



confidence and reliability. In contrast, a method less suited to the problem at hand tends to produce more diverse results, indicative of a lower level of confidence. By aggregating these outputs from different methods and applying majority voting, the _Mix Self-Consistency_ method refines these variations into a more accurate prediction. As shown in Fig. 21, This process leverages the strengths of the employed reasoning methods, thereby enhancing overall performance.

## Appendix E Results of Mix Self-Consistency on TabFact

This section presents the additional results of applying the _Mix Self-Consistency_ method to the TabFact dataset, as part of an extended investigation to verify and evaluate the method's adaptability and effectiveness in other related tasks beyond WTQ dataset.

For TabFact, a subsample of 500 data points was randomly selected from the test set. The experimental setup mirrored that of the WTQ experiments, employing the same parameters such as temperature settings for model inference. The strategy for output selection in the TabFact experiment also follows the 5+5 combination, which proves to be the best for the WTQ dataset, to aggregate the output answers from 5 instances of DP and 5 instances of PyAgent. Additionally, all the prompts (e.g., DP, PyAgent) used in the TabFact experiment were slightly modified to align with the requirements of the fact-checking scenarios.

Tab. 7 summarizes the accuracy results of the _Mix Self-Consistency_ method, StructGPT, and Dater on the TabFact dataset. _Mix Self-Consistency_ can also achieve the highest accuracy, outperforming both StructGPT and Dater in fact-checking.

\begin{table}
\begin{tabular}{l c} \hline \hline
**Method** & **Accuracy** \\ \hline StructGPT [34] & 0.708 \\ Dater [69] & 0.874 \\ \hline
**Ours** & **0.885** \\ \hline \hline \end{tabular}
\end{table}
Table 7: Accuracy results of different methods without fine-tuning on the TabFact dataset.

Figure 21: An illustration of _Mix Self-Consistency_ by aggregating outputs from multiple reasoning methods to form a unified, high-confidence prediction..