Venue: NAACL 2024

# E\({}^{5}\): Zero-shot Hierarchical Table Analysis using Augmented LLMs via

Explain, Extract, Execute, Exhibit and Extrapolate

Zhehao Zhang1, Yan Gao2, Jian-Guang Lou2

###### Abstract

Analyzing large hierarchical tables with multi-level headers presents challenges due to their complex structure, implicit semantics, and calculation relationships. While recent advancements in large language models (LLMs) have shown promise in flat table analysis, their application to hierarchical tables is constrained by the reliance on manually curated exemplars and the model's token capacity limitations. Addressing these challenges, we introduce a novel code-augmented LLM-based framework, E\({}^{5}\), for zero-shot hierarchical table question answering. This approach encompasses self-explaining the table's hierarchical structures, code generation to extract relevant information and apply operations, external code execution to prevent hallucinations, and leveraging LLMs' reasoning for final answer derivation. Empirical results indicate that our method, based on GPT-4, outperforms state-of-the-art fine-tuning methods with a 44.38 Exact Match improvement. Furthermore, we present F\({}^{3}\), an adaptive algorithm designed for token-limited scenarios, effectively condensing large tables while maintaining useful information. Our experiments prove its efficiency, enabling the processing of large tables even with models having limited context lengths. The code is available at [https://github.com/zzh=SJTU/E5-Hierarchical-Table-Analysis](https://github.com/zzh=SJTU/E5-Hierarchical-Table-Analysis).

+
Footnote †: FOOTNOTE:10]Footnote *: Work done during Zhehao Zhang’s internship at Microsoft Research Asia.

_Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 1244–1258

June 16-21, 2024 02024 Association for Computational Linguistics

## 1 Introduction

In the digital era, structured data, especially in the form of hierarchical tables, plays a crucial role in information representation. Unlike flat tables, hierarchical ones utilize multi-level headers Lim and Ng (1999); Chen and Cafarella (2014); Wang et al. (2021), offering a nuanced organization of complex datasets. They're widely used in areas like finance, scientific research, and business structures. Three main challenges with hierarchical table analysis are: (1) Complex Structure: Their multi-level, bi-dimensional indexing is intuitive for humans but confusing for models. (2) Implicit Calculation Relationships: They often have aggregated rows and columns without obvious indications. (3) Implicit Semantic Relationships: Hierarchical tables contain numerous cross-row, cross-column, and cross-level relationships that aren't clearly indicated Cheng et al. (2022).

As large language models (LLMs) Brown et al. (2020); Chowdhery et al. (2022); Chung et al. (2022); Black et al. (2022); Biderman et al. (2023); Touvron et al. (2023); OpenAI (2023a); Touvron et al. (2023b); OpeniAl (2023); Roziere et al. (2023) have exhibited significant progress in diverse NLP tasks Qin et al. (2023); Ziems et al. (2024), recent works Chen (2023); Ye et al. (2023); Jiang et al. (2023) start to apply LLMs on flat table analysis based on the models' in-context learning ability using few-shot examplers. Ye et al. (2023) introduce the Dater framework, which seeks to decompose table reasoning tasks using Codex Chen et al. (2021). Besides, Jiang et al. (2023) presents StructGPT, which utilizes an invoking-linearization-generation approach to apply LLMs on structured data. However, when transitioning to hierarchical table analysis, these methodologies encounter specific constraints: (1) these methods depend heavily on manually curated in-context exemplars, which are extremely **expensive** and inflexible for generalization (2) Given the substantial size of some hierarchical tables, directly inputting them as exemplars for in-context learning can surpass the model's token capacity, making these methods **infeasible**. In light of these challenges, our study emphasizes **zero-shot learning** without the need for exemplars, aiming for a more efficient and universally applicable approach across diverse datasets. To the best of our knowledge, we are the **first** to investigate hierarchical table

analysis using LLMs and study table analysis in a zero-shot manner.

In this work, we propose E\({}^{5}\), a code-augmented framework for hierarchical table question answering (QA) tasks in a zero-shot manner using LLMs. We formulate the process of solving this task as the following steps: (1) self-**Explaining** the hierarchical structures of the complex table to enhance LLMs' comprehension of the underlying semantic associations between headers (2) Generating code to **Extract** related rows and columns and apply appropriate logic operations (_e.g.,_ filter, sort, etc) (3) using external code interpreter to **Execute** the generated code and **Exhibit** observations to mitigate hallucinations (4) **Extrapolating** the observations to get the final answer, leveraging the reasoning capabilities of LLMs via natural language. Experiment results on benchmark datasets show that based on GPT-4, our proposal surpasses state-of-the-art (**SOTA**) fine-tuning methods by a significant margin (**44.38** on Exact Match). Besides, compared with another tool-augmented LLM agent, ReAct Yao et al. (2022), E\({}^{5}\)achieves noticeable improvement on multiple metrics with a significant increase in **efficiency** and **stability**.

To additionally study the practical token-limited scenario, we propose F\({}^{3}\), an adaptive algorithm to condense large tables while keeping as much useful information as possible within the constraints of the token length. Comprehensive experiments prove that not only does F\({}^{3}\)save computational resources while maintaining decent performances but also makes it possible to process large tables using models with relatively small context lengths.

## 2 Related Works

### Tool-Augmented LM

To address complex tasks and mitigate hallucinations, advanced external tools such as web browsers, calculators, search engines, Python interpreters, and diverse modality models have been integrated into LLMs Nakano et al. (2021); Shuster et al. (2022); Cheng et al. (2022b); Chen et al. (2022); Cobbe et al. (2021); Paranjape et al. (2023); Shen et al. (2023); Lu et al. (2024); Chern et al. (2023); Mialon et al. (2023); Song et al. (2023); Tang et al. (2023). Toolformer Schick et al. (2023) trains LLMs to use multiple tools in a self-supervised manner. However, it must fine-tune the parameters of the LLM, making it impossible to apply it to close-sourced LLMs like GPT-4. Yao et al. (2022) propose ReAct Yao et al. (2022), a prompt-based paradigm for integrating reasoning and acting for LLMs, which can be effectively applied to various NLP tasks. However, it heavily relies on meticulously curated exemplars of \((Thought,Act,Obs)\) triplets, making the method resource-intensive and constraining its adaptability across diverse tasks.

### LMs for Table Analysis

Numerous studies Yin et al. (2020); Herzig et al. (2020); Liu et al. (2021); Deng et al. (2022) have pre-trained Language Models (LMs) (such as BART Lewis et al. (2020)) on combined tabular and textual data to learn integrated representation, which can generally obtain the best-known performance on table-associated tasks Chen (2023). Chen (2023) first investigate GPT-3's Brown et al. (2020) in-context leaning ability over tabular data. Zhang et al. (2023) introduce CRT-QA, the first TableQA dataset featuring complex reasoning questions over flat tables, and propose an effective code-augmented method named ARC to solve this task.Ye et al. (2023) propose the Dater framework to decompose table reasoning tasks using Codex Chen et al. (2021). Jiang et al. (2023) propose StructGPT, employing an invoking-linearization-generation procedure to apply LLMs on structured data. However, these methods depend heavily on manually created in-context exemplars, which is extremely **expensive** or even **impractical** when the table is extremely large. To the best of our knowledge, we are the **first** to investigate large hierarchical table analysis using LLMs and study table QA in a zero-shot setting.

## 3 Task Description

The hierarchical table analysis task is a variant of the table-based QA problem, specifically tailored for hierarchical tables. Formally, the task involves processing a hierarchical table \(T\) with multi-level headers, a query \(Q\), and a prompt \(P\) that contextualizes the query, through a model \(M\). The model outputs an answer \(A\), succinctly represented by the equation: \(A=M(T,Q,P)\).

## 4 Method

As shown in Figure1, we propose E\({}^{5}\), a code-augmented pipeline for question answering over large hierarchical tables. We first let an LLM self-explain table structures to understand hierarchies. Then, the LLM generates code to extract relevant 

table parts and apply operations, making it focus on useful information. An external interpreter then executes the code to avoid hallucination, with results analyzed by the LLM itself. We also propose F\({}^{3}\), an adaptive algorithm to compress large tables in a practical context-limited scenario, which can be applied before E\({}^{5}\)to compress large tables while keeping as much useful information as possible.

### E\({}^{5}\): Explain, Extract, Execute, Exhibit and Extrapolate

ExplainInspired by CoT [26, 10] and ReAct [47], we make the LLM first explain the structure of the hierarchical table by itself. To be specific, we use the following instructions:

```
Prompt for Explain & Table Structure: you should describe the table in detail including different levels of headers and their meanings. In the end, you should clearly specify which columns AND rows and their corresponding levels are related to the question.
```

In this way, the LLM can better understand the table structure, particularly emphasizing the hierarchical header configurations and the latent semantic associations therein. This understanding provides helpful guidance to the subsequent stage of _Extract_. We independently analyze the effectiveness of _Explain_ in the main experiment, error analysis (Section 7), and a case study (Section 8).

Extract, Execute and ExhibitPrevious works [29] find that LLMs are susceptible to distractions from irrelevant information. Meanwhile, the majority of the hierarchical table analysis queries only focus on specific segments of the huge table. In response to this observation, we explicitly ask the LLM to extract relevant cells from the original table based on its internal knowledge and the headers delineated in the _Explain_ phase. The _Extract_ procedure is integrated into code generation, utilizing Python Pandas Dataframe as the container to archive the relevant data. Thereafter, the LLM can devise a range of operations (_e.g._ filter, sort, group) to manipulate the procured Dataframe.

During the code generation process, the LLM is explicitly asked to print final or intermediary outcomes(_e.g.,_ sorted rows) through instructions. This ensures the acquisition of such displayed output (_Exhibit_) for subsequent scrutiny, post-execution by an external code interpreter (_Execute_). The rationale behind employing an external code interpreter lies in reducing hallucinations in the process of generating the code output by the LLM itself.

Figure 1: An illustration of our proposed framework of E\({}^{5}\)and F\({}^{3}\). E\({}^{5}\)is a pipeline where an LLM explains table hierarchies, generates code to extract relevant data, and then interprets the results post-execution, optimizing question answering. In scenarios with token limitations (dashed line), we first apply F\({}^{3}\)to compress the large table while maintaining as useful information as possible before feeding to E\({}^{5}\).

 

ExtrapolateDuring the trial experiments, we observed that LLMs sometimes generate intermediate results instead of the final answer when executing their generated code. Consequently, we introduced an _Extrapolate_ phase to further analyze the printed outcomes and then generate the final prediction. This stage makes our framework more flexible and able to be applied to other programming languages like SQL, which does not support plenty of logic beyond its database functionalities. Furthermore, the _Extrapolate_ stage can combine the reasoning ability of LLMs through both code and natural language in order to perform more complex reasoning tasks. The complete prompt design of E\({}^{5}\)can be found in Appendix D.

### F3: Find, Filter and Fill

Original E\({}^{5}\)and all previous table analysis methods Ye et al. (2023); Jiang et al. (2023) take the whole table as part of the input to LLMs, necessitating a context length that exceeds the table's length. These approaches become infeasible when handling large tables, such as a century-long sales report of a company, or when the available models possess a limited context length. Furthermore, the substantial token count introduced to LLMs can escalate computational costs and augment carbon footprints. Consequently, we introduce F\({}^{3}\), designed to significantly reduce the input token length while trying to retain maximal pertinent information through the subsequent procedures.

Find and FilterIn most cases, cells other than headers occupy a significant portion of the table's space and do not contain much semantic information except presenting values. Namely, altering these cells does not necessitate a change in the way to address a specific query in most cases. Consequently, we input only the table headers to the LLM and instruct it to identify the pertinent rows and columns (_Find_). After that, we use a rule-based function to filter the original table and only keep the identified cells.

FillHowever, as all our prompt designs are in a zero-shot manner, even GPT-4, the most powerful LLM, can fail to accurately identify the relevant headers, potentially omitting crucial cell values related to the specified query. As a result, we propose an adaptive algorithm to integrate cells from the original table into the filtered version without using LLMs. Our proposed _Fill_ algorithm tries to preserve as many cell values as possible, adhering to the context length constraints of the LLM.

To be specific, we iterate all the cells identified in the stage of _Find_ and first add all cells from the original table in the same row and column under the assumption that cells in the same row or column may have more similarity. For example, the cells in this column of _women_ in Figure 1 table are all statistics about females. After that, we design three different strategies (as shown in Figure 2) to further add potentially useful cells. The row-wise (Figure 1(a)) and column-wise (Figure 1(b)) expansion strategy iteratively add neighboring rows and columns respectively. The spiral expansion strategy (Figure 1(c)) uses a spiral pattern to expand the extracted cells in all directions. Each strategy greedily expands until the maximum token limit is reached or complete filling all the cell values.

Although the _Fill_ procedure may bring in unrelated cells that may distract LLMs, this can be mitigated by the _Extract_ stage in E\({}^{5}\)to a certain degree. Besides, we empirically find that the effect of distraction is considerably less than the loss of useful information (described in Figure 3). The complete F\({}^{3}\)is illustrated in Algorithm 1.

## 5 Experiment Settings

DatasetAs we focus on large hierarchical table analysis, we use HiTab Cheng et al. (2022) as the primary dataset. HiTab is the only public hierarchical table QA dataset containing complex hierarchical indexing. In contrast to other table QA datasets Pasupat and Liang (2015); Chen et al. (2020); Chen et al. (2020); Pashar 

with authentic data analysis scenarios in the industry. We use the test set of HiTab, which has 1584 (table, query) pairs. For table input format, we employ HTML due to its prevalent usage online and its capability to effectively display hierarchical table architectures. To demonstrate the efficacy and adaptability of our proposed approach, we subsequently evaluate E5on an additional two prevalent table QA datasets, WikiTableQuestions Pasupat and Liang (2015) and TabFact Chen et al. (2020), as detailed in Appendix A.1.

BaselinesFor comparative analysis, we select the following methods as our baselines1:

Footnote 1: All the prompt design can be found in Appendix D

* **Zero-shot prompting**Brown et al. (2020): simply prompts LLMs with the given table and query along with instructions.
* **Zero-shot Chain-of-Thought (CoT)**Kojima et al. (2022): adds _Let's think step by step_ in the end of Zero-shot prompting.
* **ReAct**Yao et al. (2022): employs specific instructions to guide LLMs in producing \((Thought,Act,Obs)\) tuples, facilitating the combination of logical reasoning with actions. Although the original ReAct requires multiple few-shot exemplars, we use the modified zero-shot version from LangChain Chase (2023).
* **E\({}^{4}\)****(Ours)**: adopts the procedure of _Extract, Execute, Exhibit and Extrapolate_ described in Section 4.1.
* **E\({}^{5}\)****(Ours)**: adds the step of _Explain_ prior to E\({}^{4}\), instructing LLMs to first describe the hierarchical structure and identify relevant headers.

All prompting baselines are built upon GPT-4-32k OpenAI (2023b) accessed via the Azure API with the API base of 2023-05-15 for main experiments. We also compare with other LLMs with a smaller context length in Appendix A.2. In addition to LLM-based prompting methods, we also present the results of previous SOTA finetuning-based methods including table pre-trained TaPas Herzig et al. (2020) and Neural Symbolic Machine (NSM) with MAPO Liang et al. (2018). Implementation details can be found in Appendix C.

MetricsWe use the following two metrics to evaluate hierarchical table QA systems:

* **Exact Match (EM)**: Following previous works Chen (2023), we use EM score as the main metric for evaluations. EM is a common metric used in various fields such as information retrieval, machine reading comprehension, and question answering to measure how often the output from a system exactly matches the expected or ground truth value (ranging from 0 to 1). However, EM can not correctly identify semantic equivalence on the prediction and gold label (_e.g.,_ "Bhimrao Ramji Ambedkar" and "B. R. Ambedkar.") Kamalloo et al. (2023).
* **GPT-4-eval**: Inspired by the finding that LLMs have the potential to evaluate NLP models Chiang and Lee (2023); Kamalloo et al. (2023), we use GPT-4 as the evaluator to justify the correctness of predictions. We use the following prompt to input GPT-4: [ 

Token-limited ScenarioIn order to simulate conditions where the token capacity of the LLM is relatively constrained, we deliberately set a limit on the maximum token length, ranging from 2,000 to 6,000. This range is compatible with numerous LLMs, including LLaMA Touvron et al. (2023), Pythia Biderman et al. (2023) and so on. This maximum token length serves as a parameter in our proposed adaptive algorithm. In this token-limited setting, we first apply F\({}^{3}\)to get filtered-and-filled tables and subsequently feed them into E\({}^{5}\).

## 6 Experiment Results

Table 1 presents the primary experimental findings on HiTab. The observations are as follows: (1). Compared with earlier SOTA finetuning-based methods, GPT-4 with instructional prompts can substantially increase the accuracy on HiTab even in a zero-shot manner. This suggests that prompting LLMs has potential superior efficacy in hierarchical table question-answering tasks. (2). Zero-shot CoT brings about a subtle increase in EM score and a noticeable improvement on GPT-4 Eval. (3). Among all the GPT-4-based methods, those augmented with tools consistently surpass naive prompting methods, indicating that incorporating external code interpreter can effectively help LLMs for table analysis by reducing hallucinations. (4). Our proposed E\({}^{5}\)achieves the best performances on both metrics. Between our proposals, E\({}^{5}\)outperforms E\({}^{4}\) on both metrics by a noticeable margin. This combined with the case study in Figure 8 denotes that the stage of _Explain_ is useful to the system by helping the LLM better understand the tables' hierarchical structures and implicit semantic relationships.

Comparison with ReActFrom Table 1, it is evident that E\({}^{5}\)surpasses ReAct on both metrics. Furthermore, during the experimentation phase, we empirically find that ReAct requires a significantly higher number of iterations for the \((Thought,Act,Obs)\) cycle, which consequently leads to an increased time and resource expenditure. To provide a more holistic comparison, we analyze the performance variation of both ReAct and E\({}^{5}\)in relation to the maximum iteration limit. As illustrated in Figure 3, when the maximum number of iterations is reduced, E\({}^{5}\)exhibits a more marginal performance decline compared to ReAct. The disparity in performance between the two methods becomes more pronounced as the maximum number of iterations is reduced. Specifically, when the maximum number of iterations is limited to 2, this discrepancy is most evident. Under this condition, E\({}^{5}\)demonstrates decent performances on both evaluation metrics, achieving scores of **74.81** on EM and **82.95** on GPT-4-eval. Notably, the variance for E\({}^{5}\)(9.92) is substantially lower than that of ReAct (29.99), which indicates superior stability

Figure 4: Performances of three _Fill_ strategies’ with changing token limits. All three strategies achieve decent performances when the input token length is constrained. Among the three strategies, the row-wise diffusion strategy outperforms the others.

\begin{table}
\begin{tabular}{l|c c} \hline \hline
* \\ NSM w/ MAPO & 40.70 &
- \\ _Prompting using GPT-4_ & \\ Zero-shot & 78.56 & 87.84 \\ Zero-shot CoT & 78.74 & 91.36 \\ ReAct & 81.87 & 91.50 \\ \(\mathbf{E}^{4}\textbf{(Ours)}\) & 83.19 & 92.89 \\ \(\mathbf{E}^{5}\textbf{(Ours)}\) & **85.08** & **93.11** \\ \hline \end{tabular}
\end{table}
Table 1: The experiment results on HiTab. The mean \(p\)-values for the paired \(t\)-test between E\({}^{5}\)and other top-performing baselines is 0.038, indicating significant differences. We report the results averaged over three runs.

Figure 3: Performance variation of E\({}^{5}\)and ReAct in relation to maximum iteration limit. E\({}^{5}\)has more stable performances than ReAct on both metrics.

 

of E\({}^{5}\)concerning the maximum iteration number. Additionally, in the course of our experiments, we observe that in the absence of instructions to extract pertinent cells using Pandas, ReAct frequently attempts to load the complete table via multiple tools, leading to recurrent syntax errors.

Token-limited ScenarioFigure 4 illustrates the performance of three _Fill_ strategies in relation to the input token length limit. We find that GPT-4 can still achieve decent results even with token limit constraints for hierarchical table analysis, indicating the effectiveness of our proposed F\({}^{3}\). As the token limit increases from 2000 to 6000, all three strategies have considerable performance improvements. notably, the row-wise diffusion strategy outperforms the others across both evaluation metrics.

## 7 Error Analysis

Besides the main experiment, we also conduct a comprehensive error analysis of our proposals. Specifically, we run E\({}^{4}\) ten times, selecting instances with at least one recorded error. These errors were then classified into the following four categories and visualized in Figure 6:

* **Semantic Relationship Errors:** The LLM struggles to discern implicit semantic relationships between hierarchical headers, as illustrated in Figure 7.
* **Extraction Errors:** The LLM is occasionally unsuccessful in extracting relevant segments of the table to address the query.
* **Calculation Relationship Errors:** In instances where computation is required, the LLM fails to accurately determine the computational relationships between entities.
* **Grounding Errors:** The LLM exhibits challenges in aligning the intent of the query with the corresponding operations on tables.

To further investigate how well E\({}^{5}\)can address these error cases through _Explain_ phase, we separately evaluate E\({}^{4}\) and E\({}^{5}\)on them, repeating the experiment ten times. We visualize the average EM score and GPT-4-eval over ten runs in Figure 5. From this figure, it is evident that E\({}^{5}\)exhibits considerable improvement across all four error categories, with a particularly notable enhancement in the semantic relationship category (an increase of over 50%). These findings suggest that LLM can significantly benefit from self-explaining the hierarchical table structure in our tasks.

[FIGURE

Figure 5: Comparative analysis of E\({}^{5}\)and E\({}^{4}\) performance across error categories. We report the averaged results over ten runs. E\({}^{5}\)outperforms E\({}^{4}\) in all error categories by considerable margins, indicating the effectiveness of self-explaining tables’ hierarchical structures.

Figure 6: Proportional distribution of four categories of error cases generated by E\({}^{4}\)





## 8 Case Study

In order to qualitatively investigate how the stage of self-**Explaining** helps the E\({}^{5}\)system, we conduct a case study to compare three different tool-augmented approaches, ReAct, E\({}^{4}\), and E\({}^{5}\). Based on the table and query depicted in Figure 1, the reasoning trajectories of ReAct, E\({}^{4}\), and E\({}^{5}\)are illustrated in Figure 7. From this example, we observe that both ReAct and E\({}^{4}\) simply retrieve all row headers in the given table and neglect the underlying semantic correlations among them, leading to an incorrect answer. Although it is intuitive for humans that the row labeled _Total_ represents the cumulative sum of preceding rows, both ReAct and E\({}^{4}\), employing GPT-4, fail to recognize this, mistakenly interpreting _Total_ as a separate caregiver category. On the contrary, as highlighted in the paragraph of _Table Structure_, E\({}^{5}\)accurately identifies the appropriate caregiver categories, excluding the row labeled _Total_ in the generated code. This suggests that integrating a phase for self-explaining table structures can enhance the capacity of LLMs to discern implicit relationships within hierarchical headers. Besides, in this case, ReAct does not explicitly print its output, resulting in an absence of observation. Consequently, the LLM produces an inaccurate answer, a phenomenon referred to as "hallucination." Such outcomes suggest that merely instructing LLMs to use tools may not be sufficiently effective for tool-augmented agents. Alternatively, prompting LLMs to appropriately use tools in a structured manner proves to be of paramount importance and the design of such prompts should be meticulously tailored to each distinct task.

## 9 Conclusion and Future work

We introduce E\({}^{5}\), a code-augmented pipeline for zero-shot question answering over large hierarchical tables. Through comprehensive experiments, we empirically demonstrate that built upon GPT-4, E\({}^{5}\)can significantly outperform previous SOTA finetuning-based methods by a substantial margin (44.38 on Exact Match) and other prompting strategies with superior stability and adaptability. We separately analyze the phase of self-explaining through extensive error analysis and case study,

Figure 7: The complete reasoning path and generated code of ReAct, E\({}^{4}\) and E\({}^{5}\)for the (Table, Query) pair in Figure 1. The highlighted segments are related to how different methods’ reasoning over the semantic relationship between the ‘total’ row and the remaining rows. Compared with ReAct and E\({}^{4}\), E\({}^{5}\)exhibits an enhanced ability to understand the implicit relationship between headers through self-explaining. In this particular instance, E\({}^{5}\)can recognize that _Total_ is not a category of caregiver and generate the correct answer.



proving its effectiveness in helping LLMs get a deeper insight into hierarchical table structures. We also investigate the practical token-limited scenario and propose an adaptive algorithm, F\({}^{3}\), making it possible to analyze large tables using models even with models constrained by a limited context length. We believe that our contributions hold substantial promise for practical applications in real-world table analysis, such as financial statements, in the industry.

## Limitations

We identify two limitations in our work that need further investigation and improvement. (1) As our research focuses on leveraging LLMs' hierarchical table analysis ability, we only try three strategies (illustrated in Figure 2) in the _Fill_ stage of F\({}^{3}\). Other similarity-based approaches such as clustering or embedding-based search might also be applicable in this context. We leave the exploration of more _Fill_ strategies as future works. (2) In this work, as HiTab is the only public hierarchical table QA dataset comprising exclusively single-table analysis data, we do not consider hierarchical table analysis spanning multiple tables.

## References

* Workshop on Challenges & Perspectives in Creating Large Language Models_, pages 95-136, virtual+Dublin. Association for Computational Linguistics. Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _Advances in neural information processing systems_, 33:1877-1901. Chase [2023] Harrison Chase. 2023. Langchain. Accessed: 2023-07-17. Chen et al. [2021a] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021a. Evaluating large language models trained on code. _arXiv preprint arXiv:2107.03374_. Chen [2023] Wenhu Chen. 2023. Large language models are few(1)-shot table reasoners. In _Findings of the Association for Computational Linguistics: EACL 2023_, pages 1120-1130, Dubrovnik, Croatia. Association for Computational Linguistics. Chen et al. [2021b] Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Wang, and William W. Cohen. 2021b. Open question answering over tables and text. Chen et al. [2022] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2022. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Chen et al. [2020a] Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkeai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020a. Tabfact: A large-scale dataset for table-based fact verification. Chen et al. [2020b] Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020b. HybridQA: A dataset of multi-hop question answering over tabular and textual data. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 1026-1036, Online. Association for Computational Linguistics. Chen and Cafarella [2014] Zhe Chen and Michael Cafarella. 2014. Integrating spreadsheet data via accurate and low-effort extraction. In _Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_, KDD '14, page 1126-1135, New York, NY, USA. Association for Computing Machinery. Cheng et al. [2022a] Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmezi Zhang. 2022a. HiTab: A hierarchical table dataset for question answering and natural language generation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1094-1110, Dublin, Ireland. Association for Computational Linguistics. Cheng et al. [2022b] Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, et al. 2022b. Binding language models in symbolic languages. In _The Eleventh International Conference on Learning Representations_. Chern et al. [2023] I Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, Pengfei Liu, et al. 2023. Factool: Fractuality detection in generative ai-a tool augmented framework for multi-task and multi-domain scenarios. _arXiv preprint arXiv:2307.13528_
 

Cheng-Han Chiang and Hung-yi Lee. 2023: Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 15607-15631. External Links: Link Cited by: SS4.
* A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsyvashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghemawat, S. DeV, H. Michalewski, X. Garcia, V. Misra, K. Robinson, I. Fedus, D. Zhou, D. Ippolito, D. Luan, H. Lim, B. Zoph, A. Spiridonov, R. Sepassi, D. Johan, S. Agrawal, M. Omernick, A. M. Dai, T. Sakkaramarayana Pilali, M. Pellat, A. Lewkowycz, E. Moreira, R. Child, Q. Polozov, K. Lee, Z. Zhou, Xuezhi Wang, B. Saeta, M. Diaz, O. Firat, M. Catasta, J. Wei, K. Meier-Hellstern, D. Eck, J. Dean, S. Petrov, and N. Fiedel. 2022: Palm: Scaling language modeling with pathways. Cited by: SS4.
* H. Won Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. Shane Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhrey, A. Castro-Ros, M. Pellat, K. Robinson, D. Valter, S. Narang, G. Mishra, A. Yu, V. Zhao, Y. Huang, A. Dai, H. Yu, S. Petrov, E. H. Chi, J. Dean, J. Devlin, A. Roberts, D. Zhou, Q. V. Le, and J. Wei. 2022: Scaling instruction-finetuned language models. Cited by: SS4.
* K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. 2021: Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: SS4.
* X. Deng, H. Sun, A. Lees, Y. Wu, and C. Yu. 2022: Turl: Table understanding through representation learning. ACM SIGMOD Record51 (1), pp. 33-40. Cited by: SS4.
* L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig. 2023: Pal: Program-aided language models. Cited by: SS4.
* J. Herzig, P. Krzysztof Nowak, T. Muller, F. Piccinno, and J. Eisenschlos (2020)TaPas: weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 4320-4333. External Links: Link Cited by: SS4.
* Jinhao Jiang, K. Zhou, Z. Dong, K. Ye, W. Xin Zhao, and Ji-Rong Wen. 2023: StructGpt: a general framework for large language model to reason over structured data. arXiv preprint arXiv:2305.09645. Cited by: SS4.
* E. Kamalloo, N. Dziri, C. Clarke, and D. Rafiei. 2023: Evaluating open-domain question answering in the era of large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 5591-5606. External Links: Link Cited by: SS4.
* T. Kojima, S. Shane Gu, M. Reid, Y. Matsuo, and Y. Iwasawa (2022)Large language models are zero-shot reasoners. Advances in neural information processing systems35, pp. 22219-22213. Cited by: SS4.
* M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer (2020)BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 7871-7880. External Links: Link Cited by: SS4.
* C. Liang, M. Norouzi, J. Berant, Q. V. Le, and N. Lao (2018)Memory augmented policy optimization for program synthesis and semantic parsing. Advances in Neural Information Processing Systems31. Cited by: SS4.
* Seung-Jin Lim and Yi-Kai Ng. (1999)An automated approach for retrieving hierarchical data from html tables. In Proceedings of the Eighth International Conference on Information and Knowledge Management, CIKM '99, New York, NY, USA, pp. 466-474. External Links: ISBN 978-1-4503-6 Cited by: SS4.
* Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou (2021)TapeX: Table pre-training via learning a neural sql executor. In International Conference on Learning Representations, Cited by: SS4.
* P. Lu, B. Peng, H. Cheng, M. Galley, K. Chang, Y. N. Wu, S. Zhu, and J. Gao (2024)Chameleon: plug-and-play compositional reasoning with large language models. Advances in Neural Information Processing Systems36. Cited by: SS4.
* G. Mallon, R. Dessi, M. Lomeli, C. Nampantis, R. Pasunuru, R. Raileanu, B. Roziere, T. Schick, J. Dwivedi-Yu, A. Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom (2023)Augmented language models: a survey.

 

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question-answering with human feedback. _arXiv preprint arXiv:2112.09332_.
* OpenAI (2023a) OpenAI. 2023a. Chatagpt.
* OpenAI (2023b) OpenAI. 2023b. Gpt-4 technical report.
* Paranjape et al. (2023) Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. 2023. Art: Automatic multistep reasoning and tool-use for large language models. _arXiv preprint arXiv:2303.09014_.
* Pasupat and Liang (2015) Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1470-1480, Beijing, China. Association for Computational Linguistics.
* Qin et al. (2023) Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiao Chen, Michiihiro Yasunaga, and Diyi Yang. 2023. Is ChatGPT a general-purpose natural language processing task solver? In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 1339-1384, Singapore. Association for Computational Linguistics.
* Roziere et al. (2023) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jeremy Rapin, et al. 2023. Code llama: Open foundation models for code. _arXiv preprint arXiv:2308.12950_.
* Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. _arXiv preprint arXiv:2302.04761_.
* Shen et al. (2023) Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugging gpt: Solving ai tasks with chatgrpt and its friends in huggingface. _arXiv preprint arXiv:2303.17580_.
* Shi et al. (2023) Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Scharli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In _Proceedings of the 40th International Conference on Machine Learning_, volume 202 of _Proceedings of Machine Learning Research_, pages 31210-31227. PMLR.
* Shuster et al. (2022) Kurt Shuster, Mojtaba Komeili, Leonard Adolphs, Stephen Roller, Arthur Szlam, and Jason Weston. 2022. Language models that seek for knowledge: Modular search & generation for dialogue and prompt completion. _arXiv preprint arXiv:2203.13224_.
* Song et al. (2023) Yifan Song, Weimin Xiong, Dawei Zhu, Cheng Li, Ke Wang, Ye Tian, and Sujian Li. 2023. Restgpt: Connecting large language models with real-world applications via restful apis. _arXiv preprint arXiv:2306.06624_.
* Tang et al. (2023) Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xiameei Han, Qiao Liang, and Le Sun. 2023. Toolapaca: Generalized tool learning for language models with 3000 simulated cases. _arXiv preprint arXiv:2306.05301_.
* Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_.
* Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruit Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Wang et al. (2021) Zhiruo Wang, Haoyu Dong, Ran Jia, Jia Li, Zhiyi Fu, Shi Han, and Dongmei Zhang. 2021. Tutta: Tree-based transformers for generally structured table pretraining. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, KDD '21, page 1780-1790, New York, NY, USA. Association for Computing Machinery.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In _Advances in Neural Information Processing Systems_, volume 35, pages 24824-24837. Curran Associates, Inc.
* Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. In _The Eleventh International Conference on Learning Representations_.
* Ye et al. (2023) Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. 2023. Large language models are versatile decomposers: Decomposing evidence and questions for table-based reasoning. In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR '23, page 174-184, New York, NY, USA. Association for Computing Machinery.
* Yin et al. (2020) Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020. TaBERT: Pretraining for joint understanding of textual and tabular data. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8413-8426, Online. Association for Computational Linguistics 

Zhehao Zhang, Xitao Li, Yan Gao, and Jian-Guang Lou. 2023. CRT-QA: A dataset of complex reasoning question answering over tabular data. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 2131-2153, Singapore. Association for Computational Linguistics.
* Ziems et al. (2024) Caleb Ziems, William Held, Omar Shaikh, Jiaiao Chen, Zhehao Zhang, and Diyi Yang. 2024. Can Large Language Models Transform Computational Social Science? _Computational Linguistics_, pages 1-55.

## Appendix A Ablation Studies

### Experiments on Other Datasets

Although we focus on the analysis of hierarchical tables, to further demonstrate the effectiveness and flexibility of our proposal, we directly apply E\({}^{5}\)on another two commonly-used flat table QA datasets, WikiTableQuestions Pasupat and Liang (2015) and TabFact Chen et al. (2020a) using the exact same prompt design in the main experiments. We follow the exact same experiment setups with Chen (2023) for these two datasets. WikiTableQuestions is characterized by complex questions derived from Wikipedia tables. We evaluate the standard test set, encompassing approximately 4,000 questions. Within this dataset, the evaluation metric employed is the EM score. TabFact, on the other hand, has both simple and complex claims. These claims are annotated by crowd workers using Wikipedia tables as references. The simple subset predominantly lacks higher-order operations like max/min/count and the like. Conversely, the complex subset is saturated with claims that incorporate these higher-order operations. This dataset requires the QA system to determine if the provided claim is true or false based on the table. Our evaluation spans the original test set, which comprises 12,779 examples, and we present the binary classification accuracy for this dataset.

Tables 2 and 3 present the performance metrics of various baselines on WikiTableQuestions and TabFact, respectively. It is evident that our proposed E\({}^{5}\)consistently surpasses prior SOTA table pre-trained models and ReAct, yielding notable enhancements on both datasets. These results indicate the efficacy of our approach and its superior generalization capability across different datasets.

### Experiments with Models with Smaller Context Length

As we use F\({}^{3}\)to compress large hierarchical tables within the input token limit, it is possible to analyze these tables using models with limited context length limit. As a result, we experiment with another three models: text-davinci-002 Brown et al. (2020), text-davinci-003, and GPT-3.5-turbo OpenAI (2023a). These three models' context length limits are all 4,096. Following the practice in the main experiment, we first apply F\({}^{3}\)to get filtered-and-filled tables and subsequently feed them into E\({}^{5}\).

Table 4 shows the experiment results of these three models on HiTab. Combined with the results in Table 1, we have the following observations: (1) Compared with GPT-4, these three models can not reach comparative performances on both metrics, indicating a huge gap in ability between GPT-4 and the other LLMs. (2) Despite their constrained context length, these models still manifest performance metrics in close proximity to the preceding

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Model** & **EM Score** & **GPT-4 Eval** \\ \hline text-davinci-002 & 26.81 & 32.13 \\ text-davinci-003 & 35.23 & 43.11 \\ GPT-3.5-turbo & 36.78 & 41.39 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Experiment results of three models with relatively small context length

\begin{table}
\begin{tabular}{l|c} \hline \hline
**Method** & **Accuracy** \\ \hline _SOTA table pre-trained model with finetuning_ \\ TAPAS Herzig et al. (2020) & 81.00 \\ TAPEX Liu et al. (2021) & 84.20 \\ _Prompting using GPT-4_ \\ ReAct Yao et al. (2022) & 83.66 \\
**E\({}^{5}\)(Ours)** & **88.77** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Experiment results on TabFact 

state-of-the-art (SOTA) table pre-trained models. This suggests that F\({}^{3}\), facilitating extensive table analysis even with models of limited context length, possesses significant promise for this task.

## Appendix B Token Length Distribution

To count the tables' number of tokens in HiTab, we tokenize them using the same tokenizer as OpenAI LLMs with tiktoken library. We then visualize the token length distribution of HiTab datasets' tables in Figure 8. From this figure, we find that there are a proportion of tables in HiTab that contain more than 3,000 tokens. Besides, we also need to reserve enough token length for prompt instructions and model generation. As a result, it is impossible to input these large tables entirely to some LLMs with a smaller context length. To solve such issues, the stage of _Filter_ in F\({}^{3}\)can compress 93.28% of tables with more than 3,000 tokens to within 3,000. From Table 8, we can also conclude that as many hierarchical tables already exceed 3000 tokens, adding these tables with additional prompts is very likely to surpass the capacity of most LLMs, including widely used models like vanilla GPT-3.5 and GPT-4, as well as open-source models such as LLaMA, which have a limited context length of several thousand tokens. While our F3 algorithm can compress table contexts effectively, the addition of instructional exemplars, which encompass instructions and codes, is impractical for many scenarios due to these context limitations. This also makes our zero-shot setting more practical and well-motivated.

## Appendix C Implementation Details

To balance the randomness and creativity of LLMs' generations, we set the temperature of the LLMs to a small value of 0.3 for experiments. We repeat every experiment three times and report the average performances in the tables. As we find some of the answers in Hitab are incorrectly annotated during the process of error analysis, we filter them out (172 in total) manually before all experiments. We randomly sampled 100 instances for human evaluation, and a difference of less than 1% indicates that GPT-4-eval is effective in our task.

## Appendix D Propmt Design

The complete prompt designs for all prompting baselines are presented in Table 7. Table 5 and Table 6 shows the prompt design of _Find_ in F\({}^{3}\)and GPT-4-eval respectively.

 

* [145] M. C. K. McKean, J. R. Friedman, and A. G. Moore. The problem of finding a set of points in the set of points. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 37-68, 2009.
* [146] D. L. Miller, S. P. Murray, and N. W. Smith. Finding a set of points in the set of points. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [147] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [148] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [149] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [150] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [151] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [152] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [153] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [154] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [155] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [156] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [157] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [158] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [159] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [160] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [161] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [162] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [163] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [164] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [165] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [166] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [167] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [168] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [169] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [170] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [171] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [172] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [173] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [174] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [175] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [176] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [177] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [178] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [179] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [180] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [181] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [182] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [183] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [184] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [185] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [186] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [187] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [188] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.
* [189] H. T. Nguyen, Y. Zhang, and X. Wang. An efficient algorithm for finding a set of points in the set of points. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2009.

 

Table: Answer the question based on the following html of a hierarchical table Title: Table_title

Table: Html,table Question: Question: Question: Answer: []

Table: Answer the question based on the following html of a hierarchical table Title: Table_title

Table: Html,table Question: Explanation: Let's think step-by-step and output the final answer in the end.

Reace Instruction

Answer the following questions based on the html table. You have access to the following tools: python_repl

Use the following format: Question: The input question you must answer Thoughti; you should always think about what to do Action: the action to take, should be one of [python_repl] Action Input: the input to the action (python code) ... (this Thoughti/Action/ActionInput/Observation can repeat N times) Thoughti: I now know the final answer Final Answer I now know the final answer Begin

Title: Table_title

Table: Html,table Question: Question:

*

Instruction

Answer the following questions based on the html table. You have access to the following tools: python_rep1

Use the following format: Question: the input question your must answer Thoughti; you should always think about what to do Action: the action to take, should be one of [python_repl] Action Input: the input to the action (python code), you should first use pandas to first create # dataframe with relevant data and write code to accollollish the goal. To accomplish that, you should not load the raw HTML data. Instead, you should use df=pd.DataFrame(dict) as the following: df=pd.DataFrame(colum2:[],column2:[])# dict's keys are related column names and values are cell values. Note that you should not load the entire table (such as pd.read_html) and only load the related part. ... print(...)# Finally, you MUSE explicitly print the final result. For example: print(df) ... (this Thoughti/Action/ActionInput/Observation can repeat N times) Thoughti: I now know the final answer Final Answer I now know the final answer

Begin

Table_title

HubL_table Question: Question:

*

\begin{table} \end{tabular}

\end{table}
Table 7: Prompt designs of all prompting baselines in the experiment