Uncovering Limitations of Large Language Models in Information Seeking from Tables (2024)

Abstract: Tables are recognized for their high information density and widespread usage, serving as essential sources of information. Seeking information from tables (TIS) is a crucial capability for Large Language Models (LLMs), serving as the foundation of knowledge-based Q&A systems. However, this field presently suffers from an absence of thorough and reliable evaluation. This paper introduces a more reliable benchmark for Table Information Seeking (TabIS). To avoid the unreliable evaluation caused by text similarity-based metrics, TabIS adopts a single-choice question format (with two options per question) instead of a text generation format. We establish an effective pipeline for generating options, ensuring their difficulty and quality. Experiments conducted on 12 LLMs reveal that while the performance of GPT-4turbo is marginally satisfactory, both other proprietary and open-source models perform inadequately. Further analysis shows that LLMs exhibit a poor understanding of table structures, and struggle to balance between TIS performance and robustness against pseudo-relevant tables (common in retrieval-augmented systems). These findings uncover the limitations and potential challenges of LLMs in seeking information from tables. We release our data and code to facilitate further research in this field.1

heading: 1 Introduction
text: Tables are widespread and rich sources of information across the web and in various documents. Statistics show that the number of tables on internet web pages has reached several hundred million (Lehmberg et al., 2016); in the corporate environment, the number of tables in Excel-like spreadsheet files has exceeded 115 million (Wang et al., 2020). Precisely seeking relevant information from
*Corresponding author: Yixuan Cao and Ping Luo. 1https://github.com/coszero/TabIS
tables is crucial for a wide array of real-world applications, including financial analysis, scientific research, etc. Recently, the remarkable advancements of Large Language Models (LLMs) (Brown et al., 2020; Chowdhery et al., 2022; OpenAI, 2023a; Touvron et al., 2023; Google, 2023) have transformed the approach of information retrieval, moving from fetching specific passages to directly providing answers. However, the effectiveness of LLMs in seeking information from tables remains underexplored.
Some efforts have been made to evaluate the capabilities of LLMs in table information seeking (TIS), but there are unreliable evaluation issues with the used evaluation metrics. Previous studies (Zhao et al., 2023b) mainly use table-totext generation (TTG) as a test bench to assess the TIS abilities of LLMs. TTG aims at transform-
1388
ing complex tabular data into comprehensible descriptions tailored to users’ information seeking needs. The evaluation relies heavily on surfacelevel metrics such as BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004), or on metrics based on model predictions such as NLI-Acc (Chen et al., 2020a). Given that LLM responses can greatly differ in style from reference answers, using these metrics can lead to inconsistent and unreliable evaluations. An example of this issue is illustrated in Figure 1 where a fine-tuned model’s incorrect description receives higher BLEU/ROUGE scores than the correct output from GPT-3.5. This discrepancy may occur because GPT-3.5, without being fine-tuned on this specific dataset, might not mimic the style of the reference response.
To provide a more reliable evaluation, this paper introduces a new benchmark for Table Information Seeking (TabIS). We design our benchmark using a single-choice question format, motivated by popular benchmarks like MMLU (Hendrycks et al., 2020) and BBH (Suzgun et al., 2022), which utilize this format to offer a reliable and widely accepted evaluation of LLMs. We convert TTG datasets like ToTTo (Parikh et al., 2020) and Hitab (Cheng et al., 2022) into this format so that the results can be simply and reliably evaluated. A challenge during curating this benchmark is to generate high-quality options for single-choice questions. Initially, the original data’s answer could serve as the correct option. So we need to generate a deceptive wrong option. If the generated option is too simple, e.g. with obvious logical errors or unrelated to the table content, the benchmark will be too easy and fail to test LLMs’ capabilities. To address this, we devised three prompting-based methods: ModifyInput, Modify-Output, and Exam-Judge (detailed in Section 2.1) for generating wrong options. These methods together produced a variety of deceptive options. The manually verified accuracy rate of our generated data exceeds 92%. We also noted that the Exam-Judge method we proposed generated more challenging questions, which may be used for future dataset construction.
Leveraging the high-quality options, TabIS encompasses three practical scenarios with increasing difficulty for table information seeking: (1) basic TIS derived from TTG (B-TIS), (2) TIS that emphasizes structural understanding (SU-TIS), i.e. when directed to a specific table area with position information (row and column), and (3) TIS from
multiple tables (M-TIS), i.e. when confronted additional pseudo-relevant tables. These scenarios reflect common challenges in real-world applications, such as chatbots and retrieval-augmented systems.
While previous studies (Zhao et al., 2023b) that test on the basic TIS setting with unreliable metrics demonstrate the superiority of LLMs, TabIS reveals the limitations and potential challenges of LLMs in table information seeking as follows.
• Most LLMs show suboptimal TIS performance, especially in complex TIS scenarios and when handling tables with rich hierarchies. Experiments on 12 representative LLMs show that only GPT-4-turbo attained an 85.7% accuracy on average (random guess would be 50% accuracy). The top-performing 70B opensource model achieved 74.4%, with the rest falling in the 50-60% range.
• LLMs exhibit a poor understanding of table structures, with accuracy fluctuating across different cell positions. Surprisingly, we find that LLMs perform almost at random levels in basic lookup tasks, such as repeating content in a specific row. This highlights the substantial challenges in real-world SU-TIS scenarios, where models struggle to pinpoint the target table area using only positional cues.
• LLMs struggle to balance between TIS performance and robustness against pseudorelevant tables, especially for open-source models. This indicates a great challenge for LLMs in retrieval-augmented generation scenarios.
Finally, we fine-tune Llama2-13b-chat on our weakly-supervised training dataset and find that while fine-tuning can significantly improve TIS performance, boosting from 55.5 to 73.2, it still lags behind GPT-4-turbo, which has not been specifically fine-tuned. This indicates that the proposed benchmark is non-trivial, calling for further investigations and improvement in this field.

heading: 2 TabIS Benchmark
text: We curated a benchmark TabIS to investigate the table information seeking capabilities of LLMs.
We use table-to-text generation (TTG) datasets as the original data source in our benchmark. The
task of TTG is that, given a table and a set of selected cells (T,C), produce a one-sentence description of the cells, and the annotated description is called “reference” R. We transform TTG into a single-choice question with two options for objective and accurate evaluation. The format of a sample in TabIS is (T,Q,R,O) where Q is a question, R,O are correct and wrong options. In TabIS, T and R are the same as the annotation in the TTG task, O is a wrong description of the table that we generate, and Q is a question about the table that can be answered by R. So, the task of TabIS is that, given T and Q, choose an option from {R,O} as the answer.
TabIS contains three subsets: basic table information seeking (B-TIS), TIS requiring structure understanding (SU-TIS), and TIS from multiple tables (M-TIS). In the following, we will first introduce how to generate options, and then introduce these subsets respectively.

heading: 2.1 Option Generation Method
text: The option generation has three steps: 1. For each TTG sample, we generate one chal-
lenging candidate option, expecting that the option is unfaithful to the table but is similar to the golden reference. 2. We perform adversarial filtering (Zeng et al., 2023) to divide all instances into easy and hard categories. Specifically, we use three different LLMs on two different presentation orders of the options (R,O and O,R) to obtain six predicted labels. The instances in which the majority of labels are wrong are hard instances and others are easy instances. (Discussed in Appendix A.1.) 3. For hard instances, we conduct manual checking and modification on generated options to ensure correctness.
In step 1, three strategies to generate options are proposed: Modify-Input (MI). We directly prompt GPT4 to first modify the highlighted cells C slightly, resulting in a modified set C ′, and subsequently perform the TTG task using C ′ to produce an unfaithful statement O referring to R. The generated O usually has a similar syntactic structure as R but substitutes some entities. Modify-Output (MO). We directly prompt GPT4 to refer to the golden reference R and make up a new statement that contains highlighted cells C,
but is not faithful to the table fact. Exam-Judge (EJ). Given the table T and a set of cells C, we first instruct a weak LLM agent to describe the cells in natural language, yielding multiple candidate responses {O′1, O′2, . . . }. Subsequently, a more advanced LLM agent2 is employed to identify responses that are unfaithful to the table. Among these unfaithful candidates, the one that is most literally similar to the golden reference R is selected as the wrong option (detailed in Appendix A.2). The underlying idea is to automatically obtain incorrect responses from relatively weak agents, thereby producing strong false options that are diverse and deceptive. In the experiments, we find this method is good at generating difficult instances.
In step 3, for hard instances, we instruct annotators to check if the generated option is faithful to the table. If it is faithful, then it needs to be revised to an unfaithful description while ensuring the altered options are convincingly deceptive.
Finally, each instance can be categorized into four classes, MI, MO, EJ, and HA (HumanAnnotation, i.e. modified in step 3) according to how its O is generated. We put more details of the option generation pipeline in Appendix A.

heading: 2.2 B-TIS Subset
text: B-TIS mimics situations where the LLM agent is tasked with offering clear statements to users who inquire about specific real-world entities, such as celebrities and sports events, based on a table. This method could markedly diminish the necessity for users to sift through massive table data, which is crucial in areas such as sport summary writing and financial data analysis. We show an example in Figure 2.
We apply the aforementioned option generation pipeline to generate the B-TIS dataset using two public TTG datasets: (1) ToTTo (Parikh et al., 2020) is an open-domain English table-totext dataset with over 120,000 examples. The tables in ToTTo are all semi-structured HTML tables from Wikipedia pages and the reference sentences are mainly descriptive statements over the table fact. (2) HiTab (Cheng et al., 2022) is a cross-domain hierarchical table dataset with over 10,000 samples, constructed from a wealth of statistical reports. It contains hierarchical tables and accompanied de-
2We use GPT-3.5-turbo-16k and GPT-4 as the weak and strong LLM agent, respectively.
scriptive sentences collected from StatCan and NSF. Compared to ToTTo, HiTab poses a greater challenge to table information seeking since the tables are with hierarchies and the sentences may involve numerical reasoning (e.g. comparison and simple computation).

heading: 2.3 SU-TIS Subset
text: In LLM-based chat systems like ChatGPT (OpenAI, 2023b), a straightforward way for users to direct the LLM agent to a specific area of a table is by indicating positions (e.g., "row 3"). This requires LLMs to understand table structures. We mimic this scenario by introducing the TIS dataset that emphasizes structural understanding (SU-TIS). For each instance (T,Q,R,O) in B-TIS, we modify question Q by replacing the selected cells with the minimum set of rows or columns covering them, as illustrated in Figure 2.

heading: 2.4 M-TIS Subset
text: In real-world scenarios, LLM agents may be presented with additional context that, while superficially related to the golden table (the table that contains the answer), could be misleading and detrimentally affect their information seeking capabilities (Liu et al., 2023). This situation frequently arises in retrieval-augmented LLM systems oriented to documents, where in response to a query, the systems may retrieve several tables that are relevant to the query but not golden.
To mimic this scenario, we investigate the effects of adding one pseudo-relevant table, which appears relevant to the main table but does not provide useful information to answer the question. We show an example in Figure 2. For each instance in B-TIS, we add another table T ′ to the tuple (T,Q,R,O), resulting in ({T, T ′}, Q,R,O). T ′ is generated by prompting GPT-4 to create one table mirroring the structure and headers of the golden table, yet contains varied data entries. Refer to Appendix B for more details.

heading: 2.5 Dataset Statistics and Quality Assessment
text: Table 1 illustrates the data statistics of the datasets used in our experiments. We show the statistics of the strategies used for generating options in Table 2. We engage 10 sophisticated annotators to meticulously review and revise the hard instances in the test set (step 3 in Section 2.1). Out of 410 reviewed
samples, the options for 193 samples are manually adjusted. We employ two experts to assess the data accuracy on 200 samples each from ToTTo and HiTab. The overall accuracy of ToTTo and HiTab is 94.1% and 92.5%, respectively, demonstrating the high quality of the proposed TabIS.

heading: 3 Experiments on TabIS
text: Based on the curated TabIS benchmark, we evaluate the table information seeking capabilities of 12 representative LLMs.

heading: 3.1 Experimental Settings
text: Problem settings. We evaluate LLMs in a tablebased QA setting, where a linearized markdown table is presented in the context, and LLMs are required to answer a question given the context. All the questions are constructed into the single-choice form with two options, as detailed in Section 2. We use a one-shot example3 to familiarize the model with the task description and answering format, similar to previous work (Wang et al., 2023). Refer to Appendix B for more details.
We evaluate both proprietary and open-source LLMs. To enhance reproducibility, we set the temperature as 0 for proprietary models, and utilize the maximum probability of the first token as A or B to determine the outputs of open-source models. Proprietary models. We adopt three representative models: GPT-3.5 (OpenAI, 2023b), GPT4 (OpenAI, 2023a) and Gemini-pro (Google, 2023). GPTs4 is a series of popular and capable LLM systems developed by OpenAI. Recent studies (Akhtar et al., 2023; Sui et al., 2024; Zhao et al., 2023b) have shown the great potential of
3We find that more examples would often surpass the 4,096 token limit commonly used by open-source models.
4For GPTs, we investigate GPT-3.5-turbo-1106 and GPT4-turbo-1106 for more consistent evaluation. We also report results on GPT-3.5-turbo-instruct and GPT-3.5-turbo-16k, since we find their performance varies greatly.
these models on table-related tasks. Gemini-pro5 is Google’s most capable LLM which operates seamlessly across various modalities. Open-source models. Using proprietary LLM APIs as agents presents many challenges such as high costs and privacy concerns (Zeng et al., 2023). Therefore, we evaluate several popular open-source models: (1) Llama2-chat (Touvron et al., 2023) ranging from 7b to 70b parameters; (2) Mistral7b-instruct-v0.2 (Jiang et al., 2023) and Mixtral8x7b-instruct (Jiang et al., 2024), an instructiontuned sparse mixture of experts language model; (3) TableLlama-7b (Zhang et al., 2023), instructiontuned from Llama2-7b, the first large generalist models for tables; and (4) Tulu2-70b-DPO (Ivison et al., 2023), finetuned from Llama2-70b, the first 70b model aligned with DPO (Rafailov et al., 2023). These models represent the highest-quality LLMs of different architectures and alignment strategies available to the community.

heading: 3.2 Main Results on TabIS
text: We show the results of various models on the test set of TabIS in Table 3. Refer to Appendix D for more details. Overall Performance. As shown in the “Avg.” column in Table 3, both proprietary models and open-source models perform poorly in TabIS. Proprietary models are generally superior to opensource models, with the highest average accuracy recorded at 85.9 by GPT-4-turbo, compared to 74.1 by Tulu2-70b-DPO. Gemini-pro outperforms GPT3.5s but falls short of GPT-4-turbo. Regarding open-source models, a trend is observed where larger models within the same series generally outperform their smaller counterparts. For instance, Llama2-chat models with 7b, 13b, and 70b parameters achieve average accuracies of 50.7, 56.7, and 61.9, respectively. However, this trend does not hold across different model series, where a larger model size does not guarantee superior performance. For example, the 7b version of Mistralinstruct even surpasses the 70b Llama2-chat model by 1.3 points. This observation raises an important question about the impact of pre-training and alignment strategies on the TIS capabilities of LLMs which may be an interesting research topic. Performance on TabIS Subsets. The middle columns in Table 3 show that all models generally perform better in B-TIS compared to SU-TIS
5Gemini-pro is currently accessible via the Gemini API.
and M-TIS, indicating SU-TIS and M-TIS are more challenging. SU-TIS, which only provides the location of highlighted cells as hints, are inherently more difficult than B-TIS. However, models can refer to the cells contained in options to look back at the table to verify each option, therefore the performance drop is not dramatic. M-TIS introduces an extra table that is only seemingly relevant, potentially confusing the judgement of LLMs. In comparisons between datasets, all models show better performance on ToTTo than on HiTab, with improvements ranging from 5.8 to 19.0 points. This discrepancy is likely due to ToTTo predominantly featuring standard tables without merged cells, whereas HiTab includes tables with complex hierarchies, which pose greater challenges for table comprehension.
Comparing option generation strategies. As illustrated in Figure 3, models exhibit the lowest performance with options generated via ExamJudge, with average scores of only 59.2 and 50.6 for ToTTo and HiTab, respectively. This indicates that Exam-Judge is capable of producing options that are even more challenging for LLMs than those annotated by humans. Modify-Input and ModifyOutput also present significant hurdles for LLMs, with scores ranging from 65.7 to 78.3 points on average. For options generated by humans, while they are tough enough, they also lead to high expenses. Our option generation pipeline leverages the advanced instruction-following capabilities of potent LLMs, effectively balancing cost-efficiency with scalability.

heading: 4 Potential Challenges
text: In this section, we conduct in-depth analysis to investigate the LLMs’ limitations and potential challenges behind the two complex subsets: SU-TIS (Section 4.1) and M-TIS (Section 4.2). We further show error analysis on hard samples in Section 4.3.

heading: 4.1 Table Structure Understanding
text: We further investigate the table structure understanding (TSU) capabilities of LLMs, shedding light on future research on the SU-TIS sub-task.
TSU refers to the ability to perceive the twodimensional layout inherent in tables, such as the positioning of cells, rows, and columns, to access desired content based on the location within the table space. TSU is highly important to our SUTIS, which involves locating a specific region of the table. While this may seem intuitive to humans, it can be quite challenging for LLMs, especially be-
cause tables are fed to these models in a serialized format, such as markdown or HTML. To investigate the TSU capabilities of LLMs, we design six basic lookup tasks, such as "What is the content of cells in row 3/column 3?" and "What is the content of cells within the same row as the cell ’Harry Potter’?" We employ predefined templates to generate test samples from semi-structured HTML tables, transforming them into a single-choice format with two options. Each sample includes one in-context example, similar to TabIS. Refer to Appendix C.1 for more details.
Once humans understand the table structure and the task description, their TSU performance ideally remains excellent and consistent regardless of target locations. However, we find that LLMs work in a totally different manner. Specifically, we report the average accuracy on six tasks and the variation score towards target positions in Figure 4. The variation score for a TSU task is defined as the standard deviation in accuracy across different target locations. Notably, most LLMs achieve near-random performance (50) on TSU tasks. The strongest LLM, GPT-4-turbo, exhibits the lowest stability. No LLMs stand out in both performance and stability. Refer to Appendix C.2 for additional analysis.
This highlights a common challenge of table structure understanding: LLMs exhibit poor performance on TSU tasks and the accuracy varies greatly across different positions. In real-world scenarios of SU-TIS, there are no options for a user query. LLMs can only locate the target region based on the positional information (e.g. row 3). The TIS performance would be largely affected by models’ TSU capabilities. We will also release the six TSU datasets to facilitate future research.

heading: 4.2 Robustness against Pseudo-Relevant Tables
text: Based on M-TIS, we further investigate the TIS robustness of various models against pseudo-relevant tables. Specifically, to quantify a model’s robustness, we measure the deviation between the accuracy without and with the pseudo-relevant table, averaged on ToTTo and HiTab. The results are shown in Figure 5. Notably, GPT-3.5-instruct and GPT-4-turbo emerge as both effective and robust. However, the two strongest open-source models, Tulu-70b and Mixtral-7b*8, exhibit the lowest robustness levels. Besides, within the same model
series, larger models achieve better accuracy scores but worse robustness scores. This phenomenon can be observed in Llama2 series (7b, 13b, 70b) and Mistral series (Mistral-7b, Mixtral-8*7b). MTIS indicates great challenges of LLMs in balancing between TIS performance and robustness against pseudo-relevant tables, especially for open-source models. This finding calls for future research on open-source models to improve TIS robustness against pseudo-relevant tables.

heading: 4.3 Error Analysis on Hard Samples
text: To explore why LLMs fall short on TabIS, we conduct further analyses based on the hard set of B-TIS. Specifically, we sample 50 instances from the hard B-TIS set and ask an expert to analyze the reasons why these questions are difficult to answer. Finally, we categorize the main types of difficulties into four categories, as shown in Table 4. We find that current LLMs still make mistakes in distinguishing
subtle details and are more prone to commit errors on options that appear more concrete but contain errors (R1, R3). Additionally, table information seeking often requires numerical reasoning (R2) and common sense knowledge (R4), areas where current LLMs are not yet proficient.

heading: 5 Improving Table Information Seeking
text: In this section, we explore how supervised finetuning enhances table information seeking using weakly-supervised datasets.
We first utilize our proposed data generation pipeline6 (Section 2) to construct weaklysupervised B-TIS and SU-TIS training datasets without manual checking. The statistics of the training dataset are shown in Table 2. We fully finetune Llama2-13b-chat on this training set for 2 epochs to obtain TISLlama. We evaluate TISLlama on TabIS7. Refer to Appendix F for more training details.
Table 5 demonstrates that TISLlama outperforms both the base model Llama2-13b-chat and the leading open-source model Tulu2-70b-DPO, with margins of 17.7 and 5.4 points, respectively. These results demonstrate the effectiveness of TIS-oriented supervised finetuning. However, its performance does not yet match that of GPT-4-turbo, which has not undergone specialized fine-tuning. This discrepancy highlights the significant challenge TabIS presents to large language models, underscoring the need for further research in this area.
6Considering high cost of accessing GPT-4 API, we use GPT-3.5-turbo-16k instead.
7Note that training on the weakly-supervised datasets may introduce the spurious correlation between the modelgenerated options and the wrong options. Thus we only evaluate on human-annotated samples for fair comparision.

heading: 6 Related Work

heading: 6.1 Table-to-Text generation
text: Table-to-Text generation (TTG) aims at generating natural language statements that faithfully describe the information contained in the provided table region. Given its broad applications like biographical data analysis (Lebret et al., 2016) and sports game summary generation (Wiseman et al., 2017), TTG has been studied extensively in recent years (Wang et al., 2022; Zhao et al., 2023a) with the introduction of several valuable datasets (Parikh et al., 2020; Cheng et al., 2022; Chen et al., 2020a). Previous studies mainly focus on finetuning pre-trained language models on a task-specific dataset (Wang et al., 2022), which are often specialized and lack generalizability. Large language models (LLMs) have recently demonstrated remarkable performance on TTG tasks (Yang et al., 2023; Zhao et al., 2023b). However, these evaluations mainly rely on surface-level metrics, such as BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004), which may result in unreliable evaluation when the syntactic style of LLMs’ response diverges from the golden reference (Dhingra et al., 2019). In this paper, we employ the TTG tasks as a test bench for evaluating table information seeking of LLMs. To ensure a reliable assessment, we construct single-choice questions based on two high-quality TTG datasets, ToTTo (Parikh et al., 2020) and HiTab (Cheng et al., 2022).

heading: 6.2 Evaluating Table Information Seeking capabilities of LLMs
text: Short-form table QA datasets, such as WikiSQL (Zhong et al., 2017) and WikiTQ (Pasupat and Liang, 2015), contain queries seeking for information, such as "who is the manufacturer for the order year 1998?". However, these datasets focus on relational tables and evaluate the ability to comprehend table schemas and transform natural language queries into SQL queries, while our work emphasizes grasping the information conveyed by the complex table contents. Besides, Free-form table QA datasets such as FeTaQA (Nan et al., 2021) relies on text-similarity-based metrics leads to unreliable evaluations, a problem exacerbated in the era of LLMs.
Recently, Sui et al. (2024) presents a benchmark designed to measure the structural comprehension of large language models (LLMs) through the comparison of various input approaches. However, it
solely examines the capabilities of the most advanced LLM, GPT-4. Their findings suggest that GPT-4 possesses a fundamental understanding of table structures, yet there’s a noticeable absence of comprehensive evaluations across a wider range of LLMs and an examination of TSU consistency. Zhao et al. (2023b) investigates the potential of applying LLMs in real-world table information seeking scenarios, showcasing their effectiveness in producing faithful statements. Nevertheless, their analysis is significantly influenced by unreliable evaluation metrics.
To the best of our knowledge, we are the first to release a large-scale, comprehensive, reliable benchmark for evaluating TIS capabilities.

heading: 7 Conclusion
text: This paper introduces TabIS, a new benchmark designed to evaluate the table information seeking (TIS) abilities of large language models (LLMs). TabIS is comprised of three typical TIS scenarios and employs a single-choice question format to ensure reliable evaluation. Extensive experiments on 12 representative LLMs have shown that TabIS presents a significant challenge for current LLMs, with GPT-4-turbo showing only marginal satisfaction. Further analysis points out two main issues: firstly, LLMs perform almost randomly on basic tasks involving comprehension of table structures; secondly, they face difficulties in improving performance and maintaining robustness against pseudorelevant tables, which could lead to sub-optimal results in real-world TIS tasks. These observations underscore the current limitations and potential challenages in TIS, calling for further exploration and advancement in this area.

heading: 8 Limitations
text: In this paper, the benchmark adopts the form of single-choice questions, which ensures the reliability of the evaluation but may deviate from practical applications. TabIS is design with only two op-
tions, which may not sufficiently challenge LLMs, particularly when GPT-4 shows high baseline accuracy. The templates used for generating TIS questions are relatively simplistic; richer and more diverse questions would enhance the quality of the benchmark. We use GPT-4 to modify prompts and generate pseudo-relevant tables, which may introduce bias, potentially favoring GPT-series models due to their inherent familiarity with the dataset construction.
Given that the tables originate from Wikipedia, there may be concerns regarding data contamination; LLMs might still perform well without the context provided by the tables. We show a discussion on Appendix G. Besides, we mainly discuss some limitations and potential challenges of LLMs when handling table information seeking tasks, but do not explore how to address these issues or the reasons behind their observations.

heading: Acknowledgements
text: This work has been supported by the National Natural Science Foundation of China (No. 62076231, 62206265), and the China Postdoctoral Science Foundation (No. 2021M703271). We thank all the anonymous reviewers for their valuable and constructive comments.

heading: A Option Generation Details
text: For each TTG sample, we apply one of the three option generation strategies to generate one candidate option. Considering cost and quality, we use GPT3.5-turbo-16k, GPT-3.5-turbo-instruct, and GPT3.5-turbo-1106 to perform the adversarial filtering. We show the prompt of Exam-Judge, Modify-Input, and Modify-Output in Figure 6, Figure 7, and Figure 8, respectively.
A.1 Discussion on Difficulty Division In our experiments, we use adversarial filtering to determine the difficulty of samples. The classification achieved through this process is indeed accurate, with the easy samples being less challenging than the more complex ones. This assertion is supported by two pieces of evidence:
• Model performance. It’s observed that models typically fare much better on easy samples than on hard ones. To demonstrate, we present the performance of various models on the BTIS-ToTTo in Table 6. Both proprietary and open-source models exhibit superior performance on easier samples.
• Expert assessment effort. On average, experts can assess easy samples at twice the rate (32 per hour) of hard samples (14 per hour), highlighting a significant difference in the complexity and time required for evaluation.
A.2 Details on Choosing Option Candidates We simply determine the similarity by comparing the sentence lengths (measured in characters) of the golden reference and the option candidates. The premise is that an option with a length similar to that of the golden reference is likely to be more misleading. Based on our observations, these option candidates tend to be semantically similar but
differently phrased. Therefore, we just choose the option that is closer in length to the standard answer.

heading: B Benchmark Details
text: As described in Section 3.1, each test sample within our benchmark is accompanied by one in-context example. We only keep samples that contain fewer than 4000 tokens when processed by LlamaTokenizer8 (Touvron et al., 2023), in adherence to the common 4096 token limit imposed by most opensource models. SU-TIS. Given a set of row-column coordinates C = {(ri, ci)}|C|i=1 representing highlighted cells in a B-TIS sample, we construct the SU-TIS question using the smaller set between {ri | (ri, ci) ∈ C} and {ci | (ri, ci) ∈ C}. M-TIS. We use GPT-4-turbo-1106 to generate one pseudo-relevant table for each B-TIS sample. We show the prompt for generating pseudo-relevant tables in Figure 9. We randomly placed the noisy table T ′ either before or after the golden table T . In earlier trials, we attempted to retrieve similar tables from the ToTTo training set to serve as pseudorelevant tables. However, the retrieved tables often had weak relevance, likely because the dataset’s various tables are sourced from different HTML documents. Considering the challenges in acquiring noisy tables from realistic retrieval scenarios (e.g., within the same document), we opted to generate noise tables with high similarity to the golden table using LLMs.

heading: C Exploring Table Structure Understanding
text: In this section, we first introduce the construction of TSU dataset, then we show our additional experiments on TSU.
C.1 TSU Dataset Construction Understanding the structure of a table is essential for navigating among data arranged in a tabular format, interpreting the relations among data points, and understanding the table semantics. It requires to perceive the two-dimensional spatial layout inherent in tables, such as the positioning of cells, rows, and columns, to access desired content based on the location within the table space.
8Note that different LLMs may use different tokenizers. Besides, the tokenizers of proprietory models are unaccessible. Therefore, we choose the tokenizer of Llama2, one of the most popular open-source models.
To examine the table structure understanding capabilities of LLMs, we propose six probing tasks: positional cell lookup (PCL), relative cell lookup (RCL), positional row lookup (PRL), relative row lookup (RRL), positional column lookup (PLL), relative column lookup (RLL). These tasks require LLMs to acquire certain surface-level table components (cell, row and column) based on relative or absolute position information.
We generate samples for each task by applying predefined templates on high-quality tables. All question templates are shown in Table 9. We collect tables from four public datasets: WikiSQL (Zhong et al., 2017), WikiTableQuestions (Pasupat and Liang, 2015), HybridQA (Chen et al., 2020b) and FeTaQA (Nan et al., 2021). These tables are all semi-structured HTML tables collected from Wikipedia, spaning a wide array of topics such as sports and geography. After deduplicating these tables, we obtain a total of 49,561 high-quality tables. For the test set, we randomly sample 1% tables and generate one sample per table for each task.
For each sample, the options are generated by randomly sampling cells, rows and columns in proximity to the golden answer, employing a gaussian distribution N (p, 1), where p denotes the position of the golden answer.
C.2 Additional Experiments
We show the TSU performance of various models on Table 10. TSU Performance. Unexpectedly, despite TSU being straightforward for humans, all LLMs demonstrate subpar performance. The best performance of proprietary models and open-source models only achieve 66.1 (GPT-4) and 57.6 points (Tulu2-70b), respectively, while most models achieve near-random performance (50). Models do not consistently excel across all types of TSU tasks. Notably, the GPT series (GPT-4 and GPT-
3.5) tend to perform better in column-oriented tasks (PLL, RLL) relative to other tasks, whereas the Llama2 series (Llama2-7b, 13b, 70b) show greater proficiency in cell-oriented tasks (PCL, RCL). This variation in performance could be attributed to the fact that models within the same series likely undergo similar pre-training and alignment processes, resulting in comparable inductive biases. Case Study on Variations across different positions. As presented in Section 4.1, LLMs exhibit fluctuating performance across different positions. We further show case studies of RCL in Figure 10. For example, Gemini-pro exhibits large variance in different positions,with a disparity exceeding 30 points between its highest and lowest accuracy. Similar patterns are noted in other LLMs. On average, the data indicates that LLMs perform more effectively at the beginning (row 1, column 1) and ending (row 7, column 7) of tables. This pattern is likely influenced by the serialization of tables into one-dimension strings, rendering the middle part of the table more challenging to locate accurately. Effect of Cell Contents on TSU. Logically, carrying out TSU tasks should be independent of the particular content within table cells, since this doesn’t necessitate grasping the table’s underlying semantics. Thus, the performance across tables with varying content should be consistent. To test this, we altered the cell contents in our TSU test set’s real tables to random numbers (ranging from 1 to 8 digits) and random letters (also 1 to 8 characters in length), creating two new synthetic test sets named "letter" and "number."
However, we observe significant variation in performance across different table contents. As shown in Figure 11, the performance disparity be-
tween the test sets ranges from approximately 2.9 to 19.4 points. Intriguingly, GPT-4 shows markedly improved performance on the "number" set. This may be attributed to the activation of GPT-4’s numerical processing capabilities, which are particularly relevant for TSU tasks (e.g. counting rows). This observation warrants further investigation in future studies.
C.3 Summary
These findings show that current LLMs exhibit weak performance in TSU tasks, both in terms of overall performance and consistency across varying positions and cell contents.

heading: D TabIS Main Results
text: We show the main results of B-TIS, SU-TIS, and M-TIS in Table 11, Table 12, and Table 13, respectively. We also report the accuracy on each option generation strategies.
We present some results of statistical significance testing in Table 7. Specifically, we assess whether the test accuracy significantly surpasses random guessing (50% accuracy). All p-values are below 0.05, indicating that the results are statistically significant.

heading: E Training Details
text: We fully fine-tune the model Llama2-13b-chat9 with LlaMA-Factory (hiyouga, 2023). We use a learning rate of 2e-5. We train the model on 8 A800 and use a linear scheduler with a 5% warmup period for 2 epochs. To efficiently train the model, we employ DeepSpeed training with ZeRO3 stage (Rajbhandari et al., 2020). For both training and inference, we set the input length as 4096.
9https://huggingface.co/meta-llama/Llama-2-13b-chat-hf

heading: F Case Study
text: We show examples of four reason types in Figure 12, Figure 13, Figure 14, and Figure 15.

heading: G Discussion on Data Contamination
text: Given that the tables originate from Wikipedia, there may be concerns regarding data contamination; LLMs might still perform well without the context provided by the tables. This issue can be intensive especially when the correct option is factually correct with external world knowledge while the incorrect option is not. To investigate this problem, we conduct experiments to assess how models’ parametric knowledge influence their performance on choosing the correct option. We remove the tables from the test questions, allowing the models to answer based solely on their world knowledge. The results on 500 BIS samples are shown in Table 8. "w. table" denotes the model performance on the original dataset. "w/o. table" denotes performance after table removal. ∆ denotes the performance gain purely from the table understanding.
After removing the table, all models exhibit similar performance, which is nearly at random, with ToTTo slightly higher than HiTab. The ranking of different models based on the delta values is consistent with the ranking using the original dataset. Despite of data contamination, the dataset can authentically reflect the table information seeking capabilities of models.
Page title: Egor Bazin Section Title: Detailed results, with Evdokimova

heading: Future Work
text: This observation warrants further investigation in future studies. C. 3 Summary
These findings show that current LLMs exhibit weak performance in TSU tasks, both in terms of overall performance and consistency across varying positions and cell contents