RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Preliminary
In this section, we introduce the definition of table reasoning tasks. Table reasoning requires reasoning over both free-form natural language(NL) and inherently structured tables. Given the triplet (T,Q,A) , where table T=\{c_{i}\}_{i=1}^{C} , C represents the number of column features in the table. Note that we do not represent the table in cell format as we expect the table under investigation to adhere to certain norms inherently. Q signifies a query or claim related to the table, and A denotes the answer.

We specifically focus on the table question answering and fact verification tasks. In the table question answering tasks, Q and A correspond to the query and expected answers in natural language form, respectively. In the table fact verification task, Q represents a claim about the table, and the final answer A\in\{0,1\} where 0 indicates falsity and 1 indicates truth regarding the input claim.
Overview
In this work, we assume that semi-structured tabular data is rich in latent information beyond its raw data values. This information suggests that data storage adheres to certain common patterns and field semantics, facilitating the model’s inference of the overall data distribution from a minimal sample of data. Inspired by knowledge-fusion models for metadata inference (He et al., 2023 ) and the internal knowledge-retrieving ability of LLMs (Sui et al., 2024 ) , we utilize LLMs to uncover patterns and semantics within tables, which helps to understand and operate data correctly. The whole workflow is illustrated in Algorithm 1 in the appendix. In our framework, we do not include the full content of the table in a prompt; only K rows can be observed. Instead, the reasoning effect is ensured through the elaborate augmented information. The framework seamlessly accommodates large-scale tables, as the model is pre-endowed with comprehensive information about the data structure and content before encountering it.
As illustrated in Figure 1 , our proposed system ALTER , consists of three core components:

\bullet Query Augmentor : This component enhances the original query by generating multiple sub-queries, each examining the original query from different perspectives. Compared to the partial original query, this component comprehensively provides more information through the subsequent table organizer.

\bullet Table Organizer : Given the input query, this component utilizes the augment-filter-execution procedure. It first enriches the raw data with augmented table content, then filters the data to retain only highly relevant rows and columns, and finally employs an SQL executor to derive a reasonable and accessible sub-table for final inference.

\bullet Joint Reasoner : This component efficiently performs reasoning and aggregation for the query augmentor and the primary workflow.
Query Augmentor
One of the primary challenges in naive Question Answering (QA) lies in its direct reliance on the user’s original query as the basis. In tabular reasoning scenarios, an imprudent query can lead to the model focusing on one partially biased part in the table. We propose a novel improvement method for the query part, which enables the LLMs to utilize the multi-query technique to address the original query from multiple perspectives. Each sub-query undergoes the reasoning process via the table organizer module, with this process being conducted in parallel. The model can utilize each independent reasoning module to attend to different parts within the table and extract information pertinent to answering the original query.

We propose two query augmentation methods: step-back augmentation and sub-query augmentation. The step-back prompting method (Zheng et al., 2024) has been empirically validated as effective in the RAG domain. We equip it with sampled sub-table information, which aims to obtain more abstract-level comprehension within the table through query rewriting. LLMs are shown to be stronger at sequentially solving sub-problems than directly solving a complex problem (Zhou et al., 2022a). The sub-query augmentation method decomposes complex queries into sub-queries, enabling LLMs to more easily locate the relevant information within each sub-query. Specifically, we leverage LLMs to generate distinct sub-queries based on the rewrite or decomposition demand. Detailed prompts for both augmentation methods are provided in Appendix F.
Table Organizer
The table organizer is the core component of the entire reasoning process. As previously mentioned, throughout the reasoning process, we do not use the entire table data as contextual information. Instead, we further filter the column features of the table, as detailed in Section 4.3.2 , thereby simplifying the transmitted information. To maintain model performance without accessing full data, we employ the augment-filter-execution strategy. By pre-analyzing and augmenting the table’s schema, semantic, and literal information, sufficient supplementary information required by the query is provided. Given the limited table features, the augmented information does not increase commensurately with the table size. Therefore, our method exhibits strong robustness to variations in table size.

The table organizer primarily encompasses one preparatory stage and three reasoning stages, as illustrated in Figure 2. During the preparatory stage, the table augmentor augments and stores enhanced information at various levels for subsequent processing. Schema information is utilized for standardizing the table content. In stage 1, semantic information is employed to identify relevant columns. Rows are sampled based on semantic similarity. In stage 2, with the filtered sub-table, the augmented information transmitted can be further simplified. We utilize literal and semantic information and leverage text-to-SQL capabilities of LLMs to generate high-quality SQL. The SQL query is executed, and the final sub-table is retrieved.
Column Filter and Row Sample
Irrelevant table content in the prompt can lead to unnecessary computations and quality regression issues (Sui et al., 2023), especially in scenarios involving large tables. We retain a small number of columns and rows from the original table. Unless otherwise specified, we set K = 3 in this paper, meaning the model can only access three rows of data relevant to the question throughout the process. However, through the table augmentor, we can obtain globally enhanced table information. Specifically, we first store the index of the vector representation of each row content, and search for K rows based on embedding-based semantic similarity between each row and the utterance. Subsequently, a powerful LLM is utilized to select columns relevant to the query, excluding irrelevant ones. The prompt for column filtering is detailed in Appendix F. During the column pick, the augmented information is also used for comprehensive understanding. This module ensures that the scale of the sub-table passed to LLMs remains consistent regardless of the size of the original table.
Joint Reasoner
The Joint Reasoner is responsible for integrating upstream information to perform the final reasoning. To avoid interference from entirely irrelevant information, sub-queries that cannot be answered are discarded. Valid sub-queries are transformed into effective descriptions. These demonstrations are combined with the sub-table from the primary workflow to collectively aid in answering the original query. Please refer to Appendix F for further details and more comprehensive prompts. We leverage step-by-step reasoning capabilities of LLMs to arrive at the final answer.
Experiment
In this section, we first introduce the datasets and evaluation metrics. We compare ALTER with the baseline methods and present the results in Sections 5.2 and 5.3. The ablation study and analysis of large-table scenarios are discussed in Sections 5.4 and 5.5, respectively. Additional implementation details are provided in Appendix A.
Datasets and Evaluation Metrics
We evaluate our proposed method on two widely-used table-based reasoning benchmarks, WikiTQ (Pasupat and Liang, 2015 ) and TabFact (Chen et al., 2020a ) .

For the table-based fact verification task, we adopt the TabFact dataset, which contains various statements based on Wikipedia tables. We evaluate the dataset using binary classification accuracy on the small-test set containing 1998 statements with 298 different tables.

For the table reasoning task, we adopt WikiTableQuestion (WikiTQ), which contains open-domain tables accompanied by complex questions. We use denotation accuracy as our evaluation metric, which evaluates the predicted answers based on the gold ones. We evaluate our method on the test set containing 4344 samples from 421 different tables.
Baselines
We compare the proposed ALTER with a range of advanced reasoning frameworks for table-based tasks. The baseline methods for comparison can be categorized into two types: mainstream techniques following the pre-LLM era and techniques unique to the LLM era. For the techniques following the pre-LLM era, we select TAPEX Liu et al. ( 2022 ) , ReasTAP Zhao et al. ( 2022 ) , TaCube Zhou et al. ( 2022b ) , OmniTab Jiang et al. ( 2022 ) , CABINET Patnaik et al. ( 2024 ) . For the techniques unique to the LLM era, we select Binder Cheng et al. ( 2023 ) , Dater Ye et al. ( 2023 ) , ReAcTable Zhang et al. ( 2023 ) , Mix SC Liu et al. ( 2023 ) , Chain-of-Table Wang et al. ( 2024b ) . Additionally, generating multiple reasoning paths and ultimately choosing the most consistent answer through voting or self-consistency Wang et al. ( 2022 ) can enhance the performance of LLMs. Therefore, for the techniques unique to the LLM era, we report two types of results for those methods employing result ensemble techniques.
Results
We present the results on the WikiTQ and TabFact datasets. The experimental outcomes are summarized in Table 1 . From the results, we observe that our ALTER method achieves comparatively outstanding outcomes. Specifically, on the WikiTQ dataset, while the Mix SC method do marginally outperforms our results by aggregating multiple reasoning paths (with 10 sampling times), ALTER still managed to exceed the performance of all other methods under comparison. Notably, ALTER demonstrates the best performance in single-round reasoning among all other methods that utilize result ensemble techniques in the LLM era. This demonstrates the robust performance of our method in reasoning tasks, which can be attributed to the reinforced information provided by the query augmentor and our innovative modular procedure within the table organizer.
Ablation Study
We carry out an ablation study to assess the impact of various components on the performance of our methods, as well as to explore the relationship between the pure table data and the inherent augmentation information.

Analysis of the Query Augmentor .

To analyze the impact of two query augmentation methods in the query augmentor. We conducted experiments on the WikiTQ and TabFact datasets by discarding the step-back augmentation module (denoted as w/o step-back) and the sub-query augmentation module (denoted as w/o sub-query). For each dataset, we further categorized the questions based on the difficulty level, following Ye et al. ( 2023 ) . This stratification facilitates a more comprehensive evaluation of each module’s impact across different types of questions. The ablation test results are reported in Table 2 . From the results in the table, it is anticipated that employing both augmentation methods simultaneously yields the best performance under all experimental settings. For WikiTQ datasets, the accuracy of ALTER without step-back/sub-query augmentation drops by 2.9\%/2.0\% , demonstrating the necessity of augmented information from multi-queries. Furthermore, on the TabFact datasets, both augmentation methods have a much larger impact on hard questions than on simple questions. This indicates that the augmented information provided by the query augmentor is particularly effective in dealing with complex questions.

Analysis of Pure Data & Augmentation .

In our ALTER experiments, we primarily set K=3 , meaning the model can only access three rows of data relevant to the question throughout the process. To explore the relationship between pure table data and the augmented information in the table organizer, we conducted ablation experiments varying the value of K and the augmentation process. Results are shown in Table 3 . We observe that methods utilizing augmented information exhibit significant performance improvements compared to those without augmented information. We also note that the concurrent absence of augmented information and data provision leads to a catastrophic decline in model performance. Notably, on both datasets, using only one row of data with augmented information achieves comparable performance to using three rows of data. Similar trends can also be observed in other settings. This validates that when the model is limited to a small portion of data, the table augmentor serves as a beneficial auxiliary tool, providing additional insights into the table’s content.
Large Table Analysis
LLMs often struggle to interpret tables within largescale scenarios, leading to hallucinations and errors. To the best of our knowledge, nearly all methods encounter a decline in model performance as the table size increases when handling large tables.

Comparison under Large Table Scenarios. To demonstrate the effectiveness of the ALTER frame work in large-scale scenarios, we compare the performance of our framework across different table sizes in this section. We selected various table partitioning principles and different types of methods for a systematic evaluation. For table partitioning, we employed two approaches based on the token count and the number of cells. For the models, representative methods from both the LLM era and the pre-LLM era are chosen.

Figure 3 shows the comparison results of ALTER and methods following the pre-LLM era, including CABINET and OMNITAB, partitioning tables in the WikiTQ dataset by the number of cells. In Table 4 , we present the results based on different table sizes divided by the token count in the WikiTQ dataset, comparing our method with Dater, Chain-of-TABLE, and Binder unique to the LLM era. Table 4 shows that ALTER significantly outperforms all three methods in the LLM era across different table sizes. The performance improvement is particularly noteworthy when dealing with large tables. In Figure 3 , our model demonstrates a much slower performance decline as the model size increases compared to the other two methods. As the size of the table increases, both CABINET and OMNITAB exhibit a monotonous decline in performance. However, our method shows a brief reversal with an increase in performance observed in the intermediate range, indicating the robustness and insensitivity of our approach to changes in table size. Our model significantly outperforms the other two methods when the table size exceeds a certain threshold ( >300 cells). Specifically, in the 300-400 , 400-500 , and 500+ cell categories, our model exceeds their performance by at least 15\% , 19\% , and 25\% , respectively. From the results, it is evident that our method exhibits exceptional performance in large tables.
Robustness and Efficiency Analysis
We examined ALTER’s robustness to noise perturbations and token efficiency in large-scale scenarios. By adding random rows based on different perturbation factors, we introduced noise to each table in WikiTQ, details of perturbations can be found in Appendix C . From Figure 4 , we illustrate that as the degree of perturbation increases, the proportion of tokens utilized of the whole table by ALTER decreases. It can be observed that the initial fluctuation has the most significant effect, yet our model still outperforms the compared method ( 9.8\% ALTER v.s. 11.4\% CABINET). Concurrently, the decline in the framework’s performance degree slows down. This indicates that our method efficiently maintains robust performance in large-table scenarios by narrowing down the scope of larger tables.
Case Study
In Appendix B , we present a case study to elucidate the scenarios in which each component of enhanced information within ALTER can facilitate a more profound comprehension of table contents. When addressing complex problems, without the assistance of the augmentation process, the model may focus on biased information or experience hallucinations when generating SQL. However, when the augmented information is explicitly provided, the model can identify the region containing the correct information or generate syntactically correct SQL, thereby delivering accurate responses.
Conclusion
We propose a framework, namely ALTER, which significantly optimizes model performance on large-scale tables. Within this framework, we extract inherent information pertinent to the questions and tables. By leveraging an augment-filter-execution process as the core reasoning workflow, ALTER demonstrates superior performance in handling large tables. We believe ALTER can bridge the gap between table reasoning methodologies and real-world analysis and bring insights into understanding the way LLMs comprehend tables.
Limitations
ALTER is designed to generalize to large table reasoning tasks, but our method still faces some limitations. Our approach relies partly on the degree of structured and standardized storage of tables, meaning that if the table structure is totally disordered or lacks a certain level of standardization, our model’s performance will degrade, for instance, when headers and data are intermixed. Additionally, the combination methods of different augmented information can be explored further. Due to the page limits, we will leave these explorations for future work.
Figure 1: The overview of the ALTER framework for table-based reasoning. The gray background box symbolizes the primary reasoning workflow. Above it, each sub-query generated by the query augmentor is processed in parallel by the table organizer and ultimately transformed into informative demonstrations that aid in understanding the original query. The primary sub-table and relevant information is received by the joint reasoner.; The overview of the ALTER framework for table-based reasoning. The gray background box symbolizes the primary reasoning workflow. Above it, each sub-query generated by the query augmentor is processed in parallel by the table organizer and ultimately transformed into informative demonstrations that aid in understanding the original query. The primary sub-table and relevant information is received by the joint reasoner.
Figure 2: Illustration of the table organizer inside. The augmented information from the table augmentor is utilized in stage 1 and stage 2, enabling the model to correctly locate relevant columns and parse nationalities within the table, ultimately producing the correct execution sub-table.; Illustration of the table organizer inside. The augmented information from the table augmentor is utilized in stage 1 and stage 2, enabling the model to correctly locate relevant columns and parse nationalities within the table, ultimately producing the correct execution sub-table.
Figure 3: Comparison of methods following pre-LLM era with tables divided by cell count on WikiTQ. In the subplot above, the regression curves of different models are represented by dashed lines in different colors. The regression curve for ALTER exhibits a significantly slower decline rate.; Comparison of methods following pre-LLM era with tables divided by cell count on WikiTQ. In the subplot above, the regression curves of different models are represented by dashed lines in different colors. The regression curve for ALTER exhibits a significantly slower decline rate.
Figure 4: Relative performance drop and the ratio of table tokens utilized by ALTER to the total table tokens on WikiTQ as the number of rows added increases by multiples ( i.e., perturbation factor). The drop for CABINET and ALTER is specifically marked at the factor of 1 .; Relative performance drop and the ratio of table tokens utilized by ALTER to the total table tokens on WikiTQ as the number of rows added increases by multiples ( i.e., perturbation factor). The drop for CABINET and ALTER is specifically marked at the factor of 1 .
Figure 5: Intuitive example for step-back query augmentation, where ALTER correctly answers the query utilizing broader information compared to directly output SQL based on the original query.; Intuitive example for step-back query augmentation, where ALTER correctly answers the query utilizing broader information compared to directly output SQL based on the original query.
Figure 6: Intuitive example for sub-query query augmentation, where ALTER correctly answers the query utilizing sub-queries compared to directly output SQL based on the original query.; Intuitive example for sub-query query augmentation, where ALTER correctly answers the query utilizing sub-queries compared to directly output SQL based on the original query.
Figure 7: Intuitive example for table augmentor, where ALTER correctly answers the query utilizing information about data format and composition compared to directly output SQL without any augmentation information.; Intuitive example for table augmentor, where ALTER correctly answers the query utilizing information about data format and composition compared to directly output SQL without any augmentation information.
Figure 8: The prompt template for the query augmentor; The prompt template for the query augmentor
Figure 9: The prompt template for three types of augmentation in the table augmentor; The prompt template for three types of augmentation in the table augmentor
Results of different methods on WikiTQ and TabFact. 1 1 1 For the Dater method, we report the results of using the LLM-based method as backbone (We use underline to denote the second-best performance, bold to denote the best performance for each region: Pre-LLM era, LLM era with result ensemble and without ensemble)
Ablation results of query augmentor on the test sets of WikiTQ and TabFact.
Ablation results of different K values and with or without augmented information from the table augmentor on the WikiTQ and TabFact. (improvement measured against the data in the bottom-left relative position.)
Comparison of methods in the LLM era with tables divided by token count on WikiTQ. ( underline denotes the second-best performance; bold denotes the best performance)