SCITAT: A Question Answering Benchmark for Scientific Tables and Text Covering Diverse Reasoning Types
Authors: Xuanliang Zhang, Dingzirui Wang, Baoxin Wang, Longxu Dou, Xinyuan Lu, Keyan Xu, Dayong Wu, Qingfu Zhu
Accepted at: acl25

SciTAT: A Question Answering Benchmark for Scientific Tables and Text Covering Diverse Reasoning Types

Xuanliang Zhang1*, Dingzirui Wang1, Baoxin Wang2, Longxu Dou3, Xinyuan Lu4, Keyan Xu1, Dayong Wu2, Qingfu Zhu1, Wanxiang Che1*
1Harbin Institute of Technology   2iFLYTEK Research
3Individual Researcher   4National University of Singapore

Abstract

Scientific question answering (SQA) is an important task aimed at answering questions based on papers. However, current SQA datasets have limited reasoning types and neglect the relevance between tables and text, creating a significant gap with real scenarios. To address these challenges, we propose a QA benchmark for scientific tables and text with diverse reasoning types (SciTAT). To cover more reasoning types, we summarize various reasoning types from real-world questions. To reason on both tables and text, we require the questions to incorporate tables and text as much as possible. Based on SciTAT, we propose a baseline (CAR), which combines various reasoning methods to address different reasoning types and process tables and text at the same time. CAR brings average improvements of 4.1% over other baselines on SciTAT, validating its effectiveness. Error analysis reveals the challenges of SciTAT, such as complex numerical calculations and domain knowledge1.

1 Introduction

Scientific Question Answering (SQA) plays a crucial role in addressing research questions based on scientific papers (Tsatsaronis et al., 2015; Lee et al., 2023). Advancing SQA development can significantly accelerate knowledge acquisition (Taylor et al., 2022; AI4Science and Quantum, 2023). The dense technical terms and heterogeneous data representations in papers present challenges for the SQA task (Sun et al., 2024; Pramanick et al., 2024).

To evaluate and enhance the model capabilities in SQA, numerous datasets are proposed (Pampari et al., 2018; Jin et al., 2019; Pappas et al., 2020). However, existing datasets exhibit the following limitations, as shown in Table 1. Firstly, the reasoning types are relatively narrow, failing to capture the complexity of real scenarios, such as data analysis, which is frequently encountered in actual queries (Moosavi et al., 2021). Secondly, prior works focus only on split text and tables, overlooking the relevance between tables and text, thereby limiting their applicability (Wang et al., 2022). To address the limitations, in this paper: (i) We introduce a new SQA benchmark, covering diverse real-scenery reasoning types and considering tables and text simultaneously. (ii) To enhance the performance on the benchmark, we propose a baseline, which can handle multiple reasoning types and process tables and text simultaneously.

Firstly, we propose a QA benchmark for scientific tables and text (SciTAT), which are collected from papers in arXiv.org. To incorporate more reasoning types, we summarize various reasoning types from the real questions raised by researchers (see Figure 1). To ensure the questions require reasoning on both tables and text, we require questions to involve tables and text as much as possible. Overall, SciTAT contains 953 questions derived from 871 papers. Data analysis reveals that SciTAT encompasses 4 reasoning types and 13 subtypes, covering the types summarized from the

<table>
  <tr>
    <th>Dataset</th>
    <th colspan="4">Reasoning Type</th>
    <th colspan="2">Evidence</th>
  </tr>
  <tr>
    <th></th>
    <th>L</th>
    <th>N</th>
    <th>D</th>
    <th>T</th>
    <th>Text</th>
    <th>Table</th>
    <th>TaT</th>
  </tr>
  <tr>
    <td>BioRead</td>
    <td>●</td>
    <td></td>
    <td></td>
    <td></td>
    <td>✓</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>QASA</td>
    <td>●</td>
    <td>●</td>
    <td>●</td>
    <td></td>
    <td>✓</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>SciGen</td>
    <td>●</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>✓</td>
    <td></td>
  </tr>
  <tr>
    <td>SciTab</td>
    <td>●</td>
    <td>●</td>
    <td></td>
    <td></td>
    <td></td>
    <td>✓</td>
    <td></td>
  </tr>
  <tr>
    <td>SPiQA</td>
    <td>●</td>
    <td>●</td>
    <td></td>
    <td></td>
    <td></td>
    <td>✓</td>
    <td></td>
  </tr>
  <tr>
    <td>SciTAT</td>
    <td>●</td>
    <td>●</td>
    <td>●</td>
    <td>●</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
</table>

Table 1: Comparison of SciTAT to recent SQA datasets, introduced in Appendix A.1. TaT denotes Table and Text. L, N, D, and T denote the reasoning type of Look Up, Numerical Reasoning, Data Analysis, and Tabulation, with examples in Figure 1. Pie charts represent the proportion of subtypes compared with SciTAT.
Figure 1: Illustrations of the reasoning types in SciTAT. The tables and text (left) show color-coded spans for question context. The questions (right) are examples of 4 reasoning types, with their rationales and answers.

real questions in SparkRA (Wu et al., 2024a) and previous works (Lu et al., 2023; Wu et al., 2024b). SciTAT not only requires the model to look up information and numerical reasoning but also requires complex data analysis and tabulation, effectively meeting the needs of real-world researchers.

Considering the challenges of SciTAT, we propose a baseline to process scientific data by integrating reasoning methods (CAR). To handle multiple reasoning types, CAR includes two modules: Calculator and Reasoner. To process tables and text, Calculator extracts and calculates numerical information from tables and text, which is then provided to Reasoner for further reasoning.

We construct a series of baselines on SciTAT. Experimental results reveal that CAR outperforms other baselines with 4.1% on average, proving the effectiveness of the combination of Calculator and Reasoner. However, the Exact Match of CAR using gpt-4o is still below 50%, which indicates that SciTAT serves as a challenging benchmark. Error analysis reveals the main challenges of SciTAT, such as context grounding, complex numerical calculation, and the need for domain knowledge.

Our contributions are as follows:

1. To the best of our knowledge, we develop SciTAT, the first QA benchmark for scientific tables and text, covering diverse reasoning types based on real scenarios.

2. We propose CAR, a baseline to solve various reasoning types and process tables and text by integrating reasoning methods.

3. We conduct a series of experiments, providing results and error analysis to highlight the challenges of SciTAT, thereby guiding the direction for future improvements.

2 SciTAT Dataset

The input for our task consists of scientific tables, text, and a question, and the output is the answer. Moreover, we annotate the rationale of each question. For brevity, we refer to each question, its corresponding rationale, and answer, as an instance. We begin by describing the construction process of SciTAT. We employ a framework combining automatic generation with manual annotation to enhance both the quality and efficiency of the annotation process, as illustrated in Figure 2.

2.1 Paper Preparation

Source Data Collection  We select papers from the “Artificial Intelligence”, “Computation and Language”, and “Machine Learning” subfields of “Computer Science” following previous datasets (Lee et al., 2023; Moosavi et al., 2021; Lu et al., 2023). We collect LaTeX code from papers published between January 2020 and July 2023 on arxiv.org2, using a heuristic method to extract all the tables with their corresponding captions and labels, and text in each paper.

Tables and Text Selection  To ensure the inclusion of both tables and text, we filter out papers without tables. Additionally, to guarantee the relevance of the tables and text in the context, the

2https://info.arxiv.org/help/bulk_data/index.html
Figure 2: The overview of our annotation process. The blue boxes represent the data and the white boxes with solid lines represent the annotation procedures.

<table>
  <tr>
    <th colspan="3">The requirements for generating questions</th>
  </tr>
  <tr>
    <td>1. The question must meet the reasoning type.</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>2. The question is best answered by referring to both the tables and the text simultaneously.</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>3. The question should be with more reasoning and calculation.</td>
    <td></td>
    <td></td>
  </tr>
</table>

Figure 3: The requirements for generating questions.

context we provide when annotating the question is a paragraph that mentions tables and the tables mentioned. Specifically, we randomly select at least one paragraph from the paper that mentions tables and the mentioned tables as the context.

2.2 Initial Generation
Reasoning Type Design To observe the reasoning types that researchers might query, we select SParkRA (Wu et al., 2024a), a platform specifically designed to provide QA services for researchers in the context of scientific papers. We hypothesize that the reasoning types observed in these questions are comparable to those found in real-world inquiries. We randomly select 650 questions and categorize their reasoning types. To account for potentially unobserved reasoning types, we also incorporate reasoning types from previous datasets (Lu et al., 2023; Wu et al., 2024b). Finally, we summarize 4 reasoning types and 13 subtypes for SCITAT, as shown in Table 2.

Question and Answer Generation We assign a reasoning type to each context, including a paragraph and mentioned tables. Manually annotating scientific questions and answers is time-consuming and prone to introducing annotation artifacts since it requires substantial domain expertise and a deep understanding of the paper (Bender and Friedman, 2018; Pramanick et al., 2024). To address these challenges, we leverage the extensive knowledge and powerful instruction-following capabilities of gpt-4o (OpenAI et al., 2024) following previous works (Zhang et al., 2023, 2024; Yu et al., 2025).

We guide the LLM to generate questions aligned with the reasoning type based on the context using three-shot prompt, with requirements outlined in Figure 3. We guide the LLM to generate the rationale and answer for each question according to the context with a three-shot prompt. Detailed prompts are provided in Appendix B.1.

2.3 Human Refinement

Since LLMs cannot guarantee the reasonableness of the questions or the correctness of the answers, we employ manual checks and refinement. (i) For the context, annotators are tasked with verifying that the extracted tables and text are consistent with the original paper and removing any incorrect ones. (ii) For the questions, annotators should refine them following the guidelines in Figure 3. (iii) For the rationales and answers, annotators are required to verify their correctness and correct any errors. Due to the diverse reasoning types in SCITAT, our answers include both short-form and free-form types. Annotators are instructed to extract one or more tokens for short-form answers and use complete sentences for free-form answers. (iv) For the answer source, annotators are prompted to select the source of the answer, which may include Text, Table, or Table and Text, and identify the relevant tables. Annotators are compensated $1 per instance.

2.4 Quality Control

To ensure the quality of SCITAT, we implement rigorous quality control strategies.
<table>
  <tr>
    <th>Reasoning Type</th>
    <th>Subtypes</th>
    <th>Description</th>
    <th>%</th>
  </tr>
  <tr>
    <td rowspan="2">Look Up</td>
    <td>Table Look Up</td>
    <td>Search for specific tables</td>
    <td>2.7</td>
  </tr>
  <tr>
    <td>Span Look Up</td>
    <td>Search for spans in tables or paragraphs</td>
    <td>2.0</td>
  </tr>
  <tr>
    <td rowspan="6">Numerical Reasoning</td>
    <td>Arithmetic Calculation</td>
    <td>Numerical calculations</td>
    <td>11.1</td>
  </tr>
  <tr>
    <td>Comparison</td>
    <td>Comparison of values</td>
    <td>8.2</td>
  </tr>
  <tr>
    <td>Aggregation</td>
    <td>Combines multiple data points into a single metric</td>
    <td>3.9</td>
  </tr>
  <tr>
    <td>Ranking</td>
    <td>Arranges items in a specific order</td>
    <td>7.0</td>
  </tr>
  <tr>
    <td>Counting</td>
    <td>Counting occurrences</td>
    <td>9.2</td>
  </tr>
  <tr>
    <td>Domain Knowledge Calculation</td>
    <td>Calculations requiring domain knowledge</td>
    <td>6.5</td>
  </tr>
  <tr>
    <td rowspan="3">Data Analysis</td>
    <td>Descriptive Analysis</td>
    <td>Summarize or interpret to spot patterns and trends</td>
    <td>23.3</td>
  </tr>
  <tr>
    <td>Anomaly Detection</td>
    <td>Detect deviations and their causes</td>
    <td>7.0</td>
  </tr>
  <tr>
    <td>Causal Analysis</td>
    <td>Investigate cause-and-effect relationships</td>
    <td>10.6</td>
  </tr>
  <tr>
    <td>Tabulation</td>
    <td>-</td>
    <td>Standardizing the formats of tables/subtables</td>
    <td>8.4</td>
  </tr>
</table>

Table 2: The reasoning types, the description of their subtypes, and their proportion in SciTAT. Look Up, Numerical Reasoning, Data Analysis, and Tabulation account for 4.7%, 46.0%, 40.9%, and 8.4% respectively.

<table>
  <tr>
    <th>Statistics</th>
    <th>Long-context</th>
    <th>Short-context</th>
  </tr>
  <tr>
    <td>Questions</td>
    <td>953</td>
    <td>953</td>
  </tr>
  <tr>
    <td>Papers</td>
    <td>871</td>
    <td>871</td>
  </tr>
  <tr>
    <td>Avg. Tables</td>
    <td>5.2</td>
    <td>1.1</td>
  </tr>
  <tr>
    <td>Avg. Cells</td>
    <td>60.8</td>
    <td>56.6</td>
  </tr>
  <tr>
    <td>Avg. Paragraphs</td>
    <td>80.2</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>Avg. |Paragraph|</td>
    <td>83.4</td>
    <td>113.0</td>
  </tr>
</table>

Table 3: The statistics of SciTAT. Avg. Tables and Avg. Cells indicate the average number of tables and cells per table. Avg. Paragraphs and Avg. |Paragraph| indicate the average number of paragraphs and the average length of each paragraph.

<table>
  <tr>
    <th>Statistics</th>
    <th>Table</th>
    <th>Text</th>
    <th>TaT</th>
    <th>Total</th>
  </tr>
  <tr>
    <td>Short-form answers</td>
    <td>234</td>
    <td>13</td>
    <td>93</td>
    <td>340</td>
  </tr>
  <tr>
    <td>Free-form answers</td>
    <td>308</td>
    <td>67</td>
    <td>238</td>
    <td>613</td>
  </tr>
  <tr>
    <td>Total</td>
    <td>542</td>
    <td>80</td>
    <td>331</td>
    <td>953</td>
  </tr>
</table>

Table 4: Question distribution over different answers and sources in SciTAT.

Competent Annotators  The annotators we employ are all graduate students majoring in artificial intelligence. Initially, they undergo annotation training sessions to learn the task and the annotation interface (see Appendix C.1) and are required to annotate 20 questions. We retain those with Exact Match \( \geq 95\% \) and provide constructive feedback on their mistakes. Detailed annotation information is provided in Appendix C.2.

Two-round Validation  After the instances are submitted by the annotator, a two-round validation is implemented, consisting of manual verification and revision, following the previous work (Zhu et al., 2021). (i) In the first round, a verifier examines each instance to ensure that the annotations adhere to the guidelines. If errors are found, the verifier communicates with the annotator and requests the corresponding corrections. (ii) In the second round, a different annotator reviews the instances again. Any identified errors are discussed with the verifier annotator and revised as needed.

2.5 Data Analysis

Basic Statistics  To better evaluate the reasoning ability across different context lengths, we construct two settings: long-context and short-context. In the long-context setting, the model should answer questions based on the whole paper. In the short-context setting, the model is required to answer questions based on a paragraph and the tables referenced in that paragraph. We present the statistics of SciTAT in the two settings in Table 3. We also show the question distribution over different answers and sources in Table 4. Notably, over 1/3 of the questions in SciTAT require reasoning that involves both tables and text simultaneously.

Reasoning Types  We analyze the distribution of reasoning types in SciTAT, as shown in Table 2. It can be found that SciTAT has a variety of reasoning types evenly distributed. Among these types, Data Analysis and Tabulation are identified as common patterns based on observations of real queries and are rarely represented in existing datasets.

3 CAR

CAR is designed to address the questions on the context of scientific tables and text. Given that SciTAT combines diverse reasoning types, CAR is composed into two modules: Calculator and Rea-
<table>
  <tr>
    <th colspan="3">Text:</th>
    <th colspan="2">Tables 2 and 3 compare with prior works. Pre-trained embeddings used are indicated as XLNet-large* (Yang et al., 2019), BERT base* and large* (Devlin et al., 2019). Graph Recategorization, which can utilize an external entity recognizer (Lyu and Titov, 2018; Zhang et al., 2019) as (G.R.) and a* indicates the Naseem et al. (2017) oracle.</th>
  </tr>
  <tr>
    <th colspan="3">Tables:</th>
    <th colspan="2"></th>
  </tr>
  <tr>
    <th>Model</th>
    <th>UAS</th>
    <th>LAS</th>
    <th>Model</th>
    <th>AMR1.0 AMR2.0</th>
  </tr>
  <tr>
    <td>...</td>
    <td>...</td>
    <td>...</td>
    <td>...</td>
    <td>...</td>
  </tr>
  <tr>
    <td>Mini et al.(2019)*</td>
    <td>97.3</td>
    <td>96.3</td>
    <td>Cai and Lam (2020)</td>
    <td>75.4 80.2</td>
  </tr>
  <tr>
    <td>...</td>
    <td>...</td>
    <td>...</td>
    <td>(G.R.)*</td>
    <td></td>
  </tr>
  <tr>
    <td>cStack-Transform at</td>
    <td>96.2±0.1</td>
    <td>94.7±0.0</td>
    <td>cStack-Transformer</td>
    <td>75.4±0.0 79.9±0.1</td>
  </tr>
</table>

Table 2: Test-set performance prior art on the English Penn-Trebank.

<table>
  <tr>
    <th colspan="3">Table 3: Test-set performance and prior art on AMR1.0 and AMR2.0 in terms of Smatch.</th>
  </tr>
</table>

<table>
  <tr>
    <th colspan="3">Question:</th>
    <th colspan="2">Compare the performance of Stack-Transformer on the Penn-Trebank and AMR datasets with previous studies, and analyze the reasons for differences.</th>
  </tr>
</table>

Figure 4: The overview of CAR, which consists of two modules: (i) Calculator generates code to compute the numerical information required for solving the question. (ii) Reasoner continues the reasoning process based on the information provided by the Calculator to answer the question.

soner, as illustrated in Figure 4, which focus on different reasoning types. To process tables and text simultaneously, Calculator extracts and computes the numerical information from the context and Reasoner derives the final answer based on the calculated information. The prompts we use are presented in Appendix B.2.

3.1 Calculator

The input to the Calculator consists of a question and the scientific context (including tables and text), and the output provides the numerical information necessary to answer the question. Specifically, we prompt the LLM to generate a program function based on the context to answer the question. Unlike other Program-of-Thought (PoT) methods (Gao et al., 2023; Chen et al., 2023) that require the program to return the answer directly, the function is designed to return a sentence explicitly describing the numerical information, as illustrated in Figure 4. Once the function is obtained, it is executed to extract the numerical information.

3.2 Reasoner

The Reasoner takes as input a question, the scientific context, and the numerical information obtained from the Calculator to produce the final answer. Specifically, we utilize a CoT prompt (Wei et al., 2022) to guide the LLM through a step-by-step reasoning process based on the context and information, leading to the final answer. However, since the information may not always be accurate or helpful, we further prompt the LLM to engage in reflection, evaluating the correctness and relevance of the extracted information during reasoning.

4 Experiments

4.1 Settings

Metrics Due to the significant difference in token counts between free-form answers and short-form answers (see Table 12), we evaluate the two types of answers separately. For short-form answers, we use Exact Match (EM) to assess correctness, while for free-form answers, we use F1 and BERTScore F1 (BERTScore) (Zhang* et al., 2020), following previous studies (Zhu et al., 2021; Moosavi et al., 2021). EM measures the proportion of the predicted result that exactly matches the gold answer. F1 calculates the overlap between predicted and gold answers based on their bag-of-words representation. BERTScore evaluates the similarity between predicted and gold answers by calculating the cosine similarity of their embeddings.

Models We employ the open-source LLM Llama3.1-Instruct (Dubey et al., 2024), Qwen2.5-Coder-Instruct (Hui et al., 2024) and the closed-source LLM gpt-4o (OpenAI et al., 2024) to evaluate SciTAT. Llama3.1 and Qwen2.5-Coder are among the top-performing open-source general models and code models, while gpt-4o is one of the leading closed-source models.

Baselines We compare CAR with the following baselines, with prompts in Appendix B.2.

• Direct QA (Pramanick et al., 2024) prompts the LLM to directly answer the questions.

• CoT (Wei et al., 2022) prompts the LLM to perform the step-by-step reasoning process and then get the final answer.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Scale</th>
    <th rowspan="2">Method</th>
    <th colspan="3">Long-context</th>
    <th colspan="3">Short-context</th>
  </tr>
  <tr>
    <th>EM</th>
    <th>F1</th>
    <th>BERTScore</th>
    <th>EM</th>
    <th>F1</th>
    <th>BERTScore</th>
  </tr>
  <tr>
    <td rowspan="5">8B</td>
    <td rowspan="5"></td>
    <td>Direct QA</td>
    <td>0.0</td>
    <td>30.6</td>
    <td>66.4</td>
    <td>0.0</td>
    <td>30.7</td>
    <td>66.5</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>13.0</td>
    <td>29.5</td>
    <td>65.6</td>
    <td>20.6</td>
    <td>41.4</td>
    <td>71.6</td>
  </tr>
  <tr>
    <td>PoT</td>
    <td>4.4</td>
    <td>21.7</td>
    <td>54.0</td>
    <td>17.1</td>
    <td>21.5</td>
    <td>49.9</td>
  </tr>
  <tr>
    <td>Three-Agent</td>
    <td>12.6</td>
    <td>34.3</td>
    <td>68.5</td>
    <td>22.1</td>
    <td>36.5</td>
    <td>67.6</td>
  </tr>
  <tr>
    <td>CAR</td>
    <td><b>24.8</b></td>
    <td><b>37.5</b></td>
    <td><b>69.7</b></td>
    <td><b>24.2</b></td>
    <td><b>44.3</b></td>
    <td><b>73.2</b></td>
  </tr>
  <tr>
    <td rowspan="5">Llama3.1</td>
    <td rowspan="5"></td>
    <td>Direct QA</td>
    <td>0.0</td>
    <td>31.6</td>
    <td>67.5</td>
    <td>0.0</td>
    <td>33.6</td>
    <td>68.7</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>30.3</td>
    <td>36.8</td>
    <td>69.9</td>
    <td>32.1</td>
    <td>44.1</td>
    <td>73.1</td>
  </tr>
  <tr>
    <td>PoT</td>
    <td>5.3</td>
    <td>28.8</td>
    <td>61.7</td>
    <td>36.8</td>
    <td>35.6</td>
    <td>64.0</td>
  </tr>
  <tr>
    <td>Three-Agent</td>
    <td>30.3</td>
    <td>40.0</td>
    <td>71.3</td>
    <td>40.3</td>
    <td>40.0</td>
    <td>71.1</td>
  </tr>
  <tr>
    <td>CAR</td>
    <td><b>35.9</b></td>
    <td><b>41.7</b></td>
    <td><b>71.8</b></td>
    <td><b>40.7</b></td>
    <td><b>46.2</b></td>
    <td><b>74.4</b></td>
  </tr>
  <tr>
    <td rowspan="5">Qwen2.5-Coder</td>
    <td rowspan="5">7B</td>
    <td>Direct QA</td>
    <td>0.0</td>
    <td>27.7</td>
    <td>64.8</td>
    <td>1.8</td>
    <td>35.2</td>
    <td>68.9</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>15.3</td>
    <td>36.6</td>
    <td>69.3</td>
    <td>21.5</td>
    <td>44.2</td>
    <td>73.2</td>
  </tr>
  <tr>
    <td>PoT</td>
    <td>4.4</td>
    <td>6.9</td>
    <td>35.4</td>
    <td>19.7</td>
    <td>14.3</td>
    <td>42.2</td>
  </tr>
  <tr>
    <td>Three-Agent</td>
    <td>10.6</td>
    <td>35.9</td>
    <td>68.8</td>
    <td>20.3</td>
    <td>41.8</td>
    <td>72.5</td>
  </tr>
  <tr>
    <td>CAR</td>
    <td><b>16.8</b></td>
    <td><b>41.3</b></td>
    <td><b>71.5</b></td>
    <td><b>25.9</b></td>
    <td><b>44.5</b></td>
    <td><b>73.7</b></td>
  </tr>
  <tr>
    <td rowspan="5">gpt-4o</td>
    <td rowspan="5">-</td>
    <td>Direct QA</td>
    <td>0.0</td>
    <td>29.8</td>
    <td>67.3</td>
    <td>0.0</td>
    <td>39.6</td>
    <td>71.5</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>31.3</td>
    <td>41.3</td>
    <td>72.7</td>
    <td>32.2</td>
    <td>46.8</td>
    <td>75.5</td>
  </tr>
  <tr>
    <td>PoT</td>
    <td>5.0</td>
    <td>15.0</td>
    <td>44.4</td>
    <td>28.3</td>
    <td>31.2</td>
    <td>59.8</td>
  </tr>
  <tr>
    <td>Three-Agent</td>
    <td>34.7</td>
    <td>40.3</td>
    <td>72.3</td>
    <td>27.1</td>
    <td>44.9</td>
    <td>74.6</td>
  </tr>
  <tr>
    <td>CAR</td>
    <td><b>37.5</b></td>
    <td><b>41.8</b></td>
    <td><b>73.1</b></td>
    <td><b>43.7</b></td>
    <td><b>47.1</b></td>
    <td><b>75.7</b></td>
  </tr>
</table>

Table 5: Performance comparison of different models and methods. The best results of each model under each setting are annotated in bold.

• PoT (Gao et al., 2023; Chen et al., 2023) prompts the LLM to generate a program that can be executed to obtain the answer.

• Three-Agent (Fatemi and Hu, 2024) is currently the state-of-the-art method for reasoning on tables and text without considering fine-tuning models, which consists of three agents: an analyst agent that looks up relevant information and performs calculations, and two critic agents that provide feedbacks on extraction and calculation and correct errors.

Given the long-context setting, we adopt zero-shot prompts in main experiments to prevent exceeding the context limit, with the few-shot results in the short-context setting in Appendix E.1.

4.2 Main Experiments

The results of comparing CAR with other baselines on SCiTAT are shown in Table 5. The results reveal that: (i) CAR significantly outperforms other baselines across different models and settings, achieving an average improvement of 4.1% on all metrics, highlighting its effectiveness. (ii) Despite improvement, CAR demonstrates suboptimal performance, as EM and F1 remain below 50.0, and while BERTScore is relatively high (Moosavi et al., 2021; Zhao et al., 2024a), it remains under 80.0, reflecting the challenge of SCiTAT. We also observe:

Baselines (i) CAR outperforms Three-Agent, demonstrating the diversity of reasoning types in SCiTAT. It encompasses not only Look Up and Numerical Reasoning, and complex computations in SCiTAT cannot be solved solely by the models themselves. Three-Agent is more pronounced on larger-scale models, as their stronger critic capabilities allow for more precise information extraction and calculation (Pan et al., 2024; Tian et al., 2024; Lin et al., 2024). (ii) Among other baselines, CoT achieves higher performance, while Direct QA exhibits lower EM, and PoT shows lower F1 and BERTScore. Considering that diverse reasoning types in SCiTAT, CoT is relatively better at handling these types of questions (Wei et al., 2022; Wu et al., 2024b; Pramanick et al., 2024). Direct QA, due to its lack of reasoning, is prone to computational errors and longer answers for short-form answers, resulting in an EM score of zero (Snell et al., 2024). Since the program typically returns shorter answers (see Appendix E.2), the PoT method is less effective at answering free-form questions.

Context Settings CAR demonstrates a more significant improvement in the long-context setting than the short-context setting. Due to the dense knowledge presented in the paper, directly answering questions based on the entire paper may confuse the model, preventing it from focusing on the relevant tables and text (Lee et al., 2023; Pramanick
et al., 2024). In contrast, CAR uses the Calculator to extract and compute useful numerical information from the paper, effectively guiding the Reasoner and avoiding the need to search for answers directly within the whole paper.

Answer Types CAR shows more significant improvements in short-form answers than free-form answers. For short-form answers, the Reasoner typically only needs to verify the correctness of the result of the Calculator and extract the answer. In contrast, for free-form answers, the Reasoner often needs to perform additional analysis based on the numerical information provided by the Calculator.

Reasoning Types We present the average performance of four models across reasoning types in Figure 5. Specifically, the F1 and BERTScore for the type of Look Up are 0 as all corresponding answers are short-form. The EM for Data Analysis is 0, as all the answers to this reasoning type are free-form. We can observe that: (i) The models perform worst on Data Analysis, which requires more comprehensive capabilities, such as numerical computation, logical reasoning, and summarization (Wu et al., 2024b). (ii) The F1 and BERTScore on Tabulation are the highest, but the EM is the lowest, indicating the difficulty of this reasoning type. While the predicted result may be close to the gold answer, achieving an exact match remains challenging. This highlights the need for more effective evaluation metrics. (iii) There is still significant room for improvement on all the types, underscoring the challenges of SciTAT.

Figure 5: The average performance of CAR across four models on four reasoning types. Look denotes Look Up, Numerical denotes Numerical Reasoning, and Analysis denotes Data Analysis.

![Bar chart showing the average performance of CAR across four reasoning types](page_97_120_627_384.png)

Figure 6: The distribution of error types of CAR. Knowl. denotes Knowledge.

<table>
  <tr>
    <th>Scale</th>
    <th>Method</th>
    <th>EM</th>
    <th>F1</th>
    <th>BERTScore</th>
  </tr>
  <tr>
    <td rowspan="3">8B</td>
    <td>CAR</td>
    <td>24.2</td>
    <td>44.3</td>
    <td>73.2</td>
  </tr>
  <tr>
    <td>w/o <i>Calculator</i></td>
    <td>20.6</td>
    <td>41.4</td>
    <td>71.6</td>
  </tr>
  <tr>
    <td>w/o <i>Reasoner</i></td>
    <td>0.3</td>
    <td>30.8</td>
    <td>62.8</td>
  </tr>
  <tr>
    <td rowspan="3">70B</td>
    <td>CAR</td>
    <td>40.7</td>
    <td>46.2</td>
    <td>74.4</td>
  </tr>
  <tr>
    <td>w/o <i>Calculator</i></td>
    <td>32.1</td>
    <td>44.1</td>
    <td>73.1</td>
  </tr>
  <tr>
    <td>w/o <i>Reasoner</i></td>
    <td>0.0</td>
    <td>42.0</td>
    <td>70.4</td>
  </tr>
</table>

Table 6: The ablation results of CAR using Llama3.1 in the short-context setting, compared with removing Calculator (w/o <i>Calculator</i>) and removing Reasoner (w/o <i>Reasoner</i>).

4.3 Ablation Experiments

To demonstrate the effectiveness of CAR, we perform an ablation study by removing each module, with results presented in Table 6. Specifically, when removing the Calculator, it is the same as the CoT baseline. The significant performance drop confirms the validity of CAR. The results suggest that relying on a single reasoning method is insufficient to derive accurate answers due to the diverse reasoning types in SciTAT. Especially, the EM of removing Reasoner is low since we prompt the program to return the entire numerical information instead of the simple answer.

4.4 Error Analysis

To present the challenges of SciTAT, we analyze the error instances of CAR using Llama3.1-70B. Specifically, we randomly select 50 error instances with BERTScore below 60 from the results corresponding to free-form answers and another 50 instances with EM of 0 from the results corresponding to short-form answers. We manually categorize the error types, as illustrated in Figure 6, with examples of error types in Appendix D. It can be observed that the distribution of error types for
free-form answers and short-form answers differs significantly. We proceed with a detailed analysis.

(i) Miss refers to the omission of part of the answer, such as when only some sub-questions are addressed, or when data analysis is limited to summarizing phenomena without providing conclusions or insights. (ii) Locate refers to locating incorrect relevant context according to the question. (iii) Calculate denotes errors in applying formulas, programming mistakes, or computational inaccuracies. (iv) Knowledge refers to errors in responses due to the lack of domain-specific knowledge. (v) Redundancy refers to the generation of unnecessary responses that result in an EM of zero.

Compared to previous datasets, SciTAT presents the following challenges. (i) Free-form and short-form answers are associated with different error types, necessitating the design of distinct methods. (ii) SciTAT requires integration of various reasoning types and the processing of both tables and text, demanding the model to have strong domain-specific knowledge in the scientific field. We outline these challenges to inspire future work in addressing these issues, aiming to enhance model performance in SQA on tables and text.

5 Related Works

5.1 Scientific QA Datasets

Early SQA datasets were designed in a cloze-style format, limiting their difficulty (Pampari et al., 2018; Pappas et al., 2018). To address this issue, PubMedQA (Jin et al., 2019), QASPER (Dasigi et al., 2021), and QASA (Lee et al., 2023) employ humans to annotate questions and answers over papers, and SciInstruct (Zhang et al., 2024) collects questions from sources like textbooks and synthesizes answers using LLMs. However, these works primarily focus on text, without considering the tables appearing in papers. Therefore, SciGen (Moosavi et al., 2021) focuses on generating descriptions based on tables in papers, SciTab (Lu et al., 2023) concentrates on the table fact verification, and SPIQA (Pramanick et al., 2024) is designed for QA based on tables and images.

Nevertheless, the reasoning types of existing datasets are relatively limited, since they do not involve diverse reasoning types, such as Data Analysis and Tabulation, that frequently occur in real scenarios. Moreover, they overlook the relevance between tables and text, limiting their application (Chen et al., 2020; Wang et al., 2022). Therefore, we propose SciTAT, a QA benchmark for scientific tables and text with diverse reasoning types.

5.2 QA Datasets for Tables and Text

Previous QA datasets for tables and text mainly focus on Look Up and Numerical Reasoning in the Wikipedia and financial domains. For example, HybridQA (Chen et al., 2020) annotates QA pairs over Wikipedia tables and text, which primarily focuses on look up spans in the context. TAT-QA (Zhu et al., 2021), FinQA (Chen et al., 2021), DocMath-Eval (Zhao et al., 2024c), and FinanceMATH (Zhao et al., 2024b) primarily address the numerical reasoning task in the financial domain. However, previous datasets focus on limited reasoning types, mainly Look Up and Numerical Reasoning, which differ significantly from the SQA scenarios in real-world applications. Furthermore, these datasets do not require models to possess domain-specific knowledge in science. A detailed comparison SciTAT with previous datasets for tables and text is shown in Appendix A.2.

Considering the reasoning types of existing datasets, previous works introduce programs to obtain the final answer (Gao et al., 2023; Chen et al., 2023; Zhang et al., 2023). For instance, BlendSQL (Glenn et al., 2024) prompts the LLM to generate a superset of SQL that can query information from both tables and text to obtain answers. Hapropro (Shi et al., 2024) provides commonly used program functions, allowing the LLM to directly call them during program generation. However, these methods can not apply directly to SciTAT, as SciTAT also involves reasoning types, such as Data Analysis, which is challenging to be solved by the program alone (Wu et al., 2024b). Therefore, we propose CAR, which combines multiple reasoning types to enhance performance on SciTAT.

6 Conclusion

To address the limitations of previous scientific QA datasets, which involve limited reasoning types and fail to consider the relevance between tables and text, we propose SciTAT, the QA benchmark for scientific tables and text with diverse reasoning types. To incorporate diverse reasoning types, we analyze the questions posed by researchers and combine the types in prior works, summarizing 4 reasoning types with 13 subtypes. To ensure that the questions encompass both tables and text, we require the questions to include both elements when-
ever possible. For SciTAT, we introduce CAR, a baseline that combines reasoning methods to enhance the performance across various reasoning types, with handling both tables and text. Experimental results show that CAR outperforms other baselines by an average of 4.1%, demonstrating its effectiveness. Error analysis reveals the challenges in SciTAT, such as grounding relevant context, complex numerical reasoning, and the need for domain-specific knowledge.

Limitations

(i) SciTAT currently supports only the English language. Future versions will include additional languages. (ii) Currently, we focus on single-turn QA for scientific tables and texts in SciTAT. Multi-turn dialogues on scientific tables and text will be explored in future work.

Ethics Statement

All models used in this paper are publicly available, and our utilization of them strictly complies with their respective licenses and terms of use. We collect papers from ariv.org following its terms of use and regulations. During the dataset construction process, we ensure that the collected papers are publicly accessible and do not infringe on any copyrights. Additionally, we confirm that the compensation provided to annotators is significantly higher than the local minimum wage.

Acknowledge

We gratefully acknowledge the support of the National Natural Science Foundation of China (NSFC) via grant 62236004, 62206078, 62441603 and 62476073.

References

Microsoft Research AI4Science and Microsoft Azure Quantum. 2023. The impact of large language models on scientific discovery: a preliminary study using gpt-4. Preprint, arXiv:2311.07361.

Emily M Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604.

Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research.

Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020. HybridQA: A dataset of multi-hop question answering over tabular and textual data. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1026–1036, Online. Association for Computational Linguistics.

Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021. FinQA: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3697–3711, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4599–4610, Online. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, AoBo Yang, Archi Mitra, Archie Sravan Kumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Alonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esibou, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Gregoire Milon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junting Jia, Kalyan Vasudhen Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh,
Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Celebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiaodong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Giridhar, Rohit Patel, Romain Sauvestre, Ronnie Polidorio, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthi, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Barua, Spencer Whitman, Sten Sootla, Stephane Colot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaev, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vlada Petrovic, Weiwei Chu, Wenhan Xiong, Wenyu Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuexue Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesen-berg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymar, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Tessugine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcueta, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozenge1, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florrez, Gabriella Schwarz, Gada Badee, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojaianzeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gay, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelen, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Mattan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhinii Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinin, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghatham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuvigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Victor Albiero, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescuc, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. 2024. The llama 3
herd of models. Preprint, arXiv:2407.21783.

Sorourasdad Fatemi and Yuheng Hu. 2024. Enhancing financial question answering with a multi-agent reflection framework. In Proceedings of the 5th ACM International Conference on AI in Finance, pages 530–537.

Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Pal: program-aided language models. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org.

Parker Glenn, Parag Dakle, Liang Wang, and Preethi Raghavan. 2024. BlendSQL: A scalable dialect for unifying hybrid question answering in relational algebra. In Findings of the Association for Computational Linguistics: ACL 2024, pages 453–466, Bangkok, Thailand. Association for Computational Linguistics.

Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186.

Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567–2577, Hong Kong, China. Association for Computational Linguistics.

Yoonjoo Lee, Kyungjae Lee, Sunghyun Park, Dasol Hwang, Jaehyeon Kim, Hong-In Lee, and Moontae Lee. 2023. QASA: Advanced question answering on scientific articles. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 19036–19052. PMLR.

Zicheng Lin, Zhibin Gou, Tian Liang, Ruilin Luo, Haowei Liu, and Yujiu Yang. 2024. CriticBench: Benchmarking LLMs for critique-correct reasoning. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1552–1587, Bangkok, Thailand. Association for Computational Linguistics.

Xinyuan Lu, Liangming Pan, Qian Liu, Preslav Nakov, and Min-Yen Kan. 2023. SCITAB: A challenging benchmark for compositional reasoning and claim verification on scientific tables. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7787–7813, Singapore. Association for Computational Linguistics.

Nafise Sadat Moosavi, Andreas Rücklé, Dan Roth, and Iryna Gurevych. 2021. Scigen: a dataset for reasoning-aware text generation from scientific tables. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shayman Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgium, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Justin Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jiang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Hee woo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondratiuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emry Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussee, Nick Ry-
der, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillett, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Wilner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qinming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Junting Zhuang, William Zhuk, and Barret Zoph. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774.

Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng. 2018. emrQA: A large corpus for question answering on electronic medical records. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2357–2368, Brussels, Belgium. Association for Computational Linguistics.

Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. 2024. Automatically correcting large language models: Surveying the landscape of diverse automated correction strategies. Transactions of the Association for Computational Linguistics, 12:484–506.

Dimitris Pappas, Ion Androutsopoulos, and Haris Papa-georgiou. 2018. BioRead: A new dataset for biomedical reading comprehension. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).

Dimitris Pappas, Petros Stavropoulos, Ion Androutso- poulos, and Ryan McDonald. 2020. BioMRC: A dataset for biomedical machine reading comprehen- sion. In Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing, pages 140–149, Online. Association for Computational Linguis- tics.

Shraman Pramanick, Rama Challappa, and Subhashini Venugopalan. 2024. Spiqa: A dataset for multimodal question answering on scientific papers. NeurIPS.

Qi Shi, Han Cui, Haofeng Wang, Qingfu Zhu, Wanxiang Che, and Ting Liu. 2024. Exploring hybrid question answering via program-based prompting. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11035–11046, Bangkok, Thailand. Association for Computational Linguistics.

Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314.

Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu. 2024. SciEval: A multi-level large language model evaluation benchmark for scientific research. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19053–19061.

Ross Taylor, Marcin Kardas, Guillem Curcurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085.

Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Lei Han, Haitao Mi, and Dong Yu. 2024. Toward self-improvement of LLMs via imagination, searching, and criticizing. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopou- los, et al. 2015. An overview of the bioasq large-scale biomedical semantic indexing and question answering competition. BMC bioinformatics, 16:1–28.

Dingzirui Wang, Longxu Dou, and Wanxiang Che. 2022. A survey on table-and-text hybridqa: Concepts, meth- ods, challenges and future directions. arXiv preprint arXiv:2212.13465.

Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramanium, Arjun Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. 2023. SCIBENCH: Evaluating college-level scientific problem-solving abilities of large language mod- els. In The 3rd Workshop on Mathematical Reason- ing and AI at NeurIPS’23.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models. Advances in neural information processing systems, 35:24824–24837.

Dayong Wu, Jiaqi Li, Baoxin Wang, Honghong Zhao, Siyuan Xue, Yanjie Yang, Zhijun Chang, Rui Zhang, Li Qian, Bo Wang, Shijin Wang, Zhixiong Zhang, and Guoping Hu. 2024a. SparkRA: A retrieval-augmented knowledge service system based on spark large language model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing: System Demonstrations, pages 382–389, Miami, Florida, USA. Association for Computational Linguistics.
Xianjie Wu, Jian Yang, Linzheng Chai, Ge Zhang, Jiaheng Liu, Xinrun Du, Di Liang, Daixin Shu, Xianfu Cheng, Tianzhen Sun, et al. 2024b. Tablebench: A comprehensive and complex benchmark for table question answering. arXiv preprint arXiv:2408.09174.

Yiyao Yu, Yuxiang Zhang, Dongdong Zhang, Xiao Liang, Hengyuan Zhang, Xingxing Zhang, Ziyi Yang, Mahmoud Khademi, Hany Awadalla, Junjie Wang, Yujiu Yang, and Furu Wei. 2025. Chain-of-reasoning: Towards unified mathematical reasoning in large language models via a multi-paradigm perspective. Preprint, arXiv:2501.11110.

Dan Zhang, Ziniu Hu, Sining Zhoubian, Zhengxiao Du, Kaiyu Yang, Zihan Wang, Yisong Yue, Yuxiao Dong, and Jie Tang. 2024. Sciinstruct: a self-reflective instruction annotated dataset for training scientific language models. In The Thirty-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.

Zhehao Zhang, Xitao Li, Yan Gao, and Jian-Guang Lou. 2023. CRT-QA: A dataset of complex reasoning question answering over tabular data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2131–2153, Singapore. Association for Computational Linguistics.

Yilun Zhao, Lyuhao Chen, Arman Cohan, and Chen Zhao. 2024a. TaPERA: Enhancing faithfulness and interpretability in long-form table QA by content planning and execution-based reasoning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12824–12840, Bangkok, Thailand. Association for Computational Linguistics.

Yilun Zhao, Hongjun Liu, Yitao Long, Rui Zhang, Chen Zhao, and Arman Cohan. 2024b. Financemath: Knowledge-intensive math reasoning in finance domains. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12841–12858, Bangkok, Thailand. Association for Computational Linguistics.

Yilun Zhao, Yitao Long, Hongjun Liu, Ryo Kamoi, Linyong Nan, Lyuhao Chen, Yixin Liu, Xiangru Tang, Rui Zhang, and Arman Cohan. 2024c. DocMath-eval: Evaluating math reasoning capabilities of LLMs in understanding long and specialized documents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16103–16120, Bangkok, Thailand. Association for Computational Linguistics.

Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3277–3287, Online. Association for Computational Linguistics.
A  Comparison with Previous Datasets

A.1  Comparison with previous scientific QA datasets

Table 1 presents the comparison of SciTAT with previous scientific QA datasets. We first introduce the existing datasets. BioRead (Pappas et al., 2018) is a cloze-style QA dataset on biomedical papers, which only contains the reasoning type of Look Up and focuses only on the text. QASA (Lee et al., 2023) is a QA dataset on papers in AI and ML fields, but only concentrates on the text in papers, and lacks the reasoning type of Tabulation. SciGen (Moosavi et al., 2021) aims to generate descriptions according to the tables in the papers in the field of Computer Science. SciTab (Lu et al., 2023) aims to judge the claims according to the scientific tables in the field of Computer Science, which only contains the reasoning types of Lok Up and Numerical Reasoning. SPIQA (Pramanick et al., 2024) is a multimodal QA dataset on scientific papers, which only focuses on the split text and tables, ignoring the relevance between tables and text, and lacking the reasoning type of Data Analysis and Tabulation. It can be seen that SciTAT contains more diverse reasoning types and considers the relevance between tables and text.

A.2  Comparison with previous QA datasets for tables and text

Table 7 presents the comparison of SciTAT with previous QA datasets over tabular and textual data and scientific QA datasets. It can be seen that SciTAT contains more diverse and closer to real-life user questions. Moreover, SciTAT requires the model to possess domain-specific knowledge in the scientific field. It must not only understand the dense terminology commonly found in papers but also apply this knowledge to solve questions, which is not required by other datasets.

B  Prompt

In this section, we show the prompts we use to synthesize data and conduct experiments.

B.1  Prompt for generating data

Table 8 provides the prompt for generating questions, rationales, and answers when constructing SciTAT.

B.2  Prompt for experiments

Table 9 shows the prompt to build the baselines in our experiments, and Table 10 shows the prompt used by CAR. The prompts of Three-Agent we use are referred to the original paper (Fatemi and Hu, 2024).

C  Manual Annotation Procedure

C.1  Annotator training process

We recruit students from Computer Science or Artificial Intelligence programs who are willing to participate in the annotation task, offering a compensation of $1 per instance. Initially, we provide a detailed explanation of the task, including its definition, the specific responsibilities of the annotators, and how to use the annotation interface. We thoroughly explain the requirements for the questions, rationales, and answers, as well as how to select the source of the answers, as stated in §2.3. Additionally, we provide three examples and explain possible scenarios that might arise. Finally, we clarify the annotation deadline and inform them that the data will undergo additional checks. To promptly detect any errors or biases in the annotations, we sent the data in batches. After the two-round validation on the already annotated data, we communicate with the annotators to address any issues and proceed to send the next batch of data.

C.2  Statistics of the manual annotation procedure

On average, annotating a single data point required 10 minutes per annotator. The annotation process for the 953 instances was completed in approximately two months. The first round of annotations was conducted by 10 annotators, with two additional annotators performing two-round validation. Overall, 43.9% of the initially generated data is filtered out, and the remaining 43.5% of initial answers are modified. This process involved correcting incorrect answers and streamlining responses to ensure accuracy and reduce LLM-induced biases.

C.3  Annotating interface

The annotation process is conducted using a custom tool developed by us. Figure 7, Figure 8, Figure 9, and Figure 10 show the overall user interface for the manual annotation.
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Domain</th>
    <th colspan="4">Reasoning Type</th>
  </tr>
  <tr>
    <th>Look Up</th>
    <th>Numerical Reasoning</th>
    <th>Data Analysis</th>
    <th>Tabulation</th>
  </tr>
  <tr>
    <td>HybridQA (Chen et al., 2020)</td>
    <td>Wiki</td>
    <td>✓</td>
    <td>×</td>
    <td>×</td>
    <td>×</td>
  </tr>
  <tr>
    <td>TAT-QA (Zhu et al., 2021)</td>
    <td>Finance</td>
    <td>✓</td>
    <td>✓</td>
    <td>×</td>
    <td>×</td>
  </tr>
  <tr>
    <td>FinQA (Chen et al., 2021)</td>
    <td>Finance</td>
    <td>✓</td>
    <td>✓</td>
    <td>×</td>
    <td>×</td>
  </tr>
  <tr>
    <td>DocMath-Eval (Zhao et al., 2024c)</td>
    <td>Finance</td>
    <td>✓</td>
    <td>✓</td>
    <td>×</td>
    <td>×</td>
  </tr>
  <tr>
    <td>FinanceMATH (Zhao et al., 2024b)</td>
    <td>Finance</td>
    <td>✓</td>
    <td>✓</td>
    <td>×</td>
    <td>×</td>
  </tr>
  <tr>
    <td>SciTAT</td>
    <td>Science</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
</table>

Table 7: Comparison of SciTAT to recent QA datasets over tabular and textual data. Wiki denotes Wikipedia.

The prompt for Generating Questions

{Table}
{Paragraph}
You are a highly intelligent and obedient academic field question generation system.
Generate a question referring to the table and paragraph above which meets the requirements in the question description "[Type]".
The generated question must meet:
1. The question should be with fewer statements and more reasoning and calculation.
2. The question must be answerable based on the paragraph alone, and not answerable only based on the table.
3. The question must meet the question description.
4. Do not generate multiple questions or sub-questions at once.
Examples:
{Examples}

The prompt for Generating Rationales and Answers

{Table}
{Paragraph}
Based on the information in the Table and Paragraph, please answer the question "[Question]".
Represent your answer with: "Reason: <Your Reason> Answer: <Your Answer>"
If there are multiple questions, you need to answer them one by one, and the answers are separated by "
".
Examples:
{Examples}

Table 8: The prompts for generating the questions, rationales, and answers of SciTAT.

D Case Study for Error Analysis

In this section, we show examples of different error types, as shown in Figure 11, Figure 12, Figure 13, Figure 14, and Figure 15.

E Additional Experiments

In this section, we present additional experiments.

E.1 Results of Few-shot Prompts

In this subsection, we present the performance of CAR with the few-shot prompts in the short-context setting, as shown in Table 11. Specifically, we annotate 4 demonstrations, each corresponding to one of the four reasoning types in SciTAT. It can be observed that the performance of SciTAT with the few-shot prompts outperforms that under the zero-shot prompts.

E.2 Statistics of the Number of Output Tokens

In this subsection, we show the comparison of the number of tokens output by different methods and the number of tokens in the gold answers. (i) It can be found that the number of tokens output by PoT is consistently lower than that of other methods, whether it is a short-form answer or a free-form answer, which explains, to a certain extent the reason why PoT has low performance, especially on the free-form answers. (ii) On the contrary, the number of tokens output by the Direct QA is generally high, which also reveals the reason why its EM is 0 on the short-form answers. (iii) And CAR is the closest in quantity to the number of tokens of gold answer, which shows that CAR can adapt to obtain answers of various reasoning types.
The prompt for DirectQA

Based on the information in the Table and Paragraph, you should answer the question.
If there are multiple questions, you need to answer them one by one, and the answers are separated by "

".
Table (including its label, caption, and content):
{Table}
Paragraph:
{Paragraph}

Please answer the question "{Question}".

The prompt for CoT

Based on the information in the Table and Paragraph, you should answer the question.
Represent your answer with: "Reason: <Your Reason> Answer: <Your Answer>".
If there are multiple questions, you need to answer them one by one, and the answers are separated by "

".
Table (including its label, caption and content):
{Table}
Paragraph:
{Paragraph}

Please answer the question "{Question}".

The prompt for PoT

Table (including its label, caption and content):
{Table}
Paragraph:
{Paragraph}
Read the above Table and Paragraph, and then write code to answer the question "{Question}".
Please **directly use** the information such as numbers in tables and paragraphs, do not define tables and then process them.
You must return the answer 'ans = ' at the end of the code instead of 'print'.
Attention that if there are multiple questions, you need to answer them one by one, and the answers are separated by "

".

<table>
  <tr>
    <th colspan="2">Table 9: The prompts for baselines.</th>
  </tr>
</table>

E.3 How does CAR alleviate the challenges in SciTAT?

E.3.1 Challenge of handling tables and text
To mitigate the heterogeneity of evidence, the Calculator extracts relevant information from both the table and text, performs calculations, and provides numerical information. This numerical information reduces the challenges of heterogeneous evidence that the Reasoner must process to reach a final answer.

We demonstrate that CAR using Llama3.1-70B outperforms other baselines on different answer sources, as shown in Table 13. We can find that (i) It is evident that CAR consistently and significantly outperforms other baselines on almost all answer sources, especially on Table-and-Text, indicating its ability to mitigate the challenges posed by heterogeneous evidence. (ii) When the answer source is text, the performance of CAR is similar to that of CoT, as the density of numerical information in text is lower than in tables, reducing the need for the Calculator to extract and compute relevant information.

E.3.2 Challenge of diverse reasoning types
We first detail how CAR addresses each reasoning type. (i) Look Up: The calculator searches for tables or spans that meet the conditions. The reasoner further verifies the correctness of the results and returns the final outcome. (ii) Numerical Reasoning: The calculator enhances numerical computation performance, while the reasoner extracts relevant information from the returned information to derive the final answer. (iii) Data Analysis: The calculator first computes the relevant numerical information. However, it is not adept at analyzing free-form answers (Wu et al., 2024b), whereas the
The prompt for Calculator

Table (including its label, caption, and content):
{Table}
Paragraph:
{Paragraph}
Read the above Table and Paragraph, and then write code to answer the question "{Question}".
Please **directly use** the information such as numbers in tables and paragraphs, do not define tables and then process them.
You must return the answer 'ans = ' at the end of the code instead of 'print'.
You cannot return just one or a few numbers or words, you must return a complete sentence.

The prompt for Reasoner

Based on the Table and Paragraph with the Tips, you should answer the question.
Please determine whether the tips are correct, use the tips reasonably in Reason, and organize the Answer into an appropriate form.
Represent your answer with: "Reason: <Your Reason> Answer: <Your Answer>".
Attention that if there are multiple questions, you need to answer them one by one, and the answers are separated by "

".

Table (including its label, caption, and content):
{Table}
Paragraph:
{Paragraph}
Tips:
{Tips}
Please answer the question "{Question}".

<table>
  <tr>
    <th>Model</th>
    <th>Scale</th>
    <th>Method</th>
    <th>EM</th>
    <th>F1</th>
    <th>BERTScore</th>
  </tr>
  <tr>
    <td rowspan="2">Llama3.1</td>
    <td>8B</td>
    <td>CAR (zero-shot)</td>
    <td>24.2</td>
    <td>44.3</td>
    <td>73.2</td>
  </tr>
  <tr>
    <td></td>
    <td>CAR (few-shot)</td>
    <td><b>28.7</b></td>
    <td><b>48.0</b></td>
    <td><b>76.2</b></td>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td>70B</td>
    <td>CAR (zero-shot)</td>
    <td>40.7</td>
    <td>46.2</td>
    <td>74.4</td>
  </tr>
  <tr>
    <td></td>
    <td>CAR (few-shot)</td>
    <td><b>43.4</b></td>
    <td><b>46.8</b></td>
    <td><b>74.6</b></td>
  </tr>
</table>

Table 10: The prompts for CAR.

Table 11: The results of CAR with few-shot prompts compared with zero-shot prompts.

reasoner continues the inference based on the information to generate the comprehensive answer.
(iv) Tabulation: The calculator efficiently computes and extracts relevant sub-tables, and the reasoner checks the returned sub-tables to derive the final answer.

We demonstrate the performance of CAR using Llama3.1-70B in comparison to other baselines across different reasoning types, as shown in Table 14. Results reveal that: (i) CAR consistently and significantly outperforms other baselines across almost all reasoning types. (ii) For numerical reasoning, the EM of CAR is slightly lower than that of Three-Agent, as Three-Agent uses two critic agents to separately examine and provide feedback on extracting relevant information and numerical calculations, allowing for more detailed error detection and correction in numerical reasoning questions.

E.4 Influence of Order on CAR

To study the influence of order on the performance of CAR, we conduct experiments by reversing the two modules, with results presented in Table 15. Specifically, we first apply the Reasoner module and then feed its output into the Calculator, which verifies and corrects any numerical errors to produce the final result. The significant performance drop confirms the validity of CAR. The results suggest that depending on the program output for the final answer limits performance on SCITAT since the program struggles with free-form responses, as discussed in §4.2. The lower performance of PoT, as shown in Table 5, also supports our view.
Paper Information

URL: http://arxiv.org/pdf/2006.08332v1

Reasoning Type: Domain knowledge calculation

Question: Based on the information provided in the paragraph and the table, if the Statistical Machine Translation system’s dataset consists of 24k parallel sentences and achieves a BLEU score of 41, estimate the BLEU score improvement necessary for the Proposed neural machine translation system to achieve an average BLEU score improvement of 10% across similar datasets.

Predicted Answer: Reason: The Statistical Machine Translation (SMT) system has a BLEU score of 41. A 10% improvement over this score would mean a BLEU score improvement of 10% of 41. Therefore, the necessary BLEU score improvement for the Proposed neural machine translation system over the SMT system would be 0.10 * 41. Answer: 4.1 (an improvement to a BLEU score of 45.1 is needed for a 10% improvement).

Paragraph

<section (Sanskrit Hindi Machine Translation Systems) label=Sanskrit Hindi Machine Translation Systems> The comparisons of the existing and proposed system are shown in Table. \ref{tab:comp} \begin{itemize} \item[subsection(Sanskrit-Hindi Anusarka-2009)] item Approach:It is a textbf(rule based) MT system \cite{bharati2009anusarka}. item The insights for the system are taken from Panini’s Ashtadhyayi. item Developed By:Chinmaya International Foundation (CIF), Indian Institute of Information Technology, Hyderabad (IIIT-H) and University of Hyderabad at Department of Sanskrit Studies. item Tool:textbf(Samsadhan) It is a Language accessor cum machine translation system. Input can be of an of the following encodings Unicode–Devanagari, WX-alphabetic, Itrans 5.3, Velthuis (VH), Harvard Kyoto (KH), Sanskrit Library Project (SLP). Output can be displayed in either Devanaaari script or in Roman Diacritical Notation. The system has a Sanskrit language analyzer which does the analysis.</itemize></section>

Figure 7: The user interface, showing the paper information and the paragraph.

F Discussion of Details

F.1 Why papers from three subfields are selected?

(i) The three subfields selected in SciTAT follow mainstream science datasets, including QASPER (Dasigi et al., 2021), QASA (Lee et al., 2023), SciGen (Moosavi et al., 2021), and Sci-iTab (Lu et al., 2023). (ii) Our primary focus is on addressing diverse scientific questions and reasoning types, which are independent of the specific domains.

F.2 Why the baselines are chosen?
The selected baselines in the main experiments follow those used in previous datasets (Sci-iTab (Lu et al., 2023), SciEval (Sun et al., 2024), SPIQA (Pramanick et al., 2024), SciBench (Wang et al., 2023)). To further validate the necessity of SciTAT and the effectiveness of CAR, we add Three-Agent (Fatemi and Hu, 2024), which is the state-of-the-art method for reasoning on both tables and text. The previous methods in the scientific question answering (SQA) task primarily focus on model training (Dasigi et al., 2021; Lee et al., 2023; Zhang et al., 2024), which makes direct comparisons with CAR infeasible. Therefore, we do not compare CAR with the baselines in the SQA task.
<table>
  <tr>
    <th colspan="3">Table 1:</th>
  </tr>
  <tr>
    <th>Label:</th>
    <td>tab:comp</td>
    <th></th>
  </tr>
  <tr>
    <th>Caption:</th>
    <td colspan="2">Comparison of Sanskrit Hindi Translation systems</td>
  </tr>
  <tr>
    <th>\textbf{Sanskrit-Hindi (Rule Based)}</th>
    <th>\textbf{Sanskrit-Hindi (Statistical)}</th>
    <th>\textbf{Sanskrit Hindi (Proposed)}</th>
  </tr>
  <tr>
    <td>The system fails when extended to large domains. It is developed for domains like kids stories, building interactive media and e-learning substance for kids.</td>
    <td>Sanskrit-Hindi text corpora has been collected or developed manually from the literature, health, news and tourism domains.</td>
    <td>The proposed system is based on neural machine translation technique that covers all domains in general based on the dataset used.</td>
  </tr>
  <tr>
    <td>Hand crafted rules based on Panini's Ashtadhyayi.</td>
    <td>Based on Bayes theorem. Similar words assigned random numbers.</td>
    <td>Word embedding used, similar words have close numbers.</td>
  </tr>
  <tr>
    <td>Separate modules like tokenizer, sandhi splitter, morph analyser, parser, word sense disambiguation, part of speech tagger, chunker, Hindi lexical transfer and a Hindi language generator used to get translation.</td>
    <td>SMT system have three separate main components--The translation model, reordering model, and the language model.SMT would evaluate fluency of a sentence in a target language a few words at a time using N-gram language model.</td>
    <td>Proposed system learns complex relationship between languages as one single model. Proposed model considers the entire sentence.</td>
  </tr>
</table>

Figure 8: The user interface, showing the tables.

![Screenshot of a table comparing Sanskrit Hindi Translation systems](page_172_154_1102_393.png)

<table>
  <tr>
    <th colspan="2">Annotation Information</th>
  </tr>
  <tr>
    <th>Correct Reasoning Type</th>
    <td>Arithmetic Caculation</td>
  </tr>
  <tr>
    <th>Revised Question</th>
    <td>Based on the information provided in the paragraph and the table, if the Statistical Machine Translation system’s dataset consists of 24k parallel sentences and achieves a BLEU score of 41, estimate the BLEU score improvement necessary for the Proposed neural machine translation system to achieve an average BLEU</td>
  </tr>
  <tr>
    <th>Revised Rationale</th>
    <td>Reason: The Statistical Machine Translation (SMT) system has a BLEU score of 41. A 10% improvement over this score would mean a BLEU score improvement of 10% of 41. Therefore, the necessary BLEU score improvement for the Proposed neural machine translation system over the SMT system would be 0.10 * 41</td>
  </tr>
  <tr>
    <th>Revised Answer</th>
    <td>4.1</td>
  </tr>
</table>

Figure 9: The user interface, showing the annotation information.

![Screenshot of annotation information](page_172_670_1102_393.png)
Figure 10: The user interface, showing the choice for the answer source, relevant tables, and if to remain.

<table>
  <tr>
    <th></th>
    <th>SE-All</th>
    <th>SE-eco</th>
    <th>TB</th>
    <th>WP-All</th>
    <th>WP-eco</th>
    <th>WP-Top-1</th>
    <th>WP-Top-2</th>
    <th>Zeta-All</th>
  </tr>
  <tr>
    <th>x-values</th>
    <td>1.865 ± 0.347</td>
    <td>1.865 ± 0.347</td>
    <td>1.865 ± 0.347</td>
    <td>1.865 ± 0.347</td>
    <td>1.865 ± 0.347</td>
    <td>1.865 ± 0.347</td>
    <td>1.865 ± 0.347</td>
    <td>1.865 ± 0.347</td>
  </tr>
  <tr>
    <th>p-values</th>
    <td>0.996 ± 0.002</td>
    <td>0.996 ± 0.002</td>
    <td>0.996 ± 0.002</td>
    <td>0.996 ± 0.002</td>
    <td>0.996 ± 0.002</td>
    <td>0.996 ± 0.002</td>
    <td>0.996 ± 0.002</td>
    <td>0.996 ± 0.002</td>
  </tr>
</table>

Table 7: P-values of the Kolmogorov-Smirnov goodness-of-fit test applied to the pairwise combinations of the x and y values of the distributions in Fig. 6.

Figure 11: The case for the error type of Miss.

<table>
  <tr>
    <th>Age</th>
    <th>Sleep Hours</th>
    <th>awake -2H</th>
    <th>occ. sent. per day</th>
    <th>Analysis 1</th>
    <th>Analysis 2</th>
  </tr>
  <tr>
    <td>Infant</td>
    <td>1</td>
    <td>16</td>
    <td>5,760</td>
    <td>1.1</td>
    <td>1.1</td>
  </tr>
  <tr>
    <td>Young child</td>
    <td>2</td>
    <td>16</td>
    <td>5,760</td>
    <td>2.1</td>
    <td>3.2</td>
  </tr>
  <tr>
    <td>3</td>
    <td>14</td>
    <td>8,720</td>
    <td>3.5</td>
    <td>6.0</td>
  </tr>
  <tr>
    <td>4–5</td>
    <td>12</td>
    <td>10,960</td>
    <td>5.3</td>
    <td>9.5</td>
  </tr>
  <tr>
    <td>Beg.prim.school</td>
    <td>6</td>
    <td>12</td>
    <td>9,600</td>
    <td>8.8</td>
    <td>16.5</td>
  </tr>
  <tr>
    <td>7–9</td>
    <td>11</td>
    <td>11,050</td>
    <td>10.7</td>
    <td>20.3</td>
  </tr>
  <tr>
    <td>End prim. school</td>
    <td>10</td>
    <td>11</td>
    <td>11,500</td>
    <td>16.5</td>
    <td>31.9</td>
  </tr>
  <tr>
    <td>11</td>
    <td>10</td>
    <td>12</td>
    <td>11,520</td>
    <td>18.6</td>
    <td>32.3</td>
  </tr>
  <tr>
    <td>12</td>
    <td>10</td>
    <td>12</td>
    <td>11,520</td>
    <td>20.7</td>
    <td>40.3</td>
  </tr>
  <tr>
    <td>13</td>
    <td>9</td>
    <td>13</td>
    <td>12,480</td>
    <td>23.0</td>
    <td>44.9</td>
  </tr>
  <tr>
    <td>End middle school</td>
    <td>14</td>
    <td>8</td>
    <td>14,400</td>
    <td>35.4</td>
    <td>58.8</td>
  </tr>
  <tr>
    <td>End high school</td>
    <td>18</td>
    <td>8</td>
    <td>14,340</td>
    <td>35.2</td>
    <td>69.4</td>
  </tr>
  <tr>
    <td>19–20</td>
    <td>8</td>
    <td>14</td>
    <td>14,440</td>
    <td>37.7</td>
    <td>70.6</td>
    <td>1.2</td>
    <td>1.2</td>
  </tr>
  <tr>
    <td>End Bachelor</td>
    <td>21–22</td>
    <td>8</td>
    <td>14</td>
    <td>14,440</td>
    <td>42.6</td>
    <td>73.1</td>
    <td>3.7</td>
    <td>3.7</td>
  </tr>
  <tr>
    <td>End Master</td>
    <td>23</td>
    <td>8</td>
    <td>14</td>
    <td>14,440</td>
    <td>47.5</td>
    <td>75.5</td>
    <td>6.1</td>
    <td>6.1</td>
  </tr>
</table>

Table 1: Cumulative number of sentences read or heard by a professional translator

Figure 12: The case for the error type of Locate.
The results from the COMPAS dataset are detailed in Fig. 3, where minority population density, disparity, and loss measures are shown. In this scenario, the optimal control model shows comparable performance to RL-based algorithms. Specifically, TRPO and PPO reach a terminal state with a loss value of 0.574, which achieves the same level of performance compared to the proposed optimal control method. This similarity in performance is attributed to the minor disparity in representation between different gender attributes within the COMPAS dataset.

<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="6">Environment \( M^1 \)</th>
    <th colspan="6">Environment \( M^2 \)</th>
  </tr>
  <tr>
    <th>Fair-agnostic</th>
    <th>Fair-aware</th>
    <th>Dynamic-aware</th>
    <th>Fair-agnostic</th>
    <th>Fair-aware</th>
    <th>Dynamic-aware</th>
  </tr>
  <tr>
    <td>ERM</td>
    <td>Minimax</td>
    <td>DRO</td>
    <td>PG</td>
    <td>TRPO</td>
    <td>PPO</td>
    <td>Optim</td>
    <td>ERM</td>
    <td>Minimax</td>
    <td>DRO</td>
    <td>PG</td>
    <td>TRPO</td>
    <td>PPO</td>
    <td>Optim</td>
  </tr>
  <tr>
    <td>Density \( \uparrow \)</td>
    <td>0.271</td>
    <td>0.256</td>
    <td>0.184</td>
    <td>0.263</td>
    <td>0.264</td>
    <td>0.229</td>
    <td>0.271</td>
    <td>0.256</td>
    <td>0.184</td>
    <td>0.263</td>
    <td>0.264</td>
    <td>0.229</td>
    <td>0.271</td>
  </tr>
  <tr>
    <td>Disparity \( \downarrow \)</td>
    <td>0.529</td>
    <td>0.529</td>
    <td>0.141</td>
    <td>0.425</td>
    <td>0.484</td>
    <td>0.514</td>
    <td>0.529</td>
    <td>0.529</td>
    <td>0.141</td>
    <td>0.425</td>
    <td>0.484</td>
    <td>0.514</td>
    <td>0.529</td>
  </tr>
  <tr>
    <td>Loss \( \downarrow \)</td>
    <td>0.770</td>
    <td>0.803</td>
    <td>1.391</td>
    <td>0.848</td>
    <td>0.808</td>
    <td>0.588</td>
    <td>0.770</td>
    <td>0.803</td>
    <td>1.391</td>
    <td>0.848</td>
    <td>0.808</td>
    <td>0.588</td>
    <td>0.770</td>
  </tr>
</table>

Figure 13: The case for the error type of Calculate.

In particular, given a graph, we first extract all cycles out of it. Then, all edges that are not inside the cycles are considered motifs. We consider combining cycles with more than two coincident nodes into a motif. Although this method cannot extract complex motifs like single-input and multi-input motifs, it can generate the most important motifs, such as ring structures in biochemical molecules and the feed-forward loop motif. By adopting this simple but general motif extraction method, we can explain a GNN model without any domain knowledge, making our explanation model more applicable. Need to be noted that, even though the motif extraction rule cannot extract single-input and multi-input motifs, these motifs can be implicitly identified by our attention layer. Experiments in the table 2 demonstrate it.

<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="2">MUTAG</th>
    <th colspan="2">FTC</th>
    <th colspan="2">PROTEINS</th>
    <th colspan="2">IMDB</th>
    <th colspan="2">BA-Motif</th>
    <th colspan="2">BA-Shape</th>
  </tr>
  <tr>
    <th>S=0.7</th>
    <th>S=0.7</th>
    <th>S=0.7</th>
    <th>S=0.7</th>
    <th>S=0.7</th>
    <th>S=0.7</th>
    <th>S=0.7</th>
    <th>S=0.7</th>
    <th>S=0.7</th>
    <th>S=0.7</th>
    <th>S=0.7</th>
    <th>S=0.7</th>
  </tr>
  <tr>
    <td>CNNExplainer</td>
    <td>0.280</td>
    <td>0.441</td>
    <td>0.365</td>
    <td>0.483</td>
    <td>0.365</td>
    <td>0.742</td>
    <td>0.525</td>
    <td>0.525</td>
    <td>0.525</td>
    <td>0.525</td>
    <td>0.525</td>
    <td>0.525</td>
  </tr>
  <tr>
    <td>SelfgnnX</td>
    <td>0.287</td>
    <td>0.441</td>
    <td>0.365</td>
    <td>0.483</td>
    <td>0.365</td>
    <td>0.742</td>
    <td>0.525</td>
    <td>0.525</td>
    <td>0.525</td>
    <td>0.525</td>
    <td>0.525</td>
    <td>0.525</td>
  </tr>
  <tr>
    <td>MotifExplainer</td>
    <td>0.081</td>
    <td>0.129</td>
    <td>0.115</td>
    <td>-0.030</td>
    <td>0.101</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>1.0</td>
  </tr>
</table>

Figure 14: The case for the error type of Knowledge.

4 XAI Queries as Computation Problems

In this section, we consider successively the 9 XAI queries from (Audemard, Koriche, and Marquis 2020), as listed in Table 1, and we present them in formal terms.

<table>
  <tr>
    <th>XAI query</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>EMC</td>
    <td>Enumerating Minimum-Cardinality explanations</td>
  </tr>
  <tr>
    <td>DPI</td>
    <td>Deriving one Prime Implicant explanation</td>
  </tr>
  <tr>
    <td>ECC</td>
    <td>Enumerating Complementary Covering explanations</td>
  </tr>
  <tr>
    <td>CIN</td>
    <td>Counting the Instances associated with a given class</td>
  </tr>
  <tr>
    <td>EIN</td>
    <td>Enumerating the Instances associated with a given class</td>
  </tr>
  <tr>
    <td>IMA</td>
    <td>Identifying Mandatory features or forbidden features in a given class</td>
  </tr>
  <tr>
    <td>IR</td>
    <td>Identifying Relevant features in a given class</td>
  </tr>
  <tr>
    <td>IMO</td>
    <td>Identifying Monotone (or anti-monotone) features in a given class</td>
  </tr>
  <tr>
    <td>MCP</td>
    <td>Measuring Closeness of a class to a Prototype</td>
  </tr>
</table>

Table 1: Some XAI queries.

Figure 15: The case for the error type of Redundancy.

Question:
Which algorithm ranks the highest in terms of minimizing both disparity and loss across Environment $M^1$\\(last\_2\)?

Predicted Answer:
TRPO

Gold Answer:
TRPO, PPO, and Optim

Question:
Using the explanation method described in the paragraph, calculate the number of motifs in a biochemical molecule represented by a graph where all nodes form a single ring and there are no other connecting edges outside this cycle.

Predicted Answer:
1

Gold Answer:
0

Question:
If the task requires solving EMC and DPI queries successively, with EMC taking 10 minutes and DPI taking 5 minutes, what is the total time taken to perform 3 EMC queries and 2 DPI queries?

Predicted Answer:
3 EMC queries take 3 x 10 = 30 minutes, and 2 DPI queries take 2 x 5 = 10 minutes. Therefore, the total time taken to perform 3 EMC queries and 2 DPI queries is 30 + 10 = 40 minutes.

Gold Answer:
40 minutes
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Scale</th>
    <th rowspan="2">Method</th>
    <th colspan="2">Long-context</th>
    <th colspan="2">Short-context</th>
  </tr>
  <tr>
    <th>Short-form</th>
    <th>Free-form</th>
    <th>Short-form</th>
    <th>Free-form</th>
  </tr>
  <tr>
    <td>-</td>
    <td>-</td>
    <td>Gold Answer</td>
    <td>1.5</td>
    <td>45.1</td>
    <td>1.5</td>
    <td>45.1</td>
  </tr>
  <tr>
    <td rowspan="3">8B</td>
    <td></td>
    <td>Direct QA</td>
    <td>120.3</td>
    <td>197.7</td>
    <td>109.3</td>
    <td>135.1</td>
  </tr>
  <tr>
    <td></td>
    <td>CoT</td>
    <td>35.8</td>
    <td>75.4</td>
    <td>23.3</td>
    <td>74.4</td>
  </tr>
  <tr>
    <td></td>
    <td>PoT</td>
    <td>5.3</td>
    <td>26.6</td>
    <td>2.8</td>
    <td>25.1</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>CAR</td>
    <td>24.3</td>
    <td>54.8</td>
    <td>21.4</td>
    <td>53.8</td>
  </tr>
  <tr>
    <td rowspan="3">Llama3.1</td>
    <td></td>
    <td>Direct QA</td>
    <td>118.9</td>
    <td>220.1</td>
    <td>118.9</td>
    <td>220.1</td>
  </tr>
  <tr>
    <td></td>
    <td>CoT</td>
    <td>5.9</td>
    <td>44.2</td>
    <td>14.3</td>
    <td>46.2</td>
  </tr>
  <tr>
    <td></td>
    <td>PoT</td>
    <td>3.6</td>
    <td>30.0</td>
    <td>3.3</td>
    <td>36.4</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>CAR</td>
    <td>17.1</td>
    <td>43.5</td>
    <td>16.5</td>
    <td>43.0</td>
  </tr>
  <tr>
    <td rowspan="3">70B</td>
    <td></td>
    <td>Direct QA</td>
    <td>151.2</td>
    <td>213.2</td>
    <td>105.7</td>
    <td>141.6</td>
  </tr>
  <tr>
    <td></td>
    <td>CoT</td>
    <td>30.1</td>
    <td>84.5</td>
    <td>20.7</td>
    <td>69.1</td>
  </tr>
  <tr>
    <td></td>
    <td>PoT</td>
    <td>3.0</td>
    <td>24.4</td>
    <td>4.6</td>
    <td>52.4</td>
  </tr>
  <tr>
    <td rowspan="3">gpt-4o</td>
    <td></td>
    <td>Direct QA</td>
    <td>10.6</td>
    <td>67.0</td>
    <td>10.6</td>
    <td>82.4</td>
  </tr>
  <tr>
    <td></td>
    <td>CoT</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>PoT</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>CAR</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 12: Statistics of the number of tokens of gold answers and different results.

<table>
  <tr>
    <th>Method</th>
    <th>Table</th>
    <th>Text</th>
    <th>Table-and-Text</th>
  </tr>
  <tr>
    <td>DirectQA</td>
    <td>0.0, 32.7, 68.4</td>
    <td>0.0, 34.3, 68.4</td>
    <td>0.0, 34.6, 69.2</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>31.6, 45.0, 72.8</td>
    <td><b>30.8</b>, 44.9, <b>74.0</b></td>
    <td>33.3, 42.8, 73.2</td>
  </tr>
  <tr>
    <td>PoT</td>
    <td>39.7, 34.3, 62.2</td>
    <td>0.0, 36.6, 64.1</td>
    <td>34.4, 37.1, 66.3</td>
  </tr>
  <tr>
    <td>Three-Agent</td>
    <td>42.7, 44.4, 73.4</td>
    <td>23.1, 45.1, 73.7</td>
    <td>36.6, 43.2, 72.9</td>
  </tr>
  <tr>
    <td>CAR</td>
    <td><b>42.3</b>, <b>47.2</b>, <b>74.7</b></td>
    <td><b>30.8</b>, <b>45.2</b>, <b>72.8</b></td>
    <td><b>38.0</b>, <b>45.0</b>, <b>74.4</b></td>
  </tr>
</table>

Table 13: CAR using Llama3.1-70B compared with other baselines on different answer sources. The values in each cell represent EM, F1, and BERTScore, respectively.

<table>
  <tr>
    <th>Method</th>
    <th>Look Up</th>
    <th>Numerical Reasoning</th>
    <th>Data Analysis</th>
    <th>Tabulation</th>
  </tr>
  <tr>
    <td>DirectQA</td>
    <td>0.0, --, --</td>
    <td>0.0, 37.3, 32.9</td>
    <td>--, 32.9, 67.8</td>
    <td>0.0, 29.5, 72.6</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>42.2, --, --</td>
    <td>31.6, 43.8, 71.3</td>
    <td>--, 42.5, 72.0</td>
    <td>7.7, 54.4, 83.5</td>
  </tr>
  <tr>
    <td>PoT</td>
    <td>35.6, --, --</td>
    <td>37.9, 33.4, 60.8</td>
    <td>--, 34.63, 63.4</td>
    <td>15.4, 48.7, 75.0</td>
  </tr>
  <tr>
    <td>Three-Agent</td>
    <td>40.0, --, --</td>
    <td><b>41.8</b>, 44.0, 71.1</td>
    <td>--, 43.0, 72.6</td>
    <td>7.7, 49.8, 80.8</td>
  </tr>
  <tr>
    <td>CAR</td>
    <td><b>50.0</b>, --, --</td>
    <td><b>40.0</b>, <b>48.9</b>, <b>75.1</b></td>
    <td>--, <b>44.1</b>, <b>72.8</b></td>
    <td><b>23.1</b>, <b>51.6</b>, <b>83.3</b></td>
  </tr>
</table>

Table 14: CAR using Llama3.1-70B compared with other baselines on different reasoning types. The values in each cell represent EM, F1, and BERTScore, respectively. - indicates no corresponding answer type for the question.
<table>
  <tr>
    <th>Scale</th>
    <th>Method</th>
    <th>EM</th>
    <th>F1</th>
    <th>BERTScore</th>
  </tr>
  <tr>
    <td rowspan="2">8B</td>
    <td>CAR</td>
    <td><b>24.2</b></td>
    <td><b>44.3</b></td>
    <td><b>73.2</b></td>
  </tr>
  <tr>
    <td><i>Reversing</i></td>
    <td>21.5</td>
    <td>21.4</td>
    <td>47.2</td>
  </tr>
  <tr>
    <td rowspan="2">70B</td>
    <td>CAR</td>
    <td><b>40.7</b></td>
    <td><b>46.2</b></td>
    <td><b>74.4</b></td>
  </tr>
  <tr>
    <td><i>Reversing</i></td>
    <td>37.1</td>
    <td>35.9</td>
    <td>65.1</td>
  </tr>
</table>

Table 15: The results of CAR, compared with reversing the two modules (<i>Reversing</i>) in the short-context setting.

Reviewer paper summaries: This paper introduces SCITAT, a benchmark for question answering over scientific texts and tables, comprising 953 carefully curated questions from arXiv papers spanning four reasoning types. The authors propose CAR (Calculator and Reasoner), a modular framework that separates Program-of-Thoughts (PoT)-based computation from Chain-of-Thought (CoT)-based reasoning, and evaluate several large language models across various dimensions.

The paper introduces the SCITAT dataset, which focuses on the Scientific Question Answering (SQA) task. Compared to previous benchmarks, SCITAT encompasses various reasoning types, including Look Up, Numerical Reasoning, Data Analysis, and Tabulation, and involves joint reasoning over tables and text. 

The authors also propose a simple baseline called CAR, which includes a calculator (which generates the numerical information needed to answer the question given an input question and scientific context) and a reasoner (which produces the final answer based on the input question, scientific context, and the results from the calculator). Experiments were conducted on SCITAT.
Summary of strengths: 1. The paper provides a fine-grained categorization of 13 scientific reasoning types with manual annotations, enabling structured evaluation of QA models.
2. The CAR framework introduces a modular design that separates computation and reasoning, and its effectiveness is supported by ablation studies.
3. SCITAT is carefully designed to require reasoning over both tables and text, offering a realistic multi-modal QA benchmark.

1. This paper presents SCITAT, which offers a richer coverage of reasoning types compared to previous works and involves joint reasoning over tables and text. The dataset construction process is quite thorough.

2. The paper explores various baseline approaches, including CoT, PoT, and Agent, and conducts experiments using Qwen 2.5, LLama 3.1, and GPT-4o.
Summary of weaknesses: 1. The paper lacks analysis of performance differences between text- and table-based reasoning, despite emphasizing their integration.
2. The reasoning type distribution is imbalanced, with some categories underrepresented, potentially skewing evaluation results.
3. The dataset construction process is labor-intensive and difficult to scale, with over 70% of automatically generated QA pairs discarded.

1. CAR appears to be a weak baseline, implementing a standard pipeline with two prompts ("Read the table, text, and question, and return a program," and "Based on the table, text, and question, determine whether the information is correct and use it reasonably."). 
It doesn't seem to address the challenge of processing tables and text simultaneously, merely asking the LLM to read the table and text without any technical improvements. 
Moreover, it doesn't seem to tackle the challenges of the four reasoning types outlined in the introduction. 
According to lines 72-74 in the introduction, the solution is to “introduce the calculator and reasoner”, but it is not very convincing that these challenges can be addressed in this way.

2. The comparison methods are limited to basic approaches like CoT and PoT. Are there no other TableQA, TextQA and SQA methods that could be adapted to this benchmark?

3. What unique challenges does Scientific Question Answering (SQA) present compared to other TableQA and TextQA tasks? Technically, it seems it could still be categorized under TableQA/TextQA.

4. The dataset seems somewhat small, with only 953 questions from 871 papers. Does each paper only provide a few questions, and why?

5. Why is the reasoner considered a component that can be ablated if it is essential for obtaining the final answer? According to line 246, "Calculator output provides the numerical information necessary to answer the question," which implies that the calculator is used to provide intermediate results necessary to reach the final answer.

6. The description of the two components in Section 3 does not match the description in the prompt, which leaves me confused about whether the reasoner utilizes the information from the Calculator. 
Section 3.2 mentions that the reasoner's input includes the results from the calculator, yet the prompts in Figure 4 and Table 10 do not reflect this. What exactly is the Tip— is it the output of the calculator?

7. Why do EM scores drop to 0.3 and 0.0, while F1 scores are 30.8 and 42.0, after the reasoner is ablated?
Comments, suggestions and typos: NA

1. In Table 5, the font used for "gpt-4o" is inconsistent with the other methods.

2. The bar charts and x-axis in Figure 5/6 appear to be pieced together, and the caption font in Figure 5 is inconsistent with that in Figure 6.

3. Some annotation details could be moved to the appendix, allowing more space for the baseline. The introduction of the baseline in the main text is quite brief, with only eight lines in the introduction, and it abruptly ends after describing its composition. 
It is stated that it can address various reasoning problems, but no explanation is given as to why, making it less convincing.

4. There is a typo in the single quotes in Table 10.
Overall assessment scores: 3.5

3
Metareview score: 3
Metareview: This paper provides a new resource for scientific QA which emphasizes diverse reasoning processes over text and tables. Strong data quality controls are employed including annotator expertise, filtering, and training, as well as multiple validation reviews. A method using calculator and reasoning modules is proposed and compared against various recent prompting baselines, but the evaluation metrics used here are badly suited to free-form QA which comprises most of the dataset.
Summary of reasons to publish: The best reason to publish this paper is to promote its rigorous human validation and quality control processes, which demonstrates the degree of validation, correction, and revision which must be done after automatically generating datapoints even with powerful LLMs. 

Reviewers appreciate that the paper includes relevant baseline approaches using multiple contemporary LLMs. A detailed data analysis is provided which gives insights into reasoning processes in this domain. Error analysis and ablations are provided.