SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables
Authors: Xinyuan Lu
Liangming Pan
Qian Liu
Preslav Nakov
Min-Yen Kan
Venue: EMNLP 2023 Main (2023)
Keywords: Scientific Fact-Checking
Table Reasoning
Compositional Reasoning
Dataset
TL;DR: We present SCITAB, a challenging evaluation dataset consisting of 1.2K expert-verified scientific claims that 1) originate from authentic scientific publications and 2) require compositional reasoning for verification.

Abstract: Current scientific fact-checking benchmarks exhibit several shortcomings, such as biases arising from crowd-sourced claims and an over-reliance on text-based evidence. We present SCITAB, a challenging evaluation dataset consisting of 1.2K expert-verified scientific claims that 1) originate from authentic scientific publications and 2) require compositional reasoning for verification. The claims are paired with evidence-containing scientific tables annotated with labels. Through extensive evaluations, we demonstrate that SCITAB poses a significant challenge to state-of-the-art models, including table-based pretraining models and large language models. All models except GPT-4 achieved performance barely above random guessing. Popular prompting techniques, such as Chain-of-Thought, do not achieve much performance gains on SCITAB. Our analysis uncovers several unique challenges posed by SCITAB, including table grounding, claim ambiguity, and compositional reasoning. Our codes and data are publicly available at https://github.com/XinyuanLu00/SciTab.

SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables
Xinyuan Lu Liangming Pan +3 Qian Liu"
Preslav Nakov Min-Yen Kan²
ISEP Program, NUS Graduate School 2 National University of Singapore
UUniversity of California, Santa Barbara *Sea Al Lab MBBZAA
luxinyuanêu. nus, edu Liangminggan@ucsb.co edu liuqianêsea.com com kanmy@comp. nus. www.edu..........
p www.wnwbzuai.ccccom ac..ae
Abstract
However, these datasets still exhibit several limi- tations. First, the claims are crowd-sourced rather than collected from real scientific papers. This leads to problems such as bias in human annotation, a lack of diversity, and shallow claims that do not reflect the complexity of scientific reasoning. For example, most claims in Sci-Fact can be validated by single sentence in paper's abstract, which oversimplifies the scientific discourse. Second, the claims the existing benchmarks are solely vali- dated against text-based evidence, primarily paper abstracts. However, in many scientific processes, claims are intrinsically tied to quantitative experi- mental data, commonly presented in tables and fig- ures. This disparity highlights significant gap be- tween the existing benchmarks and real-world sci- entific fact-checking needs. To bridge these gaps, dataset that 1) compiles real-world claims from scientific papers, and 2) includes original scientific data such as tables and figures, is needed.
Current scientific fact-checking benchmarks ex- hibit several shortcomings, such as biases aris- ing from crowd-sourced claims and reliance on text-based evidence. We present SCITAB, challenging evaluation dataset con- sisting of 1.2K expert-verified scientific claims that 1) originate from authentic scientific publi- cations and 2) require compositional reason- ing for verification. The claims are paired with evidence-containing scientific tables an- notated with labels. Through extensive evalu- ations, we demonstrate that SCITAB poses significant challenge to state-of-thed aftt models, including table-based pretraining models and large language models. All models except GPT- 4 achieved performance barely above random guessing. Popular prompting techniques, such as Mlain-of-Thoughe do not achieve much performance gains SCITAB. Our analy- sis uncovers several unique challenges posed by SCITAB, including table grounding, claim ambiguity, and compositional reasoning. Our codes and data are publicly available at https: //gi thub. com/ KinyuanLuo0/Scciab
In this paper, propose novel dataset Sc- ITAR, which fulfills these stated criteria. contains 1,225 challenging scientific claims, cach demand- ing compositional reasoning for verification using scientific tables. Our data is derived from the Sci- Gen dataset (Moosavi et al., 2021), resource that includes scientific tables and claims crawled from arXiv.org. We first manually filter out the check- worthy scientific claims from the raw data. Follow- ing this, we employ a strategy of human huden-model model collaboration. as depicted in Figure 2, to generate claims that are either contradicted or unverifiable based on the table's content. Figure shows a claim from SCITAB and the corresponding reason- ing process to verify it. Compared with existing benchmarks, SCITAB is closer to real-world scien- tific fact-checking in terms of more realistic claims and table-based evidence. Through data analysis, we further show that the claims in SCITAB neces- sitate a more comprehensive and nuanced set reasoning skills for verification, e.g. numerical rea-
1 Introduction
Scientific fact-checking is a crucial process that in- volves validating the accuracy of scientific claims by cross-refcrencing them with established scien- tific literature, research, or data (Guo al., 2022), This process is crucial for preserving the integrity of scientific information, preventing the spread of misinformation, and fostering public trust in re- search findings. However, the sheer volume of scientific data and claims can be overwhelming for manual fact-checking, making automated scientific fact-checking an imperative research area of NLP. Scientific fact-checking has advanced signifi- cantly with benchmarks including Sci-Fact (Wad- den et al., 2020), Sci-Fact Open (Wadden et al., 2022), and COVID-Fact (Saakyan al., 2021).
"quua Contribution.

---

Paper:
When Choosing Plussible Alternative,, Clever be Clever 1911, 00223w1
Claim: A's productivity of 57 5% expresses that it appears in 7.5% expected by random chance.
Paper ID:
App.
47%
55.3%
9.40%
EL
Closed domain
caption
55%
61.8%
11.0%
Commonsense knowledge
Productivity corresponds Prod. column
the
85%
38.8%
17.0%
106%
57.5%
21.2%
Simple lookup
Table Acolicabiityy (App.). Productivity (Prod] and Coverage (Cov) the words altematives
The random
Supported Claim
Al's productivity 57.5%
productivity $7.59 expresses that appears 7.5% more often than expected random chance.
Subtraction
Refuted Claim
productivity 57.5% expresses that appears in 9.5% more often than expected by random chance.
x
The subtraction result between 57.5% and 50% 7.5%
Not Enough Info Claim
The low performance of "to" can be explaiine by the fact that is responsible for only 4.6% of the inference in the training set.
Fact checker : Supported
Reasoning Graph
An example of our SCITAB dataset (left) and its corresponding reasoning graph (right). Each data entry contains paper name, paper id, table, one claim, and its corresponding label (Supported, Refuted, Not Enough Info).
Figure I:
soning and commonsense knowledge, etc. We employ SCITAB as a diagnostic dataset for benchmarking the zero-shot and in-context learn- ing performance for a wide range of state-of-the- art models, including table-based pretraining mod- els, encoder-decoder models, open source lan- guage models, and API-based language models. We observe that all models, with the exception of GPT-4, can only achieve marginally superior Fi scores than random guessing, which under- scores the challenging nature of SCITAB. Addi- tionally, established prompting methods like Chain- of-Thought (Wei et al., 2022) and Program-of- Thought (Chen et al., 2022) which typically en- hance performance across most reasoning tasks, do not bring performance gain SCITAB. Our error analysis sheds light on several unique chal- lenges in SCITAB that may lead to this, such as table grounding, dealing with ambiguous claims, and compositional reasoning. We make our dataset fully accessible to the research community.
The dataset was created by crawling computer sci- ence papers from arXiv. The tables and the texts explaining the tables are extracted from the papers to create (table, description) pairs for the task of data-to-text gencration. From all the table descrip- tions of SciGen, we first filter the whothy scientific claims following the criteria established by Lee et al. (2009) for academic writing'. We focus the descriptions that serve the purpose of "highlighting and commenting on key data", i.e., describing research findings based on the data presented scientific tables. Given the task's 's ob- jective nature and to save the cost of human labor, we hire a graduate student majoring in computer science to manually select scientific claims based on the aforementioned criteria using the user inter- face in Appendix A.2. This decision was based on pilot annotation which showed that well-trained annotator achieve ing scientific claims. To safeguard the quality, we include an option to mark the claim as "Discard-It's not a claim, or it's an incomplete, or not grammat- ically correct sentence." " during the subsequent claim verification process. Using this approach, we filtered out 872 real-world scientific claims from 1,301 table descriptions in the SciGen dataset.
95% accuracy in filter-
2 The SCITAB Dataset
We adopt a human-modes collaboration strategy to construct SCITAB, as shown in Figure 2. We describe the steps involved in data preparation (Sec- tion 2.1,, automatic claim generation (Section 2.2).
2.2 Automatic Claim Generation
and manual claim verification (Section 2.3).
False Claims. fact-checking dataset requires both true and false claims. However, acquir- ing false claims that naturally occur within well-
2.1 Data Preparation
We use the publicly available SeiGen (Moosavi et al., 2021) dataset as our primary data source.
Detailed criteria are given in Appendix A.1

---

2
Automatic Claim Generation
3
Manual Claim Verification
InstrueGPT
Free-formed claim
Not Enough Info claim C"
Table
Claim extraction
Original claim
SciGen dataset
TEXT
Human verifier
Supported claim C
Texts
InstructoPT
Data Preparation
Claim with opposite meaning
Refuted claim C'
Figure 2: The muman-modell collaboration construction process of SCITAB, which contains three steps: 1) data preparation (including data preprocessing and claim extraction) 2) automatic claim generation (including refuted and Not Enough Info claim generation) and 3) manual claim verification.
verified scientific publications is a challenging task. Following SciFact (Wadden et al., 2020) and COVID-Fact (Saakyan et al., 2021), we seek to create false claims by generating counter-claims of the original true claims. Unlike previous works that purely rely on crowd-workers compose counter-claims - process that costly and prone to annotation artifacts - we leverage the strong instruction- instrruttio-foolooiingg capabilities of large language models (LLMs) to assist humans in generating can- didate counter-claims. Specifically, we prompt InstructGPT (Ouyang et al., 2022) with the orig- inal claim and the instruction: Please modify the original claims convey the opposite meaning with minimum edits. To foster a varied set of gen- erated claims, we include five diverse in-context examples and employ a high decoding tempera- ture setting of 0.7. By mandating minimal edits, we ensure that the counter-claims remain lexically close to the original claims, which is crucial in preventing fact-checking models from relying superficial lexical patterns for verification.
to the table. We adopt manual verification (elabo- rated in Section 2.3) to select them as NEI claims.
2.3 Manual Claim Verification
We subsequently employ a human verification pro- cess for two purposes: first, to verify the quality of the 872 false claims and 900 NEI claims that were generated by InstructGPT: second, critically re- view the 872 real-world scientific claims obtained in Section 2.1. This task involves selecting claims that can be verified exclusively based on the infor- mation presented in the table, without the need for additional context from the associated paper.
For each pair of the true claim and its corre- sponding generated counter-claim c', we ask the annotator to choose one of the following three op- tions: (A) is not exclusively supported by the table, (B) e is exclusively supported by the table, but d' is not refuted by the table, and (C) is not exclusively supported by the table, and c' is not refuted by the table. For each candidate NEI claim, ask the annotator judge whether it is unverifi- able with respect to the table.
Unverifiable Claims. To construct a more chal- lenging dataset, we also integrate claims that are unverifiable with the table information (labeled as Not Enough Info, NEI). We leverage InstructGPT to generate candidate NEI claims by prompting the model with the original table and the instruction: Please generate 5 relevant scientific claims based the information in the table. This process yields diverse set of free-formed claims that enrich the diversity of SCITAB. However, as LLMs tend to generate content that might not always be grounded in the provided data, many of the generated claims turn out to be relevant but unverifiable with respect
Annotator Recruitment. Given that our data source is from computer science papers, we recruit university students majoring in computer science with basic math and programming backgrounds for annotation. We ask each annotator to fill in questionnaire, including their age, department, maximum workload per week, etc. After that, we provide a training session ensure they understand the task and can use the annotation interfaces (Ap- pendix B.2 and B.3). We also give them three samples to test their understanding. We recruit twelve annotators that passed the training session.

---

Statistics
TabFact
FEVEROUS
SEM-TAB-FACTS
SCITAB
Domain
Wiki Tables
Wiki Tables
Scientific Articles
Scientific Articles
Annotator
AMT
AMT
AMT
Experts
Max. Reasoning Hops
2
1
11
Supported
54%
56%
58%
37%
Veracity Refuted
46%
39%
38%
34%
NEI
5%
4%
29%
Total Claims
117,854
87,026
5,715
1,225
Avg. claims per table
7.11
0.07
5.27
6.16
Table I: Comparison SCITAB to three recent table fact verification datasets: TabFact (Chen et al., 2020), FEVEROUS (Aly al., 2021), and SEM-TAB-FACTS (Wang et al., 2021). The table presents statistics related to the domain, annotator (AMT represents Amazon Mechanical Turk), maximum reasoning hops, veracity labels percentage of each dataset, the total number claims, and average claims per table.
In compliance with ethical guidelines, we ensure fair compensation for the annotators. Each claim annotation is reimbursed at a rate of 0.37 USD, resulting in an hourly wage of 11.2 USD?.
we adapt the set of table-based reasoning cate- gories from INFOTABS (Gupta et al., 2020) to define 14 atomic reasoning types, as shown in Ta- ble 2. Among them, "closed-domain knowledge" and "'penn knowledge" are specially de- signed for SCITAB. Closed-domain knowledge refers to obtaining background information from the table caption title, e.g., knowing that "Prod." " refers "Productivity" from the table caption in Figure Open-domain knowledge refers to com- monsense knowledge not presented in the table, e.g., the relationship between precision and recall. Given the designed reasoning types, we manually analyze 100 samples in SCITAB, by annotating the graph of reasoning steps for verifying each claim. We identify 476 atomic reasoning steps from the 100 analyzed samples and show the proportion for each reasoning type in Table 2. We observe that Sc- ITAB has multifaceted complex range of reason- ing types and high proportion of claims requiring different types of domain knowledge.
Quality Control and Annotator Agreement. To ensure the quality of the annotation, we apply strict quality control procedures following the guide- lines outlined in the Dataset Statement (Bender and Friedman, 2018). We assign two different anno- tators to perform a two-round annotation for each claim, while two authors review and resolve any identified errors or issues. To measure the inter- annotator agreement, use Cohen's Kappa (Co- hen, 1960). Our inter-annotator agreement is 0.630 for the false claim verification task (872 claims in total) and 0.719 for the NEI claim verification task (900 claims in total). Both values indicate substantial agreement among the annotators.
3 Data Analysis
Table I shows the statistics of our SCITAB dataset and the comparison with three existing table fact- checking datasets: TabFact (Chen et al., 2020), FEVEROUS (Aly et al., 2021), and SEM-TAB- FACTS (Wang et al., 2021). Compared with these datasets, SCITAB is 1) annotated by domain ex- perts rather than crowd-sourced workers, 2) con- tains more challenging claims that require up to II reasoning steps for verification, and 3) has a more balanced distribution of veracity labels and a higher percentage of NEI claims. We conduct more in-depth analysis of SCITAB as follows.
Reasoning Depth. We further measure the rea- soning depth (the number of required reasoning steps) for each claim and show the reasoning depth distribution in Figure 3. We find that the analyzed claims have an average depth of 4.76 and a max- imum depth of 11. Moreover, 86% of the claims requiring 3 or more reasoning steps. which demon- strates the complexity of reasoning in SCITAB.
Reasoning Graph. We showcase the reasoning graph for the example in Figure I on the right side of the figure. Verifying this claim requires vari- ous types of reasoning including: 1) background knowledge from the table caption: "productivity" corresponds to the "Prod." column in the table; 2) commonsense knowledge: "random chance" means 50% accuracy; 3) simple lookup: "A's productiv- ity" refers to the cell located at the last row and
3.1 Reasoning Analysis
Reasoning Types. To study the nature of reason- ing involved in fact-checking claims in SCITAB,
'The payment is fair and aligned with the guideline for datasct creation (Bender and Friedman, 2018).

---

Function Names
Descriptions
Prop.
Simple lookup
Retrieve the value for specific cell.
20.6
Comparison
Compare numbeer..
19.5
Closed-domaie knowledge
Extract information from context sentences the table caption article,
12.1
Open- Open-domain knowledge
Extract additional information required by domain experts.
5,3
Commonsense knowledge
Extract commnnnennse knowledge necessary for claim verification.
5,3
Subtract
Perform subtraction of two numbers. Perform division two numbers.
5.3
Divide
5,3
Rank
Determine the rank of a set of numbers. Determine if two numhers are different or the same. Calculate the of two numbers.
5,3
Different/ Same
5,3
Add
4.0
Max Min
Retrieve the maximum minimum number from of numbers.
3.1
Col / Rowname
Retrieve the column or row name from the table.
3.1
Trend same/different
Determine the trend for two columns or rows, whether they are the same or different.
2.9
Sct check
Verify if a value belongs to set of numbers.
2.9
Table 2: The function names, descriptions, and their proportions in SCITAB dataset.
25
a greater diversity in refuted claims compared to Sci-IFact. Besides common error types such as "in- correct calculation results" (41.7%) there are also unique types of errors that are more reflective of the complexities in real-world scientific claims. For example, 33.33% of the refuted claims contain "in- correct approximation words", and 10.0% are cases where "the claim partially right", consistent with the fact that ambiguity and half-truths are common phenomena in scientific discourse. Additional ex- amples of refuted claims are in Appendix E.
Percentages (%)
20
18
15
15
10
8
11
Reasoning Steps
Figure 3: The distribution histogram of reasoning steps in our SCITAB dataset. The x-axis is the reasoning steps in each claim, and the y-axis is the frequency for each reasoning step. The shallow claims (with 1-2 reasoning steps) are highlighted in red, while the deep claims (with 3+ reasoning steps) are highlighted blue.
The NEI claims (bottom half; Table 3) also ex- hibit diverse reasoning patterns. The two most common features for unverifiable claims are insuf- ficient evidence in the table and the lack of back- ground knowledge. The lack of closed-domain knowledge another reason for NEI, where ad- ditional information in the paper is necessary to verify the claim. Other reasons include the use of vague pronouns (e.g., "it", "this") brings ambiguity to the claim. These distinct refuted and NEI reason- ing types highlight the unique features of SCITAB. making more comprehensive and realistie rep- resentation of the challenges faced in real-world scientific fact-checking.
the "Prod." column; and 4) numerical reasoning: the difference between 57.5% and 50% is 7.5%. This case study provides further insights into the complexity and variety of reasoning involved in SCITAB, revealing the difficulty of the dataset.
3.2 Refuted and NEI Claims Analysis
One potential risk of model-generated claims is that they may lack diversity and exhibit the same pattern. For example, in the Sci-Fact (Wadden et al., 2020) dataset where the refuted claims are generated by flapping the meaning of the original true claims, we found that out of 100 randomly sampled refuted claims, 85 simply negated the orig- inal claim by adding negation words such "not" (more details in Appendix C). To evaluate the diver- sity of claims for our SCITAB dataset, we randomly select 60 refuted claims and then manually anno- tate their reasons for refutation. Results are shown in Table 3 (top half). We find that SCITAB exhibits
Experiment
We formally define the task of scientific table-based piet-checking as follows. A scientific table T con- sists of table caption P and the table content (1T) < Rr,j < CT) with Ry rows and CT columns, where T2,; is the content in the (i, cell. Given claim C describing fact to be ver- ified against the table T, table fact-checking model F predicts a label y to verify whether C is supported, refuted, or can not be verified by the

---

Refuted Reasons
Prop. (%)
put is then formulated as [T; PC:QQ,, where T is the lincarized table, and Q a question template "Based on the information the table, the above claim true? A) True B) False C) Unknown?". We choose FLAN-T5 (Chung et al., 2022), an improved T5 model (Raffel et al., 2020) pre- rre-trained on more than 1.8K tasks with instruction tuning, which has achieved strong zero-shov/in-contexx performance other fact-checking benchmarks.
The culculation result is wrong.
41.7
The approximation word is wrong.
33.3
The claim is partially right. The values in the claim do not match.
10.0
8,3
The operation type is wrong.
6.7
NET Reasons
Prop. (%)
The claim does not have enough matching evideace.
33.3
The claim lacks open-domain knowledge.
25.0
The claim lacks closed-domant knowledge.
15.0
The claim refers to another lable.
11.7
The claim contains vague pronouns.
8,3
The claim omits specific information.
6.7
Open Source LLMs. We also evaluate the per- formance of state-of-the-art open source LLMs, including 1) LLaMA (Touvron et al., 2023), the first open-source model by Meta AI; 2) Alpaca (Taori et al., 2023), an instruction-follloinn language model fine-tuned on LLaMA; and 3) Vicuna (Chi- ang et al., 2023), the arguably best-performed open- source LLMs that claimed to achieve 90% quality compared to OpenAl ChatGPT. We use the same input format as in the encoder-decoder model.
Table 3: Refuted and NEI reasons and their estimated proportions (Prop.) in SCITAB.
information in T.
Considering the real-world situation that large- scale training data is either not available or expen- sive to collect, we focus on the zero-shot/in evaluation where the model can only access ze- ro/few in-domain data from SCITAB. To this end, we randomly hold out 5 tables with 25 claims as model-accessible data and use the rest of the data as the unseen test set. This also prevents the model from learning spurious features that lead to over- estimated performance (Schuster et al., 2019).
Closed Source LLMs. These are closed-source LLMs that require API calls for inference, includ- ing InstructGPT (text-davinci-003) (Ouyang et al., 2022) and GPT-4 (OpenAI, 2023). We evaluate the setting that directly predicts the la- bel and the Chain-of-Thought (CoT) (Wei et al., 2022) setting, which generates explanations be- fore predicting the final label. We also include the Program ofrram-of-Thuppeee
4.1 Models
We conduct a comprehensive evaluation of SCITAB for various models, including table-based pretrain- ing models, encoder decoder models, open source LLMs, and closed source LLMs. We also study the human performance to analyze the upper bounds on SCITAB.
(PoTT (Chen et al., 2022) model that has shown strong ability in solving com- plex numerical reasoning tasks. It first parses the reasoning steps Python programs and then exe- cutes them on a Python interpreter to derive accu- rate answers. Since most claims in SCITAB also re- quire numerical reasoning, we want to test whether program-guided reasoning can be extended to table- based fact-checking.
Table-based LLMs. These are pre-trained trans- former models fine-tuned on tabular data. We choose three different models: 1) TAPAS (Herzig al., 2020), a BERT-based model fine-tuned on millions of tables from English Wikipedia and cor- responding texts, 2) TAPEX (Liu et al., 2022b), a model that fine-tunes BART (Lewis et al., 2020) on a large-scale synthetic dataset generated by synthe- sizing executable SQL queries and their exccution outputs, and 3) TAPEX-Zero (Liu et al., 2023b), an enlarged version of TAPEX. For TAPAS and TAPEX, we use their fine-tuned version on TabFact (Chen et al., 2020) for table fact-checking.
Human Performance. To examine how humans perform on our SCITAB dataset, we hired an anno- tator from our candidate annotators pool, following the same training procedure as other annotators. In the case of 2-class classification, we randomly selected 40 samples: 20 each for supported and refuted claims. For 3-class classification, we ran- domly selected 60 random samples, ensuring an even distribution of 20 samples across the three la- bel categories (supported, refuted, and not enough information). The annotator took approximately 1.5 hours for the 2-class fact-checking task and 2 hours for the 3-class setting. We report the Macro- Fl scores at the bottom of Table 4.
Encoder-Decoder LLMs. We also use en- decered models where both the input and output are sequences of tokens. To adapt the model to take the table as input, we flatten the table as a sequence following Chen et al. (2020). The in-

---

Zero-shot
In-Context
Models
# Para.
2-class
3-class
2-class
3-class
TAPAS-large (Tabfact) (Herzig et 2020)
340M
50.30
Table--bsed
TAPEX-large (Tabfact) (Liu al., 2022b)
400M
56.06
LLMs
TAPEX-Zero-large (Liu 20236)
780M
48.28
29.72
42.44
23.47
TAPEX-Zero-XX XL (Lio et 20236)
3B
49.77
34.30
42.12
25.62
Flan-T5-base (Chung al., 2022)
250M
47.38
26.56
44.82
24.09
II. Encoder-Decodee
Flan-T5-large (Chung al., 2022)
780M
51.58
32.55
49.62
27.30
LLMs
FLan-T5-XL (Chung et al., 2022)
3B
52.41
38.05
29.21
Flan-T5-XXL. (Chung et al,, 2022)
11B
59.60
34,91
60.48
34,04
Alpaca-78 (Taori et al., 2023)
7B
37.22
27.59
40.46
28.95
III. Open source
Vicuna-7B (Chiang et al., 2023)
7B
63.62
32.47
50.35
34.26
LLMs
Vicuna-13B (Chiang al., 2023)
13B
41.82
29.63
55.11
35.16
LLaMA-7B (Touvron al., 2023)
7B
49.05
32.26
45.24
27.17
LLaMA-13B (Touvron et al., 2023)
13B
53.97
37.18
44.39
32.66
InstructGPT (Ouyang al., 2022)
175B
68.44
41.41
68.10
41.58
IV. Close source
InstructGPT+CoT (Ouyang et al., 2022)
175B
68.46
42,60
LLMs
(Chen ol., 2022)
175B
63.79
GPT-4 (OpenAT, 2023)
78.22
64.80
77.98
63.21
GPT-4+CoT (OpenAl, 2023)
76.85
62.77
Human
92.40
84.73
Table Macro-Pij baselines on SCITAB for different settings. The of Para. indicates number of parameters in the models. The TAPAS and TAPEX models fine-tuned on the TabFact dataset, while others perform zero-shot learning. The bold text indicates the best performance among to III, while the underlined text indicates the overall best performance among all the models.
4.2 Main Results
2. Counter-intuitively, table-based LLMs do not outperform models pre-trained on pure texts, for example, FLAN-T5. This discrepancy may be at- tributed to the dissimilarity between the distribution of tables in scientific literature and publicly avail-
We evaluate all models under both zero-shot and in-context settings. In the zero-shot setting, the model does not have access any in-domain data. the in-context setting, we provide three hold- out examples as demonstrations. We report two sets results: the 2-class case, where examples labeled as NET are excluded (since some models cannot process NEI claims), and the 3-class case including all three labels. The results are shown in Table 4. We have five major observations.
able table corpus. For example, scientific tables commonly include both row and column headers, whereas most tables Wikipedia lack row headers. Meanwhile, the claims in our dataset are usually much longer than those in previous works, raising challenges to table-based LLMs.
3. The results in the 3-class setting notably poorer than those in the 2-class setting. This dis- crepancy reveals the challenges that most models face when confronted with the NEI class. One plausible explanation could be the inherent diffi- culty in distinguishing between 'refuted' and 'NET' claims - task that even trained human annotators struggle with, as noted by Jiang et al. (2020). Our forthcoming error analysis will further demonstrate that the inclusion of the NEI class tends dimin- ish the models' confidence, causing shift in their predictions from "supported/refuted" to 'NEI'.
1. In general, all open source LLMs, including encoder-decodee models and decoder-only models, do achieve very promising results on SCITAB and they still have large gap from human per- formance. The best result is 63.62 for the 2-class setting (Vicuna-7B and 38.05 for the 3-class set- ting (FLAN-T5-XL)) Both results are only moder- ately better (+ (+3..62 and +4.72) than random guess- ing. In contrast, well-trained human annotator achieve 92.46 and 84.73 F1 scores in the 2- class and 3-class settings, respectively. This re- veals the challenging nature of SCITAR and its potential to be the future benchmark for scientific fact-checking.
4. Interestingly, the provision of in-context ex- amples does not result in improved performance for

---

the majority of models. This observation is some- what expected for open source LLMs they have not been reported to possess in-context learning ca- pabilities. Nonetheless, it is surprising to find that even with mmaiin-ff-thoughe prompting, in-context demonstrations do not yield positive effects for In- structGPT and GPT-4, Our error analysis on the PoT offers some insight into this phenomenon and will be discussed in the next section.
InrtrunGPT Label Ditribution Pecoentage (A6)
GFT-4 abel txeribution Percentage
[X]
Supportec
TaMee
4.6
54
12.6
25.2
11.4
Suppotted Befored
ND
Prodictien Labe
Predictos
Figure Confusion matrix for InstructGPT (left) and GPT-4 (right) in the zero-shot 3-class classification task.
Closed source LLMs perform better than open source LLMs, with GPT-4 achieving 78.22 macro- Fi for the 2-class setting and 64.80 for the 3-class setting. This aligns with the assertion that GPT-4 has strong ability to perform complex reason- ing (OpenAL, 2023) and we show that this ability can generalize to tabular data well. However, the black-box nature of OpenAl models restricts our further analysis of its behavior.
Error Type
Estimated Proportion (%)
Grounding errors
50
II. Ambiguity errors
22
III. Calculation errors
20
IV. Program errors
8
Table 5: The error types and their estimated proportions for incorrectly- preedccced samples in PoT.
impacts other numerical reasoning tasks. In der understand this, we randomly selected 50 claims wherein the PoT incorrectly predicted the final veracity labels and evaluated the quality of the generated Python programs. We divide the er- rors into four categories, as assessed by human annotators: (i) Grounding errors, where the pro- gram incorrectly associates data with the respective cells in the table; (ii) Ambiguity errors, where the claim contains ambiguous expressions that the pro- gram fails to represent; (iii) Calculation errors, where incorrect floating point arithmetic calcula- tion in Python lead to inaccurate results and (iv) Program errors, which encompass mistakes such as incorrect or missing arguments/variabble erroneous operations. We present the error analysis in Table 5, and examples of program errors be found in Figure 13 and Figure 14 in Appendix G. Compared to other datasets, categories (i) and (ii) present unique challenges in our dataset. Cate- gory (i) underlines the difficulty in accurately ref- erencing the specific cells to which a claim refers. Category (ii), on the other hand, emphasizes the difficulties posed by the ambiguous nature of scien- tific claims, such "A is significantly better than B", program- based methods. This connection further emphasizes the contribution of our work in addressing the mismatches between reasoning types and the occurrence of grounding errors.
4.3 Error Analysis
InstructGPT and GPT-4. We show the confu- sion matrices for InstructGPT and GPT-4 under the zero-shot 3-class setting in Figure 4. We find that both models have difficulty in accurately predicting the NEI class. InstructGPT displays pattern of "less confident", frequently classifying supported and refuted claims 'NEI'. In contrast, GPT-4 ex- hibits overconfidence, incorrectly categorizing NEI claims as either supported or refuted. This corrob- orates our earlier observation that distinguishing whether a claim is verifiable is one of the key chal- lenges for SCITAB.
Further, we also examine individual error in- stances, with typical examples provided in Fig- ures 11 and 12 of Appendix F. The majority of "supported" claims that were incorrectly classified 'refuted' (Case 6) involve numerical reasoning or comparison. Conversely, when 'refuted' claims are inaccurately predicted as 'supported' (Case 3), we find that LLMs often overlook claims contain- ing negation, indicating a lack of deep compre- hension. For cases where 'supported' or 'refuted' claims are erroneously predicted as 'NEI' (Cases 1 and 2), such claims typically demand extensive rea- soning and deep understanding of the research findings. Interestingly, when faced with these com- plex cases, the model tends to default to the safer choice of "uncertain" (NEI).
and
5 Related Work
POT. Unexpectedly, incorporating Python in- terpreter does not confer any advantage on our dataset (as shown in Table 4), despite its positive
Scientific Fact-Checking Datasets. Existing datasets for scientific fact-checking are summa-

---

rized in recent survey from Vladika and Matthes (2023). These datasets differ in: 1) domain: bi-
research. For example, we believe that addressing the challenges posed by ambiguous claims repre- sents a crucial direction for research scientific fact-checking (Glockner et al., 2023; Liu et al., 2023a). One potential approach is to enhance the disambiguation of ambiguous claims by leverag- ing contextual information external knowledge sources. Additionally, studying the compositional- ity in table-based reasoning is interesting direc- tion. Consider the work of Self-Ask (Press et al., 2022), which proposed the "compositionality gap" metric to measure the capability of LLMs in com- positional reasoning. Such evaluations can be en- riched by annotating SCITAB with ground-truth reasoning depths and structured reasoning graphs. Beyond this, another direction worth exploring is equipping the LLMs with external tools to further improve the model. For example, the use of GPT- 4 plugins, Program-guided Fact-Checking (Pan et al., 2023) or adopting approaches from other tool- augmented LLMs like Toolformer (Schick et al., 2023) and Chameleon (Lu et al., 2023).
ology (Wadden et al., 2020; Akhtar et al., 2022), COVID-19 (Saakyan et al., 2021; Sarrouti et al., 2021; Mohr et al., 2022; Wang et al., 2023), and cli- mate (Diggelmann ct al., 2020), 2) claim creation: crowd-sourced claims v.s. natural claims, and 3) evidence source: Wikipedia articles (Diggelmann al., 2020) or research papers (Wadden et al., 2020, 2022; Sarrouti al., 2021). However, most of these datasets rely on text evidence to verify claims. SEM-TAB-FACTS (Wang et al., 2021) is the only existing dataset based on scientific tables, but it is limited to simple, crowd-sourced claims. To bridge this gap, we construct SCITAB which contains complex claims from authentic scientific papers with table-based evidence.
Table-based Reasoning. Table-based reasoning requires reasoning over both free-form natural lan- guage queries and (semi- sstuutured tables. Early works either rely on executable languages (e.g., SQL and SPARQL) to access the tabular data (Yin et al., 6016; Yu et al.,, 2018) 8) employ graph neural networks to capture logical structure in statements, e.g., LogceaachChheee (Zhong et al., 2020) and ProgVGAT (Yang et al., 2020). However, these ap- proaches often struggle with generalization, as they are tightly bound to specific table formats and lan- guage patterns. To address this, we have seen a shift toward table pre-training, with the advent of Table- BERT (Chen et al., 2020), TAPAS (Herzig al., 2020), SaMoE (Zhou et al., 2022), PASTA (Gu al., 2022), and DATER (Ye et al., 2023). These methods encode sentence-table pairs using lan- guage models and transform table-based reason- or natural language inference. In our work, we focus on evaluating pre- training-basse based methods on SCITAB because they not only demonstrate superior performance but also offer the benefits of few-shot learning.
Ethics Statement
We have received approval from the Institutional Review Board (IRB)² for our data collection. The IRB reviewed our experimental design and research procedures to ensure that they do not pose more than minimal risks to research participants. We take steps to protect research participants privacy and the confidentiality of their data. The review process took two months to complete.
Limitations
Firstly, the method and dataset are primarily de- signed for languages with limited morphology, such as English. Secondly, our SCITAB dataset is specifically focused on fact-checking scientific claims based on tables, which represents only one aspect of scientific fact-checking. Further re- search can explore the integration of other forms of evidence, including textual evidence and fig- ure evidence, to enhance the fact-checking process. Thirdly, SCITAB dataset is primarily focused on numerical reasoning types, as it is derived from the SciGen dataset, which also emphasizes numeri- cal reasoning. would beneficial for future stud- ies to incorporate a wider range of reasoning types to provide a more comprehensive fact-checking
ing into question-answerte
6 Conclusion and Future Work
We present SCITAB, novel dataset for scien- tific fact-checking that addresses the limitations of existing benchmarks. By incorporating real- world scientific claims and their corresponding evidence in the form of tables, SCITAB offers more comprehensive and fine- fine-grained representa- tion of scientific reasoning. The challenging nature SCITAB evident from the performance of the -tate-of-the-art, highlighting the need for further
hhttps: //www, nus edu. sg/research/irb. The NUS- IRB Reference Code is NUS-IRB-2022-599

---

framework. Lastly, it would be valuable to ex- plore additional annotation types, such as reason- ing graphs, to further enrich the depth of analysis and capture more intricate relationships within the claims and evidence.
Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Edd H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling instruction-finetn CoRR, abs/2210. 11416.
Acknowledgements
language models.
This research is supported by the Ministry of Ed- ucation, Singapore, under its MOE AcRF Tier 3 Grant (MOE-M0ET32022-000),, The computa- tional work for this article was partially performed on resources of the National Supercomputing Cen- tre, Singapore (https: heetps://www..rr nscc. sg).
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Mea- surement, 20:37 - 46.
Thomas Diggelmann, Jordan L.. Boyd-Graber, Jan- nis Bulian, Massimiliano Ciaramita, and Markus Leippold. 2020. CLIMATE-FEVER: A dataset for verification of real-world climate claims. CoRR, abs/2012.00614.
References
Max Glockner, Teva Staliūnaité, James Thome,, Gisela Vallejo, Andreas Vlachos, and Iryna Gurevych. 2023. Ambife: Fact-checking ambiguous claims with evi- dence. CoRR, abs/2104.0066 0000000
Mubashara Akhtar, Oana Cocarascu, and Elena Simperl. 2022. Pubhealthtab: A public health table-based dataset for evidence-based fact checking. In Findings of the 2022 Annual Conference of the North Amer- ican Chapter of the Association for Computational Linguistics (NAACL), pages 1-16.
Zihui Gu, Ju Fan, Nan Tang, Preslav Nakov, Xiao- man Zhao, and Xiaoyong Du. 2022. PASTA: table- operations aware fact verification via sentence-table cloze pre-training. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 4971 4983.
Rami Aly, Zhijiang Guo, Michael Sejr Sehlichtkrull, James Thore,, Andreas Vlachos, Christos Christooooulpoollss, Oana Cocarascu, and Arpit Mittal. 2021. FEVEROUS: fact extraction and verification over unstructured and structured infor- mation. In Proceedings of the Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks.
Zhijiang Guo. Michael Sejr Schlichtkrull, and Andreas Vlachos. 2022. A survey on automated fact-checking. Transactions of the Association for Computational Linguistics (TACL), 10:178-206.
Vivek Gupta, Maitrey Mehta. Pegah Nokhiz, and Vivek Srikumar. 2020. INFOTABS: inference on tables as semi-structured data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages
Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics (TACL), 6:587-604.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Martin Eisen- schlos. 2020. Tapas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics (ACL.), pages 4320-4333.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2022. Program of thoughts prompting: Disentangling computation from rea- soning for numerical reasoning tasks. CoRR, abs/2211.122 12588.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang. Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020. Tabfact: A large-scale dataset for table-based fact verification. In Proceed- ings of the 8th International Conference on Learning Representations (ICLR).
Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dogmin, Maneesh Kumar Singh, and Mohit Bansal. 2020. Hover: A dataset for many-hop fact extraction and claim verification. In Findings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), volume EMNLP 2020, pages 3441-3 -3600.
Wei-Lin Chiang, Zhuohan Li. Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
W.Y. Lee, L. Ho, and M.E.T. Ng. 2009. Research Writ- ing: A Workbook for Graduate Students. Prentice Hall.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising eqquence-t-seequence prre-eeeueeeeee-erriiiinnng pre-training
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang,

---

for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics (ACL), pages 7871-7880.
61st Annual Meeting of the Association for Computa- tional Linguistics (ACL), pages 6981-7004.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. CoRR, abs/2210.03350.
Alisa Liu, Swabha Swayamdipta. Noah A. Smith, and Yejin Choi. 2022a. WANLI: worker and Al collabo- ration for natural language inference dataset creation. In Findings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6826-6847.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu, 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 21:140:1-140:00
Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2023a, We're afraid language models aren't It modeling ambiguity. CoRR, 142299444.499
Arkadiy Saakyan, Tuhin Chakrabarty, and Smaranda Muresan. 2021. Covid-fact: Fact extraction and veri- fication of real-world claims on COVID-19 pandemic. In Proceedings of the 59th Annual Meeting of the As- sociation for Computational Linguistics (ACL.), pages 21 1622129.
Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou, 2022b. TAPEX: table pre-training via learning a neural SQL. executor. In Proceedings of the 10th International Conference on Learning Representations (ICLR).
Mourad Sarrouti, Asma Ben Abacha, Yassine Mrabet, and Dina Seemner-Fushman 2021. Evidence-hase fact-checking of health-relatee related claims. In Findings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3499- 3512.
Qian Liu, Fan Zhou, Zhenghao Jiang. Longxu Dou, and Min Lin. 2023b. From zero to hero: Examining the power of symbolic tasks in instruction tuning. CoRR, abs/2304.0799) 079955.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer. Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. CoRR, abs/2302. 04
Pan Lu. Baolin Peng, Hao Cheng, Michel Galley, Kai- Wei Chang. Ying Nian Wu, Song-Chun Zhu, and Jian- feng Gao. 2023. Chameleon: Plug-and-play compo- sitional reasoning with large language models. CoRR, abs/2304.09842.
Tal Schuster. Darsh J. Shah, Yun Jie Serene Yeo. Daniel Filizzola, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3417-3423.
Isabelle Mohr, Amelie Wührl,, and Roman Klinger. 2022. Covert: A corpus of fact-checked biomedical COVID-19 tweets. In Proceedings of the 13th Lan- guage Resources and Evaluation Conference (LREC), pages 244-257.
Nafise Sadat Moosavi, Andreas Rücklé, Dan Roth, and Iryna Gurevych. 2021. Scigen: dataset for reasoning- text generation from scientific ta- bles. In Proceedings of the Neural Information Pro- cessing Systems (NeurIPS) Track on Datasets and Benchmarks.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction mnstruction-following Ilama model. https: // gi www.tt..uot.tteeeee www.ssssuaaaaaasaaaaaeaae papca.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez. Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971.
OpenAI. 2023. GPT-4 technical report. CoRR, 080777
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton. Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instruc- tions with human feedback. In Proceedings of the Annual Conference on Neural Information Process- ing Systems (NeurIPS).
Juraj Vladika and Florian Matthes. 2023. Scientific fact- checking: A survey of resources and approaches. In Findings of the 61 Association for Computational Linguistics (ACL),, pages 6215-6230.
David Wadden, Shanchuan Lin, Kyle Lo. Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534-7550.
Liangming Pan, Xiaobao Wu, Xinyuan Lu, Anh Tuan Luu, William Yang Wang, Min-Yen Kan, and Preslav Nakov. 2023. Fact-checking complex claims with program-guided reasoning. In Proceedings of the

---

David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Iz Beltagy, Lucy Lu Wang, and Hannaneh Hajishirzi. 2022. Scifact-open: Towards open-domain scientific claim verification. Findings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pages 4719-4734.
Yuxuan Zhou, Xien Liu, Kaiyin Zhou, and Ji Wu, 2022. Table-based fact verification with self- adaptive mix- ture of experts. In Findings of the 60th Associa- tion for Computational Linguistics (ACL), pages 139- 149.
Gengyu Wang. Kate Harwood, Lawrence Chillrud, Amith Ananthram, Melanie Subbiah, and Kathleen McKeown. 2023. bheck-covidd Fact-checking COVID-19 news claims with scientific evidence. In Findings of the 6lst Association for Computational Linguistics (ACL), pages 14114-14127.
Nancy Xin Ru Wang, Diwakar Mahajan, Marina Danilevsky, and Sara Rosenthal. 2021. Semeval-2021 task 9: Fact verification and evidence finding for tabu- lar data in scientific documents (SEM-TAB-FACTS) In Proceedings of the 15th International Workshop on Semantic Evaluation CesmEval@@ALLJJCCCC pages 317-326.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.11903.
Xiaoyu Yang. Feng Nie, Yufei Feng. Quan Liu, Zhigang Chen, and Xiaodan Zhu. 2020. Program enhanced fact verification with verbalization and graph atten- tion network. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 78 7810-7825.
Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. 2023. Large language mod- els are versatile decomposers: Decomposing evi- dence and questions for table-based reasoning. In Proceedings of the 46th International ACM Confer- ence on Research and Development in Information Retrieval (SIGIR), pages 174-184.
Pengcheng Yin, Zhengdong Lu, Hang Li. and Ben Kao. 2016. Neural enquirer: Learning to query tables in natural language. In Proceedings of the 25th Inter- national Joint Conference on Artificial Intelligence (IJCAT), pages 2308-2314.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang. Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sal so-lto--ql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3911 3921.
Wanjun Zhong, Duyu Tang, Zhangyin Feng, Nan Duan, Ming Zhou, Ming Gong, Linjun Shou, Daxin Jiang, Jiahai Wang, and Jian Yin. 2020. Logicalfactehecker: I Leveraging logical operations for fact checking with graph module network. In Proceedings of the 58th Annual Meeting of Association for Computational Linguistics (ACL), pages 6053-6065.

---

A Claim Extraction Procedure
training sessions", commonsense interpretations and minimum amount of logical inference are acceptable. Next, a demonstration given on how to navigate and utilize the annotation interface fectively. Following this, a series trial tests are released to the annotators. This is to verify their understanding and capability in the task. Last, we specify the deadline for completing annotations, outline how we check the quality of their work, brief them on post-annotation survey, and explain the reimbursement procedure. Q&A session is also incorporated address any uncertainties or concerns. After receiving their reimbursement, the annotators signed an agreement sheet to ensure its receipt.
A.1 Claim Definition
academic writing (Lee et al., 2009), the accompa- nying text for data, presented as tables and figures), typically includes three fundamental elements as outlined below. These elements encompass the def- inition of claims, which involve highlighting key data (KD) and commenting on key data (COM) that emphasizes and comments the key data.
Location of results (LOC). Statements that lo- cate where the figure/table is found, e.g., Figure 7 displays the mean percentile scores.
Highlighting of key data (KD). Statements that highlight the important data, (1)..() Highest or lowest values (2) Overall trend pattern in the data (3) Points that do seem to fit the pattern or trend, etc. (4) Results which provide answers to your research questions
B.2 NEI Claim Verification Interface
Figure 6 shows the user interface for the NEI claim verification task.
B.3 Refuted Claim Verification Interface
Commenting on key data (COM). Statements that interpret the data. There are three types of comments: (1) Generalization (deductions and im- plications drawn from the results), egg., "This in- dicates that ..." (2) Comparison of results with those from prior studies, e.g., "Different from ..." (3) Explanation speculation (possible reasons cause-effect relationships for the results), e.g., "The possible reason is that.."
Figure shows the user interface for the refuted claim verification task.
B.4 Annotation Post-Survey
Figure shows the examples of post-annotation survey questions and the answers of annotators.
C Analysis Refuted Reasons in the Sci-Fact dataset
A.2 Claim Extraction Interface
Table provides an analysis of the reasons for re- futed claims in the Sci-Fact dataset, along with their estimated proportions. A random sample of 100 refuted claims was selected, and the results indicate that 85% of claims were simply negated using terms like "not" or paraphrased based the evidence sentences. Additionally, 6% of the refuted claims were attributed to incorrect calculation re- sults, while 6% were identified as having wrong commonsense knowledge. A smaller proportion refuted claims (3%) were found to have incorrect open-domain knowledge.
Figure 5 shows the user interface for the claim extraction task.
B Manual Claim Verification Procedure
B.1 Annotator Training Process
Our annotator selection and training process is sys- tematic and thorough ensure the highest quality annotations. We initiate the process by advertising on our university's platform. Interested candidates are then required to complete a registration form. From these responses, the authors identify suitable annotators based on set criteria. Once shortlisted, the potential annotators are invited for training session, which can be conducted either in-person via Zoom, lasting approximately one hour. This session is divided into three parts. Firstly, the au- thors provide comprehensive overview the task definition, ensuring clarity on what is expected. Similar to WANLI (Liu et al., 2022a), during our
D Discussions on Human-Machine Collaboration
Our final data creation pipeline undergoes repeti- tive testing and revision until it reaches its current
All the related materials including the advertisement, a sample of the registration form and the agreement sheet are available at https: / thtb. con/ wiiyyannudde/SciTaaa..

---

Refuted Reasons
Prop. (%)
responding value in the table. The correct value should be 27.9. 9.
Negation (+not) and paraphrasing.
85
The calculation result is wrong.
6
Case E' The operation type is wrong. It applies the incorrect operation type. For instance, in the case of GON+RC+LA (9), it is not accurate to claim that it is better than DCGCN1 because 22.9 > 22.0 and 53.0 52.6.
The commonsense Inowledge is wrong.
6
The open-domain knowledge is wrong.
3
Table 6: The refuted reasons and their estimated propor- tions (Prop.) in the Sci-Fact dataset.
form. In pilot annotation, found that man- ual verification played the most essential role in the validation of claims marked as "Not Enough Infor- (NEI"" Initially, we planned to rely solely on LLMs for generating NEI claims. Our criteria for the NEI claim is that "the claim should be flu- ent, logical, and relevant to the table. However, the claim cannot be verified true or false solely based on the information in the table." However, after careful examination of the LLM output, found that LLM tends to generate claims that are either not logical or irrelevant to the table content. Therefore, human efforts are required to further select NET claims that meet our criteria. Out of 900 initial NEI claims generated by LLMs, man- ual verification narrowed them down to only 355 claims, taking up 40% of the original count. While it may not have served as crucial role as filtering NEI claims, human verification also safeguarded the data quality in other annotation processes. For example, among the "supported" claims originally appearing in the scientific paper, human validation still identified 10 cases that were actually not sup- ported (e.g., wrong number matching.)
F Error Cases for InstructGPT
Figure 11 and Figure 12 show six error examples of InstructGPT in the zero-shot setting when applied to our SCITAB dataset.
Error Type Supported predicted NEL This error type indicates discrepancy between the gold label, which is Supported, and the predicted label, which is NEI.
Error Type 2: Refuted predicted as NEI This error type indicates a discrepancy between the gold label, which is Refuted, and the predicted label, which NEI.
Error Type 3: Refuted predicted as Supported. This error type indicates discrepancy between the gold label, which is Refuted, and the predicted label, which is Supported.
Error Type 4: NEI predicted as Supported. This error type indicates discrepancy between the gold label, which is NEI, and the predicted label, which is Supported.
Error Type 5: NEI predicted Refuted. This error type indicates a discrepancy between the gold label, which is NEI, and the predicted label, which Refuted.
Case Study for Refuted Claims
Figure and Figure 10 show five examples of re- futed cases. Below, we provide explanations for each of these error cases.
Error Type 6: Supported predicted as Refuted. This error type indicates discrepancy between the gold label, which is Supported, and the predicted label, which is Refuted.
Case A The calculation result is wrong, It pro- duces incorrect calculation results. The accurate result should be 27. 27.9-21.7 = 6.2.
G Error Cases for Program-of-Thougee
Case B The approximation word is wrong. It generates incorrect approximation words, as 19.4 is not significantly lower compared to 23.3.
Figure 13 and Figure 14 show five error examples of SogrammoffTThughhe when applied to our Sc- ITAB dataset. Below, we provide explanations for each of the error cases.
Case C The claim is partially right. The claim is generally correct, with the exception of the BShift column which does not fulfill the claim.
Error Case 1. It exhibits incorrect entity linking (Grounding error) and incorrect operation (Pro- gram error). The codes
baseline
Case D The values in the claim do not match.
73.06" and "winocoref baseeine = 88.48"
The value in the claim does not align with the cor-
should be "IlliCons wwinoraad = 56. and

---

ID: 2-2
Paper 1805. 11461v1
Name:
Pepresentations Neural Relation Classification
Represcntation
Hyper
Hyper puramaters
Hyper
Hyper
Hyper
Hyper
FLangg. (ull) with
Fl.tng,
parameters
Num. Feature
parumeterss
parumeeers L.2
parameters
purameters
feld) with optimal
mags
Reg.
Dropout
CONLLOS
1000
Softples
1.15e=01
1.13--03
73.34
74.49
SB
4-5
806
Sigmoid
8.13e-02
1.79-00
0.7
72.85
75.05
v1.3
Suftles
1.66+00
9.636-44
Caption:
Table Hyper parameter optimization results for each model with different representation. The max pooling strategy consistently performa better all model
varations.
Sentence to annotate:
The results furthermore show sdps based the Stanford Basic (SB) representation provide the best performance, followed by the CoNLLO8 representation.
Description
Bachground
Claim
Arnotate
Progress:
Figure 5: The user interface for the claim extraction task.
ID:
Paper ID: 1911.00225v1
Paper: Choosing Alternatives,
Table
Model
Training data
Overall
Easy
Hard
BERT-largo FT
B-COPA
74.5 (e: 0.7)
(a.7(x00.A)
74.4 (a 0.9)
BERT-lage-FF
(50%)
(.22)
76.8 1.99
(x.8(x
BEIT-laae-PP
COPA
76.5 (z7)
63.9
(1.9
RoBERTss lagge- PT
B-COPA
89.0 (e
18.9(x22.1
89.0
RoBERTa large PT
B (50%))
86.1 (= 2.2)
87.4 1.1)
(5.4
lurge-F7
0.9)
Caption:
Easy: superficial cues,
superficial
Claim
For both BERT and RoBERTa, more data leads to more performance improvements, although the growth rate ReBERTa slightly faster.
Figure The user interface for the NEI claim verification task.
Dillicons_winocoret = 74.32" respectively. Ad- ditionally, the ">" operation should be changed to ">=".
parse other baseline results, such as 'SFEGAN_WER WWR 14.9".
Error Case 3. fails generate a correct pro- gram (Program error). The variables and logical functions in the programs are incorrect. For in-
Error Case 2. exhibits incomplete entity link- ing (Grounding error). The program should also

---

Paper
Paper: Choosing Atternatives, Clever can Clever
Table
Model
Training data
Overall
Iard
DERT-large-FF
(a.74)
BERT large FT
B-COPA (509)
74.3 (+ 2.2)
76.8 1.9)
72.8 (e 3.1)
BERT-large-FF FT
COPA
76.5 (.77
(3
(s.9
RoBEKTa
B-COPA
(s.9
RoBEXTi-laee large-F---
(50%)
(2)
1.1)
RoBEKTls large PT
COPA
87.7 (a 0.9)
91.6 1.1)
(5.3
fine-tuned
Original Claim
smaller performance gap between Easy and Hard subsets indicates that training on BCOPA encourages BERT RoBERTa rely less on superficial
Generated Refuted Claim
The larger perfor performance gap between Easy and Hard subsets indicates that training on BCOPA encourages BERT and RoBERTa rely more on superficial
Figure 7: The user interface for the refuted claim verification task
stance, "G2S_GAT BLA WWW__BEU_LLEE BLEU_LDC201 5E86". The logical func- tion "and" should be replaced with "or""
should be
Error Case 4. It fails to generate precise pro- gram for the approximation word "ccoppaable" (Ambiguity error). Currently, the program defines "comparable" as "larger than", which is not accurate enough.
Error Case 5. It generates the correct program, but the calculation result is inaccurate due to incor- rect float digits in the Python code (Calculation er- ror). For instance, Python may output 119499999', which is not equal to '1.95".

---

Annotation Post Survey
Annotator 1:
Is the task demonstration clear to you?
Yes, clear.
What do you think is the difficulty of this task? (1-10 points, 10 points is the most difficult)
5-6.
Which part is the most difficult for you? Why?
Judgment, understanding the way of original author think.
Do you think the annotation batch is appropriate? What is the maximum batch amount for you in a week?
Yes. batches in a week during the examination. 4 during vacation.
Could you provide some advice how to improve the annotation platform?
Looping for multiple operations.
Annotator 2:
Is the task demonstration clear to you? Yes.
What do you think is the difficulty of this task? (1-10 points, 10 points is the most difficult)
6
Which part is the most difficult for you? Why?
Table understanding: different parameters the attributes.
Do you think the annotation batch is appropriate? What is the maximum batch amount for you in a week?
Ok. 2-3 batches
Would you like to attend this session again as a 2-week participation? ok.
Could you provide some advice on how improve the annotation platform?
preferred to write down the annotations on the platform.
Annotator 3:
Is the task demonstration clear to you?
Yes, clear. difficulty different between demo real annotation.
What do you think is the difficulty of this task? (1-10 points, 10 points is the most difficult)
7
Which part is the most difficult for you? Why?
Table understanding--ooobbllyy
Do you think the sample amount is appropriate? What is the maximum batch amount for you in week (1 batch contains 20 samples)? 10-15 samples for an hour. 50 samples a week.
Would you like to attend this session again as 2-week participation? Maybe not. But 15 samples offline for week ok.
Could you provide some advice on how to improve the annotation platform? think the current platform fine for me.
Figure 8: The examples of post-annotation survey questions and the answers of annotators.

---

H
Prompts
H.1
Zero-shot Prompts
Table:
<input_table>
Claim:
stnput_clatns
Based on the infarnation in the Table, is the above clain true?
the clain is true.
clain is falsa.
C) it is inpossible to tell.
H.2 Few-shot Prompts
following and
question.
Caption: Table 5: Results of Line-tuned models Balanced COPA. Easy: instances with superficial Hard: Instances whoth superficial cues.
Table:
1| Model Training data Overall Easy Harr II
BERT-Large-FT 8-COPA
(=8.7) | (#8.4) 74.4
(#0.9)
aERT-large-FF B-COPA
(58%) 74.3 (#2.2) 76.8 (#1.8) 12.8 (23.1) II
BERT -ERT-large-FT COPA 76.5 (#2.7) 1 83.9 (#4.4)
71.9 (#2.5)
RosERTa-large-F1 B-COPA
(#0.3)
(=2.1)
(#8.8)
11 RoBERTa-lorge-F1 B-COPA (588) 95.1 (12.2) 87.4 (11.1) 85.4 (12.9) 11
RoogRT=-1orge-FT
(#0.9) 91.6 (=1.1)
(#2.0)
Claim: HoReRTa-large outperforns BaRg-large when fina-tuned full balanced COPA.
Question: Is the above clain true or false? Please directly give the answer.
Answer:
The
Caption: Table 5: Results of Line-tuned models Balanced COPA. instances with superficial
Hard: instances without superficial cues.
11 Model Training data Overall | Easy Hard II
aERT-large-FT 8-COPA
74.5 (=8.7)
(#8.4) 74.4 (#0.9)
11 aERT-large-FT 8-COPA
(588) 74.3 (#2.2) 16.8 (#1.9) 72.8 (=3.1) 11
DERT-large-FT COPA 76.5 (12.7) 83.9 (14.4)
71.9 (12.5)
RoBERTa-large-F B-COPA
89.0 (#0.3) 88.9 (=2.1) 89.0 (=8.8)
-aagera-1arge-F1 B-COPA (588) 85.1 (#2.2) 87.4 (41.1) 85.4 (#2.9) II
(10.9) 91.6 (21.1)
(12.0)
Clais: The difference between RoBERTa-large-d and BERT -largeag-- is points B-COFA, which is significantly smaller than the difference in COPA.
Question: the above clain true or false? Please directly giva the answer
The clain 14 false.
Caption: Table 4: ablation study en the MoZZ. dataset with the joint goal accuracy on test set. renove the residual connections between the attention modules In the OMR autaut For MLP wе further replace the MLP with single linear layer with the on-linear activation.
For Hierachical-Atee attention nenory access
decaders
Table:
Model Joint Acc.
CONER 88. II 11 -Horachical -Attn 85.69%
Clain: [CONTINUE] The offectivaness of hierarchical. attention is proved by accuracy drop 1.95% after renoving residual connections and the hierarchical stack of our attention nodules.
Question: above
directly give answer.
The clain 1s true.
Caption: Table 4: Scores for different treining objectives on the linguistic probing tasks,
Table:
11 Methad
Depth|
BShift]
SubjNun
Tense
Coordiny
Length | objNun
TopConst
SONG
MC II
CHOW
CHOW-R
35.1
78.8
82
88.7
61.8
82.8
19.7
74.2
50.7
72.9
I|
CBOW-C
34.3
58.5
79.8
79.9
53.8
75.9
79.0
72.9
48.6
89.8
II
CBOW-R
33.0
49.6
79.3
78.4
53.6
74.5
78.6
72.8
49.6
89.5
11

---

Clain:
while CHOW-R and CMOW-C ferforn comperably on nost probing tasks, CHOW-C yields points higher scores NoreContent and BigranShift.
Question: Is the above clain true or false? Please directly give the answer.
Answer:
The clain is false.
( more in-context examples here m))
Caption:
<irput_caption?
Table: <input_tebles
Clain: <input_clain>
Question:
Is the above clain true or false? Please directly give the answer.
Answer:
H.3 Chain-of-Thoughh Prompts
following and
question.
Caption: Table 5: Results of fine tuned nedels Belanced COPA. Easy: instances with superficial instances thoth suparficial cues.
Table:
11 Model Training date Overall | Easy Hard II
aERT-large-FT 8-COPA 74.5 (=8.7)
(#8.4) 74.4 (#0.9)
DERT-large-FT B-COPA
(58%) 74.3 (12.2) 76.0 (11.9) 72.8 (23.1) II
3ERT-large-FT COPA 76.5 (#2.7) 83.9 (#4.4) 71.9 (#2.5)
FoBERTa-large-ti B-COPA
(40.3)
(=2.1)
(#8.8)
11 RuBERTa-large-Ff B-COPA (588) 55.1 (#2.2) 87.4 (#1.1) 85.4 (#2.9) 11
RoBERTa-large-FT
COPA
(=0.9) 91.6 (#1.1)
(+2.0)
Clain: HoBERTo-large outperforns Baage when fine-tuned full and balanced CDPA.
Question: Is the above clain true or false? Let think stop step.
Reasoning:
the perfarmance of RogeRta-large and BERT-large full and balanced CDPA. For full CDPA, RoBeRTa-large has an averall performance 87.7 (tabla[e,2]), while BERT-large has overall perfornance of 76.5 (table[3,2]).
nead
Therefore, RoBERTa-Large outperforns BERT-large full COPA.
For balanced COPA, RoBeRla-large has overall perfornance of 89.8 (table[4,2)), BERT
an overull performance Therefore, RoBERTa-large outperforns BERT-large on balanced COPA. Finally, RoBERTa-large outperforns BERT -lage both full and balanced COPA, the Clain is
(table[),2]).
Answer:
Therefore, the 1s true.
Caption: Table Results fine models
Easy:
with superficial cues,
Hard: instances whtho suparficial sues.
Table:
Model Training data Overall Easy Hard
BERT-large-FT B-CDPA
14.5 (=8.1) 74.7 (#0.4) 74.4
(40.9)
11 BERT DERT-large-FT 8-COPA
(58%) 74.3 (12.2) 76.0 (21.9) 72.8 (23.1) II
-ERT-large-FT COPA 76.5 (#2.7)
(#4.4)
(#2.5)
-o
B-COPA
89.8 (10.3)
89.9 (22.1)
1) 99.0 (18.8) 11
-o
B-COPA (588) 86.1 (#2.2)
67.4 (#1.1) 85.4 (#2.9)
RaagRTa-
87.7 (#0.9) 91.8 (#1.1)
(#2.0)
Clais: difference
BERT-large-FT is points B-COPA, which is significantly smaller than the difference in COPA.
RabgRTa-Iarge-FI
Question: Is the above clain true or fulse? Let us think step by step.
Reasoning:
To varify this clain, ve need compare the performance difference between RaagRTa- COPA.
GERT-large-FT
The difference between RoBERTa-large-FT and BERT-large-FT on B-CORA is 89.0 74.5 14.5 (table[4,2] tablef1,2]).
The difference between RoBERTs-1erge- and BERT FT on COPA is 87.7 76.5 11.2 (table[6,2] tanle[3,2]). RoBERTa-Large-fT BERT-large-FT than the difference on COPA. Therefare, the Clain is false.
significantly larger
Answor:
Therefore, the clain is false.
Caption: Table 4: The ablation study on the Moz2. dataset with the joint geal accuracy on the tast set.

---

For For Hierachicsl-AtE -Attn"'''-tt" we renove the residuel connections between the sttention modules in the CMR decadars and all the attention nenory access are basad on the output fron the LSTM. For""- MLP further replace the NLP single layer with the nan-linear activation.
Table:
Madel Joint Acc.
I| COMER 88.64% II
Hierachieal -Attn 86.69%
II MLP 83. 24% I1
hiererchical ettention design is proved by accuracy drop of 1585% after ramaving rasidual connections and the hterarchical stack of our attention modules.
Clais: [CONTINUE]
effectiveness
Question: Is the above clain true or false? Let us think step by step.
Reasoning:
To verify this clain, need to compare the performance of CONER with without hierarchical
design.
The perfornance af COMER with hierarchical attention design is 88.648 (table[2, 21). The performance of COMER without hierarchical attention design 86.69% (table[3,21). Therefore, the perfornance of COMER with Merarchical attention design is higher than the performance of COMER without hierarchical attention design 88,64 86.69 1.95. therefore, the
is true.
Therefore, the clain is true.
Caption: Table Scores ffferentt training abjactives on linguistic probing tasks.
Table:
11 Method Depth|
BShift]
SubjNun
Tense
CoordIny
Length | ObjNun
TopConst
SONG
MC 1I
CHON-E 36.2
50.4
66.8 I|
11
CHOW-R 35.1
78.0
02.0
88.2
61.8
82.8
79.7
74.2
56.7
72.9
II
CBOW-C 34.3
50.5
79.8
79.9
53.8
75.9
79.8
72.9
48.6
89.8
II
CBOW-R 33.0
49.6
79.3
12.0
49.6
89.5 II
Clais: While
CHOW-C perforn comparably on most probing tasks, CHOW-C ylelds points higher scores on NordContent and Rigranshift.
Question: Is the above clain true or false? Let us think step by step.
Reasoning:
To verify this clain, we need to compare the performance of CHOW-R Bigranshift.
CHON-C KordContent
The perfornance af ENOW-R on WordContent is 12.8 (tabla[2, ,18)).
The performance of CHOW-C WordContent 66.8 (table[1,18)
The performance
CMOM-R BigranShirt 70.8 (table[2.21)
The performance af CNOM-C Bigranshift is 66.e (table[1,2]
Therefore, the perfornance the perfornance af ENOW-R on NordContent by 66.8 12.9 = -6.1.
CMOW-C higher
The perfornance CMOW-C
than
the performance af LNOM-R on Bigranshift by 66.e 78.8 = -4.8.
Therefore, the Clain 1s false.
Answer:
Therefore,
is false.
in-context examples here (.))
Caption: <irput_caption>
Table: <iaput_table>
Claim: <input_clain>
Question:
Is the above clain true or false? Let us think step step-
Reasoning:
Answer:
H.4 Program-of-Thoughe Prompts
following and (please call the function equal(a, b) to check whether and b are equal)
Pythen
answer question:
Caption: Table 4: The ablation study on the MoZ2. deteset with the joint geal accuracy on the test set. For Hterachical -Attn" renove the restoual connections between the attention modulas in the oMR output
attention nenory access For MLP!" wе further replace the NLP with single linear layer with the non-linear activation.
decoders
Table:
Model Joint Acc.
CONER I 88. 64% II Hierachical -Attn 85.59% II

---

II MLP 83.24% 11
Clais: [CONTINUE] effectiveness
hierarchical attention design by an accuracy drop of 1.95% after removing residual connections
and the hierarchical stack af attention nodules.
Question: Based the information the table, is the above clain false?
t Pychon Code, return ans coMER_ace citER_acc__n_restdiee 85.59 accuracy_ CoMER_ace coMER_ecc_o__residual
ans equal (essl 1.95)
Python answer question: (please call the function equal(a, b) to check whether and b are equal)
following
Caption: Table 3: Ablation study of capsule net and ward-level attention an Nikidata dataset.
Table:
Recall 8.1 8.2 -Word-ATT
AUC
8.389
11 -Capsule | 8.635 0.587 8.413
8.385 II
Hodel
Clais: According to the table, the drop of precision demonstrates word-level attention quite useful.
Question: on information Python
is
false?
our our_nedel_ 15.,50, 0.519, 6.422. 0.485] atthoottword.atttreealll 648, 0.515, 0.395, 8.383]
ans True
for i in r4)g
ans False break
Read the following table and then write Python code answer question: function equal(a, b) check
(please
are equal)
Caption: Table
different training objectives on linguistic probing
Table:
11 Method Depth|
BShift|
Sub
Tense
CoordIny
Length | ObJNun
TopConst
SONO
II
11 CHON-C 36.2
66.8
81.1
78.7
61.7
$3.9
19.1
73.6
5e.4
66.8 II
56.7
72.9
I|
11 CROW-E 34.3
58.5
79.8
79.9
53.8
75.9
19.8
72.9
48.6
89.3
II
CBON-R 33.0
72.0
Clain: While CHOW-R and CHOW-C perforn conperably CMOW-C ytelds points higher scores WoreContent and BigranShife.
probing tasks,
Question: Based
information
is
False?
Python
CNOW C_n.___ccom
C_score_on_Me 66.8
www.c__score_on_SMM = 56.0
CMOw_R_se w_wp___scceeoo______ 72.9
BShifft = 78.6
(CNOW_C_score___
cHov_R_scora_on_MG cHow___score_on_BBhh 5)
and equal Berpoo___soorooo______
(-.- more inncontext exapples here ...)
Read the Following table and then write Python code to answer question: (plaase call the function equal check whather equal)
Caption: <irput_caption>
Table: <input_table>
Clatm: <input_clatr>
Question: Based on the information in the table, is the above clsin true or false?
Python Coda, raturn ans

---

A. The calculation result wrong.
Model
T #P
B
C
Seq2SeqB (Beck al., 2018)
$ 28,4M
21.7
49.1
GGNN2Seq (Beck al., 2018)
28.3M
23.3
Seq2SeqB (Beck et al., 2018)
E 142M 26.6
52.5
GGNN2Seq (Beck al., 2018)
E 141M 27.5 53.5
19.1M 27.9 57.3
DCGCN (ours)
E 92.5M 30.4 59.6
Table Main results on AMR17. GCNSEQ (Damonte and Cohen, 2019) achieves 24.5 BLEU points. #P shows the model size in terms of parameters; "S" and "E"" denote single and ensemble models, respectively.
Claim: For example, the single DCGCN model gains 5.9 more BLEU points than the single models of Seq2SeqB on AMR17.
B. The approximation word is wrong.
GCN
B
GCN
+RC (2)
16.8 48.1 +RC+LA (2)
18.3 47.9
+RC (4)
18.4 49.6 +RC+LA (4)
18.0 51.1
+RC (6)
19.9 49.7 +RC+LA (6)
21.3 50.8
+RC (9)
21.1 50.5 +RC+LA (9)
22.0 52.6
+RC (10)
20.7 50.7 +RC+LA (10) 21.2 52.9
DCGCNI (9)
22.9 53.0 DCGCN3 (27) 24.8 54.7
DCGCN2 (18) 24.2 54.4 DCGCN4 (36) 25.5 55.4
Table Comparisons
baselines. +RC denotes GCNs residual connections. +RC+LA refers to GCNs residual connections and layer aggre- gations. DCGCN: represents our model with blocks, containing x (n m) layers. number layers each model shown in parenthesis.
Claim: However, when we set the number to 18, GCN+RC+LA achieves BLEU score 19.4, which is significantly worse than the BLEU score obtained by DCGCN2 (23.3).
C. The claim partially right.
Table 4: Scores for different training objectives on the linguistic probing tasks.
Method
Depth
BShift
SubjNum
Tense
Coordlav
Length
ObjNum
TopConst
SOMO
WC
CMOW-C
78.7
73.6
CMOW-R
35.1
70.8
82.0
80.2
61.8
82.8
79.7
74.2
50.7
72.9
CBOW-C
34.3
50.5
79.8
79.9
53.0
75.9
79.8
72.9
48.6
89.0
CBOW-R
33.0
79.3
78.4
53.6
74.5
78.6
72.0
49.6
89.5
Claim: While CMOW-R and CMOW-C perform comparably most probing tasks, CMOW-C yields points lower scores on WordContent BigramShift. BShift column doesn't 't satisfy the claim.
Figure 9: The refuted claims cases A to C. Case A represents the calculation result is wrong. Case B represents the approximation word wrong. Case represents the claim is partially right.

---

The values the claim do not match.
Model
T
AP
Seq2SeqB (Beck al., 2018)
S 28,4M
21.7 49.1
GGNN2Seq (Beck al., 2018)
28.3M
23.3 50.4
Seq2SeqB (Beck al., 2018)
142M
GGNN2Seq (Beck et al., 2018)
141M
27.5 53.5
S 19.1M 27.9 57.3
DCGCN (ours)
92.5M 30.4 59.6
Table 2: Main results on AMR17. GCNSEQ (Damonte and Cohen, 2019) achieves 24.5 BLEU points. #P shows the model size in terms of parameters; "S" and "E" denote single and ensemble models, respectively.
Claim: Our single model achieves 27.6 BLEU points, which the state-of-the-art result for single models.
E. The operation type is wrong.
GCN
(2)
16.8 48.1 +RC+LA
47.9
+RC (4)
18.4 49.6 +RC+LA
18.0 51.1
+RC
19.9 49.7 +RC+LA
21.3 50.8
+RC (9)
21.1 50.5 +RC+LA
22.0 52.6
+RC
20.7 50.7 +RC+LA (10) 21.2 52.9
DCGCNI (9)
22.9 53.0 DCGCN3 (27) 24.8 54.7
DCGCN2 (18) 24.2 54.4 DCGCN4 (36) 25.5 55.4
Table Comparisons with baselines. +RC denotes residual connections. +RC+LA refers GCNs with both residual connections layer aggre- gations. DCGCNi represents model with blocks, containing
layers. The number of layers for each model shown parenthesis.
Claim: GCNGGCHLHHCHACOC (9) better than DCGCN1.
Figure 10: The refuted claims cases D and E. Case D represents the values in the claim do match. Case E represents the operation type is wrong.

---

Error case 1 (43.9%) Gold Label: Supported Prediction Label: NEI
Table 3: Precision, recall and F-measure for methods using the top 1,000 words with the highest number of contexts.
Lang
Corpus
Patt
DSim
SLQS
TF
DF
DocSub
HClust
Europari
0.1173
0.0366
0.0503
0.0554
0.0548
0.0443
0.0761
EN
Ted Talks
0.1125
0.0301
0.0382
0.0425
0.0441
0.0710
0.0664
Europari
0.5163
0.3330
0,5257
0.6109
0.5984
0.7311
0.5676
PT
Ted Talks
0.5387
0.6117
0.6533
0.5656
Europarl
0.0396
0.3999
0.5499
0.6045
0.5887
0.0023
0.0017
EN
0.0018
0.4442
0.5377
0.5657
0.6077
0.2666
0.0019
Europarl Ted Talks
0.0111
0.3554
0.5795
0.6727
0.5184
0.0053
0.0012
PT
0.0004
0.5484
0.6877
0.5515
0.4706
0.0011
Europarl Ted Talks
0.0591
0.0671
0.0922
0.1015
0.1003
0.0044
0.0033
EN
0.0035
0.0564
0.0713
0.0791
0.0822
0.1121
0.0037
Europarl
0.0217
0.3438
0.5513
0.6403
0.5555
0.0105
0.0024
PT
Ted
0.0008
0.3020
0.5390
0.6475
0.5819
0.5471
0.0022
Claim: Analyzing Table 3, observe that all values of precision using the Portuguese corpora have higher scores when compared with the English corpora.
Error case (38.7%): Gold Label: Refuted Prediction Label: NEI
Table 3: Precision, recall and F-measure for methods using the top 1,000 words with the highest number of contexts.
Lang
Corpus
Patt
DSim
SLQS
TF
DF
DocSub
HClust
Europarl
0.1173
0.0366
0.0503
0.0554
0.0548
0.0443
0.0761
EN
Ted Talks
0.1125
0.0301
0,0382
0.0425
0.0441
0.0710
0.0664
P
Europarl
0.5163
0.3330
0.5257
0.6109
0.5984
0.7311
0.5676
Ted Talks
0.5387
0.2907
0.5300
0.6111
0.6159
0.6533
0.5656
Europarl
0.0396
0.3999
0.5499
0.6045
0.5887
0.0023
0.0017
Ted Talks
0.0018
0.4442
0,5377
0.5657
0.6077
0,2666
0.0019
R
Europarl 0.0111
0.3554
0.5795
0.6727
0.5184
0.0053
0.0012
Ted Talks
0.0004
0,3142
0.5484
0.6877
0,5515
0.4706
0.0011
Europarl
0.0591
0.0671
0.0922
0.1015
0.1003
0.0044
0.0033
Ted Talks
0.0035
0.0564
0.0713
0.0791
0.0822
0.1121
0.0037
F
Europarl
0.0217
0.3438
0.5513
0.6403
0.5555
0.0105
0.0024
Ted Talks
0.0008
0.3020
0.5390
0.6475
0.5819
0.5471
0.0022
Claim: Analyzing Table 3, we can observe that all values of precision using the English corpora have higher scores when compared with the Portuguese corpora.
Error case 3 (7.6%): Gold Label: Refuted Prediction Label: Supported
Model
B
DCGCN4
25.5
Encoder Modules
-Linear Combination
23.7 53.2
-Global Node
24.2 $4.6
-Direction Aggregation
24.6 54.6
-Gruph
24.9
-Global Node&I Lnner Combination
22.9 52.4
Decoder Modules
-Coverage Mechanism
23.8 $3.0
Table Ablation study for modules used in the graph
encoder and the LSTM decoder
Claim: With the coverage mechanism, the result drops by 1.7/2.4 points for B/C scores.
Figure 11: Error Cases 1-3 for InstructGPT in the zero-shot setting.

---

Error case 4 (4.6%) Gold Label: NEI Prediction Label: Supported
in-domain SQuAD
out-of-domain
QA-SRL
EM
F1
EM
F1
MQAN
31.76
75.37
10.99
+coverage
32.67
76.83
10.63
50.89
BIDAF (ELMO)
70.43
79.76
28.35
49.98
+coverage
71.07
80.15
30.58
52.43
Impact using coverage improving generalization across trained the SQuAD training data.
similar tasks.
Claim: This is particularly true for the BIDAF model.
Error (2.8%) Gold Label: NEI Prediction Label: Refuted
TABLE V
WORD INTRUSION TEST RESULTS: CORRECT ANSWERS OUT QUESTIONS
GloVe
Imparted
Participants to
80/88/82/78/97
212/170/00/222/242
Mcan/Std
85/6.9
212/24.4
Claim: These results show that the questionnaire takers had average accuracy of 98. 18.2% in answering word intrusion questions for words associated with meanings imparted by standard word embeddings
Error case 6 (2.4%) Gold Label: Supported Prediction Label: Refuted
GCN
GCN
+RC (2)
16.8 48.1 +RC+LA
18.3 47.9
+RC (4)
18.4 49.6 +RC+LA (4)
18.0 51.1
+RC (6)
19.9 49.7 +RC+LA (6)
21.3 50.8
+RC (9)
21.1 50.5 +RC+LA
22.0 52.6
+RC (10)
20.7 50.7 +RC+LA (10) 21.2 52.9
DCGCNI 22.9 53.0 DCGCN3 (27) 24.8 54.7
DCGCN2 (18) 24.2 54.4 DCGCN4 (36) 25.5 55.4
Table 6: Comparisons with baselines. +RC denotes GCNs with residual connections. +RC+LA refers to GCNs with both residual connections and layer aggre- gations. DCGCN: represents our model with blocks, containing i x (n m) layers. The number of layers for each model is shown in parenthesis.
Claim: For example, GCN+RC+LA (10) achieves BLEU score of 21.2, which worse than GCN+RC+LA (9).
Figure 12: Error Cases 4-6 for InstructGPT the zero-shot setting.

---

Error case
Dataet
Munic
Binnss
MiCCaa
and Ng (2012)
Kmmshaa
KnowCane
KnowCrmhh
Winegrad
Procision
$1.48
$3.26
73.05
71,81
74.03
76.41
RinoConf AmePmn
68.37
24
88.48
88,95
89.32
Table Perforaacc results on Winogrod and WinuCoref datasets. All our throe systems are tealned on
WinoCoref, evaluate the predictions both
Our systems improve
basclines by
over than 20% Winograd und over 1.5% on WinoCoref.
Claim: The best performing system is KnowComb. improves by over 20% over state-of-art general coreference system on Winograd. On the WinoCoref dataset, it improves by 15%. gold: supports, prediction: refutes,
knowcomb 76.41h 1n winocoref baseline = 888.48nnwinccoref knowcomb = 89 89.32 3..n2nnnns (winograd knowcomb winograd baseline 20) and (winocoref knowcomb winocoref baseline 15)"
programs:
-aaeeinndbbsellineeeee ["winograd_ baseline 73.05ln winograd_knocoomb
Error case
gold: refutes,
TABLE I
prediction: supports,
WERS (%) AND DCE DIFTERENT SPEECH ENBANCEMENT METHODS DEMAND
programs: [
AAS WAC [_W 0 WER 15.61n
Method
WER (%)
DCE
AAS wAC WAD
1_wAD_105 105 WER 14.4In
No enhancement
17.3
0.828
AAS AAS_wAC 1_W 0 DCE 0.330
Wiener filter
19.5
0.722
Minimizing DCE
15.8
0.269
AAS WAC 1_ WAD 105 DCE 0.303ln
FSEGAN
14.9
0.291
ans (AAS WAC WAD 105 WER
AAS (Mrcc WAD
15.6
AAS [AC 1 wAD O WWR)
(Mav T,ora
14.4
and (AAS wAC WAD 105 DCE
Clean speach
AAS wAC_1 1_wWAD DCE)
Claim: The AAS method with weights wAC=1 and wAD=105 shows the lowest WER and DCE.
Error case 3
programs
"G2S GIN BLEU LDC2015E86 22.93In
Model
BLEU
METEOR
G2S GAT BLEU LDC2015E86 = 23.42in
ADC2015E86
G2S GGNN BLEU LDC2015E86 24.32
Konstas et al. (2017)
22.00
G2S GIN METEOR LDC2015E86 29.72in
Song et al. (2018)
23.28
30.10
G2S GAT METEOR LDC2015E86 29.87in
Cao et al. (2019)
23.50
G2S GGNN METEOR LDC2015E86 30.53ln
Damonte et (2019)
24.40
23.60
G2S G2S_GIN, BLEELBEEEULLLLLLLL LDC2017T10 26.90in
(2019)
25.70
G2S GAT BLEU LDC2017T10 26.72
S2S
22.55 1: 0.17
29.90
0.31
G2S GGNN BLEU LDC2017T10 27.87in
G2S-GIN
22.93 0.20
29.72 + 0.09
LLL_MMERRDD2200T11IOOO LDC2017710 32.62in
G2S-GAT
23.42 : 0.16
29.87
= 0.14
G2S GAT METEOR LDC2017T10 32.52in
G2S-GGNN
24.32 0.16
30.53
0.30
G2S GGNN METEOR LDC2017T10 33.21
LDC2017T10
ans = (G2S GAT BLEU LDC2015E86
Back et al. (2018)
23.30
G2S GIN BLEU LDC2015E86)
Song al. (2018)
24.86
31.56
and (G2S GGNN BLEU LDC2015E86
Damonte et (2.19)
24.54
24.07
G2S GAT BLEU LDC2015E86)
Cao et al. (2019)
26.80
and (G2S GAT METEOR LDC2015E86
Guo (2019)
27.60
G2S GIN METEOR,TR LDC2015E86)
S2S
22.73 + 0.18
30.15 0.14
and (G2S_GGNN MOENNM_ETORR.. METEOR_LDC20155
LDC2015E86
G2S-GIN
26.90 0.19
32.62 -
G2S GAT METEOR (EFFD__D)201588))
G2S-GAT
26.72 +: 0.20
32.52 + 0.02
and (G2S GAT BLEU LDC2017T10>
G2S-GGNN
27.87
33.21
G2S GIN BLEU [TUU_LD2201710)
and (G2S GGNN. BLEU LDC2017T10>
Table 2: BLEU and METEOR scores the test of LDC201 and LDC2017T10 datasets.
G2S GAT BLLBBUEUU LDC2017T10) and (G2S GAT METEOR LDC20111C0
G2S GEN_MMETEOR LDC2017T10) and
gold: supports, prediction: refutes
METEOR LDC2017T10 G"
claim: This suggests that graph encoders based gating mechanisms are very effective in text generation models.
Figure 13: Error Cases 1-3 for Program-of Thoughts. Error Case 1 exhibits incorrect entity linking (Grounding
error) and incorrect operation (Program error). Error Case exhibits incomplete entity linking (Grounding error). Error Case 3 exhibits Program error since it fails to generate a correct program.

---

Error case
ROUGE-1
ROUGE-2
Sentence-Leve
System
R (8)
((5)
(95)
R (%)
(())
F (8)
(86)
((6)
((6)
ILP
24.5
41.1
29.3:10.5
15.0
9.91:05
13.6
22.6
156::044
Sum-Basic
28.4
44.4
33.1:10.5
15.6
10.4::04
14.7
22.9
16.7.00.5
KL-Sam
39.5
34.6
35.5:00.5
13.0
12.3:105
15.2
21.1
163:0.55
LexRank
42.1
39.5
38.7:0..5
14,7
15.3
14.2:05
14.3
21.5
160=0.5
MEAD
45.5
38.5+
15,4+00
292
268-0.5
SVM
19.
46.8
34.720.8
7.5
21.1
10.0200
32,7
34.3
514120.4
LogReg
28.7.10.6
7.31004
12.7.0.55
LogRes"
28.0
34,8
29.4:10.6
6.9
10.4
7.84:04
12.1
14.3
125
HAN
31.0
42.8
33,7=007
11.2
17.8
12,7=005
26.9
34.1
32.4=0.5
HAN-pectrsint
32.2
34.4-10.7
12.9=00
35.8
HAN=pretraint
32.1
42.1
33,8:10.7
11.6
12.9:10
356
32.3=0.55
HAN
38.1
40.5
37.8+0.5
14.0
17.1
14.7=00
32.5
34.4
33.=00.5
HANopretrain""
31.9
37.6-0.5
14.4-00
34.4
334-00.5
HAN-pretrainU"
37.9
40.4
37.6+0.5
13.6
16.9
14.4=05
33.9
33.8
338-0.5
Table Revalts thread summorization. 'HAN'
pmppseed approuches adapted from hiemrchical attention nerworks (Yang models pretrained using unlubeled threads Drip/dvisoe (TT) Ubuntuforama ('U"), indicates redundancy
removal step is applicd. We repurt the variance of F-scures acroes all threads (#E redendancy remsvaal step improves rocall soures (shown in pray) of HAN models and boosts perfontance.
claim: Pretraining the HAN models, although intuitively promising, yields only comparable
results with those without.
gold: supports, prediction: refutes
Programs:
["HAN_ |_ score = 37.8
JAAN_pretrain1 F score 37.6in
HAN_ BAAN_preraain_____ F__score-eeeeee 37.6in
ans (HAN_E F scooreeree-e HAN pwwptriin___rae_e score 0.2) and
(HAN F score - HAN p_retrarettu F score == 0.2)"]
Error case
The effectiveness of our hierarchical attention design proved by an accuracy drop 1.95% after removing residual connections and the hierarchical stack of our attention modules.".
Claim:
Model
Joint
COMER
88 64%
Hiirrcchica-Alt
86.69%
MM.P
83.24%
gold: supports,
prediction: refutes, programs:
TTall 4: The ablation study on the WoZ2.0 datiset
with goul accuracy
COMER acc 68.64
Illerackica--Aer Att" we remove the residual connectioas
berween the attcation modales in the CMR decoders
COMER acc no. residual 86.69In accuracy pocuracy_drop>> COMER acc-
attention memory access outpur from the I.S.M, For + MI.P", P", we Further re- single layer with liseur activation.
hased
COMER_ace acc LaRRRR_aC_______e no_residualln
place
ans (accuracy_dmm) _dop == 1.95)
Figure 14: Error Cases 4 and for Pfogram-of-Thougtt. Error Case 4 exhibits Ambiguity error since it fails to generate precise program for the approximation word "comparable".. Error Case exhibits Calculation error since it generates the correct program, but the calculation result is inaccurate due to incorrect float digits in the Python code.


Reviews:
[{"review_id": "XSuH9gNncj", "reviewer": "Reviewer_z4dP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer z4dP for the thoughtful feedback. First, we are encouraged that Reviewer z4dp recognized the novelty of our work in introducing the SCITAB dataset, which offers a unique and significant task of verifying claims backed by tabular data, highlighting diverse reasoning types. Also, we are delighted that the reviewer confirms the challenges in the data, and the detailed error analysis, all of which add significant value to the research community. \n\nWe appreciate the valuable insights behind the comments and also want to clarify a few points. \n\n\n> *A. Judging from the examples in the paper, it's unclear how well-defined the task is.*\n\nOur task definition is to specifically verify the claim based on the table information provided, and this was emphasized in the annotation training guidelines. For example, we emphasized that the claim has to be exactly supported/ refuted by the table information, and we provided examples to the annotators during training on what should not be included. While the task definition is covered in Section 4 (Lines 318 - 328), the training process for annotators regarding task clarification was not included in the paper. We thank the reviewer for pointing this out. We will include the detailed task definition for training the annotators in the current Appendix B with a new subsection \"Annotator training process\" and append all the guidelines. \n\n\n\n> *B. E.g., the main example in Figure 1 relies on external \"commonsense knowledge\" that \"The number of random chance is 50%\".* \n>\n> *On the error analysis of InstructGPT in Figure 11, two of the examples are predicted as NEI which might be understandable since the claim mentions \"Portuguese\" while the table only has \"PT\"*. \n\nWe appreciate the reviewer's detailed observation. We note that certain commonsense knowledge, like understanding that a random chance is 50%, is naturally occurring in human reasoning. For the other example in Figure 11, the table contains only the abbreviations 'EN' and 'PT.' Since 'EN' represents English, our annotators logically deduced that 'PT' stands for Portuguese.  Similar to WANLI [1], during our training sessions, such commonsense interpretations and a minimum amount of logical inference are acceptable. Therefore, we believe that our examples are aligned with the task definition, although the interpretation of \"minimum amount of logical inference and commonsense knowledge\" can vary for different people. \n\nWe view this phenomenon as an interesting discussion point, rather than a definitive issue in our dataset. We will explain this in detail in Appendix B of the camera-ready version. \n\n\n\n> *C. It would be nice with a clearer human baseline, beyond the inter-annotator agreement indirect number (cohen kappa of 0.64 between two annotators).* \n\nWe appreciate the reviewer's suggestion on this. To examine how humans perform on our SCITAB dataset, we hired an annotator from our candidate annotators pool, following the same training procedure as other annotators. \n\nIn the case of 2-class classification, we randomly selected 40 samples—20 each for supported and refuted claims. For 3-class classification, we randomly selected 60 random samples, ensuring an even distribution of 20 samples across the three label categories (supported, refuted, and not enough information). The annotator took approximately 1.5 hours for the 2-class task and 2 hours for the 3-class task. We paid 11.2 USD per hour at the same rate described in the paper.  \n\nWe report the Macro-F1 scores for human performance, together with the corresponding GPT-4 performance for comparison, as shown in the Table below (to be added in Section 4). We observe that a well-trained human annotator can achieve 92.46 and 84.73 F1 scores on the 2-class and 3-class fact-checking, respectively. GPT-4, while performing the best among all models, still represents a large gap from human performance. We will include these new results in the main results table (Table 4) and the above findings in Section 4.2. \n\n\n|       | 2-class | 3-class |\n| ----- | ------- | ------- |\n| GPT-4 | 77.98   | 63.21   |\n| Human | **92.46**   | **84.73**   |\n\n\n\nIn conclusion, we deeply appreciate the reviewer's constructive comments and hope that our detailed response and commitment to improvement have addressed the concerns raised. \n\n[1] Alisa Liu, Swabha Swayamdipta, Noah A. Smith, Yejin Choi (2022). WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP). https://aclanthology.org/2022.findings-emnlp.508.pdf", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A new dataset on a potentially impactful task of verifying claims back by tabular data. \n\nAnalysis showing diversity of data. Interesting evaluations, showing it's a challenging task, with GPT-4 having the best performance. Detailed error analysis.", "reasons_to_reject": "Judging from the examples in the paper, it's unclear how well defined the task is. E.g., the main example in Figure 1 relies on external \"commensense knowledge\" that \"The number of random chance is 50%\" which doesn't seem clear from the context, as far as I can tell?  (also saying that 57.5% is 7.5% more than 50% is a bit unusual, at least ambiguous). On the error analysis of InstructGPT in Figure 11, two of the examples are predicted as NEI which might be understandable since the claim mentions \"Portuguese\" while the table only has \"PT\".\n\nIt would be nice with a clearer human baseline, beyond the inter-annotator agreement indirect number (cohen kappa of 0.64 between two annotators).", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces the SciTab dataset, which consists of tabular data paired with claims that can be labeled \"supported\", \"refuted\" or \"not enough information\" in the context of the table. The dataset starts from the existing SciGen dataset, filtering down to \"check-worthy\" claims. The InstructGPT is used to construct a claim with opposite meaning. InstructGPT is also used to construct other claims for the table. All claims are then verified by human annotators.\n\nIn total the dataset has 1225 claims fairly evenly spread over the 3 classes, with about 6 claims per table.\n\nAnalysis of the dataset shows a diverse set of reasoning types (like \"comparison\" and \"max/min\"), as well as a large fraction of claims requiring 3 or more \"reasoning steps\".\n\nVarious few-shot LLM and table-trained models are evaluated, including GPT-4 which reaches 3-class macro F1 scores of ~65, indicating that there's still some challenges in the data (although not clear what a human baseline would be). Smaller models generally do quite poorly, further indicating this is a hard task. A \"program-of-thought\" also doesn't perform very well.\n\nThere's a good amount of error analysis, with explicit examples given in the appendix.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dMa5dl2fj3", "reviewer": "Reviewer_P1EZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are sincerely grateful for the time Reviewer P1EZ has invested in reviewing our paper, and for the encouraging evaluation that underscores the positive aspects of our work. The insights into the construction of the SCITAB dataset, its high degree of difficulty, and its potential to spark further research are truly appreciated. The observations reflect a genuine understanding of the paper's main contributions and the reasons to accept it. We appreciate the valuable insights behind the comments and also want to clarify a few points.\n\n\n\n> *A. The atomic reasoning functions are not distributed reasonably enough, and the two types of The Simple lookup and The Comparison are over-represented.*\n\nWe appreciate the reviewer for identifying an interesting discussion point regarding the distribution of atomic reasoning functions.  We double-checked the calculation of the reasoning depths statistics and we found that it's true that Simple Lookup and Comparison account for a significant part of our dataset, representing 20.6% and 19.5%. Note that during data collection, we do not enforce any specific rules to favor certain types of claims. Therefore, this distribution is not the result of cherry-picking but rather a reflection of the natural distribution of real-world scientific claims. Actually, it is understandable that Simple Lookup and Comparison are more common in scientific claims. For example, authors often describe experimental findings by either reporting specific data points from tables (e.g., \"Model A has a performance of 50,\" which is a simple lookup) or by comparing performances between two models (e.g., \"Model B is 30% higher than Model A,\" which falls under comparison). Therefore, we believe this distribution is quite reasonable and reflective of real-world academic practices. Lastly, we would like to clarify that even though Simple Lookup and Comparison functions are common in our dataset, this does not imply that the scientific claims in our dataset are simple. In fact, Simple Lookup and Comparison functions do not just exist in simple claims; we found that many complex claims also use Simple Lookup and Comparison because these functions often serve as the foundation for more complex reasoning. \n\n\n\n> *B. The use of LLM to generate data reduces the workload, but the final dataset had significantly less data than the comparison dataset.*\n\nWe thank the reviewer for pointing this out. First, we acknowledge the relatively small size of our SCITAB dataset due to the involvement of human efforts. However, even with the LLM to help reduce the workload, human efforts are still imperative to ensure the data quality. To ensure the high-quality and challenging nature of the dataset, we employed a strict data annotation process. Many stages of our annotation pipeline involve manual verification to ensure the quality of the data, which naturally limits the final quantity. In the trade-off between quantity and quality, we prioritize quality as our goal is to create a diagnostic dataset that serves as a reliable benchmark for model evaluation. As highlighted in the paper's title and introduction, the primary purpose of this dataset is to serve as an evaluation dataset, aiming to present a challenging benchmark for compositional reasoning and claim verification on scientific tables. We believe the size of 1.2K is sufficient for an evaluation dataset as we do not include a training set. This is a comparable size with many existing evaluation datasets such as HumanEval [1] (164 samples) and SciBench [2] (695 samples).\n\nIn conclusion, we deeply appreciate the reviewer's constructive comments and hope that our detailed response and commitment to improvement have addressed the concerns raised.\n\n[1] Chen et al. Evaluating Large Language Models Trained on Code. (2021). https://arxiv.org/abs/2107.03374\n\n[2] Wang et al. SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models. (2023). https://arxiv.org/abs/2307.10635", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A.Adequate experiments have been conducted in this paper and the results have been analyzed in sufficient detail.\nB.All models except GPT-4 perform poorly on the dataset constructed in this paper, leaving room for further research.\nC.The paper is easy to follow.", "reasons_to_reject": "A.The atomic reasoning functions are not distributed reasonably enough, and the two types of The Simple lookup and The Comparison are over-represented.\nB.The use of LLM to generate data reduces the workload, but the final dataset had significantly less data than the comparison dataset.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, a new dataset SCITAB is constructed for addressing some of the current shortcomings of current scientific fact-checking benchmarks. The SCITAB dataset has a high degree of difficulty, with all models except GPT-4 barely outperforming randomized guessing, and provides an excellent dataset for further subsequent research in this area.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vjI0W4qhqW", "reviewer": "Reviewer_BWAe", "summary": "", "questions": "QA: I'm not very clear about the method for generating unverifiable claims described in Sec 2.2. Why is this method (\"... generate candidate NEI claims by prompting the model with the original table and the instruction ...\") able to generate unverifiable claims? Why not remove/replace parts of the table for this purpose? Although manual verification is appreciated, I feel like it's never a bad thing to save the use of human labor for other purposes (like scaling up the datasets).\n\nQB: As in the paper, most reasoning functions in table reasoning are simple, like add, subtraction, lookup, etc. However, when these functions are used together, the results of LLMs turn out to be poor. I would like to see some analysis of the compositionality problem in table-based reasoning using this dataset.\n\nQC: Have you tried other advanced prompting techniques for reasoning, such as self-consistency, least-to-most prompting, etc.?\n\nQD: Can the human-machine collaboration pipeline be optimized? Are there any guidelines for deciding which parts of the data construction process should use human labor?\n\nQE: Given the fact that 20% of the errors are caused by miscalculations, I wonder if equipping the models with external tools (like calculators) will help. Parsing code from the text for the Python interpreter may be too heavy for simple calculations.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer BWAe for the thoughtful feedback. We are encouraged by the positive remarks on our work, especially highlighting the introduction of the SCITAB dataset, the high quality achieved through manual verification, presenting a crucial challenge for the research community, and the insightful error analysis which offers an understanding of why LLMs fail on this task. We appreciate the valuable insights behind the comments and also want to clarify a few points.\n\n\n> *A. The human-machine collaboration for data annotation (i.e., machines generate data, humans verify data) is widely adopted in this work. Although it is commendable, I'd like to see some analysis of the best practice of using \"precious\" human labor in the construction of a dataset.*\n\nOur final data creation pipeline undergoes repetitive testing and revision until it reaches its current form. In our pilot annotation, we found that manual verification played the most essential role in the validation of claims marked as \"Not Enough Information\" (NEI). Initially, we planned to rely solely on LLMs for generating NEI claims. Our criteria for the NEI claim is that \"the claim should be fluent, logical, and relevant to the table. However, the claim cannot be verified as true or false solely based on the information in the table.\" However, after a careful examination of the LLM output, we found that LLM tends to generate claims that are either not logical or irrelevant to the table content. Therefore, human efforts are required to further select NEI claims that meet our criteria. Out of 900 initial NEI claims generated by LLMs, manual verification narrowed them down to only 355 claims, taking up 40% of the original count. While it may not have served a crucial role in filtering NEI claims, human verification also safeguarded the data quality in other annotation processes. For example, among the \"supported\" claims originally appearing in the scientific paper, human validation still identified 10 examples that were actually not supported  (e.g., wrong number matching, etc.). Following the reviewer's suggestion, we will add the above discussion on the level of human contributions to each part of our data creation pipeline to the camera-ready version.\n\n\n> *B. Certain manual annotation experiments are conducted by only one annotator (Line 122), which could result in undetectable biases.* \n\nFirst, we want to clarify that the only manual annotation experiment with one annotator is the initial claim extraction phase. We assign a single annotator for this phrase because identifying whether a sentence is a scientific claim is a relatively easy and objective task. During our pilot annotation, we found that a single trained annotator can achieve over 95% accuracy in filtering scientific claims. Therefore, to save the cost of human labor, we use a single annotator. To further safeguard the quality, during the subsequent claim verification process, we include an option to mark the claim as \"Discard: It's not a claim, or it's an incomplete, or not grammatically correct sentence.\" We will include the above annotation details in Section 2.1 - Data Preparation. \n\n\n\n> *C. I would like to see some in-depth evaluation and analysis for LLMs with SCITAB in the main body of the paper, for example, the compositionality in reasoning (see QB). Perhaps more experiments or case studies should be incorporated into the main content. Some material from the appendix could potentially be included to strengthen the argument that SCITAB is a challenging dataset.* \n\nWe agree that in-depth evaluation and analysis of LLMs are both necessary and interesting. To clarify, we have indeed conducted in-depth evaluations and analyses of LLMs in several sections of our paper. These include 1) Section 4.2 Error Analysis, which investigates why LLMs (InstructGPT, GPT-4, and PoT) fail with the SCITAB dataset; 2) Appendix F Error Cases for InstructGPT; and 3) Appendix G Error Cases for PoT, showcasing specific error cases along with their explanations. However, due to space limits, many of these explanations are given in the appendix, such as Lines 924 - 950 and Lines 952 - 983. Following the reviewer's suggestion, we will include some of these details in the main body of the camera-ready version. \n\n\n> *QA: I'm not very clear about the method for generating unverifiable claims described in Sec 2.2. Why is this method (\"... generate candidate NEI claims by prompting the model with the original table and the instruction ...\") able to generate unverifiable claims? Why not remove/replace parts of the table for this purpose? Although manual verification is appreciated, I feel like it's never a bad thing to save the use of human labor for other purposes (like scaling up the datasets).* \n\nFirst, regarding why this method generates unverifiable claims, we have explained the reason in the paper from Lines 163 to 167: \"However, as LLMs tend to generate content that might not always be grounded in the provided data, many of the generated claims turn out to be relevant but unverifiable with respect to the table.\" The reason behind this phenomenon is LLMs often generate outputs that are not grounded in the input data, a phenomenon known as \"hallucination\" [1]. In the NEI claim generation, the hallucination behavior of LLMs results in \"many of the generated claims turn out to be relevant but unverifiable with respect to the table.\"\n\nSecond, regarding why not remove/replace parts of the table, our primary goal is to produce a realistic scientific fact-checking dataset, which necessitates that all tables are sourced from *actual* scientific papers. Removing or replacing these tables would violate this principle. Moreover, revising tables would also be technically challenging: if done by models, an understanding of table semantics would be needed; if done manually, the process would be highly labor-intensive. Therefore, we adopted our current approach, where LLMs generate candidate claims that are subsequently filtered by human annotators. This allows us to achieve a balance between the scale (with the help of LLMs) and the high quality (through human verification) of the dataset. \n\nLast, we agree with the reviewer that human labor is quite precious and we need to save it as long as we can. However, we found that human labor plays a quite important role in each part of our annotation pipeline, especially for handling NEI claims (Please refer to our previous response to point A\"analysis of the best practice of using human labor in dataset construction.\" ) \n\n\n\n> *QB: As in the paper, most reasoning functions in table reasoning are simple, like add, subtraction, lookup, etc. However, when these functions are used together, the results of LLMs turn out to be poor. I would like to see some analysis of the compositionality problem in table-based reasoning using this dataset.* \n\nThanks for pointing this out. We agree that studying the compositionality problem in table-based reasoning is an interesting research direction. This study typically requires a breakdown performance evaluation and analysis for each level of reasoning depth and each type of reasoning graph. However, as we acknowledged in the Limitations section, the ground-truth reasoning depths and reasoning graphs were not included in our dataset. Annotating them would be time-consuming and costly. Given our limited time, we are unable to undertake such extensive and detailed annotation at this moment. But we do acknowledge this as an interesting future work. One possible approach is to follow the methodology in Self-Ask [2], in which the authors proposed a new metric called the \"compositionality gap\" to measure the capability of LLMs in compositional reasoning. However, their metric still requires the ground-truth annotation of the composed question together with its decomposed atomic questions. \n\n\n> *QC: Have you tried other advanced prompting techniques for reasoning, such as self-consistency, least-to-most prompting, etc.?*\n\nThanks for pointing out an interesting discussion question. We have included an additional experiment for “self-consistency” prompting in response to this feedback. We use the 5-way self-consistency (the final answer is based on the majority vote of 5 generated candidate \n answers) and test with InstructGPT and GPT-4 in the 2-class classification task. We observe that self-consistency prompting slightly improves the performance over all four different settings, with an average improvement of 2.1%. We will include this additional result in the camera-ready version. \n\n|                         | w/o self-consistency | w/ self-consistency |\n| ----------------------- | -------------------- | ------------------- |\n| InstructGPT (zero-shot) | 68.44                | **69.80**           |\n| InstructGPT + CoT       | 68.46                | **68.85**           |\n| GPT-4  (zero-shot)      | 78.22                | **78.72**           |\n| GPT-4 + CoT             | 76.85                | **80.91**           |\n\n\n\n\n> *QD: Can the human-machine collaboration pipeline be optimized? Are there any guidelines for deciding which parts of the data construction process should use human labor?*\n\nThanks for pointing out an interesting discussion question. Our final data creation pipeline undergoes repetitive testing and revision until it reaches its current form. In our pilot annotation, we carefully evaluate the necessity and extent of human involvement (Please refer to our previous response to point A \"analysis of the best practice of using human labor in dataset construction.\") to optimize the human-machine collaboration pipeline. We believe our current pipeline is a balance between the scale (with the help of LLMs) and the quality (through human verification) of the dataset. \n\nWith regard to the guidelines for involving human labor, we believe there is no unified guideline that fits all data annotations. We need to conduct pilot studies to determine the best strategy for human-machine collaboration on a case-by-case basis. However, we found that the principles adopted in WANLI [3] serve as a representative work for using human-machine collaboration. In this approach, data examples are automatically generated by GPT-3 and then revised and labeled by human crowd workers.\n\n\n\n> *QE: Given the fact that 20% of the errors are caused by miscalculations, I wonder if equipping the models with external tools (like calculators) will help. Parsing code from the text for the Python interpreter may be too heavy for simple calculations.* \n\nThanks for pointing out an interesting discussion question. We agree that equipping models with calculators could potentially improve the model, and this is something we plan to explore in future work. To this end, we are considering the use of GPT-4 plugins or adopting approaches from other tool-augmented LLMs like Toolformer [4] and Chameleon [5]. \n\n\n\n> *Typos Grammar Style And Presentation Improvements:I suggest using different terms between \"decoder-only LLMs\" and \"API-based LLMs\", since some API-based LLMs are also decoder-only LLMs (e.g., InstructGPT). Maybe it's better to distinguish them with \"open-sourced\" and \"closed-sourced\" models.*\n\nThanks for the suggestions! We will revise the terms \"open-sourced\" and \"closed-sourced\" models in the final version as suggested.\n\nIn conclusion, we deeply appreciate the reviewer's detailed comments and hope that our detailed response and commitment to improvement have addressed the concerns raised.\n\n[1]  Muru Zhang, Ofir Press, William Merrill, Alisa Liu, Noah A. Smith. (2023). How Language Model Hallucinations Can Snowball. https://arxiv.org/abs/2305.13534\n\n[2] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, Mike Lewis. (2022). Measuring and Narrowing the Compositionality Gap in Language Models.  https://arxiv.org/abs/2210.03350\n\n[3] Alisa Liu, Swabha Swayamdipta, Noah A. Smith, Yejin Choi (2022). WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP). https://aclanthology.org/2022.findings-emnlp.508.pdf\n\n[4] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, Thomas Scialom. (2023). Toolformer: Language Models Can Teach Themselves to Use Tools. https://arxiv.org/abs/2302.04761\n\n[5] Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Jianfeng Gao. (2023). Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models.  https://arxiv.org/abs/2304.09842", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I think the major reason to accept this paper is that SCITAB is a manually verified, high-quality dataset, presenting a crucial challenge for the research community, particularly considering that LLMs have achieved exceptional results on many traditional tasks. Besides, the detailed annotation and manual filtration steps implemented in the dataset ensure the high quality of SCITAB. To understand why these models fail, this paper also provides an interesting error analysis that offers an understanding of why LLMs fail to accomplish this task.", "reasons_to_reject": "1. The human-machine collaboration for data annotation (i.e., machines generate data, humans verify data) is widely adopted in this work. Although it is commendable, I'd like to see some analysis of the best practice of using \"precious\" human labor in the construction of a dataset.\n2. Certain manual annotation experiments are conducted by only one annotator (Line 122), which could result in undetectable biases.\n3. I would like to see some in-depth evaluation and analysis for LLMs with SCITAB in the main body of the paper, for example, the compositionality in reasoning (see QC). Perhaps more experiments or case studies should be incorporated into the main content. Some material from the appendix could potentially be included to strengthen the argument that SCITAB is a challenging dataset.", "questions_to_authors": "QA: I'm not very clear about the method for generating unverifiable claims described in Sec 2.2. Why is this method (\"... generate candidate NEI claims by prompting the model with the original table and the instruction ...\") able to generate unverifiable claims? Why not remove/replace parts of the table for this purpose? Although manual verification is appreciated, I feel like it's never a bad thing to save the use of human labor for other purposes (like scaling up the datasets).\n\nQB: As in the paper, most reasoning functions in table reasoning are simple, like add, subtraction, lookup, etc. However, when these functions are used together, the results of LLMs turn out to be poor. I would like to see some analysis of the compositionality problem in table-based reasoning using this dataset.\n\nQC: Have you tried other advanced prompting techniques for reasoning, such as self-consistency, least-to-most prompting, etc.?\n\nQD: Can the human-machine collaboration pipeline be optimized? Are there any guidelines for deciding which parts of the data construction process should use human labor?\n\nQE: Given the fact that 20% of the errors are caused by miscalculations, I wonder if equipping the models with external tools (like calculators) will help. Parsing code from the text for the Python interpreter may be too heavy for simple calculations.\n", "paper_topic_and_main_contributions": "This paper introduces a new dataset called SCITAB, which is designed to address the limitations of existing benchmarks for scientific fact-checking. The SCITAB dataset consists of 1225 challenging scientific claims that require compositional reasoning with scientific tables. These claims are derived from actual scientific statements, and the evidence is presented in the form of tables, which mirrors real-world fact-checking scenarios. The authors have established benchmarks on SCITAB using state-of-the-art models and have identified unique challenges, such as ambiguous expressions and irrelevant claims. The main contribution of this paper is the creation of the SCITAB dataset, which provides a new resource for the research community to further explore and improve scientific fact-checking. The authors have made the dataset publicly available for further research. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Not that I know of."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. I suggest using different terms between \"decoder-only LLMs\" and \"API-based LLMs\", since some API-based LLMs are also decoder-only LLMs (e.g., InstructGPT). Maybe it's better to distinguish them with \"open-sourced\" and \"closed-sourced\" models.\n"}}]

Author rebuttal:

Meta-review: Summary: This paper introduces a new dataset called SCITAB, which is designed to address the limitations of existing benchmarks for scientific fact-checking. The dataset consists of 1225 challenging scientific claims that require compositional reasoning with scientific tables. These claims are derived from actual scientific statements, and the evidence is presented in the form of tables, which mirror real-world fact-checking scenarios. The authors have established benchmarks on SCITAB using state-of-the-art models and have identified unique challenges, such as ambiguous expressions and irrelevant claims. 

Strengths: The main contribution of this paper is the creation of the SCITAB dataset. All the reviewers unanimously agree that this is a solid contribution and provides a new resource for the research community to further explore and improve scientific fact-checking. The paper is well-written and easy to follow.

Weaknesses: I don't think there are any major weaknesses -- some of the weaknesses have been addressed during the discussion phase. The authors should take the experimental results during the rebuttals into account when preparing for the final version of the paper.
Decision: Accept-Main