SlowGuess/NAACL_2022_OCR huggingface dataset:
Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Doug Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Michael Kinney, Yunyao Li, Ziyang Liu, William Merrill, Paul Mooney, Dewey A. Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex D. Wade, Kuansan Wang, Nancy Xin Ru Wang, Christopher Wilhelm, Boya Xie, Douglas M. Raymond, Daniel S. Weld, Oren Etzioni, and Sebastian Kohlmeier. 2020. CORD-19: The COVID-19 open research dataset. In Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020.
Dustin Wright, David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Isabelle Augenstein, and Lucy Lu Wang. 2022. Generating Scientific Claims for Zero-Shot Scientific Fact Checking. In ACL.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big Bird: Transformers for Longer Sequences. In NeurIPS.
Zhiwei Zhang, Jiyi Li, Fumiyo Fukumoto, and Yanming Ye. 2021. Abstract, Rationale, Stance: A Joint Model for Scientific Claim Verification. In EMNLP.
A Data processing and statistics
SCIFACT We use SCIFACT in its original form, as it was released by the paper authors (Wadden et al., 2020).
HealthVer The HealthVer (Sarrouti et al., 2021) data release available at https://github.com/sarrouti/HealthVer appears in NLI format, pairing claims with evidence-containing sentences; the documents from which the sentences were extracted are not provided. We match evidence-containing sentences to their abstracts in the CORD-19 corpus (Wang et al., 2020) using a simple substring search, after normalizing for capitalization and whitespace differences. Evidence for which no match was found in the corpus is discarded.
We then segment the abstracts into sentences. Any sentence in the abstract with a string overlap of  $>50\%$  with the evidence provided in the original data is marked as a rationale. A small number of claims in HealthVer had both supporting and refuting evidence in the same abstract; we remove these claims as well to conform to our task definition. Modeling conflicting evidence is a promising direction for future work.
COVIDFact The COVIDFact data available at https://github.com/asaakyan/covidfact is released in a similar format to HealthVer. Like HealthVer, we perform string search over CORD-19 to identify the abstracts containing evidence, and use the same procedure for assigning rationale labels to sentences from the abstract. COVIDFact also includes evidence from sources scraped from the web that are not contained in CORD-19, such as news articles. These sources are not provided with the data release; we discard evidence from non-CORD-19 sources<sup>7</sup>.
Refuted claims in COVIDFact are generated automatically by replacing words in the original claim. Based on a manual inspection, we found this process to generate a truly refuted claim roughly a third of the time; in most other cases, it generated a claim that was either ungrammatical or for which the provided evidence was irrelevant. A few cases are provided in Table 6.
FEVER We use the FEVER dataset as-is.
EVIDENCEINFERENCESHEET consists of "ICO" (intervention / comparator / outcome) prompts, paired with labels indicating whether the intervention leads to an increase, decrease, or no change in the outcome with respect to the comparator. The dataset is available at https://evidence-inference.ebm-nlp.com/. We use templates to convert these prompts to claims. See Figure 2 for an example. Rationale annotations are provided for this dataset. Additional examples of templates are below; the full list will be included in the code release. Refuted claims are generated by swapping "increase" and "decrease" templates.
- Increase: [intervention] raises [outcome] relative to [comparator]
- No change: [intervention] and [comparator] have very similar effects on [outcome]
- Decrease: [intervention] results in a decrease in [outcome], relative to [comparator]
PUBMEDQA We use the PQA-A subset released at https://pubmedqa.github.io/, which is filtered for "claim-like" titles. We generate negations by identifying titles with the phrases "does not",
Figure 2: An example showing how an evidence inference prompt (top) can be converted into a claim (bottom) using templates. A refuted claim could be generated by substituting "increases" for "decreases" in the prompt text.
“do not”, “are not”, “is not”. “Does not” and “do not” are removed and the relevant verbs are modified to have the correct inflection; for instance “smoking does not cause cancer” is converted to “smoking causes cancer”. Similarly, “are not” and “is not” are replaced by “are” and “is”.
To generate rationales needed to train pipeline models on PUBMEDQA, we employ the following procedure. First, we encode the claim and all abstract sentences using the all-MiniLM-L6-v2 model from the Sentence-Transformers package https://www.sbert.net/. Then, we rank abstract sentences by cosine similarity with the claim and label the top- $k$  sentences as rationales, where  $k$  is randomly sampled from  $\{1,2,3\}$  with a 4:2:1 frequency ratio (this matches the distribution of  $k$  in SCIFACT).
A.2 Dataset statistics
Table 7 provides counts showing the number of claim / evidence pairs with each label (SUPPORTS, REFUTES, NEI), in each of our target datasets. Note that a given claim may be (and often is) paired with more than one abstract containing evidence. HealthVer is the largest dataset. COVIDFact is the smallest, in part due to the aggressive evidence filtering described in §A.1.
A.3 Examples of context-dependent rationales
Table 8 provides an example of a context-dependent rationale (as defined in §7.1), as well as an example of a rationale with an undefined acronym. The latter occurs when an acronym appears in a rationale but its full expansion does not; an analysis of undefined acronyms is included in Appendix C.2. The code and data release will contain full annotations indicating which of the 128 human-annotated examples described in §7.1 are context-dependent, and which contain undefined acronyms.
Upon request, the paper authors did kindly provide us with scraped evidence documents. Unfortunately, we did not have time to re-run our experiments on these additional sources.
Original claim	Automatic negation	Comment
Sars-cov-2 reactive t cells ... are likely expanded by beta-coronaviruses	Sars-cov-2 reactive t cells ... are not expanded by beta-coronaviruses	Successful negation
Regn-cov2 antibody cocktail prevents and treats sars-cov-2 ...	On-cov2 antibody cocktail prevents and treats sars-cov-2 infection ...	Ungrammatical; “On-cov2” isn’t a scientific entity.
...immunity is maintained at 6 months following primary infection	...immunity is maintained at 6 weeks following primary infection	Not refuted; The original claim entails the negation. Immunity at 6 months implies immunity at 6 weeks.
Table 6: Automatic negations from COVIDFact. Some are successful, in the sense that the attempted negation contradicts the original claim. Others are either ungrammatical or are entailed by the original claim.
Fold	Dataset	SUPPORTS	NEI	REFUTES
Train	SCIFACT	508	485	265
COVIDFact	299	-	641
HealthVer	2384	2384	1464
Eval	SCIFACT	113	127	109
COVIDFact	102	-	215
HealthVer	374	304	225
Table 7: Evidence distribution by dataset.
A.4 Annotators
In §7, we report an analysis based on annotations performed on the SCIFACT dataset. These annotations were performed by students and / or professional annotators associated with the authors' research institutions. Annotators were paid between $15 and $20 / hour.
B Modeling details
We implement MULTIVERS using PyTorch Lightning (https://www.pytorchlightning.ai/), which relies on PyTorch (https://pytorch.org/).
Pretraining For pretraining, we train for 3 epochs on FEVER, EVIDENCEINFERENCE, and PUBMEDQA, with the data randomly shuffled. We train on 4 negative samples (i.e. abstracts containing no evidence) per claim, which we find improves precision. We train on 8 NVIDIA RTX 6000 GPUs with a batch size of  $1 / \mathrm{gpu}$  (effective batch size of 8), using a learning rate of  $1e - 5$ , using the PyTorch Lightning implementation of the AdamW optimizer with default settings. We initialize from a Longformer-large checkpoint pretrained on the S2ORC corpus (Lo et al., 2020).
Finetuning For finetuning, we train for 20 epochs on the target dataset (SCIFACT, HealthVer,
or COVIDFact). For SCIFACT, we train on 20 negative samples / claim. To create "hard" negatives — i.e. abstracts that have high lexical overlap with the claim — we create a search index from 500K abstracts randomly selected from the biomedical subset of the S2ORC corpus. For each claim, we obtain negative abstracts by using the VERT5ERINI retrieval system from §3.1 to retrieve the top-1000 most-similar abstracts from this index, removing abstracts that are annotated as containing evidence, and randomly sampling 20 abstracts to be used as negatives during training.
Since HealthVer and COVIDFact do not have a retrieval step, they do not require negative sampling, and we train on the original datasets as-is.
Retrieval For SCIFACT, we performed dev set experiments retrieving 10, 20, or 50 abstracts / claim, and found that 10 was the best. We use that in our final experiments.
B.3 Model hyperparameters
No organized hyperparameter search was performed. We consulted with the authors of the Longformer paper for suggestions about good model parameters, and generally followed their suggestions.
The loss function in Section 3.1 requires a weight  $\lambda_{\mathrm{rationale}}$ . This is set to 15 for all final experiments. We informally experimented with values of 1, 5, and 15; no organized hyperparameter search was performed. We selected the learning rate from the values  $[9e - 5, 5e - 5, 1e - 5]$ .
We performed all experiments with the same random seed, 76, used by invoking the seedeverything function in PyTorch Lightning.
All reported results are from a single model run.
VERT5ERINI For prediction on SCIFACT, we use the checkpoint available at
Category	Example
Context-dependent	Claim:	Errors in peripheral IV drug administration are most common during bolus administration
Context:	OBJECTIVES: To determine the incidence of errors in the administration of intravenous drugs ...
Evidence:	... Most errors occurred when giving bolus doses
Explanation:	The evidentiary sentence reporting the finding does not specify the type of error.
Undefined acronym	Claim:	Hematopoietic stem cells segregate their chromosomes randomly.
Context:	we tested these hypotheses in hematopoietic stem cells (HSCs)...
Evidence:	... indicated that all HSCs segregate their chromosomes randomly.
Explanation:	HSCs is an acronym for Hematopoietic stem cells.
Table 8: Examples from the SCIFACT dataset showcasing rationales that are context-dependent (top example), or include an undefined acronym (bottom example).
https://github.com/castorini/pygaggle/tree/master/experiments/vert5erini. For COVIDFact and HealthVer, we follow the instructions in that repository to convert the data to the required format, and train using the available training code as-is, beginning from the available SCIFACT checkpoint. We used Google Cloud TPU for training and inference.
PARAGRAPHJOINT We use the code available at https://github.com/jacklxc/ParagraphJointModel. For predictions on SCIFACT, we make predictions using the publicly available checkpoint. For the other two target datasets, we use the training code in the repo without modification.
We used PARAGRAPHJOINT as our baseline for zero / few-shot learning experiments, and hence also performed pretraining on PARAGRAPHJOINT. The repository provides code to train on the FEVER dataset, which we used for pretraining with EVIDENCEINFERENCE and PUBMEDQA added to the data.
C Additional results and analysis
C.1 Full ablation results
In Table 3, we presented F1 scores for ablations comparing pretraining data, model architecture, and encoder used. Table 9 presents the full results, including precision and recall.
C.2 Performance on rationales with undefined acronyms
In §7.1, we examined the difference in performance on instances with self-contained vs. context-dependent evidence. Here, we show the results of evaluation on instances containing an undefined acronym vs. cases without one. We find that undefined acronyms do not pose a challenge for Multi-
task and Pipeline, but do cause a small performance drop on MT / PI.
C.3 Negative sampling
In §5.1 we described how, for SCIFACT, we trained on 20 negative abstracts per claim. The effect of training on these additional negative samples is shown in Figure 11. In the abstract-provided setting, negative sampling is not very beneficial. However, when the model must select evidence from retrieved abstracts, precision drops off dramatically without negative sampling. This is worth noting since it suggests that performance reported when models are provided with "gold" candidate abstracts may not offer an accurate estimate of the accuracy these systems would achieve when deployed in a real-world setting, which could require systems to verify claims over hundreds of thousands of documents.
C.4 Cross-dataset generalization
In §5, we discussed how the available scientific fact-checking datasets differ in a number of important respects. Here, we explore whether models trained on one system are able to generalize to another despite these differences. We train MULTIVERS on each of our three datasets and then evaluate its performance on the other two. We also train a version of MULTIVERS on all three datasets together, and evaluate on each one. Since COVIDFact has no NEI instances, during evaluation we remove all NEI instances from the other two datasets, and evaluate in the abstract-provided setting.
The results are shown in Table 12. The sentence-level evaluation results (Table 12b) indicate that none of the datasets generalize well to each other in their ability to identify rationales. The situation is better for abstract labeling (Table 12a). SCIFACT and HealthVer each generalize reasonably well to
Pretraining	HealthVer	COVIDFact	SCIFACT
Abstract	Sentence	Abstract	Sentence	Abstract	Sentence
P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Zero	FEVERSCI	60.6	20.5	30.7	25.0	4.6	7.8	48.8	45.7	47.2	32.7	18.5	23.6	49.0	44.6	46.7	39.0	21.6	27.8
FEVER	80.0	0.7	1.3	66.7	0.4	0.7	95.8	14.5	25.2	63.5	6.2	11.2	83.8	14.0	23.9	64.9	6.5	11.8
Few	FEVERSCI	63.6	47.9	54.7	41.9	31.0	35.7	71.3	68.1	69.7	39.5	35.4	37.4	76.4	54.1	63.3	51.7	40.3	45.3
FEVER	56.4	50.8	53.4	34.8	29.4	31.9	74.4	74.4	74.4	39.3	45.3	42.1	72.4	43.7	54.5	48.8	32.4	39.0
No-Pretrain	38.5	40.4	39.4	28.5	25.7	27.0	67.8	67.8	67.8	24.9	20.7	22.6	20.0	30.6	24.2	9.5	12.7	10.8
Full	FEVERSCI	78.9	76.3	77.6	71.4	67.0	69.1	77.3	77.3	77.3	41.5	46.1	43.7	73.8	71.2	72.5	67.4	67.0	67.2
FEVER	77.5	76.6	77.1	70.8	69.8	70.3	77.5	77.3	77.4	40.6	46.5	43.3	64.3	72.1	67.9	57.1	67.0	61.7
No-Pretrain	75.0	74.0	74.5	71.8	67.8	69.7	69.7	69.7	69.7	35.3	38.1	36.6	64.9	61.7	63.3	62.7	54.6	58.4
(a) Effect of pretraining data.
Encoder	HealthVer	COVIDFact	SCIFACT
Abstract	Sentence	Abstract	Sentence	Abstract	Sentence
P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Zero	Longformer	60.6	20.5	30.7	25.0	4.6	7.8	48.8	45.7	47.2	32.7	18.5	23.6	49.0	44.6	46.7	39.0	21.6	27.8
RoBERTa	59.5	24.0	34.2	25.4	5.6	9.2	49.3	47.3	48.3	35.2	20.9	26.2	45.5	45.0	45.2	34.4	20.8	25.9
Few	Longformer	63.6	47.9	54.7	41.9	31.0	35.7	71.3	68.1	69.7	39.5	35.4	37.4	76.4	54.1	63.3	51.7	40.3	45.3
RoBERTa	55.0	47.9	51.2	39.0	35.0	36.9	72.5	71.6	72.1	39.7	42.5	41.0	59.0	44.1	50.5	36.8	31.6	34.0
Full	Longformer	78.9	76.3	77.6	71.4	67.0	69.1	77.3	77.3	77.3	41.5	46.1	43.7	73.8	71.2	72.5	67.4	67.0	67.2
RoBERTa	77.8	80.0	78.8	73.4	72.0	72.7	78.2	78.2	78.2	40.8	46.3	43.4	67.1	68.0	67.6	62.7	61.9	62.3
(b) Effect of base encoder.
Approach	HealthVer	COVIDFact	SCIFACT
Abstract	Sentence	Abstract	Sentence	Abstract	Sentence
P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Zero	Multitask	60.6	20.5	30.7	25.0	4.6	7.8	48.8	45.7	47.2	32.7	18.5	23.6	49.0	44.6	46.7	39.0	21.6	27.8
Pipe	58.8	1.7	3.2	29.4	0.5	0.9	67.3	11.0	19	57.4	5.8	10.5	80.6	13.1	22.5	72.2	7.0	12.8
MT / PI	60.9	2.3	4.5	41.7	0.9	1.8	78.5	16.1	26.7	57.7	7.6	13.5	80.9	17.1	28.3	75.5	10.0	17.7
Few	Multitask	63.6	47.9	54.7	41.9	31.0	35.7	71.3	68.1	69.7	39.5	35.4	37.4	76.4	54.1	63.3	51.7	40.3	45.3
Pipe	56.3	49.7	52.8	32.6	27.0	29.5	69.4	67.2	68.3	40.6	36.0	38.2	54.8	51.4	53.0	43.7	36.8	39.9
MT / PI	67.0	35.9	46.7	44.5	25.3	32.3	72.6	50.2	59.3	40.2	29.7	34.1	85.3	41.9	56.2	54.7	33.0	41.1
Full	Multitask	78.9	76.3	77.6	71.4	67.0	69.1	77.3	77.3	77.3	41.5	46.1	43.7	73.8	71.2	72.5	67.4	67.0	67.2
Pipe	78.7	78.1	78.4	70.2	68.3	69.2	79.9	75.4	77.6	48.2	47.2	47.7	68.5	73.4	70.9	64.5	68.1	66.2
MT / PI	77.6	64.8	70.6	70.0	59.5	64.3	77.7	69.4	73.3	43.6	44.4	44.0	80.5	48.2	60.3	70.5	47.8	57.0
(c) Effect of model architecture.
each other, but not to COVIDFact. COVIDFact generalizes well to SCIFACT, but not to HealthVer. In general, SCIFACT appears the "easiest" dataset to generalize to; this could be explained by the fact that SCIFACT claims were written to be atomic and therefore simple to verify.
Finally, a model trained on all datasets combined manages to achieve reasonable performance across all three datasets, though falling short of the performance of models trained specifically for each individual dataset.
Table 9: Full ablation results.
Approach	No undefined acronym	Undefined acronym	%Δ
P	R	F1	P	R	F1
Multitask	88.1	73.8	80.3	86.0	77.1	81.3	1.2%
Pipeline	89.9	77.5	83.2	88.6	81.2	84.8	1.9%
MT / PI	97.1	42.5	59.1	85.0	35.4	50.0	-15.4%
Count	80	48
Table 10: Performance of different modeling approaches on instances with vs. without an undefined acronym. We perform evaluation on the same data as reported in Table 4.
Retrieval	Neg. sample	Abstract	Sentence
P	R	F1	P	R	F1
Abstract-provided	X	81.9	85.6	83.7	69.5	69.7	69.6
✓	85.2	75.2	79.9	79.0	70.3	74.4
Open	X	38.9	80.6	52.5	35.4	65.1	45.9
✓	73.8	71.2	72.5	67.4	67.0	67.2
Table 11: Effect of negative sampling on SCIFACT.
Eval → Train ↓	HealthVer	COVIDFact	SCIFACT
F1	Δ	F1	Δ	F1	Δ
HealthVer	86.1	0.0	50.2	-24.0	73.4	-15.8
COVIDFact	50.6	-35.6	74.1	0.0	76.1	-13.1
SCIFACT	70.5	-15.7	54.6	-19.6	89.2	0.0
Combined	83.0	-3.2	64.3	-9.8	87.8	-1.3
(a) Abstract-level evaluation. SCIFACT and HealthVer generalize fairly well to each other. COVIDFact generalizes well to SCIFACT, but not HealthVer.
Eval → Train ↓	HealthVer	COVIDFact	SCIFACT
F1	Δ	F1	Δ	F1	Δ
HealthVer	74.2	0.0	28.0	-12.6	39.7	-32.4
COVIDFact	14.6	-59.5	40.6	0.0	41.6	-30.6
SCIFACT	20.5	-53.7	33.9	-6.7	72.1	0.0
Combined	71.4	-2.8	39.8	-0.9	70.5	-1.6
(b) Sentence-level evaluation. None of the datasets generalize particularly well to each other. HealthVer generalizes better to SCIFACT than vice versa.
Table 12: Cross-dataset generalization performance. The rows and columns indicate the training and evaluation datasets, respectively. The  $\Delta$  values indicate the loss in performance from evaluating on a dataset different from the one the model was trained on. The "Combined" row indicates training on all datasets combined.