SlowGuess/NAACL_2022_OCR huggingface dataset:
In the fully-supervised setting, we compare against both baselines. For prediction on SCIFACT, we use publicly available model checkpoints as is. For training on HealthVer and COVIDFact, we use the code provided by the authors, starting from the available checkpoints trained on SCIFACT. Model hyperparameters (learning rate, batch size, epoch number, etc.) for all systems including MULTIVERS were tuned based solely on SCIFACT and not adjusted further. Additional details can be found in Appendix B.4.
Evaluation in the few-shot and zero-shot settings requires pretraining and finetuning as described in §5.1. Due to the expense of pretraining T5-3B, we do not perform these experiments for VERT5ERINI, and compare only against PARAGRAPHJOINT (which shows comparable performance in the fully-supervised setting). We pretrain PARAGRAPHJOINT on the data described in §4.2.
Since PARAGRAPHJOINT and VERT5ERINI differ from MULTIVERS along a number of important dimensions (e.g. model architecture, number of parameters, and base encoder), we conduct ablations to characterize the performance contributions of three key components of MULTIVERS.
Pretraining data We compare the results of three different pretraining strategies. For FEVERSCI, we
pretrain on all available data as described in  $\S 5.1$ . For FEVER, we pretrain on FEVER only. For NoPretrain, we perform no pretraining.
Base encoder We compare the performance achieved using LongFormer as the encoder for MULTIVERS, compared to the results when we swap in RoBERTa but keep other settings identical. We use Longformer-large and RoBERTa-large.
Modeling approach We compare three modeling approaches: (1) The Multitask approach is the method used by MULTIVERS as described in §3.1. (2) The Pipeline approach consists of two separate Longformer modules. The first selects rationales as described in §3.1, but with  $L_{\mathrm{label}}$  removed from Eq. 1, and the second module predicts a label given the text of the rationales selected by the first module as input. When pretraining on PUBMEDQA, we train on the rationales chosen by Sentence-BERT as described in §4.2. (3) The Multitask train / Pipeline inference (MT / PI) approach takes the model trained using the Multitask approach, and performs inference using the Pipeline approach. Specifically, MT / PI is trained to make label predictions based on full abstracts, but must make test-time label predictions based on predicted rationales only. By contrast, the Pipeline model makes label predictions based on gold and predicted rationales at train and test time, respectively.
We compare MULTIVERS performance relative to our baseline systems, and present ablation results.
Table 2 compares the performance of MULTIVERS against PARAGRAPHJOINT and VERT5ERINI. A few trends are apparent. First, MULTIVERS outperforms the baselines on all datasets, with relative improvements — averaged over the three datasets and two evaluation methods — of  $26\%$ ,  $14\%$ , and  $11\%$  in the zero-shot, few-shot, and fully-supervised settings respectively. We examine possible causes of this improvement in §6.2. Second, while all models score within roughly six points of each other on HealthVer and SCIFACT, variability is much greater on COVIDFact. We suspect that this is due to the automatically-generated nature of COVIDFact negations. Third, we observe that HealthVer appears to be the most challenging dataset of the three. Few-shot abstract-level F1 scores for COVIDFact and SCIFACT are generally
Setting	Model	HealthVer	COVIDFact	SCIFACT
Abstract	Sentence	Abstract	Sentence	Abstract	Sentence
P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Zero	PARAGRAPHOINT	72.3	14.4	24.0	22.9	2.7	4.9	51.3	37.9	43.6	31.5	16.0	21.3	52.9	32.4	40.2	36.4	14.9	21.1
MULTIVERS	60.6	20.5	30.7	25.0	4.6	7.8	48.8	45.7	47.2	32.7	18.5	23.6	49.0	44.6	46.7	39.0	21.6	27.8
Few	PARAGRAPHOINT	62.7	41.6	50.0	46.0	29.3	35.8	73.3	60.6	66.3	44.3	30.6	36.2	44.4	51.4	47.6	33.0	35.1	34.0
MULTIVERS	63.6	47.9	54.7	41.9	31.0	35.7	71.3	68.1	69.7	39.5	35.4	37.4	76.4	54.1	63.3	51.7	40.3	45.3
Full	VERT5ERINI	71.3	74.0	72.6	65.6	61.2	63.3	76.6	52.7	62.4	44.8	27.2	33.9	64.0	73.0	68.2	60.6	66.5	63.4
PARAGRAPHOINT	75.0	68.3	71.5	69.9	60.6	64.9	71.5	68.1	69.8	41.4	40.3	40.8	75.8	63.5	69.1	68.9	54.6	60.9
MULTIVERS	78.9	76.3	77.6	71.4	67.0	69.1	77.3	77.3	77.3	41.5	46.1	43.7	73.8	71.2	72.5	67.4	67.0	67.2
Table 2: Performance of MULTIVERS and baselines. In the fully-supervised setting, we compare to PARAGRAPHJOINT and VERT5ERINI, which exhibit comparable performance. In the zero and few-shot settings, we compare to PARAGRAPHJOINT only due to the high cost of pretraining VERT5ERINI. We report performance using abstract-level and sentence-level evaluation as defined in §2.1.
within 10 F1 of their fully-supervised values, while the gap is roughly 20 F1 for HealthVer. This may be due to the high complexity of HealthVer claims.
6.2 Ablations
The results of all ablations are shown in Table 3. We report abstract and sentence-level F1 scores in the main text; full results can be found in Table 9 in Appendix C.
In-domain pretraining substantially improves zero / few-shot performance In Table 3a, we compare the performance of models pretrained on FEVERSCI, FEVER, and No-Pretrain. In the zero-shot setting, removing scientific data during pretraining results in a relative performance decrease of  $65\%$ , averaged over the three datasets and two evaluation methods. The decrease is driven primarily by very low recall (see Table 9a).
In the few-shot setting, FEVER pretraining scores within  $4\%$  of FEVERSCI, while No-Pretrain results in a  $39\%$  decrease relative to FEVERSCI. This suggests that training on a handful of target examples is sufficient to recalibrate a model trained for a different domain, but not to learn the task from scratch. In the fully-supervised setting, FEVER pretraining is only slightly worse than FEVERSCI, while No-Pretrain lags by roughly  $9\%$ . Overall, the results indicate that pretraining always helps, and pretraining on weakly-labeled in-domain data helps especially when target data are scarce.
Longformer improves performance on datasets with long documents Table 3b compares the performance of MULTIVERS when Longformer and RoBERTa are used as the base encoder. Using Longformer consistently helps on SCIFACT, but
does not help on the other two datasets. This is unsurprising, since  $27\%$  of SCIFACT instances exceed the RoBERTa token limit, compared to less than  $15\%$  for the other two datasets (Table 1).
Multitask modeling improves zero / few-shot performance Results comparing our three different modeling approaches are shown in Table 3c. In the zero-shot setting, we find that Multitask performs best, with both MT / PI and Pipeline exhibiting performance drops greater than  $50\%$ . The Multitask approach of predicting rationales conditioned on the predicted label leads to improved recall (see Table 9c). Similarly, in the few-shot setting, both Pipeline and MT / PI perform roughly  $10\%$  worse than Multitask. Collectively, the results indicate that Multitask makes the best use of the available data when target annotations are limited.
We also find that MT / PI outperforms Pipeline in the zero-shot setting. This supports our intuition from §3.2 that, while training on weakly-supervised document-level labels improves zero-shot performance, training on weakly-supervised sentence-level rationales (as for Pipeline) leads to worse performance than not training on these rationales (as for MT / PI).
In the fully-supervised setting, Multitask performs best on SCIFACT, while Pipeline slightly outperforms Multitask on HealthVer and COVIDFact. MT / PI performs substantially worse than the other approaches on all datasets. We investigate these findings further in §7.1; our results indicate that Pipeline may, in effect, be trained to make predictions based on insufficient evidence.
Pretraining	HealthVer	COVIDFact	SCIFACT
Zero	FEVERSCI	30.7 / 7.8	47.2 / 23.6	46.7 / 27.8
FEVER	1.3 / 0.7	25.2 / 11.2	23.9 / 11.8
Few	FEVERSCI	54.7 / 35.7	69.7 / 37.4	63.3 / 45.3
FEVER	53.4 / 31.9	74.4 / 42.1	54.5 / 39.0
No-Pretrain	39.4 / 27.0	67.8 / 22.6	24.2 / 10.8
Full	FEVERSCI	77.6 / 69.1	77.3 / 43.7	72.5 / 67.2
FEVER	77.1 / 70.3	77.4 / 43.3	67.9 / 61.7
No-Pretrain	74.5 / 69.7	69.7 / 36.6	63.3 / 58.4
(a) Effect of pretraining data. In-domain pretraining is very effective in the zero- and few-shot settings. In the zero-shot setting, "No-Pretrain" metrics are not shown since this would correspond to no training at all.
Encoder	HealthVer	COVIDFact	SCIFACT
Zero	Longformer	30.7 / 7.8	47.2 / 23.6	46.7 / 27.8
RoBERTa	34.2 / 9.2	48.3 / 26.2	45.2 / 25.9
Few	Longformer	54.7 / 35.7	69.7 / 37.4	63.3 / 45.3
RoBERTa	51.2 / 36.9	72.1 / 41.0	50.5 / 34.0
Full	Longformer	77.6 / 69.1	77.3 / 43.7	72.5 / 67.2
RoBERTa	78.8 / 72.7	78.2 / 43.4	67.6 / 62.3
(b) Effect of base encoder. Longformer improves performance on SCIFACT, which has the largest fraction of instances exceeding the RoBERTa token limit.
Approach	HealthVer	COVIDFact	SCIFACT
Zero	Multitask	30.7 / 7.8	47.2 / 23.6	46.7 / 27.8
Pipe	3.2 / 0.9	19.0 / 10.5	22.5 / 12.8
MT / PI	4.5 / 1.8	26.7 / 13.5	28.3 / 17.7
Few	Multitask	54.7 / 35.7	69.7 / 37.4	63.3 / 45.3
Pipe	52.8 / 29.5	68.3 / 38.2	53.0 / 39.9
MT / PI	46.7 / 32.3	59.3 / 34.1	56.2 / 41.1
Full	Multitask	77.6 / 69.1	77.3 / 43.7	72.5 / 67.2
Pipe	78.4 / 69.2	77.6 / 47.7	70.9 / 66.2
MT / PI	70.6 / 64.3	73.3 / 44.0	60.3 / 57.0
(c) Effect of model architecture. The Multitask approach performs best in the zero- and few-shot settings. We examine the fully-supervised setting in detail in  $\S 7.1$ .
Table 3: Ablations examining the effects of pretraining data, base encoder, and modeling approach. Entries are formatted “{Abstract-level F1} / {Sentence-level F1}”.
7.1 Fully-supervised Pipeline performance
In §6.2, we found that the Pipeline approach (but not the MT / PI approach) performed on par with the Multitask approach in the fully-supervised setting. To understand this finding, we collected detailed annotations for 128 claim / evidence instances from the SCIFACT test set. For each instance, an annotator indicated whether the annotated rationales were "self-contained" — i.e. sufficient to justify the fact-checking label when taken
Approach	Self-contained	Context-dependent	%Δ
P	R	F1	P	R	F1
Multitask	86.1	82.9	84.5	90.3	60.9	72.7	-14.0%
Pipeline	92.4	89.0	90.7	82.4	60.9	70.0	-22.8%
MT / PI	91.8	54.9	68.7	100.0	13.0	23.1	-66.4%
Count	82	46
Table 4: Performance of the Multitask, Pipeline, and MT / PI modeling approaches on SCIFACT instances with rationales that are self-contained (can be interpreted in isolation) or context-dependent (must be interpreted in the context of the abstract). Evaluation is performed in the abstract-provided setting. We report abstract-level metrics; sentence-level results are similar. The  $\% \Delta$  indicates the drop in F1 score on context-dependent instances relative to self-contained instances. Multitask suffers the smallest performance loss, while MT / PI suffers the largest.
in isolation, or “context-dependent” — i.e. only sufficient when taken in the context of the abstract. Figure 1 and Table 8 provide examples; see Choi et al. (2021) for a detailed discussion of different forms of context-dependence. $^6$
Table 4 compares the performance of the three modeling approaches on instances with self-contained vs. context-dependent evidence. We find that all approaches have lower performance on context-dependent instances relative to self-contained instances, but the size of the performance drop varies widely. The Multitask approach performs  $14.0\%$  worse on context-dependent instances, while the Pipeline approach performs  $22.8\%$  worse. Most interestingly, MT / PI performs  $66.4\%$  worse, driven predominantly by low recall. The MT / PI model frequently (and correctly) predicts that context-dependent rationales are not sufficient to justify a SUPPORTS / REFUTES decision. These findings suggest that (1) the Multitask approach is, as expected, best at verifying claims with context-dependent evidence, and (2) the Pipeline approach has, in effect, over-fit to context-dependent rationales and learned to make predictions based on insufficient evidence.
7.2 Performance upper bound
To determine an "upper bound" on the achievable performance of scientific fact-checking models, we
Unlike Choi et al. (2021), we do not include the presence of acronyms as "context-dependent," since an acronym can be matched with its expansion based on surface-level textual features. See Appendix C.2 for further analysis of acronyms.
Abstract	Sentence
P	R	F1	P	R	F1
VERT5ERINI	90.7	74.3	81.7	79.6	62.2	69.8
PARAGRAPHJOINT	87.2	64.4	74.1	76.7	55.1	64.1
MULTIVERS	87.4	75.2	80.9	80.5	70.3	75.0
Human	94.8	84.1	89.1	67.4	67.4	67.4
Table 5: Performance on SCIFACT in the "abstract-provided" setting. Models exceed human agreement as measured by sentence-level F1, but not abstract-level.
assign 151 claim-evidence pairs from SCIFACT for independent annotation by two different annotators. We estimate human-level performance by treating the first annotator's results as "gold," and the second annotator's results as predictions. For comparison, we make predictions using MULTIVERS and our two baseline models, with candidate abstracts provided as input. The results are shown in Table 5. Existing systems already exceed human agreement for sentence-level evaluation, but not abstract-level, indicating that experts tend to agree on the overall relationship between claim and abstract, but may disagree about exactly which sentences contain the best evidence. This constitutes another reason not to rely solely on selected rationales when predicting a fact-checking label: the choice of rationales is itself somewhat subjective.
In addition, these results suggest that one key subtask of scientific claim verification — namely, predicting whether an evidence-containing abstract SUPPORTS or REFUTES a claim — may be nearly "solved" in the setting where (1) the claims are atomic and (2) roughly 1,000 in-domain labeled claims are available for training.
Background on scientific claim verification is covered in  $\S 2$ ; we discuss other relevant work here. Nye et al. (2020) have previously observed that document-level context is often required to properly interpret scientific findings.
DeYoung et al. (2020) use an "extract-then-label" pipeline for the original EVIDENCEINFERENCE task. Multitask label prediction and rationale selection was proposed by Pruthi et al. (2020) and applied to sentiment analysis and propaganda detection. As in this work, the authors condition on the predicted label when predicting rationales. Another alternative to supervised rationale selection is to treat rationales as latent variables (Lei et al., 2016; Paranjape et al., 2020).
Long-document encodings for fact verification have been explored by Stammbach (2021), who use Big Bird (Zaheer et al., 2020) for full-document evidence extraction from FEVER. Domain adaptation for scientific text has been studied in a number of works, including Gururangan et al. (2020); Beltagy et al. (2019); Lee et al. (2020); Gu et al. (2021). In those works, the primary focus is on language model pretraining. Here, we focus on training on the target task using out-of-domain and weakly-labeled data.
This work points to a number of promising future directions for scientific claim verification. These include applying the approach presented here to develop scientific claim verification models for new scientific sub-domains or other specialized fields given a handful of labeled examples, and extending the task definition to verify claims against longer contexts (e.g. full scientific papers) or larger corpora. Our task formulation also offers an opportunity to study the effects of rationale decontextualization (Choi et al., 2021), especially in cases where models may be making predictions based on insufficient evidence.
In presenting the MULTIVERS system, we addressed two challenges associated with scientific claim verification: incorporating relevant information beyond rationale boundaries by modeling full-document context, and facilitating zero / few-shot domain adaptation through weak supervision enabled by a multitask modeling approach. Our experiments show that MULTIVERS outperforms existing systems across several scientific claim verification datasets. We hope that the task, data, and modeling resources presented in this paper will encourage further work and progress towards the broader goals of identifying and addressing scientific mis- and disinformation.
10 Ethical considerations and broader impact
One long-term goal of research on scientific claim verification is to build systems that can automatically identify mis- and dis-information, which we believe would be socially beneficial given the current prevalence of mis- and dis-information online.
In the shorter term, this work presents two potential risks. First, automated systems for scientific fact-checking are not mature enough to inform real
world medical decisions. We will include a disclaimer with released software to this effect. Second, bad actors could potentially use this work to develop disinformation generators trained to "fool" automated fact-checkers. While this risk cannot be ruled out, we believe that the benefits of publishing this work and making our models available to the community to facilitate further research outweigh the risks that this work will be misused by malicious actors.
This research was supported by the ONR MURI N00014-18-1-2670, ONR N00014-18-1-2826, DARPA N66001-19-2-4031, NSF (IIS 1616112), NSF Convergence Accelerator Award #2132318, an Allen Distinguished Investigator Award, and the Sloan fellowship. We thank the Semantic Scholar team at AI2, UW-NLP, and the H2lab at UW for helpful comments and feedback. Thanks to Arkadiy Saakyan and Tuhin Chakrabarty for help with COVIDFact, to Mourad Sarrouti for help with HealthVer, and to Xiangci Li and Ronak Pradeep for help with PARAGRAPJOINT and VERT5ERINI, respectively.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A Pretrained Language Model for Scientific Text. In EMNLP.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The Long-Document Transformer. ArXiv, abs/2004.05150.
Daniel Fernando Campos, T. Nguyen, M. Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, L. Deng, and Bhaskar Mitra. 2016. MS MARCO: A Human Generated MAchine Reading CComprehension Dataset. NeurIPS.
Eunsol Choi, Jennimaria Palomaki, Matthew Lamm, Tom Kwiatkowski, Dipanjan Das, and Michael Collins. 2021. Decontextualization: Making Sentences Stand-Alone. TACL.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL.
Jay DeYoung, E. Lehman, B. Nye, I. Marshall, and Byron C. Wallace. 2020. Evidence Inference 2.0: More Data, Better Models. In Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing.
T. Diggelmann, Jordan L. Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. 2020. CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims. In Tackling Climate Change with ML workshop @ NeurIPS.
Yuxian Gu, Robert Tinn, Hao Cheng, Michael R. Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing. ACM Transactions on Computing for Healthcare (HEALTH).
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't Stop Pretraining: Adapt Language Models to Domains and Tasks. In ACL.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. 2019. PubMedQA: A Dataset for Biomedical Research Question Answering. In EMNLP.
Neema Kotonya and F. Toni. 2020. Explainable Automated Fact-Checking for Public Health Claims. In EMNLP.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics.
Nayeon Lee, Yejin Bang, Andrea Madotto, Madian Khabsa, and Pascale Fung. 2021. Towards Few-shot Fact-Checking via Perplexity. In NAACL.
Eric P. Lehman, Jay DeYoung, R. Barzilay, and Byron C. Wallace. 2019. Inferring Which Medical Treatments Work from Reports of Clinical Trials. In NAACL.
Tao Lei, R. Barzilay, and T. Jaakkola. 2016. Rationalizing Neural Predictions. In EMNLP.
Xiangci Li, G. Burns, and Nanyun Peng. 2021. A Paragraph-level Multi-task Learning Model for Scientific Fact-Verification. In Workshop on Scientific Document Understanding @ AAAI.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. ArXiv, abs/1907.11692.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Michael Kinney, and Daniel S. Weld. 2020. S2ORC: The Semantic Scholar Open Research Corpus. In ACL.
Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. 2020. Document Ranking with a Pretrained Sequence-to-Sequence Model. In EMNLP.
Benjamin E. Nye, Jay DeYoung, E. Lehman, A. Nenkova, I. Marshall, and Byron C. Wallace. 2020. Understanding Clinical Trial Reports: Extracting Medical Entities and Their Relations. ArXiv, abs/2010.03550.
Bhargavi Paranjape, Mandar Joshi, John Thickstun, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. An Information Bottleneck Approach for Controlling Conciseness in Rationale Extraction. In EMNLP.
Ronak Pradeep, Xueguang Ma, Rodrigo Nogueira, and Jimmy Lin. 2021. Scientific Claim Verification with VerT5erini. In Proceedings of the 12th International Workshop on Health Text Mining and Information Analysis @EACL.
Danish Pruthi, Bhuwan Dhingra, Graham Neubig, and Zachary Chase Lipton. 2020. Weakly- and Semi-supervised Evidence Extraction. In EMNLP Findings.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence Embeddings using Siamese BERT-Networks. In EMNLP.
S. Robertson and H. Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval.
Arkadiy Saakyan, Tuhin Chakrabarty, and Smaranda Muresan. 2021. COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic. In ACL.
Mourad Sarrouti, Asma Ben Abacha, Yassine Mrabet, and Dina Demner-Fushman. 2021. Evidence-based Fact-Checking of Health-related Claims. In EMNLP.
Dominik Stammbach. 2021. Evidence Selection as a Token-Level Prediction Task. In Proceedings of the Fourth Workshop on Fact Extraction and VERIFICATION (FEVER) @ EMNLP.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for Fact Extraction and VERIFICATION. In NAACL.
Ellen M. Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R. Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2020. TREC-COVID: Constructing a Pandemic Information Retrieval Test Collection. In SIGIR.
David Wadden and Kyle Lo. 2021. Overview and Insights from the SciVer Shared Task on Scientific Claim Verification. In Proceedings of the Second Workshop on Scholarly Document Processing @ NAACL.
David Wadden, Kyle Lo, Lucy Lu Wang, Shanchuan Lin, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or Fiction: Verifying Scientific Claims. In EMNLP.