SlowGuess/NAACL_2022_OCR huggingface dataset:
MULTIVERS: Improving scientific claim verification with weak supervision and full-document context
David Wadden† Kyle Lo‡ Lucy Lu Wang‡ Arman Cohan‡ Iz Beltagy‡ Hannaneh Hajishirzi†‡
† University of Washington, Seattle, WA, USA  
‡ Allen Institute for Artificial Intelligence, Seattle, WA, USA
dwadden,hannaneh}@cs.washington.edu
{kyle1,lucyw,armanc,beltagy}@allenai.org
The scientific claim verification task requires an NLP system to label scientific documents which SUPPORT or REFUTE an input claim, and to select evidentiary sentences (or rationales) justifying each predicted label. In this work, we present MULTIVERS, which predicts a fact-checking label and identifies rationales in a multitask fashion based on a shared encoding of the claim and full document context. This approach accomplishes two key modeling goals. First, it ensures that all relevant contextual information is incorporated into each labeling decision. Second, it enables the model to learn from instances annotated with a document-level fact-checking label, but lacking sentence-level rationales. This allows MULTIVERS to perform weakly-supervised domain adaptation by training on scientific documents labeled using high-precision heuristics. Our approach outperforms two competitive baselines on three scientific claim verification datasets, with particularly strong performance in zero / few-shot domain adaptation experiments. Our code and data are available at https://github.com/dwadden/multivers.
The proliferation of scientific mis- and disinformation on the web has motivated the release of a number of new datasets (Saakyan et al., 2021; Sarrouti et al., 2021; Wadden et al., 2020; Kotonya and Toni, 2020) and the development of modeling approaches (Pradeep et al., 2021; Li et al., 2021; Zhang et al., 2021) for the task of scientific claim verification. The goal of the task is to verify a given scientific claim by labeling scientific research abstracts which SUPPORT or REFUTE the claim, and to select evidentiary sentences (or rationales) reporting the findings which justify each label.
A common approach to this task is to first extract rationales from the larger document context, and then make label predictions conditioned on the
Claim:
Ibuprofen worsens COVID-19 symptoms
Evidence abstract:
Covid-19 and avoiding Ibuprofen.
a potential increased risk of COVID-19
infection was feared with ibuprofen use
At this time, there is no supporting evidence
to discourage the use of ibuprofen
Figure 1: A claim from the HealthVer dataset, refuted by a research abstract. The sentence in red is a rationale reporting a finding that REFUTES the claim. However, this finding cannot be interpreted properly without the context in blue, which specifies that the finding applies to Ibuprofen as a treatment for COVID. MULTIVERS incorporates the full context of the evidence-containing abstract when predicting fact-checking labels.
selected rationales. This "extract-then-label" approach has two important drawbacks, which we aim to address in this work. First, the rationales may lack information required to make a prediction when taken out-of-context; for instance, they may contain acronyms or unresolved coreferences, or lack qualifiers that specify the scope of a reported finding (Figure 1 provides an example). Second, the "extract-then-label" approach requires training data annotated with both sentence-level rationales and abstract-level labels. While sentence-level rationale annotations are costly and require trained experts, abstract-level labels can be created cheaply using high-precision heuristics, e.g., the titles of research papers sometimes make claims that are supported by their abstracts.
Motivated by these challenges, we introduce MULTIVERS (Multitask Verification for Science): Given a claim and evidence-containing scientific abstract, MULTIVERS creates a shared encoding of the entire claim / abstract context, using the Longformer encoder (Beltagy et al., 2020) to accommo
Findings of the Association for Computational Linguistics: NAACL 2022, pages 61 - 76
date long sequences. Then, it predicts an abstract-level fact-checking label and sentence-level rationales in a multitask fashion, enforcing consistency between the outputs of the two tasks during decoding. This modeling approach ensures that label predictions are made based on all available context, and enables training on instances derived via weak supervision for which abstract-level labels are available, but sentence-level rationales are not.
In experiments on three scientific claim verification datasets, we find that MULTIVERS outperforms two state-of-the-art baselines, one of which has more than  $10\mathrm{x}$  the parameters of our system. In addition, we show that training MULTIVERS on weakly-labeled in-domain data substantially improves performance in the zero / few-shot domain adaptation settings. The ability to achieve reasonable performance given limited labeled data is especially valuable in specialized domains, due to the high cost of collecting expert annotations.
1. We introduce MULTIVERS, a multitask system for full-context scientific claim verification. MULTIVERS improves fully-supervised fact-verification performance by an average of  $11\%$  on three datasets over two state-of-the-art baselines, with improvements of  $14\%$  and  $26\%$  in the few-shot and zero-shot settings.
2. We present weak supervision heuristics to assign fact-checking labels to two large scientific datasets, and show that training on these annotations more than doubles zero-shot domain adaptation performance.
3. Through ablations and analysis, we demonstrate that our multitask modeling approach achieves our goals of incorporating full-document context into label predictions, and facilitating zero / few-shot domain adaptation.
2.1 The scientific claim verification task
We use the definition of scientific claim verification from the SCIFACT task (Wadden et al., 2020), and provide a brief overview of the task here. Other works have cast scientific claim verification as a sentence-level natural language inference (NLI) task; in §4.1, we describe how we process these datasets to be compatible with the task as considered in this work.
Task definition Given a claim  $c$  and a collection of candidate abstracts which may contain
evidence relevant to  $c$ , the scientific claim verification task requires a system to predict a label  $y(c, a) \in \{\text{SUPPORTS}, \text{REFUTES}, \text{NEI}^1\}$ , which indicates the relationship between  $c$  and  $a$  for each candidate  $a$ . For all abstracts labeled SUPPORTS or REFUTES, the system must also identify rationales  $R(c, a) = \{r_1(c, a), \ldots, r_n(c, a)\}$ , where each  $r_i(c, a)$  is a sentence from  $a$  that either entails or contradicts the label  $y(c, a)$ . The rationales may not be self-contained, and may require additional context from elsewhere in the abstract to resolve coreferential expressions or acronyms, or to determine qualifiers specifying experimental context or study population. Examples of these situations are provided in Figure 1 and Appendix A.3.
Evaluation The SCIFACT task reports four evaluation metrics. We have found that two of these metrics are sufficient to convey the important findings for our experiments: (1) abstract-level label-only evaluation computes the model's F1 score in identifying abstracts that SUPPORT or REFUTE each claim. Predicting the correct label  $y(c, a)$  is sufficient; models do not need to provide rationales. (2) Sentence-level selection+label evaluation computes the point-wise product of the model's F1 score in identifying the rationales  $R(c, a)$ , with the model's abstract-level label  $y(c, a)$ ; this metric rewards precision in identifying exactly which sentences contain the evidence justifying the label. In this work, we refer to these two metrics as "abstract" and "sentence" evaluation respectively.
Retrieval settings For open scientific claim verification, the system must retrieve candidate abstracts from a corpus of documents. In the abstract-provided setting, candidate abstracts for each claim are given as input. We describe the retrieval settings for all datasets in §4.1.
Supervision settings We consider three supervision settings. In the zero-shot domain adaptation setting, models may not train on any in-domain fact-checking data, though they may train on general-domain fact-checking data and other available scientific datasets. In the few-shot domain adaptation setting, models may train on 45 claims from the target dataset. In the fully-supervised setting, models
$^{1}$ NEI stands for "Not Enough Info".
2This rationale definition is simplified slightly from the one presented in Wadden et al. (2020).
This convention is consistent with related tasks in rationalized NLP for biomedical literature, such as Lehman et al. (2019) and DeYoung et al. (2020).
may train on all claims from the target dataset.
While most existing work on scientific fact-checking has focused on the fully-supervised setting, some recent work has examined the zero-shot setting. Lee et al. (2021) use language model perplexity as a measure of claim veracity. Wright et al. (2022) generate claims based on citation sentences, and verify each generated claim against the abstracts mentioned in the claim's source citation. Given the high potential impact of fact verification systems for specialized domains, combined with the substantial cost of creating these datasets, we believe that the development of techniques for zero / few-shot domain adaptation represents an important area for continued research.
2.2 Scientific claim verification datasets
Several scientific claim verification datasets have been released in the past few years. COVIDFact (Saakyan et al., 2021) and HealthVer (Sarrouti et al., 2021) verify COVID-19 claims against scientific literature. PUBHEALTH (Kotonya and Toni, 2020) verifies public health claims against news and web sources. SCIFACT (Wadden et al., 2020) verifies claims made in citations in scientific papers. CLIMATE-FEVER (Diggelmann et al., 2020) verifies claims about climate change against Wikipedia. In this work, our focus is verifying claims against scientific literature. We therefore perform experiments on the COVIDFact, HealthVer, and SCIFACT datasets. Preprocessing details and summary statistics for these datasets are included in §4.1.
2.3 Models
Motivated in part by the SCIVER shared task (Wadden and Lo, 2021) and leaderboard, a number of models have been developed for SCIFACT (the focus of the shared task). The two strongest systems on the shared task were VERT5ERINI (Pradeep et al., 2021) and PARAGRAPHJOINT (Li et al., 2021), which we adopt as baselines. More recently, ARSJOINT (Zhang et al., 2021) achieved performance competitive with these two systems.
Given a claim  $c$  and candidate abstract  $a$ , these models make predictions in two steps. First, they predict rationales  $\widehat{R}(c,a) = \{\widehat{r}_1(c,a),\ldots,\widehat{r}_n(c,a)\}$  likely to contain evidence. Then, they make a label prediction  $\widehat{y}(c,f_R(\widehat{R}(c,a))$  based on the claim and predicted rationales, where  $f_R$  is a function which creates a representation of the predicted rationales.
While existing models share this general approach, they use different functions  $f_{R}$  to construct rationale representations. For VERT5ERINI, rationale selection and label prediction are performed by two separate T5-3B models, and  $f_{R}$  concatenates the text of the selected rationales. As a result, the label predictor may not have access to all context needed to make a correct label prediction. PARAGRAPHJOINT and ARSJOINT attempt to address this issue by encoding the claim and full abstract (truncating to 512 tokens), and using these representations as the basis for both rationale selection and label prediction. The function  $f_{R}$  consists of self-attention layers over the (globally-contextualized) token representations of the predicted rationales. Thus, PARAGRAPHJOINT and ARSJOINT can incorporate abstract-level context into label decisions. However, the mechanism by which this occurs is more complex than for our proposed system and requires rationale supervision for all training instances.
3 The MULTIVERS model
We propose the MULTIVERS model for full-context claim verification. In §3.1, we describe our modeling approach. Rather than predicting rationales  $\widehat{R}(c,a)$  followed by the overall fact-checking label  $\widehat{y}(c,f_R(\widehat{R}(c,a)))$ , we predict  $\widehat{y}(c,a)$  directly based on an encoding of the entire claim and abstract, and enforce consistency of  $\widehat{R}(c,a)$  with  $\widehat{y}(c,a)$  during decoding. A similar idea has been shown to be effective on sentiment analysis and propaganda detection with token-level rationales (Pruthi et al., 2020). In §3.2, we explain how our approach facilitates few-shot domain adaptation using weakly-labeled scientific documents.
3.1 Full-context claim verification
Long-document encoding Given a claim  $c$  and candidate abstract  $a$  consisting of title  $t$  and sentences  $s_1, \ldots, s_n$ , we concatenate the inputs separated by  $
$  tokens. The  $
$  token following each sentence  $s_i$  is notated as  $
_{i}$ :
$$
<   s > c <   / s > t <   / s > s _ {1} <   / s > _ {1} \dots s _ {n} <   / s > _ {n}
$$
The model input sometimes exceeds the 512-token limit common to transformer-based language models like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019); see Table 1 for details on how frequently this occurs. Therefore, we use the Longformer model (Beltagy et al., 2020) as our encoder.
4Recent progress can be found on the SciFact leaderboard.
We assign global attention to the  $< s >$  token, as well as all tokens in  $c$  and all  $< / s >$  tokens.
Multitask rationale selection and label prediction Given the full-context Longformer encoding, we predict whether sentence  $s_i$  is a rationale via a binary classification head, consisting of two feedforward layers followed by a two-way softmax, on top of the globally-contextualized token  $\langle /s\rangle_{i}$ .
Similarly, we predict the overall fact-checking label  $\widehat{y}(c, a)$  by adding a three-way classification head over the encoding of the  $
$  token. Since the  $
$  token is trained with global attention, the model makes predictions based on a representation of the entire claim and abstract.
During training, we compute the cross-entropy losses for the label and rationale predictions, and train to minimize the multitask loss:
$$
L = L _ {\text {l a b e l}} + \lambda_ {\text {r a t i o n a l e}} L _ {\text {r a t i o n a l e}} \tag {1}
$$
where  $\lambda_{\mathrm{rationale}}$  is tuned on the dev set.
At inference time, we first predict  $\widehat{y}(c, a)$  to be the label with the highest softmax score. If the predicted label is NEI, we predict no rationales. If the predicted label is either SUPPORTS or REFUTES, then we predict rationales as all sentences with an assigned softmax score of greater than 0.5. If no sentences have a rationale softmax over 0.5, then we predict the highest-scoring sentence as the sole rationale. In §6.2, we show that this ability to condition the rationale predictions on the label prediction (as opposed to conditioning the label on the predicted rationales) leads to substantial improvement in the zero-shot domain adaptation setting.
Candidate abstract retrieval For datasets that require retrieval of candidate abstracts, we rely on the VERT5ERINI (Pradeep et al., 2021) retrieval system, which achieved state-of-the-art performance on the SCIVER shared task (SCIVER used the SCIFACT dataset for evaluation). This model first retrieves abstracts using BM25 (Robertson and Zaragoza, 2009), then refines the predictions using a neural re-ranker based on Nogueira et al. (2020), which is trained on the MS MARCO passage dataset (Campos et al., 2016).
3.2 Training for domain adaptation
Three types of data are available to train scientific claim verification systems. (1) In-domain fact-checking annotations are the "gold standard", but they are expensive to create and require expert annotators. (2) General-domain fact-checking
datasets like FEVER (Thorne et al., 2018) are abundantly available, but generalize poorly to scientific claims (see §6.1). (3) Scientific documents – either unlabeled or labeled for different tasks – are abundant, and high precision heuristics (described in §4.2) can be used to generate document-level fact-checking labels  $y(c, a)$  for these data.
We train MULTIVERS as follows: we first pretrain on a combination of general-domain fact-checking annotations, combined with weakly-labeled in-domain data. Then, we finetune on the target scientific fact-checking dataset. The multitask architecture of MULTIVERS is well-suited to this strategy, since the model can be trained on data with or without rationale annotations. When no rationales are available, we set  $\lambda_{\mathrm{rationale}} = 0$  in the loss function and train as usual. By contrast, training an "extract-then-label" model on weakly-supervised data requires creating rationale annotations  $R(c,a)$ , which is quite noisy (see §4.2).
4.1 Scientific claim verification datasets
We experiment with three scientific claim verification datasets. Table 1 provides a summary of important dataset characteristics. Preprocessing steps and additional statistics can be found in Appendix A. HealthVer and COVIDFact were originally released in an NLI format, pairing claims with (out-of-context) evidentiary sentences. We convert to our task format by identifying the abstracts in the CORD-19 corpus (Wang et al., 2020) containing these sentences.
We use the following terminology: an atomic claim makes an assertion about a single property of a single entity, while a complex claim may make assertions about multiple properties or entities.
SCIFACT Claims in SCIFACT (Wadden et al., 2020) were created by re-writing citation sentences occurring in biomedical literature into atomic claims, which were verified against the abstracts of the cited documents. REFUTED claims were created by manually negating the original claims. Abstracts that were cited but which annotators judged not to contain evidence were labeled NEI. SCIFACT requires retrieval of candidate abstracts.
HealthVer (Sarrouti et al., 2021) consists of COVID-related claims obtained by extracting snippets from articles retrieved to answer questions
<sup>5</sup>We use "pretraining" as shorthand for "training on the target task with out-of-domain and/or weakly-supervised labels."
Dataset	Domain	Claim source	Open	Has NEI	Claim complexity	Negation method	Train claims	Eval claims	>512 tokens
HealthVer	COVID	TREC-COVID	×	✓	Complex	Natural	1,622	230	14.9%
COVIDFact	COVID	Reddit	×	×	Complex	Automatic	903	313	12.4%
SCIFACT	Biomed	Citations	✓	✓	Atomic	Human	1,109	300	27.4%
FEVER	Wiki	Wikipedia	-	✓	Atomic	Human	130,644	-	33.2%
PUBMEDQA	Biomed	Paper titles	-	✓	Complex	Automatic	58,370	-	12.1%
EVIDENCEINFERENCE	Biomed	ICO prompts	-	✓	Atomic	Automatic	7,395	-	42.7%
Table 1: Summary of datasets used in experiments. The top group of datasets are scientific claim verification datasets, and the bottom group are for pretraining. Datasets with a  $\checkmark$  for "Open" require that candidate abstracts be retrieved from a corpus; those with a  $\times$  provide candidate abstracts as input. Datasets with a  $\checkmark$  for "Has NEI" require three-way (SUPPORTS / REFUTES / NEI) label prediction, while those with an  $\times$  are (SUPPORTS / REFUTES) only. The  $>512$  tokens' column indicates the percentage of claim / abstract contexts that exceed 512 tokens.
from TREC-COVID (Voorhees et al., 2020), verified against abstracts from the CORD-19 corpus (Wang et al., 2020). Claims in HealthVer may be complex. REFUTED claims occur naturally in the article snippets. HealthVer provides candidate abstracts for each claim, but some of these candidates do not contain sufficient information to justify a SUPPORTS / REFUTES verdict and are labeled NEI.
COVIDFact (Saakyan et al., 2021) collects claims about COVID-19 scraped from a COVID-19 subreddit, and verifies them against linked scientific papers, as well as documents retrieved via Google search. Claims in COVIDFact may be complex, and candidate abstracts for each claim are provided. All candidates either SUPPORT or REFUTE the claim. Claim negations were created automatically by replacing salient words in the original claims, and as a result the labels  $y(c, a)$  are somewhat noisy (see Appendix A).
4.2 Pretraining datasets
We briefly describe our pretraining datasets and the weak supervision heuristics used to construct them. Detailed descriptions of these heuristics can be found in Appendix A.1.
FEVER (Thorne et al., 2018) consists of claims created by re-writing Wikipedia sentences into atomic claims, verified against Wikipedia articles.
EVIDENCEINFERENCE (Lehman et al., 2019; DeYoung et al., 2020) was released to facilitate understanding of clinical trial reports, which examine the effect of an intervention on an outcome, relative to a comparator ("ICO" elements). The dataset contains ICO prompts paired with (1) labels indicating whether the outcome increased or decreased due to the intervention, and (2) rationales justifying each label. We use rule-based heuristics to convert
these prompts into claims – for instance “[intervention] increases [outcome] relative to [comparator]”.
PUBMEDQA (Jin et al., 2019) was released to facilitate question-answering over biomedical research abstracts. We use the PQA-A subset, which is a large collection of abstracts with "claim-like" titles – for instance, "Vitamin B6 supplementation increases immune responses in critically ill patients." We treat the paper titles as claims and the matching abstracts as the evidence sources.
To train models requiring rationale supervision, we create weakly-supervised rationales by selecting the sentences with highest similarity to the claim as measured by cosine similarity of SentenceBERT embeddings (Reimers and Gurevych, 2019). These annotations are not used to train MULTIVERS. To estimate the precision of our rationale labeling heuristic, we predict rationales in the same fashion for our supervised datasets and compute the Precision@1 with which this method identifies gold rationales. The scores are relatively low: 49.4, 48.8, and 43.4 for SCIFACT, COVIDFact, and HealthVer respectively.
We describe our model training procedure, the systems against we compare MULTIVERS, and our ablation experiments.
5.1 Model training
Our complete training procedure consists of pretraining on the three datasets from §4.2, followed by finetuning on one of the target datasets from §4.1. We conduct experiments with three different levels of supervision. For zero-shot experiments, we perform pretraining only. For few-shot experiments, we pretrain followed by finetuning on 45
target examples. For fully-supervised experiments, we pretrain and then train on all target data.
Following Li et al. (2021), we found that negative sampling was important to achieve good precision on SCIFACT, which requires document retrieval. We train with 20 negative samples per claim and retrieve 10 abstracts per claim at inference time. Appendix C.3 shows results without negative sampling. For the other datasets, no negative sampling was used. Additional details including batch sizes, learning rates, number of epochs, etc. can be found in Appendix B.
During model development, we experimented with training on all three target datasets combined before predicting on each one, but found that this did not improve performance; see Appendix C.4.
5.2 Baseline systems
We use PARAGRAPHJOINT and VERT5ERINI as baselines. VERT5ERINI is the largest model, with 5.6B parameters. MULTIVERS and PARAGRAPHJOINT are comparably-sized, with 440M and 360M parameters, respectively.