Multiple Evidence Combination for Fact-Checking of Health-Related Information
Year: 2023

Abstract:
Fact-checking of health-related claims has become necessary in this digital age, where any information posted online is easily available to everyone. The most effective way to verify such claims is by using evidences obtained from reliable sources of medical knowledge, such as PubMed. Recent advances in the field of NLP have helped automate such fact-checking tasks. In this work, we propose a domainspecific BERT-based model using a transfer learning approach for the task of predicting the veracity of claim-evidence pairs for the verification of health-related facts. We also improvise on a method to combine multiple evidences retrieved for a single claim, taking into consideration conflicting evidences as well. We also show how our model can be exploited when labelled data is available and how backtranslation can be used to augment data when there is data scarcity.

text: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 237–247 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: In today’s age of easy access to the internet, information exchange among people has increased rapidly, which has also resulted in the spread of misinformation (Vosoughi et al., 2018) within the society. Misinformation has been found to spread faster than real news, and the rise of social media popularity has aided the spread of misinformation (Vosoughi et al., 2018). Research on health misinformation is still an ongoing area, as it is different from political misinformation on the basis of the complexity level of fact-checking (Deka et al., 2022b). Manual fact-checking of health information requires domain-specific experts, which increases both time taken and cost incurred. Automated fact-checking of health information found online has been aided by the release of datasets such as SCIFACT (Wadden et al., 2020), HEALTHVER (Sarrouti et al., 2021), COVIDFACT (Saakyan et al., 2021). Fact-checking of
health information comprises of retrieving evidences from reliable resources which either supports or refutes the key claim (Zeng et al., 2021; Guo et al., 2022). Recent works have focused on building end-to-end fact-checking models evaluating on the aforementioned datasets (Pradeep et al., 2020; Zhang et al., 2021; Li et al., 2021; Wadden et al., 2022). However, they do not take into account conflicting evidences retrieved for a single claim. Any claim can have more than one evidence, and these evidences can be conflicting in real-world scenarios wherein one evidence would be supporting the claim and another evidence may be refuting the claim.
In this work, we have focused on the subtask of classifying a claim-evidence pair as either supporting, refuting, or neutral as shown in Figure 1. We assume in this work that evidences for claims are already retrieved. We proposed a domain specific BERT-based model using a transfer learning approach where the model is trained over textual entailment data which can then be applied directly over fact-checking data. We have also used the Dempster-Shafer theory (Dempster et al., 2008; Shafer, 1976) of evidence combination for mitigating the conflicting evidences issue to provide an end result. We then extend our work by showing
237
how data augmentation techniques can help in a more robust training for smaller datasets with the help of neural machine translation language models. We also analyse the performance of our model when it is trained over other similar datasets. We further share our trained model publicly for further research1.

heading: 2 Related work
text: In this section, we will discuss the research work that has been done for fact-checking scientific claims using evidences retrieved from existing medical article repositories. With the release of the SCIFACT dataset, various transformer-based methods of predicting the veracity labels using evidences for scientific claims have been proposed and evaluated using the dataset. (Wadden et al., 2020) established a pipeline model using a RoBERTa-large (Liu et al., 2019) model to retrieve evidences from PubMed abstracts. The retrieved evidence sentences are then passed along with the claims to predict whether the evidences SUPPORT or REFUTE the claims using a RoBERTa-large model fine-tuned over the training set of SCIFACT.
VerT5erini (Pradeep et al., 2020) uses a T5 (Raffel et al., 2020) model-based pipeline for their work. For the evidence sentence selection task for the claims, as well as for label prediction from PubMed abstracts, they used two different T5 models. For the sentence selection task, the T5 model used is fine-tuned over the MS-MARCO (Bajaj et al., 2016) dataset and then further trained on SCIFACT. For the label prediction task, the T5 model is trained on the SCIFACT dataset.
PARAGRAPH-JOINT (Li et al., 2021) uses a RoBERTa-large model similar to (Wadden et al., 2020) for both the evidence sentence selection as well as the label prediction task which is fine-tuned over SCIFACT. However, the training approach is different, as (Li et al., 2021) uses a multitask learning approach for model training. Both the tasks of sentence selection and label prediction are done using a joint cross-entropy loss as the training objective. For the label prediction task, the authors have also used two different approaches which includes a simple sentence-level attention and KGAT which is a Kernel Graph Attention Network (Wang et al., 2019a).
Similarly, ARSJOINT (Zhang et al., 2021) also
1https://huggingface.co/pritamdeka/ PubMedBERT-MNLI-MedNLI
uses a joint approach where their proposed method jointly learns the three tasks of abstract retrieval, sentence selection, and label prediction. Similar to (Wadden et al., 2020), they have also used RoBERTa-large for their work together with BioBERT-large (Lee et al., 2020).
All the above works focus on the three tasks of abstract retrieval, evidence sentence selection and label prediction as a pipeline approach. However, there is a difference in the sentence selection task as well as the label prediction. VerT5erini selects sentences independently, whereas PARAGRAPHJOINT and ARSJOINT use the abstracts to select the sentences. The label prediction also differs as both PARAGRAPH-JOINT and ARSJOINT use a joint approach unlike VerT5erini. The models used in the tasks also differ as PARAGRAPH-JOINT and ARSJOINT use BERT-based models whereas VerT5erini uses a much larger T5 model having superior performance. However, the current stateof-the-art method, MULTIVERS (Wadden et al., 2022) differs from these works in the approach and the transformer model used. MULTIVERS uses a Longformer (Beltagy et al., 2020) architecture to encode both claims and abstracts together so that there is a minimum loss of information. The authors have used a weak supervision approach, in which the Longformer model is trained on available scientific data before fine-tuning on SCIFACT. However, the overall pipeline training method is a multi-task approach similar to (Li et al., 2021). It outperforms the other approaches in the label prediction task of SCIFACT.
Contrary to the above works, our approach is different in the way that for a given pair of claim and evidence, our model can predict the labels in a zero-shot approach, surpassing the state-of-theart results without the need for any supervision. The above mentioned works have the end goal of predicting the labels of the claim-evidence pairs. However, to have a final prediction for the claims whether it is a “True” claim or “False” claim, we need to have a combined judgement of all the evidence sentences for that claim which is not addressed by the above works. We have extended our approach to include the final prediction for the claims taking into consideration conflicting evidences as well.

heading: 3 Task Formulation
text: In this section, we will first discuss the problem statement and then proceed with the formulation of the tasks. The problem statement is “Given a claim and a number of evidence sentences, determine whether the claim is True or False or Neutral”. We can formulate the problem as two tasks:
• Classification of claim-evidence pair Given a claim c and an evidence sentence s for that claim, classify the claim-evidence sentence pairs as supporting, refuting or neutral.
[c, s] classify−−−−→ (support, refute, neutral)
• Prediction of the claims Given a list of supporting or refuting evidence sentences S where S = [s1, s2 . . . sn] for a claim c, the task is to predict whether the claim is True or False or Neutral by combining all the evidences.
[c, S] predict−−−→ c(true, false, neutral)

heading: 4 Methodology
text: In this section we will describe in detail the proposed methods we have adopted for the formulated tasks.

heading: 4.1 Classification of claim-evidence pair
text: Previous studies have focused on using factchecking datasets such as FEVER (Thorne et al., 2018) for training models for the task of factchecking. However, we have modelled the classification task as a natural language inference (NLI) problem, since (Pradeep et al., 2020) found in their study that models learn better from NLI data than datasets such as FEVER. Textual entailment or NLI is defined as the task of determining if, given a “premise”, a “hypothesis” is true (entailment) or false (contradiction) or not determined (neutral) (Williams et al., 2017). Fact-checking has similarities with the NLI task, in which premises can be modelled as evidences and the hypothesis as claims (Thorne et al., 2018). The idea is to train domain specific BERT (Devlin et al., 2018) model using NLI data to see if the model can learn knowledge that can be transferred to fact-checking task in biomedical domain. In order to achieve this, we have trained PubMedBERT2 (Gu et al., 2021)
2https://huggingface.co/microsoft/ BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext
which is a domain specific BERT model on the multi-NLI (MNLI) dataset (Williams et al., 2017) by minimizing the cross-entropy loss. We also experimented with other models such as BioBERT (Lee et al., 2020) and SciBERT (Beltagy et al., 2019), however, we achieved the best performance with the PubMedBERT model which is why we chose this model.
In order to fine-tune PubMedBERT on the MNLI data, the sentence pair of hypothesis-premise is used as the initial input sequence. Once the model is trained over the MNLI dataset, we can directly transfer the model to the claim-evidence sentence pairs for the fact-checking task. We then trained it on the MedNLI (Romanov and Shivade, 2018) dataset, which is a domain-specific NLI dataset, as previous research (Phang et al., 2018; Wang et al., 2018) has shown that further training over a similar domain-specific dataset increases model performance. The learned model can then make predictions whether an evidence supports or refutes a claim or if it is undetermined. The model can also be adapted in a supervised way when fact-checking datasets are available. We have done extensive experiments to show how the model can also be adapted for unseen data.

heading: 4.2 Prediction of the claims
text: The classifier can assign a claim-evidence sentence pair as support or refute. In order to further predict the claims as true or false, we need to combine all the evidences for each of the claims. In more complex scenarios, when some evidences support a claim whereas others refute the same claim and yet others may be undetermined, this task is not trivial. To resolve such conflicting cases, we have used an improvised Dempster-Shafer (D-S) theory of evidence combination. Research has mainly focused on using the D-S theory for multi-sensor domain where evidences from multiple sources are combined to achieve a final decision (Xiao, 2019; Khan and Anwar, 2019; Smets, 2000; Jiang et al., 2016). This is similar to our work and we have used the D-S theory for combining multiple conflicting evidences for claims in order to achieve a final decision for the claims. The D-S theory is mathematically defined as follows (Dempster et al., 2008): Definition 1. The set of all the possible sets of the hypotheses or class categories is known as the frame of discernment (FOD). A frame of discernment consisting of N elements where each element
ai is mutually exclusive to each other can be defined as: θ = {a1, a2, a3 . . . , aN} (1) Definition 2. If A is a subset of P (θ) where P (θ) = 2θ, the basic probability assignment (BPA) or mass function, m(A) is a function that maps A → [0, 1] and satisfies the following conditions:
m(ϕ) = 0, ∑
A⊆θ m(A) = 1 (2)
Definition 3. If m1(Ai) and m2(Aj) are the BPA of two bodies of evidence (BOE), then according to D-S combination rule, they can be combined as follows:
m(A) = 1 1−K ∑
Ai∩Aj=A m1(Ai)m2(Aj), A ̸= 0 (3)
where K is a normalization factor defined as follows:
K = ∑
Ai∩Aj=ϕ m1(Ai)m2(Aj) (4)
Definition 4. The combination formula can be extended for n terms as well which is defined as:
m(A) = 1 1−K ∑
Ai1 ...∩Ain=A m1(Ai1) . . .mn(Ain),
A ̸= 0 (5)
and K is defined as follows:
K = ∑
Ai1 ...∩Ain=ϕ m1(Ai1) . . .mn(Ain) (6)

heading: 4.3 Illustrative example
text: In order to understand the working of the D-S theory, let us take a few examples. According to our work, let us take three classes for the FOD, θ = {a, b, c} where a, b, c are “support”, “refute” and “neutral” respectively. Example 1. Let us take two conflicting evidences with respective probabilities for a, b and c.
E1 : m1(a) = 0.062 m1(b) = 0.937 m1(c) = 0.001 E2 : m2(a) = 0.952 m2(b) = 0.048 m2(c) = 0
We can see that for both E1 and E2, equation 2 is fulfilled. According to equation 4, we get
K = m1(a)m2(b)m2(c) +m1(b)m2(a)m2(c)
+m1(c)m2(a)m2(b)
Putting the respective values, we get K = 0.896. Using equation 3, we get the following
m(a) = m1(a)m2(a) (1−K) , m(b) = m1(b)m2(b) (1−K) and
m(c) = m1(c)m2(c)
(1−K)
After calculation, we get m(a) = 0.436, m(b) = 0.563 and m(c) = 0. We can see that m(b) has the highest probability value using the D-S combination theorem.
In certain situations, the D-S theorem fails. Let us look at one such example. Example 2. Let us take four conflicting evidences with respective probabilities for a, b and c.
E1 : m1(a) = 0.889 m1(b) = 0.106 m1(c) = 0.005 E2 : m2(a) = 0.0 m2(b) = 0.999 m2(c) = 0.0 E3 : m3(a) = 1.0 m3(b) = 0.0 m3(c) = 0.0 E4 : m4(a) = 0.481 m4(b) = 0.515 m4(c) = 0.004
We can see that for both E1 and E2, equation 2 is fulfilled. However, here, we find that K = 1 which means that the denominator is 1−K = 0. In such situations, the D-S combination rule will fail as division by zero is mathematically undefined. Definition 5. In order to overcome such situations, we adapted the base belief function from (Wang et al., 2019b) which is defined as follows: Let δ be a set of N possible values that are mutually exclusive. The power set of δ is 2δ, in which the number of elements is 2N . According to (Wang et al., 2019b), the base belief function mbase is then defined as:
mbase(Ai) = 1
2N − 1 (7)
where Ai is the subset of δ except for the empty set ϕ. The modified BPA then becomes
m′(Ai) = m1(Ai) +mbase(Ai)
2 (8)
where m1(Ai) is the original BPA. This modified BPA allows us to mitigate situations when BPA values are 0. However, this leads to the violation of the condition ∑ A⊆θ m(A) = 1. In order to preserve the condition, we normalize the value of m′(Ai) and therefore the final BPA is:
m′norm(Ai) = m′(Ai)∑ m′(Ai)
(9)
Example 3. Using the modifications, from Example 2, the modified BPAs are as follows
E1 : m1(a) = 0.723 m1(b) = 0.174 m1(c) = 0.104 E2 : m2(a) = 0.100 m2(b) = 0.801 m2(c) = 0.100 E3 : m3(a) = 0.801 m3(b) = 0.100 m3(c) = 0.100 E4 : m4(a) = 0.437 m4(b) = 0.461 m4(c) = 0.102
We can see that for both E1 and E2, equation 2 is fulfilled. From equation 6, we can calculate K = 0.0318 and 1−K = 0.968. After that we can use equation 5 and get the values of m(a) = 0.795, m(b) = 0.201 and m(c) = 0.0033. Using the
modified BPAs have helped overcome situations where the D-S combination rules fail.
From the illustrative examples, we have shown how the modified D-S method can be used for the task of combining evidences for a claim taking into account conflicting evidences as well.

heading: 5 Experimental Details
text: In this section, we will describe the experimental details for our tasks.

heading: 5.1 Dataset used
text: For evaluation purposes, we have used the SCIFACT dataset. The SCIFACT dataset has a train file, a dev file and a test file. However, the test file is part of a shared task and as such the labels are not available. This is why we are evaluating directly on the dev file. However, some claims in the dev set do not have the evidence sentences and as such we cannot evaluate on those claims which is why we have dropped those claims.

heading: 5.2 Classification of claim-evidence pair
text: For this task, we have experimented on different scenarios by doing an evaluation study over different classification settings. First we experimented directly on the dev set where we use our PubMedBERT fine-tuned model directly on the dev set examples by passing the claim-evidence pair and predicting the labels. The results are shown in Table 1 showing improvements over other models where P, R and F-1 are the precision, recall and f score respectively.
For the second experiment, we fine-tuned the MNLI fine-tuned model over MedNLI to see if there is any performance difference. Experiments using this model yielded a very good result which shows that in order to achieve an increased performance while fine-tuning over a smaller dataset, it is better to first fine-tune over a larger dataset and then use that model to further fine-tune over the smaller dataset. To confirm the results, we also compared the performance of a few more models from Table 1 by further fine-tuning these models
over MedNLI. We can see from Table 2 that there is a performance increase in all models. This is in line with the findings by (Phang et al., 2018; Wang et al., 2018; Clark et al., 2019; Sap et al., 2019).
We also experimented in a zero-shot setting for the SCIFACT pipeline where we first retrieve relevant PubMed abstracts for the claims in the dev set using the corpus provided in the dataset (Deka et al., 2022b). After that, the top n evidence sentences are extracted from the abstracts (Deka et al., 2022a) and then we use the claim-evidence pairs to predict whether the evidence supports or refutes the claim. We compared our model with stateof-the-art zero-shot as well as few-shot baselines evaluated on the SCIFACT dataset. However, it should be noted that the baselines have different trade-off points in calculating the results due to our method being different from theirs. The results for the experiment are shown below in Table 3 where top n sentences are the evidence sentences from the relevant abstracts For each setting, we retrieve the top 2, 3, 5 and 10 evidence sentences and then the label for claim-evidence pair is predicted.
The baselines use a supervised approach where they use the train set of the SCIFACT dataset. In our method, however, we are using a transfer learning approach where we directly use our method over the dev set without using the train set. From Table 3, we can see that we have outperformed the baseline models in the zero-shot setting. We can also see that our best-performing model setting outperforms even the few-shot baselines as well as the fully fine-tuned VERISCI (Wadden et al., 2020) baseline.

heading: 5.3 Prediction of the claims
text: In order to use the D-S theory for our work in resolving conflicting evidences, we first need to calculate the probabilities of the classes. As we have approached our task as an NLI problem, we have three different classes: SUPPORT, REFUTE and NEUTRAL. For each claim-evidence pair, our
classifier calculates the probabilities of the three classes. We are using these probabilities as the BPAs from Equation 2. Once we have the BPAs, we then use equation 9 to calculate the final modified BPAs to mitigate the denominator error. Once we have calculated the modified BPAs, we use equations 5 and 6 to combine the BPAs according to the D-S combination rules.
The SCIFACT dataset does not have labels that can be used for the evaluation of the combination method. In order to infer these labels for the evaluation, we give the final class label for a claim as either “Fake”, “Truth” or “Neutral”. This label is based on the gold standard label of the evidences. We have seen that in the SCIFACT dataset, all evidences for a claim can either be “SUPPORT” or “REFUTE”. Based on this, we label claims as “True” which has evidences labelled as “SUPPORT” and “False” for claims that have evidence labels as “REFUTE”. Some of the evidences do not enough information to either “SUPPORT” or “REFUTE” claims. These are labelled as “Neutral”. This will be our gold standard and the results from the D-S combination theory will be evaluated against this gold standard. As evaluation metrics, we have used macro precision, recall and f-1 score. We experimented in two different scenarios. Initially, we experimented directly on the dev set using our model from Table 2. We got the results as follows: Precision = 0.898, Recall = 0.893, F-1 score = 0.894.
For the next experiment, we have used the whole pipeline process where we first retrieve top n abstracts and then from these abstracts we retrieve the top n evidences. Once we have the evidence sentences, we then use the classifier to classify them accordingly. The results are shown in Table 4.

heading: 6 Supervised approach using augmented data
text: In situations where we have labelled data, our model can be used to train over such data in a supervised way which means that the knowledge from our model can be transferred over such data. However, a problem with the available datasets such as SCIFACT, is the fact that it has very less labelled data for training which may not lead to improved performance of the model. To improve it, there should be more data for training, and data augmentation is one way of increasing the number of training examples (Shorten et al., 2021; Feng et al., 2021). There are various ways of augmenting data such as rule-based, interpolation-based and model-based (Shi et al., 2022). In rule-based methods, words and phrases are manipulated in order to generate augmented text. But a problem with such methods is that changing words or phrases may lead to change in the meaning of the sentences (Niu and Bansal, 2018). In the context of biomedical text, if the meaning of the sentence changes then the sampled augmented data may lead to negative performance in model training. By performing interpolation operations directly on the source text (Chawla et al., 2002; He et al., 2008) or latent space representations (Chen et al., 2020), interpolation-
based approaches produce new instances. However, such methods can be error-prone due to noisy generated data (Chawla et al., 2002). Model-based methods use language models such as BERT to generate new training examples. One popular way of using these models to generate new training examples is back translation (Edunov et al., 2018). Recent research works have explored these language models for data augmentation via back translation (Melton et al., 2022).
For our work, we explored the following research questions:
RQ 1. Can we use back-translation method for data augmentation on domain specific fact checking task without loss of context?
RQ 2. How well does the model fare without using augmented examples vs the model which uses augmented examples?
In order to answer the research questions, we explored two different ways: using Google Translate and transformer-based language models. Using Google Translate for back translation has been studied in previous research (Pappas et al., 2022). We have used Google Translate to convert the claimevidence pairs to different languages such as German, French, Russian, Chinese and Spanish. Each language has a different language structure and since biomedical text is different than general text, a comparison of all the different languages would show which languages can be better suited for such tasks in the medical domain. We have used the deep translator python API 3 for the Google Translate method.
Transformer-based language models have been proven to be very good in neural machine translation tasks (Przystupa and Abdul-Mageed, 2019; Uhrig et al., 2021). For the study, we have used the OpusMT (Tiedemann and Thottingal, 2020) models which are pretrained transformer models for the neural machine translation task based on the Marian MT framework (Junczys-Dowmunt et al., 2018). We have used the models from the HuggingFace repository for the OpusMT 4 models.
For the experiment using the Google translator API, we translate all the claims as a batch to different languages and then back to English and the same approach is taken for the evidences as well. We then merge the synthetic data with the original
3https://deep-translator.readthedocs.io/en/ latest/
4https://huggingface.co/Helsinki-NLP
data by removing any duplicates. However, for NMT models, all claims and evidence are backtranslated one at a time using the HuggingFace pipeline5. Once we get the back-translated examples, we then merge them with the original data. For evaluation, we use our model from Table 3 with the best results and train it over the augmented data. The results are shown below in Table 5.
As seen from the table above the model without fine-tuning on train set performs poor which is expected. Training the model with the train set but without augmentation results in slight improvement on the results. However, we can see there is a significant improvement in the results once we augment the train file using the back translation approaches. Out of the two different approaches that we have experimented with, the transformer-based NMT models perform better than Google translate API. However, these models are also time consuming while performing the back-translation task unlike the Google translate approach. The results show that data augmentation using back translation gives us better results for such domain-specific fact checking tasks which answer both RQ1 and RQ2.

heading: 7 Transferring over other datasets
text: In order to know how well our model generalizes over other similar data, we experimented with two similar fact-checking datasets on biomedical data, HEALTHVER (Sarrouti et al., 2021) and COVIDFACT (Saakyan et al., 2021). Both datasets focus on Covid-19 data, however, the way claims and evidences are collected in both these datasets differ. HEALTHVER claims are collected from CORD-19 (Wang et al., 2020) corpus article snip-
5https://huggingface.co/docs/transformers/ main_classes/pipelines
pets which were retrieved to answer questions for TREC-COVID (Voorhees et al., 2021). The claims in HEALTHVER are complex and evidences are provided for each claim. The dataset has three labels, SUPPORT, REFUTE and NEUTRAL based on the evidences which are collected from the article snippets itself. COVIDFACT, on the other hand, has claims collected from Covid-19 subreddit and evidences are collected from linked scientific papers and documents collected from Google search. The claims in COVIDFACT are also complex and has two labels for the evidences collected, SUPPORT and REFUTE. Both HEALTHVER and COVIDFACT have one annotated evidence per claim, whereas in SCIFACT, there may be more than one evidence for one claim. Also, in SCIFACT, relevant abstracts are needed to be retrieved first from the corpus provided and then evidence sentences are needed to be retrieved from those abstracts.
Although our model has not been trained over Covid-19 specific text, we wanted to experiment how well it generalizes over such data by performing two different experiments. In the first experiment, we applied our model to the test set of both HEALTHVER and COVIDFACT without using the training set in a zero-shot approach. For SCIFACT, we have used the dev set.
We can see from Table 6 that in a zero-shot setting, our model performs better with the SCIFACT dataset. This can be attributed to the fact that SCIFACT data contain PubMed abstracts and PubMedBERT (Gu et al., 2021) has been trained over PubMed text which is why it performs better. HEALTHVER and COVIDFACT, on the other hand do not contain PubMed data and as such the model does not generalise well over the other datasets.
For the second experiment, we have transferred a trained model over one dataset to the other two to see how well models trained on one dataset generalize to other datasets. We use the train sets of the datasets to train the model and then use the test sets of the other datasets to evaluate the model per-
formance. Since the HEALTHVER dataset has the NEUTRAL label, we have dropped instances from its test set having that label in order to maintain consistency over all the datasets when the model was trained over SCIFACT and COVIDFACT train sets since these datasets only have SUPPORT and REFUTE labels. The results of the experiment are shown in Table 7.
From Table 7, it can be seen that when the model is trained on SCIFACT and HEALTHVER, transferring to the COVIDFACT test set does not give very good results. This is due to the fact that COVIDFACT contains both scientific as well as non-scientific claim-evidence pairs and therefore a model trained on either SCIFACT or HEALTHVER does not generalize well as they are based on scientific data. We can also see that the model trained on HEALTHVER generalizes better on the SCIFACT data and vice-versa as both these datasets are based on scientific claims and evidences, they learn better and generalize well on each other. However, we can also see that the model trained on COVIDFACT generalizes well on the other datasets since it contains both scientific and non-scientific data. These results confirm the findings by (Saakyan et al., 2021) that models trained on scientific data do not generalize well on data that contain non-scientific data as well. This is important as real-world health misinformation data may contain both scientific and non-scientific claims. In such situations, we need to have both scientific as well as non-scientific data so that models
can learn to generalize on such data.

heading: 8 Conclusion and future work
text: We have explored the prediction of veracity for health-related fact-checking tasks that can be learned from NLI data. By doing experiments, we showed that training domain-specific BERT-based models on domain-specific NLI data improves the model performance for fact-checking task. We also explored a method that can be used to combine different evidences for a claim, even for situations that have conflicting evidences for the same claim. We have also shown by experiments that augmenting data using back-translation helps in situations where there is a lack of training data. Although fact-checking of scientific claims is still a new task, there is a potential for improvement of the current methods being used for the task. With the advent of more capable large language models, new research direction such as prompt based methods can also be explored. As future work, we are interested in exploring such prompt-based approaches along with multimodal data in this space.

heading: Future Work
text: As future work, we are interested in exploring such prompt-based approaches along with multimodal data in this space