Venue: NAACL 2025

# CORRECT: Context- and Reference-Augmented Reasoning and Prompting

for Fact-Checking

Delvin Ce Zhang

The Pennsylvania State University

delvincezhang@gmail.com &Dongwon Lee

The Pennsylvania State University

dongwon@psu.edu

###### Abstract

Fact-checking the truthfulness of claims usually requires reasoning over multiple evidence sentences. Oftentimes, evidence sentences may not be always self-contained, and may require additional contexts and references from elsewhere to understand coreferential expressions, acronyms, and the scope of a reported finding. For example, evidence sentences from an academic paper may need contextual sentences in the paper and descriptions in its cited papers to determine the scope of a research discovery. However, most fact-checking models mainly focus on the reasoning within evidence sentences, and ignore the auxiliary contexts and references. To address this problem, we propose a novel method, Context- and Reference-augmented Reasoning and Prompting. For evidence reasoning, we construct a three-layer evidence graph with evidence, context, and reference layers. We design intra- and cross-layer reasoning to integrate three graph layers into a unified evidence embedding. For verdict prediction, we design evidence-conditioned prompt encoder, which produces unique prompt embeddings for each claim. These evidence-conditioned prompt embeddings and claims are unified for fact-checking. Experiments verify the strength of our model. Code and datasets are available at [https://github.com/cezhang01/correct](https://github.com/cezhang01/correct).

## 1 Introduction

The proliferation of misinformation has posed growing challenge in the realm of information reliability. There is a need to develop automated fact-checking methods Guo et al. (2022) to verify the truthfulness of real-world claims using evidence.

Existing fact-checking models Zhou et al. (2019); Liu et al. (2020) have shown promise in aggregating and reasoning over multiple evidence sentences to verify a claim. However, the evidence sentences retrieved from a large corpus may contain incomplete information when they are taken out-of-corpus. We need to refer to additional contexts and references from elsewhere to understand coreferential expressions, acronyms, and the scope of a reported finding. For example, Fig. 1(a) illustrates context-dependent evidence, where undefined acronym "MNC" in evidence sentence from a paper abstract requires additional context from the abstract to jointly interpret the meaning of acronym "MNC". Fig. 1(b) presents reference-dependent evidence, where we need to check the cited paper to understand that "SARS-CoV-2 infection" and "COVID-19 infection" are coreferential expressions, so that we could accurately fact-check the claim. Such scenario also exists in general domain where evidence sentences from a Wikipedia page may need contextual sentences in the same page and text in the hyperlinked pages to complement the insufficient information in the evidence.

**Challenges and Approach.** To overcome the limitations of existing methods, we propose **C**ontex- and **R**eference-augmented **Re**asoning and **r**ompting for **fact-**checking (CORRECT), to address two open questions.

First, _how to aggregate both contextual and referential documents into evidence reasoning?_ Some models are proposed to capture contextual documents, e.g., MultiVerS Wadden et al. (2022). Some others are designed for referential documents, e.g.,

Figure 1: Illustration of (a) context-dependent and (b) reference-dependent evidence from BearFact dataset.



Transformer-XH Zhao et al. (2020) and HESM Subramanian and Lee (2020). However, they incorporate either contextual or referential documents, failing to aggregate both of them into unified evidence embedding. Moreover, most of them simply concatenate evidence with contextual or referential documents, and inefficiently input the long text to language models for evidence encoding. Though they have shown that modeling either contexts or references helps fact-checking, integrating both of them for evidence reasoning is still unexplored. In our model, we construct a three-layer graph with evidence, context, and reference layers. We design intra- and cross-layer reasoning to aggregate three graph layers into unified evidence embedding.

Second, _how to integrate evidence reasoning and claim for accurate verdict prediction?_ Previous fact-checking methods, e.g., ProToCo Zeng and Gao (2023), rely on natural language as input prompt to language model for claim verification. However, discrete natural language prompts are difficult to design and may result in suboptimal results Zhou et al. (2022b). Recently, prompt tuning Lester et al. (2021) uses continuous and learnable prompt embeddings to replace discrete prompt and has achieved decent result, but no one has explored its design for claim verification. We propose evidence-conditioned prompt encoder, which takes evidence embedding as input, and produces unique prompt embeddings for each claim. We combine prompt embeddings with claim token embeddings to unify evidence and claim for verdict prediction.

**Contributions.** First, we propose a novel model, Context- and Reference-augmented Reasoning and Promptting (CORRECT), to integrate both contextual and referential documents into evidence reasoning. Second, we design a three-layer evidence graph, and propose intra- and cross-layer reasoning to learn unified evidence embedding. Third, we propose evidence-conditioned prompt embeddings, which are combined with claims to integrate evidence reasoning with claim for fact-checking.

## 2 Related Work

**Multi-hop fact-checking.** Complex claims usually require reasoning over multiple evidence sentences. Many methods are based on Language Models Vaswani et al. (2017); Devlin et al. (2019) and Graph Neural Networks Hamilton et al. (2017), such as GEAR Zhou et al. (2019), KGAT Liu et al. (2020), DREAM Zhong et al. (2020), SaGP Si et al. (2023), DECKER Zou et al. (2023), CausalWalk Zhang et al. (2024a), etc. However, they mainly focus on the reasoning within evidence sentences. They ignore the auxiliary contextual and referential documents. Methods incorporating contextual documents are proposed, e.g., ParagraphJoint Li and Peng (2021), ARSJOint Zhang et al. (2021), MultiVerS Wadden et al. (2022), etc. Some others integrating referential documents include Transformer-XH Zhao et al. (2020) and HESM Subramanian and Lee (2020). However, they incorporate either contextual or referential documents, but not both. In contrast, we construct a three-layer evidence graph to model evidence sentences, contexts, and references. There are fake news detection models where auxiliary graph with Wikidata is used Hu et al. (2021); Whitehouse et al. (2022). Fake news detection aims to detect the whole article with meta-data, while fact-checking focuses on claim sentences with retrieved evidence.

Some fact-checking works are based on retrieval-augmented generation Zeng and Gao (2024). They unify evidence retrieval and claim verification as a joint approach, while our model mainly focuses on verification, and relies on external tool for evidence retrieval. Our setting is consistent with existing works Wadden et al. (2022); Zhang et al. (2024a).

**Prompt-based fact-checking.** Some models verify claims by prompting LLMs Achiam et al. (2023). ProToCo Zeng and Gao (2023) inputs both evidence sentences and claim to T5 Raffel et al. (2020). ProgramFC Pan et al. (2023) decomposes complex claims into simpler sub-tasks and uses natural language to prompt LLMs. Varifocal Ousidhoum et al. (2022) formulates fact-checking as question generation and answering. They rely on handcrafted natural language as prompt. The performance heavily relies on the choice of prompt, and it is difficult to design a prompt that produces a decent result, as shown in Zhou et al. (2022b). Our model is designed with learnable prompt embeddings where the prompting instruction is naturally learned by embeddings through optimization.

**Prompt learning.** Prompting Brown et al. (2020) uses natural language as the input to language models to fulfill certain tasks. Many prompting models have been proposed, including natural language prompt Gao et al. (2021); Shin et al. (2020) and prompt embeddings Lester et al. (2021); Liu et al. (2023, 2022); Li and Liang (2021). Prompting also benefits many tasks Zhou et al. (2022a); Tan et al. (2022). However, no one has explored 

prompt embeddings for fact-checking.

**Text-attributed graph.** Texts are usually connected in a graph structure, termed text-attributed graph (Zhang et al., 2024). Various methods have been developed to learn text embeddings in an unsupervised manner (Zhang and Lauw, 2020, 2023, 2021; Zhang et al., 2023; Yang et al., 2021; Jin et al., 2023; Yang et al., 2024). Though both our model and these works construct a text-attributed graph, our work is different from them, since our model is a supervised model for fact-checking.

## 3 Model Architecture

We introduce \({\bf Context}\)- and \({\bf Ref}\)erence-augmented \({\bf Reasoning}\) and prompting for fact-checking (CORRECT). Table 1 summarizes math notations.

### Problem Formulation

We are given a fact-checking dataset \(\mathcal{D}=\{\mathcal{X},\mathcal{E},\mathcal{C},\mathcal{R}\}\). Claim set \(\mathcal{X}=\{x_{i}\}^{N}_{i=1}\) contains a set of \(N\) claims. Evidence set \(\mathcal{E}=\{e_{j}\}^{E}_{j=1}\) is a corpus of \(E\) evidence sentences. For each evidence sentence \(e\in\mathcal{E}\), we have its contextual document \(c\in\mathcal{C}\). Usually, an evidence sentence has only one contextual document, from which this sentence is retrieved. We also have \(e\)'s referential documents \(\mathcal{N}_{\textrm{ref}}(e)=\{r_{e,n}\}^{R_{e}}_{n=1}\subset\mathcal{R}\). Here \(R_{e}\) is the number of \(e\)'s referential documents. Evidence sentence \(e\) may have multiple referential documents, such as papers cited by \(e\)'s paper or Webpages hyperlinked by \(e\). We use \(\mathcal{N}_{\textrm{ref}}(e)\) to represent the set of \(e\)'s referential documents. We use \(\mathcal{N}_{\textrm{evid}}(x)\subset\mathcal{E}\) to denote the set of evidence sentences for a claim \(x\).

Given \(\mathcal{D}\) as input, we design a model that uses evidence sentences from \(\mathcal{E}\) together with their contextual documents in \(\mathcal{C}\) and referential documents in \(\mathcal{R}\) to verify claims. Eventually, for each claim \(x\in\mathcal{X}\), we output its predicted label \(\hat{y}\in\mathcal{Y}=\{\mbox{SUPPORT, REFUTE, NEI}\}\), indicating whether the evidence supports, refutes, or does not have enough information to verify the claim.

As shown in Fig. 2, CORRECT has two modules: (a-c) context- and reference-augmented evidence reasoning on three-layer graph, (d) evidence-conditioned prompting for claim verification.

### Three-layer Evidence Graph Reasoning

**Graph construction.** For each claim \(x\in\mathcal{X}\) and its evidence sentences \(\mathcal{N}_{\textrm{evid}}(x)\subset\mathcal{E}\), we construct a three-layer graph with evidence, context, and reference layers in Fig. 2(a). We consider evidence sentences, contextual documents, and referential documents as three types of vertices. Each type of vertices reside on their own layer. Cross-layer links between evidence layer and context layer connect each evidence sentence with its contextual document. Each evidence sentence and its referential documents are connected by cross-layer referential links. Green links in Fig. 2(a) are cross-layer links. For multi-evidence reasoning, we add intra-layer links on evidence layer where evidence sentences of a claim are fully connected, shown by black links in Fig. 2(a). The purpose of constructing three layers instead of mixing all vertices into one layer is to better differentiate three types of vertices.

**Intra-layer reasoning.** Evidence reasoning includes intra- and cross-layer reasoning. We first show intra reasoning (orange arrows in Fig. 2(b)).

For each evidence sentence \(e\in\mathcal{N}_{\textrm{evid}}(x)\), we let \({\mathbf{H}}^{(l)}_{e}=[{\mathbf{h}}^{(l)}_{e,\mathsf{CLS}},{\mathbf{h}}^{( l)}_{e,1},{\mathbf{h}}^{(l)}_{e,2},...]\) denote the output from the \(l\)-th Transformer step. Note that previous works call it the \(l\)-th layer, but to distinguish it from our three-layer graph, we instead call it the \(l\)-th step. \({\mathbf{h}}^{(l)}_{e,i}\in\mathbb{R}^{d}\) is \(d\)-dimensional token embedding. We use graph neural network to aggregate different evidence sentences of a claim. For each evidence sentence \(e\), we first project it by

\[\tilde{\mathbf{h}}^{(l)}_{e,\mathsf{CLS}}={\mathbf{W}}_{1}{\mathbf{h}}^{(l)} _{e,\mathsf{CLS}}. \tag{1}\]

The [CLS] token is taken as the evidence sentence embedding, and \({\mathbf{W}}_{1}\in\mathbb{R}^{d\times d}\) is type-specific parameter. We design type-specific attention,

\[a_{e,e^{\prime}}=\textrm{softmax}\Big{(}\textrm{LeakyReLU}({\mathbf{b}}^{ \top}_{1}[\tilde{\mathbf{h}}^{(l)}_{e,\mathsf{CLS}}||\tilde{\mathbf{h}}^{(l) }_{e^{\prime},\mathsf{CLS}}])\Big{)}. \tag{2}\]

\(e^{\prime}\in\mathcal{N}_{\textrm{evid}}(x)\backslash e\) is another evidence sentence for the same claim \(x\), \([\cdot||\cdot]\) is concatenation, and \({\mathbf{b}}_{1}\in\mathbb{R}^{d}\) is the corresponding sentence.

\begin{table}
\begin{tabular}{p{85.4pt}|p{76.9pt}} \hline Notation & Description \\ \hline \(\mathcal{D}\) & a fact-checking dataset \\ \(\mathcal{X}\) & a set of \(N=|\mathcal{X}|\) claims \\ \(\mathcal{E}\) & a corpus of evidence sentences \\ \(\mathcal{C}\) & a set of contextual documents \\ \(\mathcal{R}\) & a set of referential documents \\ \(\mathcal{N}_{\textrm{ref}}(e)\) & evidence sentence \(e\)’s referential documents \\ \(\mathcal{N}_{\textrm{evid}}(x)\) & claim \(x\)’s retrieved evidence sentences \\ \(\mathcal{Y}\) & a set of labels \\ \({\mathbf{h}}^{(l)}_{e,\mathsf{CLS}}\) & evidence sentence \(e\)’s [CLS] token embedding \\ \(\tilde{\mathbf{h}}^{(l)}_{e}\) & aggregated contextual document embedding \\ \(\tilde{\mathbf{h}}^{(l)}_{e}\) & aggregated referential document embedding \\ \(\tilde{\mathbf{h}}^{(l)}_{e}\) & evidence sentence \(e\)’s augmented embedding matrix \\ \(\mathbf{\pi}_{m,y}\) & the \(m\)-th prompt embedding for class \(y\) \\ \({\mathbf{h}}_{m,y}\) & the \(m\)-th base prompt embedding for class \(y\) \\ \hline \end{tabular}
\end{table}
Table 1: Summary of mathematical notations.

 

\({\mathbb{R}^{2d}}\) is learnable parameter. Finally, we aggregate evidence sentences to \(e\) by mean pooling.

\[\hat{\mathbf{h}}_{e}^{(l)}=\text{mean}\Big{(}\tilde{\mathbf{h}}_{e,\text{CLS}}^{( l)},\sum_{e^{\prime}\in\mathcal{N}_{\text{eval}}(x)\backslash e}a_{e,e^{\prime}} \tilde{\mathbf{h}}_{e^{\prime},\text{CLS}}^{(l)}\Big{)}. \tag{3}\]

The aggregated sentence embedding \(\hat{\mathbf{h}}_{e}^{(l)}\) captures information of both itself and other evidence sentences. To summarize Eqs. 1-3, we have

\[\hat{\mathbf{h}}_{e}^{(l)}=f_{\text{GNN}}\Big{(}\mathbf{h}_{e,\text{CLS}}^{(l) },\{\mathbf{h}_{e^{\prime},\text{CLS}}^{(l)}|e^{\prime}\in\mathcal{N}_{\text{ evid}}(x)\backslash e\};\mathbf{W}_{1},\mathbf{b}_{1}\big{)}. \tag{4}\]

To integrate intra-layer aggregation into the encoding of each evidence sentence, we introduce a _virtual token_ to represent the aggregated sentence embedding \(\hat{\mathbf{h}}_{e}^{(l)}\). For evidence sentence \(e\), we concatenate \(\hat{\mathbf{h}}_{e}^{(l)}\) with \(e\)'s text token embeddings by \(\widehat{\mathbf{H}}_{e}^{(l)}=\hat{\mathbf{h}}_{e}^{(l)}||\mathbf{H}_{e}^{( l)}\). After concatenation, \(\widehat{\mathbf{H}}_{e}^{(l)}\) contains information of both evidence sentence \(e\)'s text and the aggregated embedding from \(e\)'s intra-layer neighbors. We aim to propagate the aggregated sentence embedding to other text tokens of sentence \(e\), so that the text tokens can fully unify other sentences for multi-evidence reasoning. We will introduce _asymmetric_ multi-head self-attention to achieve this goal. But before that, we first discuss cross-layer reasoning.

**Cross-layer reasoning.** We present cross-layer reasoning, which aggregates contextual and referential documents into evidence sentences (green arrows in Fig. 2(b)). The aggregation from referential documents to evidence sentence is similarly defined by Eq. 5. We use reference-specific parameters, \(\mathbf{W}_{2}\) and \(\mathbf{b}_{2}\), to preserve graph heterogeneity.

\[\hat{\mathbf{h}}_{r}^{(l)}=f_{\text{ONN}}\Big{(}\mathbf{h}_{e,\text{CLS}}^{( l)},\{\mathbf{h}_{r,\text{CLS}}^{(l)}|r\in\mathcal{N}_{\text{ref}}(e)\}; \mathbf{W}_{2},\mathbf{b}_{2}\Big{)}. \tag{5}\]

Each referential document \(r\in\mathcal{N}_{\text{ref}}(e)\) is also encoded, and its [CLS] token is passed to Eq. 5 for aggregation. Similarly, we have \(\hat{\mathbf{h}}_{c}^{(l)}\) as contextual document embedding. To integrate both embeddings into evidence sentence for cross-layer reasoning, we introduce two more _virtual tokens_.

\[\widehat{\mathbf{H}}_{e}^{(l)}=\hat{\mathbf{h}}_{c}^{(l)}||\hat{\mathbf{h}}_{ r}^{(l)}||\hat{\mathbf{h}}_{e}^{(l)}||\mathbf{H}_{e}^{(l)}. \tag{6}\]

The augmented embedding matrix, i.e., \(\widehat{\mathbf{H}}_{e}^{(l)}\), contains both intra-evidence reasoning as well as cross-layer context and reference augmentation.

To fully unify all three graph layers into evidence sentence \(e\), we input \(\widehat{\mathbf{H}}_{e}^{(l)}\) at Eq. 6 to the \((l+1)\)-th Transformer step with our proposed _asymmetric_ multi-head self-attention (MSAasy).

\[\text{MSAasy}(\mathbf{H}_{e}^{(l)},\widehat{\mathbf{H}}_{e}^{(l)},\!\widehat{ \mathbf{H}}_{e}^{(l)})=\text{softmax}\Big{(}\frac{\textbf{QK}^{\top}}{\sqrt{ d}}\Big{)}\textbf{V}, \tag{7}\]

Keys **K** and values **V** are augmented with virtual tokens, but queries **Q** are not, to avoid context and reference embeddings being overwritten by evidence sentence embedding. The result of asymmetric MSA is passed to a multi-layer perceptron and layer normalization Vaswani et al. (2017). Finally, we obtain the output from the \((l+1)\)-th step, \(\mathbf{H}_{e}^{(l+1)}\), integrating evidence sentence \(e\), other evidence sentences of the same claim, \(e\)'s contextual and referential documents, see Fig. 2(c).

We conduct such intra-layer and cross-layer reasoning inside each Transformer step to allow different graph layers to fully communicate with each other. We repeat such nested and graph-augmented encoding for \(L\) times, and obtain \(\mathbf{h}_{e}=\mathbf{h}_{e,\text{CLS}}^{(L)}\) as the graph-augmented embedding for evidence sentence \(e\). This nested architecture is shown by Fig. 2(c). For claim \(x\), we have \(\{\mathbf{h}_{e}\}_{e\in\mathcal{N}_{\text{eval}}(x)}\), a set of graph-augmented embeddings for its evidence

Figure 2: Model architecture. (a) A three-layer graph for a claim. (b) Intra- and cross-layer reasoning. (c) A nested architecture with language model and graph reasoning for evidence encoding. (d) Evidence-conditioned prompting.

 

sentences. Finally, we aggregate them by mean pooling and obtain a single evidence embedding.

\[{\textbf{h}_{E}=\text{mean}({\textbf{h}_{e}|e\in\mathcal{N}_{\text{ evid}}(x)}).} \tag{8}\]

### Evidence-conditioned Prompting

Now we integrate evidence reasoning into claim embedding to fully integrate their information for fact-checkcing. Prompting Liu et al. (2023) is a powerful method in fact-checking Zeng and Gao (2023). However, existing models are mainly based on natural language as input prompt to language models for verdict prediction. Handcrafted discrete prompt has two disadvantages: First, it is difficult to manually design a prompt that provides a decent performance. Previous works Zhou et al. (2022b) have shown that the change of a single word in the prompt may lead to significant deterioration of the results, and it is time-consuming to enumerate every prompt. Second, discrete natural language prompt is difficult to optimize, since language models are intrinsically continuous.

To mitigate these problems, we explore learnable and continuous prompt embedding. Below we design a prompt encoder, which takes evidence embedding **h\({}_{E}\)** as input, and produces evidence-conditioned prompt embeddings. See Fig. 2(d).

**Evidence-conditioned prompt encoder.** We consider below continuous embeddings as prompt.

\[{\textbf{P}_{x}=[{\textbf{h}_{x,\text{CLS}}},{\boldsymbol{\pi}_{1}},{ \boldsymbol{\pi}_{2}},...,{\boldsymbol{\pi}_{M}},{\textbf{h}_{x,1}},{ \textbf{h}_{x,2}},...]}.} \tag{9}\]

Here \(\{{\boldsymbol{\pi}_{m}}\}_{m=1}^{M}\) where \({\boldsymbol{\pi}_{m}}\in\mathbb{R}^{d}\) is a set of \(M\) learnable evidence-conditioned prompt embeddings to be explained shortly, and \(M\) is a hyperparameter, indicating the number of prompt embeddings. Each \({\textbf{h}_{x,i}}\in\mathbb{R}^{d}\) is a \(d\)-dimensional embedding of the \(i\)-th text token in claim \(x\). In language models, there is an embedding look-up table before language model encoder. In this look-up table, input text tokens are first mapped to the vocabulary to obtain their token embeddings, which are then summed up with positional encodings. \({\textbf{h}_{x,i}}\) in Eq. 9 is obtained by this look-up table.

Now we explain prompt embeddings \(\{{\boldsymbol{\pi}_{m}}\}_{m=1}^{M}\), output from an evidence-conditioned prompt encoder. We first initialize \(M\)_base_ prompt embeddings, \(\{{\textbf{h}_{m}}\}_{m=1}^{M}\). We then project evidence embedding \({\textbf{h}_{E}}\) in Eq. 8 to the prompt embedding space, followed by element-wise product and summation.

\[{\boldsymbol{\alpha}_{x}=\text{tanh}\Big{(}\frac{{\textbf{W}_{\alpha}{\textbf {h}_{E}}+{\textbf{b}_{\alpha}}}}{\tau}\Big{)}},~{}~{}~{}\beta_{x}=\text{ tanh}\Big{(}\frac{{\textbf{W}_{\beta}{\textbf{h}_{E}}+{\textbf{b}_{\beta}}}}{ \tau}\Big{)}, \tag{10}\]

\[{\boldsymbol{\pi}_{m}}={\textbf{h}_{m}}\odot({\boldsymbol{\alpha}_{x}+{ \textbf{1}}})+\beta_{x}. \tag{11}\]

\(\odot\) is element-wise product, and \({\textbf{1}}\in\mathbb{R}^{d}\) is a vector of ones to ensure that the scaling of \({\textbf{h}_{m}}\) is centered around one. \(\tau\) is a temperature to scale the shape of tanh function. \({\boldsymbol{\pi}_{m}}\) is thus conditioned on evidence embedding, and different claims with their own evidence sentences should have their unique claim-specific prompt embeddings, shown by Fig. 2(d).

Given the label set \(\mathcal{Y=}\{\text{SUPPORT, REFUTE, NEI}\}\), which usually has three types of labels, we apply above evidence-conditioned prompt encoder and correspondingly obtain three sets of prompt embeddings, \(\{{\boldsymbol{\pi}_{m,y}}\}_{m=1}^{M}\) where \(y\in\mathcal{Y}\). As in Eq. 9, we concatenate each set of prompt embeddings with token embeddings of claim \(x\), and obtain three sets of inputs \(\{{\textbf{P}_{x,y}}\}_{y\in\mathcal{Y}}\) to claim encoder.

\[{\textbf{H}_{x,y}^{(L)}} =f({\textbf{P}_{x,y}}) \tag{12}\] \[=f([{\textbf{h}_{x,\text{CLS}}},{\boldsymbol{\pi}_{1,y}},{ \boldsymbol{\pi}_{2,y}},...,{\boldsymbol{\pi}_{M,y}},{\textbf{h}_{x,1}},{ \textbf{h}_{x,2}},...])\]

\({\textbf{H}_{x,y}^{(L)}}\) is the output from the claim encoder, and its [CLS] token is taken as claim embedding \({\textbf{h}_{x,y}}={\textbf{h}_{x,y,\text{CLS}}^{(L)}}\). Claim encoder shares parameters with evidence encoder. Due to contextualized modeling, claim token embeddings and evidence-conditioned prompt embeddings fully exchange information, and the output claim embedding captures both claim \(x\) and evidence reasoning for fact-checkching.

Finally, we use contrastive loss function to predict the veracity of claim \(x\) by

\[\small\mathcal{L=-\sum_{x\in\mathcal{X}_{\text{train}}}\log\frac{\exp({ \textbf{h}_{x,y}^{\mathsf{T}}{\textbf{h}_{E}}})}{\exp({\textbf{h}_{x,y}^{ \mathsf{T}}{\textbf{h}_{E}}})+\sum_{y^{\prime}\in\mathcal{Y}_{\text{hy}}}\exp( {\textbf{h}_{x,y}^{\mathsf{T}}{\textbf{h}_{E}}})}}. \tag{13}\]

\({\textbf{h}_{E}}\) is evidence embedding of claim \(x\) obtained by Eq. 8. \(\mathcal{V}_{\text{train}}\) is a set of training claims. Though we use three types of labels in \(\mathcal{Y}\), more types of labels in \(\mathcal{Y}\) can also be modeled. Algorithm 1 summarizes the learning process.

``` Input: The dataset \(\mathcal{X}\), \(\mathcal{Y}\), \(\ 

each label in the label set \(y\in{\cal Y}\), we have training claims belonging to this label \({\cal X}_{\text{train},y}=\{x\in{\cal X}_{\text{train}}|y_{x}=y\}\). For each of these claims, we truncate its evidence sentences, contextual and referential documents to \(M\) words, and obtain their \(M\) word embeddings in the look-up table of language model. We then take mean pooling for evidence sentences, contextual and referential documents, and obtain \(M\) pooled word embeddings for each claim. Finally, we average all training claims belonging to the same label \({\cal X}_{\text{train},y}\), and obtain \(M\) word embeddings, which are used to initialize \(M\) base prompt embeddings \(\{\textbf{h}_{m,y}\}^{M}_{m=1}\). They are derived from training claims of the same label, thus provide a more informative starting point than random initialization for verdict prediction. We repeat this process for every label \(y\in{\cal Y}\), and obtain initialization for each set of base prompt embeddings.

## 4 Experiments

We conduct extensive experiments and ablation analysis to evaluate the effectiveness of the proposed model CORRECT.

**Datasets.** We use 4 datasets in Table 2. FEVEROUS [3] is a general-domain dataset. Each claim is annotated in the form of sentences and/or cells from tables in Wikipedia pages. Since we focus on textual fact-checking, we follow [50] and select claims that only require sentences as evidence. We call this subset **FEVEROUS-S. BearFact**[69] is a biomedical dataset with sentences from papers as evidence. Its original dataset does not have evidence for claims in NEI class. We follow [78] and select sentences that have the highest _tf-idf_ similarity with those claims as evidence. **Check-COVID**[104] contains claims about COVID-19. **SciFact**[103] is a dataset with sentences in papers as evidence. As in its original paper, for claims in NEI class, we choose sentences from the cited abstract with top-3 highest _tf-idf_ similarity with the claim as evidence. Appendix B contains data preprocessing details.

**Baselines.** We have 4 categories of baselines.

_i) Multi-hop fact-checking_, KGAT [105], HESM [106], Transformer-XH [104], MultiVerS [104], and the recent CausalWalk [104]. MultiVerS models contextual documents, and HESM and Transformer-XH incorporate referential documents. By comparing to them, we highlight the advantage of three-layer graph for modeling both contextual and referential documents. Since our model is built on Transformer-XH, we further extend it by modeling both contextual and referential documents, and name it Transformer-XH++. The comparison showcases the effect of evidence-conditioned prompting.

_ii) Few-shot fact-checking_, GPT2-PPL [107], ProToCo [78], and ProgramFC [50]. They are mainly designed for few-shot setting. By increasing their training set, we could also compare to them on fully supervised setting. ProToCo and ProgramFC are proposed with handcrafted natural language prompt. By comparison, we verify the usefulness of our evidence-conditioned prompt embedding.

_iii) Prompt tuning_ is not for fact-checking. But for completeness, we convert P-Tuning v2 [104], a continuous prompting, to our task.

_iv) Retrieval-augmented generation for fact-checking_. Though our model is not designed with retrieval-augmented generation, we still compare to JustiLM [78] for completeness.

**Implementation details.** Following [104], we set \(L\) to 12 and \(d\) to 768. Number of prompt embeddings \(M\) is 8. Temperature \(\tau\) in Eq. 10 is 100. For both our model and language model-based baselines, we initialize the model with pre-trained parameters in biomedical domain [104] for scientific datasets, and in general domain [104] for FEVEROUS-S. Each result is obtained by 5 independent runs. Experiments are done on 4 NVIDIA A100 80GB GPUs. More details are in Appendix C.

We present two experimental settings below.

**Fully supervised v.s. Few-shot.** For fully supervised setting, we train the model on the training set. If the dataset provides data split, we follow the split and obtain training and test sets. Otherwise, we split the dataset into 80:20 for training and test, respectively. Among training set, we further reserve 10% for validation. For few-shot setting, we report 5-shot experiments as the main results, i.e., for each class in the label set \(y\in{\cal Y}\), we randomly sample

\begin{table}
\begin{tabular}{c|c c|c|c} \hline \multirow{2}{*}{Name} & \#Claims & \#Contextual & \#Referential \\  & Train & Test & Documents & Documents \\ \hline FEVEROUS-S & 23,912 & 5,978 & 19,546 & 21,579 \\ BearFact & 1,158 & 290 & 1,166 & 12,938 \\ Check-COVID & 1,275 & 229 & 347 & 3,132 \\ SciFact & 809 & 300 & 1,189 & 9,617 \\ \hline \end{tabular}
\end{table}
Table 2: Dataset statistics.

 

5 instances from training set, obtaining \(5\times|{\cal Y}|\) training instances. This setting is consistent with existing work Zeng and Gao (2023). For a fair comparison, we sample instances using 5 random seeds. We keep the same sampling for our model and baselines. We report the results on test set.

**Gold v.s. Retrieved evidence.** For gold evidence setting, we observe the ground-truth evidence sentences, and we verify the claim based on the gold sentences. For retrieved evidence setting, we do not observe any evidence sentences, and retrieve sentences from an evidence corpus, based on which we make prediction. We follow Pan et al. (2023) and use BM25 Robertson et al. (2009) to retrieve top-3 evidence sentences for each claim. In the original Check-COVID dataset, if a claim is labeled as REFUTE based on the evidence, this claim is _reused_ in NEI class with another random evidence. Thus, there are two claims with the same content, but different evidence and labels. However, in our retrieved evidence setting, both claims will receive the same retrieved evidence, but they are labeled differently, making model training inconsistent. Thus, for retrieved evidence setting, we remove claims in NEI class for Check-COVID.

### Empirical Evaluation

**Fully supervised setting.** We follow Wadden et al. (2022) and report Macro F1 score for both gold and retrieved evidence settings in Table 3. We also show Micro F1 score in Table 4. Transformer-XH++ consistently outperforms Transformer-XH, verifying that contextual and referential documents bring useful information. By comparing CORRECT to Transformer-XH++, we design evidence-conditioned prompting to integrate evidence and claim embeddings, and further improve the performance. Models with handcrafted prompt do not predict verdict as accurately as our model, which showcases the advantage of continuous prompt embeddings. Overall, the results on gold evidence setting are higher than on retrieved evidence setting, because the retrieved evidence sentences may not be always correct and may contain noisy information. The only exception is Check-COVID, because the retrieved evidence setting has only two labels, making the prediction task easier. MultiVerS is slightly better than CORRECT on SciFact, because the evidence sentences in SciFact contain sufficient information for fact-checking as shown in Wadden et al. (2020, 2022), and referential documents do

\begin{table}
\begin{tabular}{c|c c|c c|c c|c c} \hline \multirow{2}{*}{Model} & \multicolumn{2}{c|}{BearFact} & \multicolumn{2}{c|}{Check-COVID} & \multicolumn{2}{c|}{SciFact} & \multicolumn{2}{c}{FEVEROUS-S} \\ \cline{2-8}  & Gold & Retrieved & Gold & Retrieved & Gold & Retrieved & Gold & Retrieved \\ \hline KGAT & 69.42\(\pm\)0.87 & 57.36\(\pm\)0.52 & 72.05\(\pm\)1.58 & 76.47\(\pm\)0.65 & 74.44\(\pm\)0.96 & 62.33\(\pm\)0.88 & 86.21\(\pm\)0.28 & 67.99\(\pm\)0.78 \\ HESM & 63.68\(\pm\)1.39 & 58.62\(\pm\)0.32 & 63.47\(\pm\)0.25 & 71.90\(\pm\)1.85 & 72.44\(\pm\)0.77 & 53.36\(\pm\)2.33 & 83.30\(\pm\)0.75 & 68.36\(\pm\)0.86 \\ Transformer-XH+ & 61.26\(\pm\)0.72 & 56.55\(\pm\)1.50 & 68.56\(\pm\)0.87 & 76.91\(\pm\)1.51 & 75.89\(\pm\)0.51 & 58.67\(\pm\)1.53 & 85.61\(\pm\)0.80 & 69.78\(\pm\)0.41 \\ Transformer-XH++ & 64.02\(\pm\)1.44 & 58.39\(\pm\)1.11 & 70.60\(\pm\)0.67 & 78.65\(\pm\)0.38 & 77.78\(\pm\)0.69 & 60.56\(\pm\)1.83 & 85.52\(\pm\)0.39 & 70.37\(\pm\)0.77 \\ MultiVFS & 62.93\(\pm\)1.17 & 50.69\(\pm\)1.46 & 66.65\(\pm\)1.71 & 70.70\(\pm\)1.73 & 83.68\(\pm\)1.40 & **66.77\(\pm\)0.14** & 83.57\(\pm\)1.54 & 67.66\(\pm\)1.65 \\ CausalWalk & 69.31\(\pm\)1.69 & 60.90\(\pm\)0.69 & 71.86\(\pm\)1.54 & 71.68\(\pm\)2.48 & 77.34\(\pm\)2.30 & 59.00\(\pm\)1.20 & 86.42\(\pm\)0.92 & 71.51\(\pm\)1.66 \\ \hline GPT2-PPL & 40.00\(\pm\)2.43 & 39.49\(\pm\)0.73 & 32.75\(\pm\)0.62 & 32.54\(\pm\)0.93 & 31.50\(\pm\)0.71 & 31.24\(\pm\)1.56 & 54.33\(\pm\)0.06 & 54.23\(\pm\)0.01 \\ ProToCo & 56.03\(\pm\)0.24 & 35.57\(\pm\)2.35 & 37.12\(\pm\)1.10 & 32.75\(\pm\)2.31 & 60.00\(\pm\)1.53 & 31.17\(\pm\)0.71 & 54.21\(\pm\)0.67 & 44.71\(\pm\)1.95 \\ ProgramFC & 62.00\(\pm\)2.83 & 54.63\(\pm\)3.66 & 65.50\(\pm\)2.12 & 72.36\(\pm\)1.85 & 65.40\(\pm\)3.78 & 59.76\(\pm\)3.54 & 86.48\(\pm\)0.33 & 69.50\(\pm\)2.12 \\ \hline P-Tuning v2 & 70.69\(\pm\)0.49 & 60.34\(\pm\)0.15 & 72.93\(\pm\)1.85 & 77.32\(\pm\)1.96 & 80.34\(\pm\)0.94 & 57.44\(\pm\)1.83 & 87.16\(\pm\)0.33 & 70.56\(\pm\)0.54 \\ TwistLM & 62.41\(\pm\)1.25 & 48.49\(\pm\)2.21 & 59.71\(\pm\)2.56 & 61.71\(\pm\)2.05 & 72.20\(\pm\)1.85 & 54.74\(\pm\)0.82 & 81.60\(\pm\)0.33 & 68.38\(\pm\)1.45 \\ \hline CORRECT & **74.60\(\pm\)1.11** & **61.84\(\pm\)0.11** & **75.33\(\pm\)0.93** & **80.83\(\pm\)0.76** & **85.17\(\pm\)0.71** & 63.50\(\pm\)1.17 & **88.51\(\pm\)0.19** & **75.35\(\pm\)0.28** \\ \hline \end{tabular}
\end{table}
Table 4: Verdict prediction results on _fully supervised_ setting with _Micro F1_ score. Results are in percentage.

\begin{table}
\begin{tabular}{c|c c|c c|c c|c c} \hline \multirow{2}{*}{Model} & \multicolumn{2}{c|}{BearFact} & \multicolumn{2}{c|}{Check-COVID} & \multicolumn{2}{c|}{SciFact} & \multicolumn{2}{c}{FEVEROUS-S} \\ \cline{2-8}  & Gold & Retrieved & Gold & Retrieved & Gold & Retrieved & Gold & Retrieved \\ \hline KGAT & 53.11\(\pm\)2.25 & 36.55\(\pm\)1.95 & 71.97\(\pm\)1.31 & 75.83\(\pm\)0.74 & 70.23\(\pm\)1.08 & 59.83\(\pm\)0.68 & 86.10\(\pm\)0.32 & 67.76\(\pm\)0.93 \\ HESM & 44.90\(\pm\)2.20 & 42.93\(\pm\)0.27 & 62.85\(\pm\)0.59 & 71.58\(\pm\)0.19 & 86.66\(\pm\)0.69 & 50.91\(\pm\)2.57 & 83.12\(\pm\)0.80 & 67.43\(\pm\)0.81 \\ Transformer-XH & 45.28\(\pm\)1.08 & 38.39\(\pm\)0.80 & 67.81\(\pm\)0.93 & 76.51\(\pm\)0.29 & 72.01\(\pm\)0.86 & 56.26\(\pm\)0.64 & 85.44\(\pm\)0.75 & 68.13\(\pm\)0.52 \\ Transformer-XH++ & 46.81\(\pm\)1.52 & 41.06\(\pm\)1.70 & 70.52\(\pm\)0.55 & 78.49\(\pm\)0.52 & 73.92\(\pm\)0.58 & 57.82\(\pm\)2.29 & 85.35\(\pm\)0.45 & 69.76\(\pm\)0.73 \\ MultiVerS & 51.56\(\pm\)1.30 & 38.71\(\pm\)1.96 & 

not bring much additional benefit.

**Few-shot setting.** We report 5-shot results in Table 5 for Macro F1 score and Table 6 for Micro F1 score. Overall, HESM and Transformer-XH perform better than others, since referential documents contain useful information to complement evidence sentences for accurate prediction. Our model further improves them, verifying the strength of both contextual and referential documents. P-Tuning v2 outperforms models with handcrafted prompt, since continuous prompt embeddings can better adapt to the training data. By comparing to it, we design an evidence-conditioned prompt encoder to integrate contextual and referential documents into prompt embeddings, and produce more accurate results. We vary the number of shots in \(\{1,3,5,10,15\}\) in Fig. 3. Though our model is competitive with MultiVerS on SciFact, we are still better than it on other datasets, due to the advantage of both contextual and referential documents.

### Model Analysis

**Effect of intra- and cross-layer reasoning.** We respectively remove each graph layer from the complete model. Macro F1 score is shown in Fig. 4(a). The model with all three layers performs the best, indicating that all three layers bring useful information. Intra-layer reasoning on evidence sentence layer plays the most important role, since evidence sentences provide the most immediate information for verification. Contexts and references are important, and disregarding them deteriorates the results.

**Different numbers of prompt embeddings \(M\).** We vary the number of prompt embeddings \(M\) in Fig. 4(b). When \(M=2\), we cannot fully capture the interaction between evidence and claims, causing a low accuracy. After we increase \(M\), we observe an improvement. An overly high \(M\) hurts the result, because overfitting problem appears.

**Prompt initialization and encoder.** Our prompt encoder has both initialization of base prompt embeddings and evidence-conditioned prompt encoder. _i_) To test the effect of initialization, we replace it with random initialization and report the results in Fig. 4(c). Our initialization produces better results, because evidence graph separates different sets of prompt embeddings and provides a more informative starting point. _ii_) We remove

\begin{table}
\begin{tabular}{r|c c|c c|c c|c c} \hline \multirow{2}{*}{Model} & \multicolumn{2}{c|}{BearFact} & \multicolumn{2}{c|}{Check-COVID} & \multicolumn{2}{c|}{ScFiact} & \multicolumn{2}{c}{FEVEROUS-S} \\ \cline{2-9}  & Gold & Retrieved & Gold & Retrieved & Gold & Retrieved & Gold & Retrieved \\ \hline KGAT & 44.66\(\pm\)0.25 & 36.78\(\pm\)3.86 & 37.55\(\pm\)2.86 & 50.98\(\pm\)1.13 & 42.22\(\pm\)3.06 & 36.78\(\pm\)3.52 & 51.13\(\pm\)1.55 & 51.66\(\pm\)1.38 \\ HESM & 48.85\(\pm\)2.42 & 28.97\(\pm\)3.26 & 36.68\(\pm\)5.15 & 50.11\(\pm\)3.22 & 39.56\(\pm\)2.45 & 35.89\(\pm\)4.67 & 56.33\(\pm\)0.91 & 53.68\(\pm\)0.37 \\ Transformer-XH & 32.53\(\pm\)3.13 & 40.80\(\pm\)3.32 & 41.67\(\pm\)2.19 & 51.63\(\pm\)1.16 & 48.89\(\pm\)2.84 & 35.11\(\pm\)1.95 & 52.94\(\pm\)2.65 & 51.56\(\pm\)0.73 \\ Transformer-XH+ & 37.93\(\pm\)4.10 & 35.06\(\pm\)3.66 & 41.40\(\pm\)1.78 & 27.41\(\pm\)1.22 & 50.00\(\pm\)0.85 & 36.67\(\pm\)1.65 & 59.97\(\pm\)1.50 & 53.23\(\pm\)1.19 \\ MultiVers & 40.86\(\pm\)0.74 & 39.49\(\pm\)1.70 & 41.11\(\pm\)2.99 & 49.82\(\pm\)1.56 & **54.99\(\pm\)1.90** & 43.33\(\pm\)1.74 & 51.39\(\pm\)1.33 & 51.84\(\pm\)1.54 \\ CausalWalk & 45.52\(\pm\)3.47 & 41.38\(\pm\)3.24 & 37.70\(\pm\)4.59 & 43.79\(\pm\)3.25 & 44.02\(\pm\)2.70 & 44.78\(\pm\)2.71 & 60.60\(\pm\)1.98 & 55.20\(\pm\)4.18 \\ \hline GPT2-PPL & 36.38\(\pm\)4.14 & 40.69\(\pm\)0.49 & 33.19\(\pm\)0.67 & 34.62\(\pm\)0.72 & 29.44\(\pm\)2.50 & 29.00\(\pm\)1.46 & 53.02\(\pm\)1.11 & 53.11\(\pm\)1.12 \\ ProToCo & 51.03\(\pm\)2.44 & 33.80\(\pm\)2.44 & 41.05\(\pm\)2.47 & 34.50\(\pm\)2.31 & 51.55\(\pm\)2.27 & 36.78\(\pm\)1.35 & 54.72\(\pm\)1.32 & 42.45\(\pm\)2.73 \\ ProgramFC & 38.45\(\pm\)2.19 & 38.28\(\pm\)0.49 & 37.55\(\pm\)0.38 & 50.00\(\pm\)1.08 & 49.93\(\pm\)0.36 & 36.17\(\pm\)1.25 & 52.94\(\pm\)2.67 & 51.94\(\pm\)0.25 \\ \hline P-Tuning v2 & 48.70\(\pm\)1.95 & 41.90\(\pm\)3.10 & 41.05\(\pm\)3.88 & 52.07\(\pm\)3.09 & 45.78\(\pm\)1.07 & 37.67\(\pm\)1.85 & 58.02\(\pm\)1.68 & 52.06\(\pm\)0.76 \\ \hline JustilLM & 42.70\(\pm\)1.64 & 37.72\(\pm\)5.08 & 40.82\(\pm\)2.20 & 46.73\(\pm\)1.53 & 47.41\(\pm\)1.07 & 33.00\(\pm\)1.58 & 52.54\(\pm\)1.71 & 49.38\(\pm\)1.60 \\ \hline CORRECT & **51.72\(\pm\)1.04** & **42.76\(\pm\)0.73** & **43.37\(\pm\)1.82** & **54.46\(\pm\)0.76** & 53.00\(\pm\)1.30 & **44.36\(\pm\)0.84** & **63.33\(\pm\)0.91** & **57.14\(\pm\)0.82** \\ \hline \end{tabular}
\end{table}
Table 6: Verdict prediction results on _5-shot_ setting with _Micro F1_ score. Results are in percentage.

\begin{table}
\begin{tabular}{r|c c|c c|c c|c c} \hline \multirow{2}{*}{Model} & \multicolumn{2}{c|}{BearFact} & \multicolumn{2}{c|}{Check-COVID} & \multicolumn{2}{c|}{ScFiact} & \multicolumn{2}{c}{FEVEROUS-S} \\ \cline{2-9}  & Gold & Retrieved & Gold & Retrieved & Gold & Retrieved & Gold & Retrieved \\ \hline KGAT & 36.62\(\pm\)2.28 & 29.92\(\pm\)3.99 & 35.65\(\pm\)4.56 & 47.81\(\pm\)2.40 & 39.07\(\pm\)2.06 & 35.12\(\pm\)2.79 & 50.21\(\pm\)0.95 & 50.68\(\pm\)1.21 \\ HESM & 35.40\(\pm\)3.77 & 26.00\(\pm\)2.34 & 35.41\(\pm\)1.48 & 42.82\(\pm\)6.50 & 38.87\(\pm\)1.69 & 34.03\(\pm\)5.61 & 51.36\(\pm\)0.35 & 51.92\(\pm\)0.39 \\ Transformer-XH & 29.45\(\pm\)2.49 & 31.69\(\pm\)2.06 & 40.48\(\pm\)2.73 & 49.24\(\pm\)1.60 & 47.65\(\pm\)3.99 & 33.47\(\pm\)1.11 & 52.45\(\pm\)2.71 & 49.41\(\pm\)1.93 \\ Transformer-XH+ & 31.34\(\pm\)4.07 & 29.74\(\pm\)1.30 & 38.73\(\pm\)1.35 & 50.56\(\pm\)0.64 & 47.53\(\pm\)0.65 & 33.79\(\pm\)1.87 & 58.19\(\pm\)0.75 & 52.78\(\pm\)1.60 \\ MultiVers & 24.34\(\pm\)3.12 & 20.20\(\pm\)0.48 & 32.16\(\pm\)2.50 & 50.80\(\pm\)1.78 & **52.29\(\pm\)1.92** & **29.69\(\pm\)1.53** & 38.26\(\pm\)0.18 & 38.82\(\pm\)0.37 \\ CausalWalk & 32.01\(\pm\)3.35 & 31.10\(\pm\)1.56 & 31.

 

prompt encoder from our model, and only retain base prompt embeddings. Fig. 4(c) shows that removing prompt encoder hurts the results, indicating that prompt encoder is necessary to combine evidence and claim for accurate prediction.

**Effect of prompting.** We design two ablated models. _i_) We replace prompting with an MLP classifier, which concatenates evidence and claim embeddings as input, and produces predicted label. Here claim embedding is obtained without prompt embeddings. _ii_) We directly consider evidence embedding as prompt embedding, and do not assume base prompt embeddings. Fig. 4(d) shows that prompting performs better than MLP classifier, because prompt embeddings and claim token embeddings are input to claim encoder together, and the contextualized encoding helps exchange information between evidence and claim for accurate prediction. Base prompt embeddings are also helpful, since they store general fact-checking knowledge and help generalize across different claims.

**Case study.** To intuitively understand that our model captures useful information in referential documents, we conduct a case study and visually show the attention values between an evidence sentence and its cited papers in graph neural networks. Fig. 5 shows that the highest attention scores appear between the evidence sentence and referential documents that indeed contain useful information. This visualization verifies that referential documents are crucial to improve claim verification.

## 5 Conclusion

We propose a context- and reference-augmented reasoning and prompting model for fact-checking. To model contextual and referential documents, we construct a three-layer graph with intra- and cross-layer reasoning. To integrate evidence into claims, we design evidence-conditioned prompting, which produces unique prompt embeddings for each claim. A future work is to extend three-layer graph to a multi-modal graph for fact-checking.

#### Acknowledgments

This work was in part supported by NSF awards #1934782 and #2114824. Some of the research results were obtained using computational resources provided by NAIRR award #240336.

Figure 4: Model analysis on Check-COVID and FEVEROUS-S.

Figure 5: Case study on BearFact dataset.

Figure 3: Few-shot veracity prediction with different number of shots.

 



## Limitations

Here we identify two limitations of our work in terms of dataset and evidence type.

**Dataset.** Our model is proposed to incorporate contextual and referential documents of evidence sentences. We assume that the contextual and referential documents of evidence sentences are available in the dataset, or the dataset provides identifiers for evidence sentences, such as PubMed ID, so that we can use these identifiers to search their contextual and referential documents online. In Appendix B, we provide details on how to use identifiers to obtain contextual and referential documents. If the given dataset does not provide contextual or referential documents, or the identifiers of evidence sentences are not available, our model will reason within evidence sentences for fact-checking.

**Evidence type.** Following existing textual fact-checking models, we propose our model to reason over textual evidence sentences only. Our model is not proposed for tabular or multi-modal evidence, thus cannot reason over these types of evidence for fact-checking. One potential future work would be to extend our three-layer evidence graph to a multi-modal graph for evidence reasoning.

## Ethics Statement

We do not foresee any undesired implications stemming from our work. Conversely, we hope that our work can advance AI Ethics research.

## References

* Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: SS1.
* Aly et al. (2021)FEVEROUS: Fact extraction and VERification over unstructured and structured information. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, Cited by: SS5.
* T. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. (2020)Language models are few-shot learners. Advances in neural information processing systems33, pp. 1877-1901. External Links: ISSN 1674-709X, Link, Document Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)Bert: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171-4186. Cited by: SS1.
* T. Gao, A. Fisch, and D. Chen (2021)Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pp. 3816-3830. Cited by: SS1.
* Y. Gu, R. Tinn, H. Cheng, M. Lucas, N. Usuyama, X. Liu, T. Naumann, J. Gao, and H. Poon (2021)Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH)3 (1), pp. 1-23. Cited by: SS1.
* Z. Guo, M. Schlichtkrull, and A. Vlachos (2022)A survey on automated fact-checking. Transactions of the Association for Computational Linguistics10, pp. 178-206. Cited by: SS1.
* W. Hamilton, Zhitao Ying, and J. Leskovec (2017)Inductive representation learning on large graphs. Advances in neural information processing systems30, pp. 754-763. Cited by: SS1.
* L. Hu, T. Yang, L. Zhang, W. Zhong, D. Tang, C. Shi, N. Duan, and M. Zhou (2021)Compare to the knowledge: graph neural fake news detection with external knowledge. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pp. 754-763. Cited by: SS1.
* Bowen Jin, Wentao Zhang, Yu Zhang, Yu Meng, Xinyang Zhang, Qi Zhu, and Jiawei Han (2023)Patton: language model pretraining on text-rich networks. In The 61st Annual Meeting Of The Association For Computational Linguistics, Cited by: SS1.
* N. Lee, Y. Bang, A. Madotto, and P. Fung (2021)Towards few-shot fact-checking via perplexity. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1971-1981. Cited by: SS1.
* B. Lester, R. Al-Rfou, and N. Constant (2021)The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059. Cited by: SS1.
* X. L. Li and P. Liang (2021)Prefix-tuning: optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pp. 4582-4597.



Xiangci Li and Nanyun Peng. 2021. A paragraph-level multi-task learning model for scientific fact-verification.
* Liu et al. (2022) Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_, pages 61-68.
* Liu et al. (2023) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2023. Gpt understands, too. _AI Open_.
* Liu et al. (2020) Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2020. Fine-grained fact verification with kernel graph attention network. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7342-7351.
* Lo et al. (2020) Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel S Weld. 2020. S2orc: The semantic scholar open research corpus. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4969-4983.
* Ousidhoum et al. (2022) Nedjima Osudhoum, Zhangdie Yuan, and Andreas Vlachos. 2022. Varifocal question generation for fact-checking. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 2532-2544.
* Pan et al. (2023) Liangming Pan, Xiaobao Wu, Xinyuan Lu, Anh Tuan Luu, William Yang Wang, Min-Yen Kan, and Preslav Nakov. 2023. Fact-checking complex claims with program-guided reasoning. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 6981-7004.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of machine learning research_, 21(140):1-67.
* Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. _Foundations and Trends@ in Information Retrieval_, 3(4):333-389.
* Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4222-4235.
* Si et al. (2023) Jiasheng Si, Yingjie Zhu, and Deyu Zhou. 2023. Exploring faithful rationale for multi-hop fact verification via salience-aware graph learning. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 37, pages 13573-13581.
* Subramanian and Lee (2020) Shyam Subramanian and Kyumin Lee. 2020. Hierarchical evidence set modeling for automated fact extraction and verification. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7798-7809.
* Tan et al. (2022) Zhixing Tan, Xiangwen Zhang, Shuo Wang, and Yang Liu. 2022. Msp: Multi-stage prompting for making pre-trained language models better translators. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_, pages 6131-6142.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. _Advances in neural information processing systems_, 30.
* Wadden et al. (2020) David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishizri. 2020. Fact or fiction: Verifying scientific claims. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7534-7550.
* Wadden et al. (2022) David Wadden, Kyle Lo, Lucy Lu Wang, Arman Cohan, Iz Beltagy, and Hannaneh Hajishirzi. 2022. Multivers: Improving scientific claim verification with weak supervision and full-document context. In _Findings of the Association for Computational Linguistics: NAACL 2022_, pages 61-76.
* Wang et al. (2023) Gengyu Wang, Kate Harwood, Lawrence Chillrud, Amith Ananthram, Melanie Subbiah, and Kathleen McKeown. 2023. Check-covid: Fact-checking covid-19 news claims with scientific evidence. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 14114-14127.
* Wen and Fang (2023) Zhihao Wen and Yuan Fang. 2023. Augmenting low-resource text classification with graph-grounded pretraining and prompting. In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_, pages 506-516.
* Whitehouse et al. (2022) Chenxi Whitehouse, Tillman Weyde, Pranava Madhyastha, and Nikos Komninos. 2022. Evaluation of fake news detection with knowledge-enhanced language models. In _Proceedings of the international AAAI conference on web and social media_, volume 16, pages 1425-1429.
* Whehrl et al. (2024) Amelie Whehrl, Yarik Menchaca Resendiz, Lara Grimminger, and Roman Klinger. 2024. What makes medical claims (un) verifiable? analyzing entity and relation properties for fact verification. In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics_, pages 2046-2058.
* Yang et al. (2017) Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, 

Guangzhong Sun, and Xing Xie. 2021. Graphformers: Gnn-nested transformers for representation learning on textual graph. In _Advances in Neural Information Processing Systems_, volume 34, pages 28798-28810. Curran Associates, Inc.
* Yang et al. (2024) Menglin Yang, Harshit Verma, Delvin Ce Zhang, Jiahong Liu, Irwin King, and Rex Ying. 2024. Hypformer: Exploring efficient transformer fully in hyperbolic space. In _Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_, pages 3770-3781.
* Zeng and Gao (2023) Fengzhu Zeng and Wei Gao. 2023. Prompt to be consistent is better than self-consistent? few-shot and zero-shot fact verification with pre-trained language models. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 4555-4569.
* Zeng and Gao (2024) Fengzhu Zeng and Wei Gao. 2024. Justilm: Few-shot justification generation for explainable fact-checking of real-world claims. _Transactions of the Association for Computational Linguistics_, 12:334-354.
* Zhang and Lauw (2020) Ce Zhang and Hady W Lauw. 2020. Topic modeling on document networks with adjacent-encoder. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 34, pages 6737-6745.
* Zhang et al. (2024a) Congzhi Zhang, Linhai Zhang, and Deyu Zhou. 2024a. Causal walk: Debiasing multi-hop fact verification with front-door adjustment. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 38, pages 19533-19541.
* Cez Zhang and Lauw (2021) Delvin Ce Zhang and Hady W Lauw. 2021. Semi-supervised semantic visualization for networked documents. In _Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part III 21_, pages 762-778. Springer.
* Zhang and Lauw (2023) Delvin Ce Zhang and Hady W Lauw. 2023. Topic modeling on document networks with dirichlet optimal transport barycenter. _IEEE Transactions on Knowledge and Data Engineering_.
* Zhang et al. (2024b) Delvin Ce Zhang, Menglin Yang, Rex Ying, and Hady W Lauw. 2024b. Text-attributed graph representation learning: Methods, applications, and challenges. In _Companion Proceedings of the ACM on Web Conference 2024_, pages 1298-1301.
* Zhang et al. (2023) Delvin Ce Zhang, Rex Ying, and Hady W Lauw. 2023. Hyperbolic graph topic modeling network with continuously updated topic tree. In _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_, pages 3206-3216.
* Zhang et al. (2021) Zhiwei Zhang, Jiyi Li, Fumiyo Fukumoto, and Yanming Ye. 2021. Abstract, rationale, stance: A joint model for scientific claim verification. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 3580-3586.
* Zhao et al. (2020) Chen Zhao, Chenyan Xiong, Corby Rosset, Xia Song, Paul Bennett, and Saurabh Tiwary. 2020. Transformer-xh: Multi-evidence reasoning with extra hop attention. In _International Conference on Learning Representations_.
* Zhong et al. (2020) Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2020. Reasoning over semantic-level graph for fact checking. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6170-6180.
* Zhou et al. (2019) Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2019. Gear: Graph-based evidence aggregating and reasoning for fact verification. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 892-901.
* Zhou et al. (2022a) Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022a. Conditional prompt learning for vision-language models. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pages 16816-16825.
* Zhou et al. (2022b) Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022b. Learning to prompt for vision-language models. _International Journal of Computer Vision_, 130(9):2337-2348.
* Zou et al. (2023) Anni Zou, Zhuosheng Zhang, and Hai Zhao. 2023. Decker: Double check with heterogeneous knowledge for commonsense fact verification. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 11891-11904.

## Appendix A Pseudo-code of Training Process

We summarize the training process at Algo. 1.

## Appendix B Dataset Preprocessing Details

Here we present details of dataset preprocessing.

**FEVEROUS1**[Aly et al., 2021] is a general-domain dataset, and each claim is annotated in the form of sentences and/or cells from tables in Wikipedia pages. In this paper we mainly focus on textual evidence sentences, thus we follow ProgramFC [Pan et al., 2023] and obtain claims that only require textual evidence for verification, and name this subset **EVERROUS-S**. Claims in this dataset have two labels only, SUPPORT and REFUTE. In the original dataset, each evidence sentence may contain hyperlinks to other Wikipedia pages, and such hyperlinks in sentences are indicated with double square brackets. We thus retrieve words or phrases inside double square brackets, and use them as entries to query Wikipedia dump 

to obtain their corresponding pages as referential documents. FEVEROUS uses the December 2020 dump, including 5.4 million full Wikipedia articles. If a Wikipedia page has overly many sentences, we reserve its top-20 sentences, since almost all the evidence sentences appear within top-20 sentences in FEVEROUS. Similarly, the full content of the Wikipedia page is contextual document of each evidence sentence. If a page has overly long content, we reserve its top-20 sentences.

**BearFact2**[Wuehrl et al., 2024] is a biomedical claim verification dataset. Evidence sentences are obtained from paper abstracts in PubMed database3. Original dataset does not provide evidence sentences for claims in NEI class. Thus we follow existing work Zeng and Gao (2023) and select evidence sentences that have the highest _tf-idf_ similarity with claims as their evidence. We consider the full abstract as the contextual document for each evidence sentence, as in MultiVerS Wadden et al. (2022). In addition, we use S2ORC Lo et al. (2020) to obtain cited papers with abstracts as referential documents. Specifically, the original dataset provides PubMed ID for each evidence sentence. We use PubMed IDs as identifiers to search in S2ORC database and obtain cited papers. If a paper has overly many citations, we reserve its top-20 citations to avoid data redundancy.

Footnote 3: [https://pubned.ncbi.nlm.nih.gov/](https://pubned.ncbi.nlm.nih.gov/)

**Check-COVID4** specifically focuses on COVID-19 claims taken from news articles. Each evidence sentence is from a paper abstract with CORD ID as identifier. We thus use CORD IDs to search in S2ORC database and obtain cited papers. Similarly, we consider the full abstract as contextual document. The original dataset provides sentences for claims in NEI class.

Footnote 4: [https://github.com/posuer/Check-COVID/tree/main/Check-COVID](https://github.com/posuer/Check-COVID/tree/main/Check-COVID)

**SeiFact5**Wadden et al. (2020) is another biomedical fact-checking dataset with sentences in paper abstracts as evidence. Similarly, the full content of the abstract is considered as contextual document. In addition, each evidence sentence is coupled with S2ORC ID, which is used to obtain its citations using S2ORC database. The original dataset does not have sentences for claims in NEI class. Thus we follow the original paper Wadden et al. (2020) and choose top-3 sentences in the same abstract with the highest _tf-idf_ similarity with the claim as evidence.

Footnote 5: [https://github.com/allenai/scifact/tree/master](https://github.com/allenai/scifact/tree/master)