Document-level Claim Extraction and Decontextualisation for Fact-Checking (2024)

Abstract: Selecting which claims to check is a timeconsuming task for human fact-checkers, especially from documents consisting of multiple sentences and containing multiple claims. However, existing claim extraction approaches focus more on identifying and extracting claims from individual sentences, e.g., identifying whether a sentence contains a claim or the exact boundaries of the claim within a sentence. In this paper, we propose a method for documentlevel claim extraction for fact-checking, which aims to extract check-worthy claims from documents and decontextualise them so that they can be understood out of context. Specifically, we first recast claim extraction as extractive summarization in order to identify central sentences from documents, then rewrite them to include necessary context from the originating document through sentence decontextualisation. Evaluation with both automatic metrics and a fact-checking professional shows that our method is able to extract check-worthy claims from documents more accurately than previous work, while also improving evidence retrieval.

heading: 1 Introduction
text: Human fact-checkers typically select a claim in the beginning of their day to work on for the rest of it. Claim extraction (CE) is an important part of their work, as the overwhelming volume of claims in circulation means the choice of what to fact-check greatly affects the fact-checkers’ impact (Konstantinovskiy et al., 2021). Automated approaches to this task have been proposed to assist them in selecting check-worthy claims, i.e., claims that the public has an interest in knowing the truth (Hassan et al., 2017a; Guo et al., 2022).
Existing CE methods mainly focus on detecting whether a sentence contains a claim (Reddy et al., 2021; Nakov et al., 2021b) or the boundaries of the claim within a sentence (Wührl and Klinger, 2021; Sundriyal et al., 2022). In real-world scenarios
though, claims often need to be extracted from documents consisting of multiple sentences and containing multiple claims, not all of which are relevant to the central idea of the document, and verifying all claims manually or even automatically would be inefficient.
Moving from sentence-level CE to documentlevel CE is challenging; we illustrate this with the example in Figure 1. Sentences in orange are claims selected by a popular sentence-level CE method, Claimbuster (Hassan et al., 2017b), that are worth checking in principle but do not always relate to the central idea of the document, and multiple sentences with similar claims are selected, which would not all need to be fact-checked (e.g., sentences 1 and 6).
Claims extracted for fact-checking are expected to be unambiguous (Lippi and Torroni, 2015; Wührl and Klinger, 2021), which means that they cannot be misinterpreted or misunderstood when they are considered outside the context of the document they were extracted from, consequently allowing them to be fact-checked more easily (Schlichtkrull et al., 2023). Figure 1 shows an example of claim decontextualisation, where the claim “Bird is scrapping thousands of e-scooters in the Middle East ...... ” requires coreference resolution to be understood out of context, e.g., “Bird” refers to “California scooter sharing start-up Bird”. However, existing CE methods primarily focus on extracting sentence-level claims (i.e., extracting sentences that contain a claim) from the original document (Reddy et al., 2021) and ignore their decontextualisation, resulting in claims that are not unambiguously understood and verified.
To address these issues, we propose a novel method for document-level claim extraction and decontexualisation for fact-checking, aiming to extract salient check-worthy claims from documents that can be understood outside the context of the document. Specifically, assuming that salient
11943
https://web.archive.org/web/20210722180850/https://misbar.com/en/factcheck/2020/06/18/arebird-e-scooters-leaving-the-middle-east
claims are derived from central sentences, i) we recast the document-level CE task into the extractive summarization task to extract central sentences and reduce redundancy; ii) we decontextualise central sentences to be understandable out of context by enriching them with the necessary context; iii) we introduce a QA-based framework to obtain the necessary context by resolving ambiguous information units in the extracted sentence.
To evaluate our method we derive a CE dataset3
containing decontextualised claims from AVeriTeC (Schlichtkrull et al., 2023), a recently proposed benchmark for real-world claim extraction and verification. Our method achieves a Precision@1 score of 47.8 on identifying central sentences, a 10% improvement over Claimbuster. This was verified further by a fact-checking professional, as the sentences returned by our method were deemed central to the document more often, and check-worthy more often than those extracted by Claimbuster. Additionally, our method achieved a character-level F score (chrF) (Popović, 2015) of 26.4 against gold decontextualised claims, outperforming all baselines. When evaluated for evidence retrieval potential, the decontextualised claims obtained by enriching original sentences with the necessary context, are better than the original claim sentences, with an average 1.08 improvement in precision.
1https://www.cnbc.com/2020/06/03/ bird-circ-scooters-middle-east.html
2https://misbar.com/en/factcheck/2020/06/18/ are-bird-e-scooters-leaving-the-middle-east
3https://github.com/Tswings/AVeriTeC-DCE

heading: 2 Related Work
text: Claim Extraction Claim extraction is typically framed either as a classification task or claim boundary identification task. The former framing focuses on detecting whether a given sentence contains a check-worthy claim. Claimbuster (Hassan et al., 2017b), the most popular method in this paradigm, computes the score of how important a sentence is to be fact-checked. More similar to our work are studies that formulate the task of checkworthy claim detection as a sentence ranking task. For example, Zhou et al. (2021) present a sentencelevel classifier by combining a fine-tuned hatespeech model with one dropout layer and one classification layer to rank sentences. However, these methods were not able to handle the challenges of document-level claim extraction, e.g., avoid redundant claim sentences.
The framing of claim extraction as boundary identification focuses on detecting the exact claim boundary within the sentence. Nakov et al. (2021a) propose a BERT-based model to perform claim detection (Levy et al., 2014) by identifying the boundaries of the claim within the sentence. Sundriyal et al. (2022) tackle claim span identification as a token classification task for identifying argument units of claims in the given text. Unlike the above methods, where the claims are extracted from given sentences, our work aims to extract salient checkworthy claims from documents, thus addressing the limitations of sentence-level methods in extracting salient claim sentences and avoiding redundancy.
Decontextualisation Choi et al. (2021) propose two different methods for decontextualisation, based on either a coreference resolution model or a seq2seq generation model. Both methods use the sentences in the paragraph containing the target sentence as context to rewrite it. Newman et al. (2023) utilize an LLM to generate QA pairs for each sentence by designing specific prompts, and then use an LLM with these QA pairs to rewrite each sentence. Sundriyal et al. (2023) propose to combine chain-of-thought and in-context learning for claim normalization. Unlike the above methods, we generate declarative sentences for potentially ambiguous information units in the target sentence based on the whole document, and combine them into context to rewrite the target sentence.

heading: 3 Method
text: As illustrated in Figure 2, our proposed documentlevel claim extraction framework consists of four components: i) Sentence extraction (§3.1); extracts the sentences related to the central idea of the document as candidate claim sentences; ii) Context generation (§3.2), extracts context from the doc-
ument for each candidate sentence; iii) Sentence decontextualisation (§3.3), rewrites each sentence with its corresponding context to be understandable out of context; iv) Check-worthiness estimation (§3.4), selects the final check-worthy claims from candidate decontextualised sentences.

heading: 3.1 Sentence Extraction
text: The claims selected by human fact-checkers are typically related to the central idea of the document considered. Thus we propose to model sentence extraction as extractive summarization. For this purpose, we concatenate all the sentences in the document into an input sequence, which is then fed to BertSum (Liu and Lapata, 2019), a document-level extractive summarization method trained on the CNN/DailyMail dataset. Specifically, given a document consisting of n sentences D = {s1, s2, ..., sn}, we first formulate the input sequence C as “[CLS] s1 [SEP] [CLS] s2 [SEP] ... [CLS] sn[SEP]”, where [CLS] and [SEP] denote the start and end token for each sentence, respectively, and then feed them into a pre-trained encoder BERT to obtain the sentence representation s. Finally, a linear layer on sentence representations
S = {s1, ..., si, ..., sn} is used to score sentences. S = BERT(C)
scorei = σ(W si + b0) (1)
where σ is a sigmoid function, si denotes the representation of the i-th [CLS] token, i.e., the representation of the i-th sentence, and scorei denotes the score of the i-th sentence. All sentences are constructed into an ordered set S = {s′1, ..., s′i, ..., s′n} according to their scores. Since all sentences are ranked by sentence-level scoring, some top-scoring sentences may have the same or similar meaning. To avoid redundancy, we add an entailment model DocNLI (Yin et al., 2021), a more generalizable model trained on five datasets from different benchmarks, on top of the output of BertSum to remove redundant sentences by calculating the entailment scores between sentences, e.g., we first remove the sentences that have an entailment relationship with the top-1 sentence in S, and then repeat this process for the remaining top-2/3/... sentence until we extract k central sentences. Following previous work (Liu and Lapata, 2019), we only select the top-k sentences with the highest scores in Equation 1 as candidate central sentences.
S′ = DocNLI(S) (2)
where S′ = {s′1, s′2, ..., s′k} is a set of central sentences that do not contain the same meaning.

heading: 3.2 Context Generation
text: After sentence extraction, the next step is to clarify the (possibly) ambiguous sentences in S′ by rewriting them with their necessary context. Unlike Choi et al. (2021) where the context consists of a sequence of sentences in the paragraph containing the ambiguous sentence, we need to consider the whole document, i.e., sentences from different paragraphs. We propose a QA-based context generation framework to produce a specific context for each ambiguous sentence, which contains three components: i) Question Generation: extracts potentially ambiguous information units from the sentence and generates questions with them as answers; ii) Question Answering: finds more information about ambiguous information units by answering generated questions with the whole document; iii) QAto-Context Generation: converts question-answer pairs into declarative sentences and combines them into context specific to the sentence. In the following subsections, we describe each component in detail.
Question Generation. To identify ambiguous information units in candidate central sentences, we first use Spacy4 to extract named entities, pronouns, nouns, noun phrases, verbs and verb phrases in the sentence i as the potentially ambiguous information units Ui = {u1i , u2i , ..., uji , ..., umi }, i ∈ [1, 2, ..., k], where uji denotes the j-th information unit of the i-th candidate sentence s′i.
Once the set of information units for a sentence Ui is identified, we then generate a question for each of them. Specifically, we concatenate uji and s′i in which u j i is located as the input sequence and feed it into QG (Murakhovs’ka et al., 2022), a question generator model trained on nine question generation datasets with different types of answers, to produce the question qji with u j i as the answer.
Qi = {qji }mj=1 = {QG(s′i, u j i )}mj=1 (3)
where Qi denotes the set of questions corresponding to Ui in the sentence s′i.
Question Answering. After question generation, our next step is to clarify ambiguous information units by answering corresponding questions with the document D. Specifically, following Schlichtkrull et al. (2023), we first use BM25 (Robertson et al., 2009) to retrieve evidence E related to the question qji from D, and then answer qji with E using an existing QA model (Khashabi et al., 2022) trained on twenty datasets that can answer different types of questions.
E = BM25(D, qji ) aji = QA(E, q j i )
(4)
where aji denotes a more complete information unit corresponding to uij , e.g., a complete coreference. We denote all question-answer pairs of the i-th sentence as Pi = {(q1i , a1i ), (q2i , a2i ), ..., (qmi , ami )}.
QA-to-Context Generation. After question answering, we utilize a seq2seq generation model to convert QA pairs Pi into the corresponding context C ′i. Specifically, we first concatenate the question qji and the answer a j i as the input sequence, and then output a sentence using the BART model (Lewis et al., 2019) finetuned on QA2D (Demszky et al., 2018). QA2D is a dataset with over 500k
4https://spacy.io
NLI examples that contains various inference phenomena rarely seen in previous NLI datasets. More formally,
s̃ji = BART(q j i , a j i ) (5)
where s̃ji is a declarative sentence corresponding to the information unit uji . Finally, all generated sentences are combined into high-quality context C ′i = {s̃1i , s̃2i , ..., s̃mi } corresponding to the information units Ui in sentence s′i, which is then used in the next decontextualisation step to enrich the ambiguous sentences.

heading: 3.3 Sentence Decontextualisation
text: Sentence decontextualisation aims to rewrite sentences to be understandable out of context, while retaining their original meaning. To do this, we use a seq2seq generation model T5 (Raffel et al., 2020) to enrich the target sentence with the context generated in the previous step for it. Specifically, we first formulate the input sequence as “[CLS] s̃1i [SEP] s̃ 2 i ...... [SEP] s̃ m i [SEP] s ′ i”, where s′i denotes the potential ambiguous sentence and [SEP] is a separator token between the context sentences generated. We then feed the input sequence to D (Choi et al., 2021), a decontextualisation model was trained on the dataset annotated by native speakers of English in the U.S. that handles various linguistic phenomena, to rewrite the sentence. Similarly, we set the output sequence to be [CAT] [SEP] y.
yi =    D(s′i, C ′ i), if CAT = feasible s′i , if CAT = infeasible
s′i , if CAT = unnecessary
(6)
where CAT = feasible or infeasible denotes that s′i can or cannot be decontextualised, CAT = unnecessary denotes that s′i can be understood without being rewritten, yi denotes the i-th decontextualised sentence.

heading: 3.4 Check-Worthiness Estimation
text: Unlike existing CE methods that determine whether a sentence is worth checking without considering the context, we estimate the check-worthiness of a sentence after decontextualisation because some sentences may be transformed from not checkworthy into check-worthy ones in this process. Specifically, we use a DeBERTa model trained on the ClaimBuster dataset (Arslan et al., 2020) to classify sentences into three categories: Check-worthy Factual Sentence (CFS), Unimportant Factual Sentence (UFS) and Non-Factual Sentence (NFS). Formally,
score(yi) = DeBERTa(class = CFS | yi) claim = argmax{score(yi)}ki=1 (7)
where score(yi) reflects the degree to which the decontextualised sentence yi belongs to CFS, and claim denotes that the final salient check-worthy claim that can be understood out of context.

heading: 4 Dataset
text: We convert AVeriTeC, a recently proposed dataset for real-world claim extraction and verification (Schlichtkrull et al., 2023), into AVeriTeC-DCE, a dataset for the document-level CE task. AVeriTeC is collected from 50 different fact-checking organizations and contains 4568 real-world claims. Each claim is associated with attributes such as its type, source and date. In this work, we mainly focus on the following attributes relevant to claim extraction: i) claim, the claim as extracted by the fact-checkers and decontextualised by annotators, and ii) source url: the URL linking to the original web article of the claim. The task of this work is to extract the salient check-worthy claims from the source url. We also consider whether claims need to be decontextualised when extracting them from documents, as this will directly affect the subsequent evidence retrieval and claim verification.
To extract claim-document pairs from AVeriTeC that can be used for document-level CE, we perform the following filtering steps: 1) Since we focus on the extraction of textual claims, we do not include source urls containing images, video or audio; 2) To extract the sentences containing the claims from source urls, we build a web scraper to extract text data in the source url as the document. We found that the attribute source url is not always available in samples, thus we only select those samples where the web scraper can return text data from source urls. We obtain a dataset AVeriTeC-DCE, containing 1231 available samples, for document-level CE. We do not divide the dataset into train, dev and test sets, as all models we rely on are pre-trained models (e.g., BertSum) and approaches that do not require training (e.g., BM25). Statistics for AVeriTeC-DCE are described in Table 1.

heading: 5 Experiments
text: Our approach consists of four components: sentence extraction, context generation, sentence decontextualisation and check-worthiness estimation. As such, we conduct separate experiments to evaluate them, as well as an overall evaluation for document-level CE.

heading: 5.1 Sentence Extraction
text: We compare our sentence extraction method, the combination of BertSum and DocNLI stated in Section 3.1, against other baselines through automatic evaluation and human evaluation.
Baselines 1) Lead sentence: the lead (first) sentence of most documents is considered to be the most salient, especially in news articles (Narayan et al., 2018); 2) Claimbuster (Hassan et al., 2017b): we use this well-established method to compute the check-worthiness score of each sentence and we rank sentences based on their scores; 3) LSA (Gong and Liu, 2001): a common method of identifying central sentences of the document using the latent semantic analysis technique; 4) TextRank (Mihalcea and Tarau, 2004): a graph-based ranking method for identifying important sentences in the document; 5) BertSum (Liu and Lapata, 2019): a BERT-based document-level extractive summarization method for ranking sentences.
Automatic Evaluation Since the central sentences of documents are not given in AVeriTeC,
we cannot evaluate the extracted central sentences by exact matching. Thus, we instead rely on the sentence that has the highest chrF (Popović, 2015) with the claim, as the claim is the central claim annotated by human fact-checkers. We use Precision@k as the evaluation metric, which denotes the probability that the first k sentences in the extracted sentences contain the central sentence. Table 2 shows the results of different sentence extraction methods. We can see that our method outperforms all baselines in identifying the central sentence, achieving a P@1/P@3/P@5/P@10 score of 47.8/63.1/68.6/73.8, which indicates that the combination of the extractive summarization (BertSum) and entailment model (DocNLI) can better capture the central sentences and avoid redundant ones. We found that the common extractive summarization methods (e.g., Lead Sentence, TextRank and BertSum) are better than Claimbuster on P@1, confirming what we had stated in the introduction, that sentence-level CE methods have limitations when they are applied at the document-level CE. Moreover, we observe that the lead sentence achieves a P@1 score of 42.3, indicating that there is a correlation between the sentences selected for factchecking and the lead sentence that often served as the summary. We list the source URLs of the samples for claim extraction in Appendix A1.
Human Evaluation To further compare sentences extracted by our method and Claimbuster,
we asked a fact-checking professional to evaluate the quality of extracted sentences on the following two dimensions: 1) IsCheckWorthy: is the sentence worth checking? 2) IsCentralClaim: is the sentence related to the central idea of the article? We randomly select 50 samples, each containing at least 5 sentences. For simplicity, we only select the top-1 sentence returned by each method for comparison. As shown in Table 3, we observed that 68% of the central sentences extracted by our method are related to the central idea of the document compared to 24% of Claimbuster, which further supports our conclusion obtained by automatic evaluation, i.e., the sentences extracted by our method were more often central to the document, and more often check-worthy than those that extracted by Claimbuster. This indicates that when identifying salient check-worthy claims from documents, it is not enough to consider whether a sentence is worth checking at the sentence level, but also whether the sentence is related to the central idea of the document. Thus, we believe that the claims related to the central idea of the document are the ones that the public is more interested in knowing the truth.

heading: 5.2 Decontextualisation
text: To evaluate the effectiveness of decontextualisation on evidence retrieval, for a fair comparison, we select the sentence that has the highest chrF with the claim as the best sentence as we considered in Section 5.1, and conduct a comparison between it and its corresponding decontextualised sentence. The evidence set used for evaluation is retrieved from the Internet using the Google Search API given a claim, each containing gold evidence and additional distractors (Schlichtkrull et al., 2023). We use Precision@k as the evaluation metric.
Baselines 1) Coreference model: decontextualisation by replacing unresolved coreferences in the target sentence, e.g., (Joshi et al., 2020); 2) Seq2seq model: decontextualisation by rewriting the target sentence with necessary context (Choi et al., 2021).
Retrieval-based Evaluation For a given claim different decontextualisations could be considered correct, thus comparing against the single reference in AVeriTeC would be suboptimal. Thus we prefer to conduct the retrieval-based evaluation, assuming that better decontextualisation improves evidence retrieval, as it should provide useful context for fact-checking. Following previous work (Choi et al., 2021), we compare our QA-based decontextualisation method against other baselines through retrieval-based evaluation. We use BM25 as the retriever to find evidence with different sentences as the query. Table 4 shows the results for evidence retrieval with different decontextualised sentences on the dev set of AVeriTeC-DCE (AVeriTeC only publicly released the train and dev sets). We found that our method outperforms all baselines, and improves the P@3/P@5/P@10 score over the original sentence by 1.42/0.82/0.99, achieving an average 1.08 improvement in precision. After further analysis, we found that only 21/149 sentences are decontextualised by our method, and 17/21 of these sentences obtain better evidence retrieval, with an average improvement of 1.21 in precision over the original sentences, proving that decontextualisation enables evidence retrieval more effectively.
Case Study Figure 3 illustrates three case studies of sentence decontextualisation. The first case is an example that requires coreference resolution. To make the sentence understandable out of context, these words (e.g., “He”, “their”) need to be rewritten with the context. After decontextualisa-
tion, we can see that “He” is rewritten to “President Obama”, which helps us understand the sentence better without context. As for “their”, we cannot decontextualise it because there is no information about this word in the document. This supports the claim that providing a high-quality context is necessary for better decontextualisation. The second case is an example that requires global scoping, which requires adding a phrase (e.g., prepositional phrase) to the entire sentence to make it better understood. In this case, we add “on Washington D.C.” as a modifier to “During the attack” to help us understand where the attack took place. The third case is an example that requires a bridge anaphora, where the phrase noun “The government” becomes clear by adding a modifier “India”. In summary, decontextualising the claim is helpful for humans to better understand the claim without context.

heading: 5.3 Document-level Claim Extraction
text: In this section, we put four components together to conduct an overall evaluation for document-level CE, e.g., extracting the claim in green from the document in Figure 1. We first select top-3 sentences returned by different sentence extraction methods as candidate central sentences, and then feed them into our decontextualisation model to obtain decontextualised claim sentences, finally use a claim check-worthiness classifier to select the final claim. We evaluate performance by calculating the similarity between our final decontextualised claim sentence and the claim decontextualised by factcheckers. We use the chrF as the evaluation metric. The chrF computes the similarity between texts using the character n-gram F-score. Other metrics are reported in Appendix A2.
Statistics for decontextualisation are described in Table 5, we observe that 122 out of 1231 (10%) sentences can be decontextualised (feasible); 122
out of 1231 (10%) sentences cannot be decontextualised (infeasible); and 987 out of 1231 (80%) sentences can be understood without being decontextualised (unnecessary), including the lead sentence. Since the lead sentence of the document is often considered to be the most salient, we do not decontextualise the lead sentence. In Table 6, we show the results of original sentences and decontextualised sentences for claim extraction, and our method achieves a chrF of 26.4 on gold claims decontextualised by the fact-checkers, outperforming all baselines. We observe that the performance of claim extraction and sentence extraction is positively correlated, i.e., the closer the extracted sentence is to the central sentence, the more similar the extracted claim is to the claim, which supports our assumption that salient claims are derived from central sentences. For this reason, the performance of our document-level CE is limited by the performance of sentence extraction, i.e., if our sentence extraction method cannot find the gold central sentence, decontextualisation may not improve the performance of CE, and may even lead to a decrease in the performance of CE due to noise caused by decontextualisation.
Moreover, empirically, we found that central sentences led to improved overall performance. This might be a consequence of the dataset – social media sites such as Twitter or Facebook are common
sources of claims in AVeriTeC (along with more traditional news), and a Twitter or Facebook thread often contains only a few key points. AVeriTeC was collected by reverse engineering claims which fact-checkers from around the world chose to work on. As such, the distribution of source articles represents what journalists found to be check-worthy and chose to work on. Our work as such reflects the contexts wherein real-world misinformation appears (but indeed, may have a bias towards what works well in those contexts).
To further verify the effectiveness of our method, we conduct a comparison on the document-level CE dataset (CLEF-2021, subtask 1B (Shaar et al., 2021)) using our method and Claimbuster. Table 7 shows the results of two different methods for identifying check-worthy claims on the dev set of subtask 1B. We observe that our method outperforms Claimbuster on P@1/3/5/10, indicating that our document-level CE method can better identify check-worthy claims than Claimbuster (sentencelevel CE). This supports our conclusion that the document-level check-worthy claims extracted by our method are the claims that the public is more interested in knowing the truth.

heading: 6 Conclusions and Future work
text: This paper presented a document-level claim extraction framework for fact-checking, aiming to extract salient check-worthy claims from documents that can be understood out of context. To extract salient claims from documents, we recast the claim extraction task as the extractive summarization task to select candidate claim sentences. To make sentences understandable out of context, we introduce a QA-based decontextualisation model to enrich them with the necessary context. The experimental results show the superiority of our method over previous methods, including document-level claim extraction and evidence retrieval, as indicated by human evaluation and automatic evaluation. In future work, we plan to extend our document-level claim extraction method to extract salient checkworthy claims from multimodal web articles.

heading: Acknowledgements
text: This project has received funding from the European Research Council
(ERC) under the European Union’s Horizon 2020 Research and Innovation programme grant AVeriTeC (Grant agreement No. 865958). We thank David Corney from Full Fact for his help with evaluating the output of our models, and Yulong Chen for his helpful comments and discussions. We would also like to thank the anonymous reviewers for their helpful questions and comments that helped us improve the paper.
Limitations
While our method has demonstrated superiority in extracting salient check-worthy claims and improving evidence retrieval, we recognize that our method is not able to decontextualise all ambiguous sentences, particularly those that lack the necessary context in the source url. Also, human fact-checkers have different missions, thus checkworthiness claims to one fact-checking organization may not be check-worthiness to another organization (i.e., some organizations check parody claims or claims from satire websites, while others do not). Furthermore, since the documents we use are extracted from the source url, a powerful web scraper is required when pulling documents from source urls. Moreover, our method assumes salient claims are derived from central sentences. Although this assumption is true in most cases, it may be inconsistent with central claims collected by human fact-checkers. Besides, we use the chrF metric to calculate the similarity between claims extracted from the source url and the gold claim, while gold claims are decontextualised by the factcheckers with fact-checking articles and may contain information that is not in the original article, thus the metrics used to evaluate document-level claims are worth further exploring.
Ethics Statement
We rely on fact-checks from real-world factcheckers to develop and evaluate our models. Nevertheless, as any dataset, it is possible that it contains biases which influenced the development of our approach. Given the societal importance of factchecking, we advise that any automated system is employed with human oversight to ensure that the fact-checkers fact-check appropriate claims.

heading: A1 Statistic of Source URLs
text: We described the statistic of the source URLs of the samples for document-level CE in Table A1.
URL #sample
twitter.com 241 facebook.com 235 perma.cc 63 channelstv.com 37 aljazeera.com 34 president.go.ke 27 gov.za 25 instagram.com 18 c-span.org 16 factba.se 13 axios.com 12 youtu.be 12 rumble.com 12 abcnews.go.com 11 rev.com 11 cnn.com 10 news24.com 10 punchng.com 10 washingtonpost.com 10 cbsnews.com 8 foxnews.com 8 misbar.com 7 thegatewaypundit.com 7 politifact.com 7 nypost.com 7 nbcnews.com 7 telegraph.co.uk 7 wisn.com 6 tatersgonnatate.com 6 bustatroll.org 6 dailymail.co.uk 5 whitehouse.gov 5
Table A1: Statistic of the source URLs of the samples for document-level CE. We only list URLs with a total number number greater than 5.

heading: A2 Evaluation Metrics
text: We use the following metrics to assess the similarity between the claim decontextualised by our method and the claim decontextualised by fact-checkers. SARI (Xu et al., 2016) is developed to compare the claim with the reference claim by measuring the goodness of words that are added, deleted and kept. BERTScore (Zhang et al., 2019) is utilized to compute the semantic overlap between the claim and the reference claim by sentence representation. Since most claims in AVerTeC are decontextualised by fact-checkers with fact-checking articles, they may contain some information that is not in the source url, making it challenging for SARI and BERTScore to be used as evaluation metrics in this task. Thus, we use the chrF as our main evaluation metric for claim extraction.
Method Sentence⋆ Dec. Sentence⋆
SARI BERTScore chrF SARI BERTScore chrF
Claimbuster 6.23 82.7 24.3 6.24 82.8 24.5
Lead Sentence 6.41 83.4 23.8 - - - LSA 5.56 83.2 24.1 5.57 83.2 24.3 TextRank 6.60 83.1 24.5 6.61 83.1 25.4 BertSum 6.54 83.6 25.6 6.55 83.6 25.9
Ours 6.56 83.7 25.9 6.70 83.8 26.4
Table A2: Results of Document-level CE on three different metrics.
A3 Implementation Details
All models we use in this paper are pre-trained models (e.g., BertSum) or approaches that do not require training (e.g., BM25). The hyperparameters of each model can be found in the original paper. To help readers reproduce our method, we have released our code on GitHub5.

heading: A4 ChatGPT for Decontextualisation
text: To verify how well ChatGPT would do on decontextualisation, we use ChatGPT to decontextualise three claim sentences in Figure 3. The ChatGPT prompt for decontextualisation is as follows:
ChatGPT Prompt
Claim: [claim] Context: [context]
To rewrite the Claim to be understandable out of context based on the Context, while retaining its original meaning.
Decontextualised sentences produced by ChatGPT: • Sentence 1: Barack Obama publicly expressed sympathy for ISIS and hinted at providing them with American resources during his 2008 State of the Union address. • Sentence 2: During a specific event, there was a delay in obtaining approval from a certain authority for assistance requested by the Capitol Police. • Sentence 3: The Indian government lacks adequate storage facilities for managing the large surplus of grain it possesses.
5https://github.com/Tswings/AVeriTeC-DCE
From the results, we can see that ChatGPT can produce well-formed claims that can be understood out of context, but it tends to rephrase the claim. In AVeriTeC (Appendix J.3.1), the decontextualised claims are required to be as close as possible to their original form. Our method tends not to change the original claims, but to rewrite only the ambiguous information units in claims, thus our generated claims are closer to the claims decontextualised by annotators than ChatGPT.

heading: Future Work
text: In future work, we plan to extend our document-level claim extraction method to extract salient checkworthy claims from multimodal web articles