$\textit{From Chaos to Clarity}$: Claim Normalization to Empower Fact-Checking
Authors: Megha Sundriyal
Tanmoy Chakraborty
Preslav Nakov
Venue: EMNLP 2023 Findings (2023)
Keywords: Claim Normalization
Social Media
Claims
Misinformation
TL;DR: We introduce a novel task, Claim Normalization (aka ClaimNorm), which aims to decompose complex and noisy social media posts into more straightforward and understandable forms, termed normalized claims.

Abstract: With the rise of social media, users are exposed to many misleading claims. However, the pervasive noise inherent in these posts presents a challenge in identifying precise and prominent claims that require verification. Extracting the important claims from such posts is arduous and time-consuming, yet it is an underexplored problem. Here, we aim to bridge this gap. We introduce a novel task, Claim Normalization (aka ClaimNorm), which aims to decompose complex and noisy social media posts into more straightforward and understandable forms, termed normalized claims. We propose CACN, a pioneering approach that leverages chain-of-thought and claim check-worthiness estimation, mimicking human reasoning processes, to comprehend intricate claims. Moreover, we capitalize on the in-context learning capabilities of large language models to provide guidance and to improve claim normalization. To evaluate the effectiveness of our proposed model, we meticulously compile a comprehensive real-world dataset, CLAN, comprising more than 6k instances of social media posts alongside their respective normalized claims. Our experiments demonstrate that CACN outperforms several baselines across various evaluation measures. Finally, our rigorous error analysis validates CACN’s capabilities and pitfalls.

From Chaos to Clarity: Claim Normalization to Empower Fact- Pact-Checking
Megha Sundriyal Tanmoy Chakraborty", Preslav Nakov
Indraprastha Institute of Information Technology Delhi, India 2 Indian Institute of Techholoog Delhi, India
Mohammed Bin Zayed University of Artificial Intelligence
meghas@iiitd.acc ac. in, tanchakēi itd. ac. in, preslav. www.wvnbbzaii.ccc.mmmmmm
Abstract
Cyanocabalaminn is synthetic form Vitamin 612.
With the rise of social media, users are exposed many misleading claims. However, the per- vasive noise inherent in these posts presents a challenge in identifying precise and promi- nent claims that require verification. Extracting the important claims from such posts is ardu- ous and cine-consuming, yet is an underex- plored problem. Here, aim to bridge this gap. We introduce novel task, Claim Normal- ization (aka ClaimNorm), which aims de- compose complex and noisy social media posts into more straightforward and understandable forms, termed normalized claims. We propose CACN, pioneering approach that leverages chain-of-thought and claim worth-sss estimation, mimicking human reasoning pro- cesses, to comprehend intricate claims. More- over, we capitalize on the in-context learning capabilities of large language models provide guidance and to improve claim normalization. To evaluate the effectiveness of our proposed model, we meticulously compile a comprehen- sive world dataset, CLAN, comprising more than 6k instances of social media posts along- side their respective normalized claims. Our ex- periments demonstrate that CACN outperforms several baselines across various evaluation mea- sures. Finally, our rigorous eиror analysis vali- dates CACN's capabilities and pitfalls.'
most common cost-affactive
B12.
Syanocobalaminn most often found In supploments & some fortificd foods.
Media
It's absolute garbage bargagg your body. you'rr on B22 supolements, throw them away.
Cyanzcobulantin which is the most common form Normalized Vitamin 812, toel.
Claim
They gave you MASK cover your face AND then gave you George Flayd's signature phrass cen't breath"...
Social
Media
realiaatio that EVERYTHING on Flayd and Convid never existed
years
struggle
Normalized Claim
George Floyd and CZV D-10 never exited
Figure Illustration our proposed Claim Normaliza- tion task, highlighting the normalized claims authored by fact-checkers for social media posts from distinct social media platforms.
False claims are an intrinsic aspect of fabricated news, rumors, propaganda, and misinformation. Journalists and fact-checkers work tirelessly to as- sess the factuality of such claims in spoken and/or written form, sifting through avalanche of claims and pieces of evidence determine the truth. To further address this pressing issue, several indepen- dent fact-checking organizations have emerged in recent years, such as Snopes, FullFact, and Poli- tiFact, which play a crucial role in verifying the accuracy of online content. However, the rate at which online information is being disseminated far outpaces the capacity of fact-checkers, making it difficult to verify every single claim. This, in turn, leaves numerous unverified claims circulating on- line, potentially reaching millions before they can be verified.
1 Introduction
Social media have enabled a new way commu- nication, breaking down geographical barriers and bringing unprecedented opportunities for knowl- edge exchange. However, this has also presented a growing threat to society, c.g., during the 2016 US Presidential Election (Allcott and Gentzkow, 2017), the COVID-19 pandemic (Alam al., 2021; Rocha et al., 2021; Nakov al., 2022a), the Ukraine- Russia conflict (Khaldarova and Pantti, 2016), etc.
Phttps://www swwpes.oppccoooooo com hhttps://ullfactt.om "https://ww.poll http://www.oliiiffcc..oo.o.
'Wo release our dataset and code at https://githhn com/LCS2-IIITD/CAA EMNLP- 2823

---

While the complete automation of the fact- checking pipeline may pose hazards to account- ability and reliability, several recent studies have targeted identifying downstream tasks suitable for automation, such as detecting claims (Daxenberger et al., 2017; Chakrabarty et al., 2019; Gangi Reddy et al., 2022b), evaluating their worthiness for fact- checking (Gencheva et al., 2017: Jaradat et al., 2018; Wright and Augenstein, 2020), making sure they were not fact-checked before (Shaar et al., 2020, 2022a.b; Hardalov et al., 2022), and vali- dating them by retrieving relevant shreds of evi- dence (Zhi et al., 2017; Hanselowski et al., 2018; Soleimani et al., 2020; Pan et al., 2023). See also a recent survey automated fact-checking for as- sisting human fact-checkers (Nakov et al., 2021 a).
Our contributions are as follows:
We introduce the novel task of claim normaliza- tion, which seeks to detect the core claim in a given piece of text.
We present a meticulously curated high-quality dataset specifically tailored for claim normaliza- tion of noisy social media posts.
We propose a robust framework for claim nor- malization, incorporating chain-of-thought, in- context learning, and claim check-worthiness es- timation to comprehend intricate claims.
We conduct a thorough error analysis, which can inform future research.
Related Work
In light of the growing challenges faced by fact- checkers in verifying the factuality of social media claims, we propose the novel task of claim normal- ization. This task aims extract and to simplify the central assertion made in a long, noisy social media post. This can improve the efficacy and curtail the workload of fact-checkers while main- taining high precision and conscientiousness. We provide a more detailed explanation of why the claim normalization task is essential and illustrate its significance in Appendix A.I.
Claim Analysis. Previous work has focused on distinct aspects of claims, including claim detec- tion (Daxenberger et al., 2017: Gupta et al., 2021; Sundriyal et al., 2021; Gangi Reddy et al., 2022a,b), claim theck-worthiness, estimation (Hassan et al., 2017: Gencheva et al., 2017; Barrón-Cedeño al.. 2018; Jaradat al., 2018; Vasileva et al., 2019; Barrón-Cedeño et al., 2020; Konstantinovskly al.. 2021), claim span identification (Sundriyal et al., 2022), etc. By curating the AAWD corpus, Bender et al. (2011) pioneered the efforts in claim detec- tion, the foremost step in the fact-checking tasks. Following this, linguistically motivated features, including sentiment, syntax, context-free gram- mar, and parse trees, were frequently used (Dax- enberger et al., 2017; Lippi and Torroni, 2015; Levy et al., 2017; Sundriyal et al., 2021). Re- cently, large language models (LLMs) have also been used for claim detection (Chakrabarty et al.. 2019; Barrón-Cedeño et al., 2020; Gupta et al., 2021; Gangi Reddy et al., 2022a.b).
In our problem formulation, given input so- cial media claim, the system needs to simplify it in concise form that contains the post's central assertion that fact-checkers can easily verify. To better understand our motivation, we illustrate the task in Figure The first social media post reads, Cyanocobalaminn is a synthetic form of Vitamin 1f you're re on B12 supplements, throw them away.. This post contains some extraneous infor- mation that has no relevance for fact-checkers. As result, they distil the information and summarize it as, "Cyanocobalamin, the most common form of Vi- tamin B72, is toxic.' Fact-checkers tasked with veri- fying the accuracy of such noisy posts need to read through them and condense their content to obtain a concise claim that can be easily fact-checked. Un- fortunately, this process can be exceedingly time- consuming. By automating the claim normalization process, fact-checkers Another aspect is that fact-checkers often choose what fact-check based on the virality of a claim, for which they need to be able to recognize when the same claim appears in a slightly different form, and claim normalization is essential for this.
Most previous work claim detection and ex- traction primarily concentrated on adapting to text that comes from similar distributions or topics. Moreover, it often relied on well-structured for- mal writing. In contrast, our objective to develop a system that specifically addresses the challenges posed by posts in social media and aims to ex- tract the central claim in a more simplified manner, which goes beyond extracting text subspan in a cial media post and aims at abstractive claim extrac- tion that mimics what professional fact-chhccaccchhkkke fact-checkers do. To the best of our knowledge, we are the first address the task of claim extraction in this very practical formulation.
work more efficiently.

---

Text Summarization. The task of claim normal- ization is closely related to the task of text summa- rization. In the latter, given a lengthy document, the goal is to summarize it into a much shorter sum- mary. Previous work on text summarization has explored various approaches, including large pre- trained seq2seq models to generate high-quality summaries (Radford et al., 2019; Lewis et al., 2020; Raffel al., 2020).
3 Dataset
Existing text summarization datasets have not specifically addressed the need for claim-oriented summaries. To address this gap, we propose a novel dataset CLAN (Claim Normalization), consist- ing of fact-checked social media posts paired with concise claim-oriented summaries (known as nor- malized claims), created by fact-checkers as part of the verification process. As result, our dataset is not subjected to external annotation, thus averting potential biases and ensuring its high quality.
One issue has been the faithfulness of the sum- mary with respect to the source. To address this, Kryscinski al. (2020) introduced FactCC, weakly-supervised BERT-based entailment model, which augments the dataset with artificially in- troduced faithfulness errors. Similarly, (Utama et al., 2022) trained model for detecting factual inconsistencies in data from controllable text gen- summaries,
3.1 Data Collection
We gathered our fact-checked post and claim pairs from two sources: (i) Google Fact-Check Explorer and (ii) ClaimReview Schema.
eration that perturbs introducing varying types of factual inconsisten- cies. Durmus et al. (2020) proposed question- answering framework that compares answers from the summary to those from the original text.
Google Fact-Check Explorer. We acquired list of fact-checked claims from multiple reputed fact- check sources via Google Fact-Check Explorer's API (GFC). This data collection pipeline followed a three-step process. First, we extracted the title, which is usually a single-sentence short summary of the information being fact-checked, and the fact- checking site's URL. This step yielded total of 22,405 unique fact-checks. We then proceeded to retrieve the social media post and the associated claim review if they were available on the fact- checking site. Due to the collected posts having already undergone fact-checking and containing misleading claims, a significant number of them were unavailable for inclusion in our dataset. More- over, a significant number of the posts only con- tained images or videos, which were unsuitable for our task at hand. As result, we were left with a considerably smaller number of relevant instances. We also noted that in certain instances, the title in the Google Fact-Check Explorer and the claim review were identical; consequently, we included only one in the final dataset.
All these approaches primarily focused on general-purpose summarization and did not provide means for models to generate summaries primarily focusing specific needs. To address this limi- introduced
tation, controlled summarization (Fan al., 2018). One aspect of controlled sum- marization is length control, in which users can set their preferred summary length (Rush et al., 2015; Kikuchi et al., 2016). Recent research has dis- covered that, despite their fluency and coherence, state-of-the-arr abstractive summarization systems produce summaries with contradictory information.
While text summarization systems can assist in condensing social media posts into shorter sum- maries, their primary goal is not ensure verifia- bility. It aims capture the key points of the text rather than emphasizing the specific claims within the text that need to be fact-checked. Our task of claim normalization, on the other hand, works at an entirely different level. It needs a thorough understanding of the claims made in the social me- dia post and strives to ensure that the normalized claims are not only consistent with the original post, but are also self-contained and verifiable.
The ClaimReview Schema. We targeted the ClaimReview Schema elements with an entry for reviewed items as they were relevant to our require- ments. Out of 44,478 entries, only 22,428 had this partieular field. Therefore, we had filter out the remaining entries. Next, we extracted all the links to social media posts and their corresponding claim reviews provided by the fact-checkers.
Despite the progress in text summarization, the task of claim normalization remains underexplored. In this work, we aim to tackle this challenging prob- lem by developing a robust approach specifically tailored to the unique aspects of this task.
htttps:/toollox. http:///tooloxx.oogleeeeo httpp:///oolbbbeeeeeeeeoooooocer cowm//factcchecc///llleeer hetttsss/ssheen.org/llllaineeee

---

Social Media Post
Normalized Clain
Research into dangers of cooking with aluminum foil has found that of the toxic metal contaminate food. Increased levels of aluminumm the body have been linked osteoponsis, and Alzheimer's 's diseuse.
Cooking in Aluminum foil Alzheimer's Disease.
6. can add them as authorized credit cards. Never give them card, & all payments make from goes child credit too..ur kid will have unbelievable credit score from years of payment history.
Did you know when child
of
G-ycar-old kids added authorizee all credit cards.
As if couldn't I get any worse. #HopelCancer says #RootCanal causes #CANCER Solution... Rip Cancer patients teeth out. Monsters #aalseHopesCanner PanrieeCCanceraaiiens
Having root canal cause cancer.
Rooi canall Treaiment cancer
Table 1: Examples of social media posts and their corresponding normalized claims from CLAN. The first two examples come from the training set and each has one reference normalized claim, while the last one comes from the test set, and thus it has two reference normalized claims.
3107
Figure 2 shows an analysis of the cosine sim- ilarities between the social media posts and the corresponding normalized claims. We can see that the cosine similarities consistently low for most examples, demonstrating that claim normalization involves more than just summarizing the social me- dia post. This highlights the need for a specialized effort to accurately identify, extract, and normalize the claims within social media posts.
1753
Number of instencces
1500
$250-
1003
759
233
116
10
201
301
401
500
660
701
301
(90 100)
Cosine Similanty Range
Figure 2: Histogram of the cosine similarity between the social media posts and the corresponding normalized claims from CLAN dataset.
4 Proposed Approach
In this section, explain proposed approach, Check-worthiness Aware Claim Normalization (CACN), which aims to integrate task-specific information with large language models (LLMs). We focus our experiments on GPT-3 (text-davinci-003)) (Brown et al., 2020). Our approach amalgamates two key ideas: (i) chain-of-thought prompting and (ii) reverse check-
As mentioned above, we only processed textual claims and excluded other modalities, such as audio or video. Further, we ensured that all the entries were in English.
3.2 Data Statisties and Analysis
By using both of these data collection methods and by exercising careful consideration, we curated total of 6,388 instances. To ensure the creation of diverse and high-quality test set, chose posts that comprised not one, but two reference normalized claims (c.f. Sec 3.1). This enabled us to capture different aspects and perspectives of the normalized claims by including multiple refer- ences, thereby increasing the test set's robustness and reliability. Representative examples from our dataset are shown in Table 1, and the final dataset statisties are shown in Table 2.
Chain-of- -houuugh Prompting. The realm of chain-off of of-thhught (CoT) prompting has emerged as a veritable tour de force within LLMs (Wei et al., 2022). Instead of undergoing the laborious process of fine-tuning individual model checkpoints for ev- ery new task, we use CoT to navigate the complex- ity of claim normalization by using step-by-step reasoning. To accomplish this, we use claim check- worthiness, as described in the following subsec- tion. This enables the model to iteratively enhance its comprehension and effectively generate precise normalized claims while eliminating the need for extensive fine-tuning.
Dataset
Train
Val
Test
Overall
Total number of pairs
5,341
594
6,388
Avg. length of posas
39.52
37.12
57.97
44.87
Our proposed prompt example is shown in Fig- ure 3. Chain-of-thoon of-thutghtttt approaches a complicated problem by efficiently breaking it into a sequence of simpler intermediate stages.
Avg. longth of claims
16.47
17.24
15.41
16.37
Table 2: Statistics about our CLAN dataset.

---

Identify the central claim the given post: Did you know that E20 BILLION a year tax on Scottish whisky goes straight to England and is spent on priorities like London's Crossroil, 2nd Heathvow runway, Yacht? also revenues BILLION) year goes to England as well? And that is just the tip of the iceberg. Scotland subsidises England, NOT the other way around!
Let's think step by step.
The given post claims that Scottish taxes are sent to England to spent on their priorities. claim verifiable through data statistics. This also huge impact on society and seems contain false information. The claim here needs to be fact-checked. Hence the central claim here 'income tax and tax on whisky go to England to be spent on English priorities.'
more context examples here
Identify the central claim in the given post: <input post>
Let's think step by step.
Figure 3: Illustration of our proposed approach. To generate a normalized claim, we use the CACN prompt template, which encompasses explicit task instruction and relevant in-context examples, well chain-of-thought reasoning.
Reverse Check-Worthines Worthiness. The idea about re- verse worthewor originates from the task of worthh-sssss estimation, which in turn is an integral part of the manual fact-checking pro- cess (Nakov et al., 2021b). We leverage check- worthiness to steer the model's attention toward salient and pertinent information. By giving the model the ability produce rationales in natural language that clearly explain the sequence of rea- soning stages leading to the solution, we strengthen its capacity for cogmitive reasoning with unwaver- ing efficacy. Based on prior research on claim check-worthines. (Barrón-Ceeeño et al., 2018; Shaar et al., 2021; Nakov et al., 2022b), we direct our model to prioritize claims that meet specific criteria within the given social media post. These criteria include identifying claims within social me- dia posts that (i) contain verifiable factual claims, (ii) have higher likelihood of being false, (iii) are general public interest, (iv) are likely to be harm- ful, and (v) are worth fact-checking. For instance, in Figure 3, the claim normalization process be- gins by identifying the central claim within the in- put social media post. Subsequently, we reckon the claim's verifiability, i.e., whether it is self- contained and verifiable (e.g., opposed not containing a claim or expressing an opinion, etc.). We further evaluate the likelihood of the claim be- This
Experimental Sctup
Baseline Models. For comparison, we use sev- eral state-of-the-art generative systems and catego- rize them into two groups: (i) Pre-trained Large Language Models (PLMs): T5 (Raffel et al., 2020), BART (Lewis et al., 2020), FLAN-T5 (Chung al., 2022), and PEGASUS (Zhang al., 2020). For T5, BART, and FLAN-T5, we use base and large model sizes. For PEGASUS, we use the reddit t model. (ii) In-context Learning Model: GPT-3 (text (text-davinci-003 (Brown et al., 2020).
Evaluation Measures. To evaluate lexical over- lap, we use ROUGE (1,2, L) and BLEU-4 (Pa- pineni et al., 2002). We further METEOR (Banerjee and Lavie, 2005) and BERTScore (Zhang et al., 2019) assess the similarity between the gold and the generated normalized claims.
Zero-Shot Learning. Zero-shot learning aims to apply the previously acquired capabilities of PLMs to similar tasks in a low domain. We hereby assess its suitability for the claim normalization task.
Few-Shot Learning, We adopt few-shot learning with 10, 20, 50, and 100 training examples. This gradual exposure to additional labeled data aims to enhance the models' ability to generate accurate and contextually appropriate normalized claims.
ing false and its overall thhek-wwrthinng step-by-step process ensures comprehensive anal- ysis of the central claim's 's characteristics, allowing for effective claim normalization. By incorporating these aspects into our approach, we aim to improve the model's ability identify and prioritize claims that require scrutiny and verification.
Prompt Tuning. Prompt-tuming entails adding specific prefix to the model's input customized to the downstream tasks (Zhang et al., 2023). We investigate the impact of affixing different prompts to the given posts on the performance of T5-based and GPT-3 models. To exert control over the gener- ated normalized claims, we use five control aspects:

---

tokens, abstractness, number of sentences, claim- centricity, and entity-centriiity. A comprehensive description of all these prompts given in the Ap- pendix (A.2).
We can see that the models exposed to in-context examples showcase superior performance, high- lighting their efficacy in capturing task-specific patterns. While the trained models exhibit ex- cellence in lexical metrics, their performance semantic metrics is noticeably lower. Notably, BART trained on our dataset, outperforms other trained models by sizable margins. These results strongly underline that, within the realm of LLMs, incorporating prompt-tuning with in- context learning holds more promise, leading to enhanced generalization capabilities.
In-Context Learning. LLMs have the remark- able ability to tackle diverse tasks with mini- mal amount of examples given in-context learn- ing prompts (Brown et al., 2020). We use GPT-3 (text-davinci-893) with three different prompts: (i) direct prompt (DIRECT), (ii) question-guided prompt (Q-GUIDED), and (iii) zero-shot chain-of- thought (ZS-CoT)). Detailed prompt templates are given in Appendix A.3.
Do models demonstrate inherent proficiency in generating normalized claims with minimal or no prior training? We examine the potential benefits of zero-shot and few-shot learning to investigate the inherent proficiency in generating normalized claims. The zero-shot and the few-shot results are shown in Table Zero-shot learning, which relies solely on the pre-trained language model without any task-specific fine-tuning, performs quite well. On the other hand, few-shot learning does not re- sult in significant improvements. Surprisingly, the models trained using few-shot learning perform slightly worse than zero-shot learning, where the models have exposure to task-specific data. Af- ter training on ten examples, the performance of LLAN-T5LARGE drops by BERTSScre points absolute, and it continues to decline as more ex- amples are provided. This unexpected result sug- gests that few-shot learning may be unsuitable for this intricate and complex task. The limited num- ber of examples provided during few-shot learning may have been insufficient for the models to gen- eralize and capture the underlying patterns of nor- malized claims effectively. Moreover, introducing task specifp data might have introduced conflicting information as these models were never trained on this task, leading to a degradation in performance.
6 Experiments and Evaluation
Our experiments reveal that our CACN outperforms all baselines across most evaluation measures. We further examine all systems aiming to answer the research questions listed below.
Do meticulously crafted prompts enhance the performance of generative models? The find- ings exhibit a significant performance improvement when using prompt-tuning, specifically with in- context examples. Table shows the effectiveness various prompts across all evaluation measures. However, a notable enhancement of approximately 2-3 points absolute is observed for all semantic measures when transitioning from conventional prompts to proposed approach when using the same in-context examples. This emphasizes the im- portance of our framework tailored for the specific task. Moreover, an upsurge in ROUGE-FI scores (1, 2, and L) emphasizes the resemblance between the generated normalized claims and such created by humans. This, in turn, validates the incorpo- ration of the "reverse check-worthiness" chain-of- thought process, which effectively integrates task- specific information into the generative system. We also attempt prompt-tuning in a zero-shot setup; the results are shown in the Appendix (A.2). To sum- marize, the deliberate design of prompts, along with in-context learning, substantially enhances the performance of generative models.
Qualitative Analysis
Error Analysis. To comprehend the performance of CACN, we strive to qualitatively analyze the errors committed by our model in this section. Table 5 shows some randomly selected instances from our test dataset, along with gold normalized claims and predictions from CACN. For comparison, also show predictions from best-performing baselines. BART LARGE and DIRECT.
Is training models specific task less effective than in-context learning with a few examples? We observe substantial disparities in the perfor- mance of models trained on task-specific data com- pared to using in-context learning with a limited number examples, as shown in Table
"See Appendix (A.4) 4) for 50-shot and for 100-shot results.

---

ROUGE-1
ROUGE-2
ROUGE-L
Model
BLEU-4
METEOR
BERTScore
P
Fl
R
F1
P
R
F1
TSBASE
23.65
43.60
28,99
11.38
20.70
13.90
20.45
37.91
25.15
4.57
28.86
85.13
T5, ARGS
23.37
44.81
28.99
10.98
21.11
13.66
20.07
38.46
24.97
4.43
29.33
85.09
Finetune
BART BASS
33.41
41.64
34.11
17:57
21.05
17:55
29.70
36.61
30.25
6.69
29.57
86.32 32.32
BAKTLARSE
35.83
42.88
36.12
19.25
21.73
18.97
31.64
37.65
31.93
7.71
31.07
86.92
FLAN-T5BASE
22.50
47.38
28.79
10.65
22.28
13.61
19.15
40.14
24.50
4.44
29.07
84.66
FLAN- 15LARGE
28.70
45,98
31,35
15.32
22.63
16.24
25.46
39.89
27.63
6.62
29.36
84.65
In-conext
DIRECT
32.19
46.43
35.52
14.19
20.60
15.65
27.32
39.75
30.31
6.52
33.25
88.87
Q-GUIDED
33.48
44.50
35.40
15.34
20.42
16.15
29.10
38.88
30.90
6,56
32.13
88.81
75-CoT
27.77
32.42
13.18
22.33
23.84
41.51
27.79
32.93
88.44
(ours)
37.54
46.10
38.64
23.08
19.32
33.14
34.30
9.66
35.10
89.00
ACKON-BESTIR)
14:77
3 4.73
6.98
L 2.08
1.90
t 1.85
4.74
J.
1.142
7:42
25.20
1 5.56
1.15
Table 3: Experimental results of CACN and baseline systems on CLAN. We report ROUGE (1, I.), BLEU-4, METEOR, and BEoTe The best scores shown in bold, while the second-best scores underlined, across each metric. The last row gives the percentage increase in performance between CACN and the best baseline.
ROUGE-1
ROUGE-2
ROUGE-L
N-shot
Model
BLEU-4
METEOR
BERTScore
R
F1
F1
TSBASE
23.38
42.25
27.79
11.16
19.54
13.14
20.29
36.52
24.10
4,30
28.45
85.36
ISLARGS
24.80
43.71
29.08
12.44
20.93
14.31
21.58
37.82
25.30
5.01
29.36
85.65
BART BASE
22.57
47.76
27.96
22.60
13:60
19.58
24.14
30.44
BARTLARGE
20.80
39.68
24.41
17.20
11.16
18.23
33.85
21.20
24.67
84.78
FLAN-T5RASE
30.78
32.33
28.73
15.03
15.17
13.78
28.08
28.99
26.02
4.76
23.27
83,80
FLAN-T3/ARGE
31,20
34.06
30.22
15.67
16.83
15.07
28.41
30.57
27.40
5.76
25.82
84,89
PEGASUS
23.96
36.07
26. 16
11.69
16.76
12.48
21.14
31,50
22.99
4,87
24.87
83.07
T5
21.55
44.22
27 44
9.87
20.36
12.57
18.31
37.77
23.40
4,04
28
84.97
TSLARGE
21.92
45.73
28.04
10.17
21.04
12.98
18.64
38.81
23.85
4.19
29.52
85.11
BARTBASE
20.09
53.08
25.65
10.14
24.94
12.60
17.34
44.77
22.00
4.11
29.67
84.72
10
LARGL
19.71
53.22
30
9.83
24.87
12:27
17.01
45.22
21.70
4,00
29.51
84.65
FLAN-T5RASK
23.05
42.60
27.97
19.48
12.84
19.83
24.14
28.04
85,34
29.66
19.99
13.18
9.49
17.84
18.87
PEGASUS
15.26
40.32
20.86 86
7.15
18.29
9.58
13.39
35.87
18.35
3.44
24.73
80.02
15
21.56
44.22
20.36
12.59
37.78
23.42
28.64
T5LARGE
21.97
45.80
28.07
10.24
21.11
13:03
18.69
38.77
23.86
29.46
85.12
BART
20.08
53.16
25.63
10.14
25.03
12.60
17.34
44.89
21.99
4.11
29.63
84.72
20 BART
19:66
53.11
25.25
9.80
24.84
12.24
16.96
45.08
21.64
3.97
29 29.46
82.69
FLAN-TSHASE
23.15
42.80
28.11
10.63
19.58
12.91
19.93
36.76
24.27
4.44
28.20
85,34
FLAN-TSZARGE
22.38
29.79
20.00
10.69
13:25
9.51
20.48
26.00
17.85
3.04
18.96
78.01
PEGASUS
15.25
40.30
20.84
7.15
18.29
9.58
13.38
35.87
18.35
3.44
24.72
80.01
Table Zero-shot and few-shot performance on our dataset CLAN. We report ROUGE (1, L), BLEU-4, METEOR, and BERTScore.
Naturally, the predictions in the fine-grained analysis are much more intricate than in the coarse- grained quantitative setup. During our manual qualitative analysis, we unveiled several interest- ing patterns and errors in the generated responses. For example, although BART LARGE generated re- sponses with high BERTScore in example 1, we noticed that the factual alignment is incorrect, mak- ing this model untrustworthy for downstream tasks such as claim check-worthinee and claim verifi- cation. In contrast, our proposed model produced a response that is both correct and precise. The response generated by DIRECT is also accurate, but is excessively long, which contradicts the objective of the normalized claims being concise and straightforward.
This problem is also evident in example 3, where DIRECT produces a factually correct claim but is overly long.
In example observe that the BARTLARGE LARGE model demonstrates the lowest number of hallu- cinations and adheres closely to the input social media post. In contrast, our model's BERTScore performed the worst for this example. However, upon closer inspection, we noticed that the normal- ized claim that our model generated was indeed correct and most relevant for fact-checking. These findings highlight the complexity and the trade-offs involved in generating normalized claims. While certain models may excel in certain cases, there is often a compromise in other aspects, such as factual accuracy and conciseness.

---

Social Media Post
Normalized Claim
BS
They gave you a MASK cover your Bace AND then gave you George Flayd's signature phrase II can't hreath" Yer 2.5
GOLD: George Floyd and COV1D-11 "never existed".
Sanple
years later, most still struggle with the realization that
BART: Gearge Floyd died of COVID-19
901
EVERYTIIING on TV is fake. Flayd and ConVid never existed
DIRECT: George Floyd's death highlighted the lack of understanding of the reality of events pertrayed on television.
$6.84
CACN: George Floyd and Covid never existed.
90.98
Good News, Finally, CAJY VAZ. Indian EX Student from St.
GOLD: Honey, Ginger, Black Pepper
Comoavins.
Xavier's High School from Mumbui, presenlly based in Gon,
found home remedy for "Covid 19*, which was appeoved by the
BART: One teaspoon of pepper powder, two reaspoons of honey, little gingar juice taken for consecutive days can climinate the effect of Covid up to 100%.
84.89
WOll first
leaspoon of pepper powder, two tesspoons of honey, little ginger juice taken for
proved
Sample
consecutive days climinate the effect of corces up 100%* whole world starting finally a happy experience of 2021.
DIRECT; Indian ex-student Cajy found home remedy pepper powder, honey and ginger juice that can reduce the effects of Covid. 19 up 100%.
precautionary treatment,
CACN: A home remedy of pepper powder, honey, and ginger juice can elimi- 78.47
the elfect of COVID-19 19 up 100%.
Zelensky sold mallion hectares of agricultural land Monsanto, Dupont, and Cargil. Yes, you read it well, 17 million hectares GiMIcchemical companies. This very bad for the eatire world since Ulraain is largest exporter of wheat and other grains. Zelensky sold 28% of the entire Ukrainian arable land. Australian National Review reports three major US cross-burde border consettiums have bought million hoctares of Ukrainian famaland. To compare: In all of million boctares agricultural land. tims out that three. American companies in Ukriine baught valuable agricullural land than in all of Iialy. The entire
GOLD: Ukraine President Volodymyr Zelensky sold million hectares of farmland to US corporations.
BART: Three American companies bought million bectures of farmland in Ukraine.
$8.03
Sample
DIRECT: Three major US companics have bought 17 million hoctares of U Ukrainian agricultural land, which is more than in all of Italy These companies are Cargill, Dupont and Monsanto, with their
83.57
sharcholders being American venture capitalists Blackrock, Vanguard and Blackstone.
Italy, there
CACH: Ulraainian President 7elensky sold million hectares of agricultural 90.48
area of Ukraine 600,000 sqm, 170,000 sgm built.
land to Momsatto Dupons, and Cargill.
Table 5: Examples of generated normalized claims along with the gold reference. BART refers to BART BARTLARGE.
Model
Fluency
Coherence
Relevance
Consisteccy
SC
The average scores are presented in Table 6. For comparison, we also included the results from the best-performing baseline systems, namely LARGE and DIRECT. Our analysis reveals that the outputs generated by CACN exhibit qualita- tive superiority compared to the baseline systems across all dimensions.
BART
3.44
3.74
3.66
3.82
3.77
DIRECT
4.48
4.58
4.03
4.26
4,38
CACN
4.59
4.63
4.17
4.34
4,39
Table 6: Human evaluation on the generated normalized claims. SC denotes Self-Comtextualized, while BART refers to BARTLARGE
Human Evaluation. We conducted an extensive human evaluation to assess the linguistic profi- ciency of the generated normalized claims. Build- ing upon the measures proposed by van der Lee et al. (2021), we evaluated the generated claims based on four aspects: fluency, coherence, rele- vance, and factual consistency. We further intro- duced the parameter of self-contexttaaizzation to measure the extent to which the normalized claims included the necessary context for fact-checking within themselves. Each of these measures played a unique and vital role in evaluating the quality of the generated claims. To conduct the evaluation, we randomly selected 50 instances from our test set and assigned five human evaluators to rate every normalized claim on a scale of I to 5 for each of these five aspects. All evaluators were fluent En- glish speakers with a Bachelor's 's or Master's degree. To ensure reliability, each example was evaluated by all five evaluators independently, and then averaged their scores.
8 Conclusion and Future Work
We introduced the novel task of claim normaliza- tion, which holds substantial value on multiple fronts. For human fact-checkers, claim normal- ization is a useful tool that can assist them in effec- tively removing superfluous texts from subsequent processing. This also benefits downstream tasks such as identifying previously fact-checked claims, estimating claim check worth-wott,,ssss, etc. We further compiled a dataset of social media posts compris- ing over 6k posts and their normalized claims. We further benchmarked this dataset with a novel ap- proach, CACN, and showed its superior performance compared to different state-of-the generative models across multiple assessment measures. We also documented our data collection process, pro- viding valuable insights for future research in this domain. In future work, we plan to extend the dataset, including with new languages. We also plan to use more powerful LLMs.
*See Appendix A.6 for more detail.

---

Limitations
References
While our study has made major contributions to claim normalization, is critical to recognize and to address its potential limitations. During our data collection process, we excluded claims about im- ages and videos. Yet, we believe that including multimodal information may help improve claim normalization. Another key problem is that each fact-checking organization adheres to its own set of editorial norms, procedures, and subjective in- terpretations of claims. These variations in writing style and judgments make it challenging to estab- lish standardized claim normalization. Address- ing this issue will necessitate attempts to develop consensus or guidelines among fuct-checking or- ganizations in order to ensure greater consistency and coherence in claim normalization. By acknowl- edging and addressing these limitations, we may endeavor to improve the reliability and soundness of claim normalization systems in the future.
Firoj Alam, Shaden Shaar, Fahim Dalvi, Hassan Saj- jad, Alex Nikolov. Hamdy Mubarak, Giovanni Da San Martino, Ahmed Abdelali, Nadir Durrani, Kareem Darwish, Abdulaziz Al-Homaid, Wajdi Za- ghouani, Tommaso Caselli, Gijs Danoe, Friso Stolk, Britt Bruntink, and Preslav Nakov. 2021. Fighting the COVID-19 infodemic: Modeling the perspective of journalists, fact-checkers, social media platforms, policy makers, and the society. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 611-649, Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Hunt Allcott and Matthew Gentzkow. 2017. Social media and fake news in the 2016 election. Journal of Economic Perspectives, 31(2):211
Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D. Manning. 2015. Leveraging linguis- tic structure for open domain information extraction. In Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 344-354, Beijing, China. Association for Computational Lin- guistics.
Ethics and Broader Impact
Satanjeev Banerjee and Alon Lavie. 2005. METEOR:
Data Bias. It is important to acknowledge the possibility of biases within our dataset. Our data collection process involves gathering normalized claims from multiple fact-checking sites, each with its own set of editorial norms, procedures, and sub- jective interpretations. These elements can intro- duce systemic biases that impact the overall assess- ment of normalized claims. However, it is impor- tant to acknowledge that these biases are beyond our control.
An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACL Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Transla- tion and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguis- tics.
Alberto Cedéno, Tamer Elsayed, Preslav Nakov, Giovanni Da San Martino, Maram Hasanain, Reem Suwailch, and Fatima Haouari. 2020. CheckThat! at CLEF 2020: Enabling the automatic identifica- tion and verification of claims in social media. In Proceedings of the 42nd European Conference on Information Retrieval, ECIR 20, pages 499-507, Lisbon, Portugal.
Environmental Footprint. Large language mod- els (LLMs) require a substantial amount of energy for training. which can contribute to global warm- ing (Strubell et al., 2019). Our proposed approach, on the other hand, leverages few-shot in-context learning rather than training models from seratch, leading lower carbon footprint. It is worth mentioning that using LLMs for inference still con- sumes a considerable amount of energy, and we are actively seeking to reduce it by using more energy-efficient techniques.
A Alberto Barrón-Cedeñoo, Tamer Elsayed, Reem Suwailch, Lluís Mårquez, Pepa Atamasova, Wajdi Zaghouani, Spas Kyuchukov, Giovanni Da San Mar- tino, and Preslav Nakov. 2018. Overview of the CLEF-2018 CheckThat! Lab automatic identi- fication and verification of political claims. Task 2: Factuality. In CLEF 2018 Working Notes. Working Notes of CLEF 2018 Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings, Avignon, France. CEUR-WS.org
Our model interest the general public and save time for human fact-checkers. Its applications extend be- yond fact-checking to other downstream tasks such as detecting previously fact-checked claims, claim matching, and even estimating claim check- worthiness of new claims.
Broader Impact and Potential Use.
Emily M. Bender, Jonathan T. Morgan, Meghan Oxley, Mark Zachry, Brian Hutchinson, Alex Marin, Bin Zhang, and Mari Ostendorf. 2011. Annotating so- cial acts: Authority claims and aligmment moves in Wikipedia talk pages. In Proceedings of the Work- shop Language in Social Media (LSM 2011), pages 48-57 57, Portland, Oregon. Association for Com- putational Linguistics.

---

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-199..
Processing, pages 6002-6018, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Pepa Gencheva, Preslav Nakov, Lluís Märquez, Alberto and Ivan Koychev. 2017. A context- aware approach for detecting worth-checking claims in political debates. In Proceedings of the Interna- tional Conference on Recent Advances in Natural Language Processing, RANLP "1N,P'17, pages 267-276. Varna, Bulgaria.
Tuhin Chakrabarty, Christopher Hidey, and Kathy McK- cown. 2019, IMHO fine-tuning improves claim de- tection. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 558-563, Minneapolis, Minnesota. Association for Computational Linguistics.
Shreya Gupta, Parantak Singh, Megha Sundriyal, Md. Shad Akhtar, and Tanmoy Chakraborty. 2021. LESA: Linguistic encapsulation and semantic amal- gamation based generalised claim detection from on- line content. In Proceedings of the 16th Conference of the European Chapter of the Association for Com- putational Linguistics: Main Volume, pages 3178- 3188. Online. Association for Computational Lin- guistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph. Yi Tay. William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. language models.
2022. Scaling instructionffinete arXiv preprini arXiv:2210. /1416.
Johannes Daxenberger, Steffen Eger, Ivan Habernal, Christian Stab, and Iryna Gurevych. 2017. What the essence of a claim? Cross-domain claim identi- fication. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2055-2005 Copenhagen, Denmark. Associa- tion for Computational Linguistics.
Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. 2018. UKP-Athene: Aultensentence textual entailment for claim verification. In Proceed- ings of the First Workshop Fact Extraction and VERification (FEVER), pages 103-108. 108, Brussels, Bel- gium. Association for Computational Linguistics.
Markus Dreyer, Mengwen Liu, Feng Nan, Sandeep Atluri, and Sujith Ravi. 2023. Evaluating the tradeoff' between abstractiveness and factuality abstractive summarization. In Findings of the Association for Computational Linguistics: EACL 2023, pages 2089- 2105, Dubrovnik. Croatia, Association for Computa- tional Linguistics.
Momchil Hardalov, Anton Chemyavskiy, Ivan Koychev, Dmitry llvossky, and Preslav Nakov. 2022. Crowd- Checked: Detecting previously fact-checked claims in social media. In Proceedings of the 2nd Confer- ence of the Asia-Pucific Chapter of the Association for Computational Linguistics and the 12th Interna- tional Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), pages 266-285, Online only, Association for Computational Linguis- tics.
Esin Durmus, He He, and Mona Diab. 2020. FEQA: question answering evaluation framework for faith- fulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 5055- 5070. Association for Computational Linguisties.
Naeemul Hassan, Gensheng Zhang, Fatma Arslan, Jo- sue Caraballo, Damian Jimenez, Siddhant Gawsane, Shohedul Hasan, Minumol Joseph, Aaditya Kulkarni. Anil Kumar Nayak, et al. 2017. Claimbuster: The first-ever end-to-end fact-checking system. Proceed- ings of the VLDB Endowment 2017, 10(12):1945- 1948.
Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In Proceed- ings of the 2nd Workshop on Neural Machine Transla- tion and Generation, pages 45-54, Melbourne. Aus- tralia. Association for Computational Linguistics.
Revanth Gangi Reddy, Sai Chetan Chinthakindi, Yi R. Fung. Kevin Small, and Heng Ji. 2022a. zero-shot claim detection framework using question answer- ing. In Proceedings of the 29th International Con- ference on Computational Linguisties. pages 6927- 6933, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Israa Jaradat, Pepa Gencheva, Alberto Barrón-Cedefio, Lluís Marquez, and Preslav Nakov. 2018. Claim- Rank: Detecting
claims in Arabic and English. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 26-30, New Orleans, Louisiana. Association for Computational Linguistics.
Revanth Gangi Reddy, Sai Chetan Chinthakindi, Zhen- hailong Wang. Yi Fung, Kathryn Conger, Ahmed EL- sayed, Martha Palmer, Preslav Nakov, Eduard Hovy, Kevin Small, and Heng Ji. 2022b. NewsClaims: A new benchmark for claim detection from news with attribute knowledge. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language
Irina Khaldarova and Mervi Pantti. 2016. Fake news. Journalism Practice, 10(7):891-911
Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. 2016. Controlling

---

output length in neural mecodr-ddccoder. In Pro- ceedings of the 2016 Conference on Empirical Meth- ods in Natural Language Processing, pages 1328- 1338, Austin, Texas. Association for Computational Linguistics.
fighting the COVID-19 infodemic and fake news de- tection. In Proceedings of the 13th International Conference of the CLEF Association: Information Access Evaluation meets Multilinguality, Multimodal- ity, and Visualization, CLEF 2022, Bologna, Italy.
Lev Konstantinovskiy, Oliver Price, Mevan Babakar, and Arkaitz Zubiaga. 2021. Toward automated factchecking: Developing an annotation schema and benchmark for consistent automated claim detection. Digital threats: research and practice, 2-2):1-16.
Preslav Nakov, David Corney, Maram Hasanain, Firoj Alam, Tamer Elsayed, Alberto Barrón-Codeñi,, Paolo Papotti, Shaden Shaar, and Giovanni Da San Martino. 2021a. Automated fact-checking for assisting hu- man fact-checkers. In Proceedings of the 30th Inter- national Joint Conference on Artificial Intelligence. IJCAI '21, pages 4551-4558.
Wojeiech Kryscinski. Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 9332-9346, Online. Association for Computa- tional Linguisties.
Preslav Nakov, Giovanni Da San Martino, Tamer Elsayed, Alberto Barrón-Codeño, Rubén Míguez. Shaden Shaar, Firoj Alam, Fatima Haouari, Maram Hasanain, Watheq Mansour, et al. 202 Ib. Overview of the CLEF-2021 CheckThat! lab on detecting check-worthy claims, previously fact-checked claims, and fake news. In Proceedings of the 12th Interna- tional Conference of the CLEF Association: Infor- mation Access Evaluation Meets Multiliguality, Mul- rimodality, and Visualization, CLEF '2021, pages 264-291, Bucharest, Romania (online).
Ran Levy, Shai Gretz, Benjamin Sznajder, Shay Hum- mel, Ranit Aharonov, and Noam Slonim. 2017. Un- supervised corpus-wide claim detection. In Proceed- ings of the 4th Workshop on Argument Mining, pages 19-84, Copenhagen. Denmark. Association for Com- putational Linguisties.
Liangming Pan, Xiaobao Wu, Xinyuan Lu, Anh Tuan Luu, William Yang Wang. Min-Yer Kan, and Preslav Nakov. 2023. Fact-checking complex claims with loogram-guide reasoning. In Proceedings of the 6 Anmual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 6981-7004, Toronto, Canada. Association for Com- putational Linguistics.
Mike Lewis, Yinban Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online, Association for Computa- tional Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318 1-318. Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Marco Lippi and Paolo Torroni. 2015. Context- independent claim detection for argument mining. In Proceedings of the 24th International Conference on Artificial Intelligence, IJCAT 15, page 185-191. AAAI Press.
Preslav Nakov, Alberto Barrón-Cedeñho, Giovanni Da San Martino, Firoj Alam, Julia Maria Struß, Thomas Mandl, Rubén Míguez, Tom- maso Caselli, Mucahid Kutlu, Wajdi Zaghouani, Chengkai Li, Shaden Shaar, Gautam Kishore Shahi, Hamdy Mubarak, Alex Nikolov, Nikolay Babulkov, Yavuz Selim Kartal, and Javier Beltrán. 2022a. The CLEF-2022 TheckThat! lab on fighting the COVID- 19 infodemic and fake news detection. In Proceed- ings of the 44th European Conference on IR Re- search: Advances in Information Retrieval, ECIR '22, pages 416-428 Berlin, Heidelberg. Springer- Verlag.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Suiskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1)55485-551.
Preslav Nakov, Alberto Barrón-Cedehn Giovanni Da San Martino, Firoj Alam, Julia Maria Struß, Thomas Mandl, Rubén Míguez, Tom- maso Caselli, Mucahid Kutlu, Wajdi Zaghouani. Chengkai Li, Shaden Shaar, Gautam Kishore Shahi, Hamdy Mubarak, Alex Nikolov, Nikolay Babulkov, Yavuz Selim Kartal, and Javier Beltrán. 2022b. Overview of the CL2F-2022 CheckThat! lab on
Yasmim Mendes Rocha, Gabriel Acácio de Moura, Gabriel Alves Desidério, Carlos Henrique de Oliveira, Francisco Dantas Lourenço, and Larissa Deadame de Figueiredo Nicolete. 2021. The impact of fake news on social media and its influence on health during the COVID-19 pandemic: A systematic review. Journal of Public Health, pages 1-10.

---

Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sen- tence summarization. In Proceedings of the 2015 Conference on Empirical Methods Natural Lan- guage Processing, pages 379-389, Lisbon, Portugal. Association for Computational Linguistics.
Information & Knowledge Management, CIKM '21, page 1673,,77,,,0 Virtual Event, Queensland, Aus- tralia. Association for Computing Machinery.
Prasetya Utama, Joshua Bambrick, Nafise Moosavi, and Iryna Gurevych. 2022. Falsesum: Generating document-level NLI examples for recognizing fac- tual inconsistency in summarization. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 2763-2776. Seattle, United States. Association for Computational Linguistics.
Shaden Shaar, Firoj Alam, Giovanni Da San Martino, and Preslav Nakov. 2022a. The role of context in detecting previously fact-checked claims. In Find- ings of the Association for Computational Linguis- tics: NAACL 2022, pages 1619-1631, Seattle, United States. Association for Computational Linguisties.
Shaden Shaar, Nikolay Babulkov, Giovanni Da San Mar- tino, and Preslav Nakov. 2020. That is a known lie: Detecting previously fact-checked claims. In Pro- ceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics, pages 3607-3618. Association for Computational Linguistics.
Chris van der Lee, Albert Gatt, Emiel van Miltenburg, and Emiel Krahmer. 2021. Human evaluation of automatically generated text: Current trends and best practice guidelines. Computer Speech Language, 67:101151.
Slavena Vasileva, Pepa Atanasova, Lluis Märquez, Al- berto Barrón-Cedeñio, and Preslav Nakov. 2019, It takes nine to smell a rat: Neural multi-task learning for check-worthiness prediction. In Proceedings of the International Conference on Recent Advances in Natural Language Processing, RANLP '19, pages 1229-1239, Varn,, Bulgaria. INCOMA Ltd.
Shaden Shaar, Nikola Georgiev, Firoj Alam, Giovanni Da San Martino, Aisha Mohamed, and Preslav Nakov. 2022b. Assisting the human fact-checkers: Detect- ing all previously fact-checked claims in a document. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2069-2080, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Fed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thouge prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824-24837
Shaden Shaar, Maram Hasanain, Bayan Hamdan, Zien Sheikh Ali, Fatima Haouari, Alex Nikolov, Mucahid Kutlu, Yavuz Selim Kartal, Firoj Alam, Giovanni Da San Martino, Alberto Barrón-Cedeñio, Rubén Míguez, Tamer Elsayed, and Preslav Nakov. 2021. Overview of the CLEF-2021 CheckThat! lab task on check-worthiness estimation in tweets and political debates. In Working Notes of CLEF 202 Conference and Labs of the Evaluation Fo- rum, CLEF 2021, Bucharest, Romania (online).
Dustin Wright and Isabelle Augenstein. 2020. Claim check worthiness detection as positive unlabelled learning. In Findings of the Association for Compu- tational Linguistics: EMNLP 2020, pages 476-488, Online. Association for Computational Linguistics.
Amir Soleimani, Christof Monz, and Marcel Worring. 2020. BERT for evidence retrieval and claim verifica- tion. Advances in Information Retrieval, 12036:359.
Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song. 2023. A survey of controllable text generation using transformer-based pre-trained language models, ACM Comput. Surv., 56(3).
Emma Strubell, Ananya Ganesh, and Andrew McCal- lum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645-3650, Florence, Italy. Asso- ciation for Computational Linguistics.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. PEGASUS: Pre-training with extracted cap-sentences for abstractive summarization. In Pro- ceedings of the International Conference on Machine Learning, pages 1 1338-11339, PMLR.
Megha Sundriyal. Atharva Kulkarni, Vaibhav Pulastya, Md. Shad Akhtar, and Tanmoy Chakraborty. 2022. Empowering the fact-checkers! Automatic identifi- cation of claim spans on Twitter. In Proceedings of the 2022 Conference on Empirical Methods in Nat-
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- berger, and Yoav Artzi 2019. BERTScore: Evalu- ating text generation with BERT. Proceedings of the International Conference on Learning Represen- tations, ICLR '9'.
ural Language Processing, pages 7701-7715, 1711, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Shi Zhi, Yicheng Sun, Jiayi Liu, Chao Zhang, and Jiawei Han. 2017. CeaifVerif: A real-time claim verifica- tion system using the web and fact databases. In Proceedings of the 2017 ACM on Conference In- formation and Knowledge Management, CIKM '17 page 2555-2558, New York, NY, USA. Association for Computing Machinery.
Megha Sundriyal, Parantak Singh, Md. Shad Akhtar, Shubhashis Sengupta, and Tanmoy Chakraborty. 2021. DESYR: Definition and syntactic represen- tation based claim detection on the web. In Proceed- ings of the 30th ACM International Conference on

---

A
Appendix
A.1 Task Motivation
000000000
Claim normalization holds significant promise for combating the spread of misinformation by stream- lining fact-checking processes and enhancing the reliability of retrieved evidence. To substantiate our hypothesis regarding the effectiveness of claim normalization, we conducted a well-structured re- trieval experiment using the Google API. The ob- jective was to demonstrate the practical benefits of claim normalization in assisting fact-checkers. We randomly selected sample of 35 instances from our dataset, encompassing social media posts and their normalized claims. Leveraging the capabili- ties of the Google API, we sought the top-5 most relevant articles for each post and its normalized claim. In a meticulous evaluation process, three annotators individually assessed the relevance (0 or 1) of each retrieved article to the input (post or normalized claim). We then used majority voting to determine the final relevance score for each re- trieved article. As depicted in Table 7, the results of our experiment consistently demonstrated the ad- vantage of normalized claims in evidence retrieval. In top-k precision evaluations for various values of k (1,3, and 5), normalized claims consistently out- performed their corresponding source posts. This observation indicates that claim normalization is not merely& thcoretical concept, but significantly enhances the efficiency of evidence retrieval, result- ing more concise and effective tools for aiding the fact-checking process.
Number of tokens
Figure 4: Box-plot for the number of tokens in normal- ized claims in CLAN.
Uncontrolled. We investigated the use of the traditional prompt 'summarize' for uncontrollable models. This prompt lacks specifie control sig- nals, making uncontrolled as it does not provide explicit guidance with specific attributes.
Token Limit. We found that normalized claims written by fact-checkers typically adhered to around ten words, as shown in Figure 4. Thus, in order control the length of the normalized claims, used the following prompt: "summarize within the length of 10 tokens".
A Abstractness. Abstractness quantifies how much the generated text's words and phrases differ from those extracted directly from it: a fully ab- stractive summary expresses the central points of the input in very different words and sequences of words compared to the original input. Precisely, the more n-grams overlap between summary and its original document, the less abstractive a summary is. Thus, in order to control the abstractness of the generated normalized claims, we use the prompt "summarize with abstractness of a", where a repre- sents a value within the range [0:1], denoting the desired level of abstractness. Inspired by Dreyer et al. (2023), we compute the abstractness score o, for each pair of post p; and a normalized claim si. and then we take an average across examples:
P@
P@3
P@5
Post Normalized Claim
0.82
0.64
0.58
0.88
0.73
0.69
Table 7:
Comparative top-k precision evaluations of normalized claim vs. original posts in evidence retrieval.
a; = (1- X (pe 8:))
A.2 Prompt-Tuning
where X is the harmonic mean of unigram overlaps precision, bi-gram overlap precision, and longest sub-sequence overlap precision. We found the av- erage abstractness (a) to be around 0.8.
Raffel et al. (2020) demonstrated that prompt- tuning could enable controllable text generation in basba models. We also investigate the im- pact of affixing different prompts to the given in- put the performance of b5-based models for normalized claim generation, along with GPT-3 (text-davinci-003) (Brown al., 2020). We ex- perimented with various prompts suffixed to the input text before inference, in a zero-shot setting, We discuss our different prompts P. below.
Single Sentence. The normalized claim written by fact-checkers often consists of concise single- sentence summary of the post. We the prompt "summarise one sentence" in order limit the normalized claims to a single-sentence summary, ensuring brevity and conciseness delivering the pivotal assertion.

---

ROUGE-1
ROUGE-2
ROUGE-L
Model
BLEU-
METEOR
BeRT
P
R
Fl
P
R
F1
P
UNCONTROLLED
23.38
42.25
27.79
11.16
19.54
13.14
20.29
36.52
24.10
4.30
85.36
TOKEN-LIMIT
22.87
42.71
27.59
11.42
20.61
13.60
20.21
37.47
24.34
85.31
TSRASE
ABSTRACTNESS
27.58
11.40
13.47
85.41
SINGLE SENTENCE
21.06
31.19
23.07
14.02
10.46
18.30
26.75
3.57
84.41
CLAIM-CENTRIC
19.76
38.89
24.20
8.90
17.94
10.96
17.37
34.33
3.51
25.77
84,58
ENTITY-CENTRIC
19.08
32.88
22.63
8.37
14.29
9.91
16.66
28.70
19.81
3.26
20.11
83,20
UNCONTROLIITE
24.80
43.71
29.08
20.93
21.58
85.65
TSLANCE
TOKEN-LIMIH
14.51
36.24
19.53
6.24
16.01
8.45
12.79
2.56
82.08
ABSTRACTNESS
23.53
44.44
28.62
11.69
20.98
13.96
20.66
38.64
25.10
5.02
29.74
85.58
SINGLE SENTENCE
23.92
44:73
28.92
11.81
21.04
14.04
20.82
38.60
25.16
4.90
29.94
85,65
CLAIM-CENTRIC
11.61
33.36
16.44
4,95
14.77
7.08
10,50
30.70
14.97
2.38
19.64
80,99
INTITY-CENTTIC
12.35
30.61
16.55
5.13
13.42
7.01
10.91
27.22
14.65
2.20
18.07
82.
UNCONTROLLED
19.17
48.22
24.77
7.17
19.69
9.51
16.09
20.76
TOKEN-LIMII
26.29
40.66
28.92
9.75
10.59
22.24
3.75
GPT3
ABSTRACTNESS
15.74
54.44
23.27
12.52
2.84
86.91
SINGLE SENTENCE
22.81
44.71
28.46
8.78
17.72
11.07
18.60
37.37
4.27
29.02
87.85
CLAIM-CENTRIC
14.03
56.41
21.72
5.53
23.93
8.68
11.11
46.18
17.33
2.59
25.88
86.50
ENTITY-CENTRIC 18.92
53.52
26.34
7.74
22.41
10.82
15.26
43.90
21.34
3.67
30.98
87.16
Table 8: Zero-shot prompt-tuning results for T5 and GPT-3 our datatset CLAN.
Claim-Centrics The task at hand more than just text summarization; transcends conventional text summarization by seeking not only to condense the input social media post, but also to discern and to encapsulate the central claim within that input post conciscly. Thus, we use the prompt "summa- rice the texl identifying the central assertion" to helm the model focus on the main assertion in the input text.
A.3 In-Context Learning Templates
In Figure 5, show the templates used for the three in-context learning methods used for GPT-3 (text-davinci-003) as mentioned in Section 5.
A.4 Few-Shot Additional Results
We report 50-shot and 100-shot experimental re- sults in Table 9. Interestingly, we observe that introducing more examples to the model did not help it much.
Entity-Centricity. Similarly to the claim-centric prompt, we investigated the technique of creat- ing entity-centered summaries using the prompt, "summarize the text focusing the given keywords (kw1, kw2,, use Open Information Extraction (Angeli et al., 2015) order extract subject-verb-objec triples from the input texts." Subsequently, we compile a keyword list encompassing all subjects and all objects within the text. The objective is to direct the model to produce summaries that align well with the subjects and the objects mentioned in the input text.
A.5 Implementation Details
We performed basic data cleaning, e.g., removing alnaaaphaauerri characters, removing links and hashtags, etc. on our dataset CLAN, using nlik. For standardized evaluation, relied on widely rec- ognized evaluation libraries such py-rouge, nlik-bleu," nlik-meteor," and hugging-face bert- score." We trained all models for 50 epochs, with early stopping based on validation loss. We set the patience value at 5, and we optimized the models using the Adam optimizer. We set the weight decay to 0.01. I For our proposed approach CACN, we used as base model. Fi- nally, set the maximum length of the generated response 120 with temperature of 0.6.
For this approach,
Results. We report the results for our zero- shot prompt-tuning experiments in Table 8. We can see that for the T5-based models, prompt- tuning did not yield any major improvements; rather, decreased the performance compared the uncontrolled prompt. However, GPT-3 ftext-davinci. -033) showed some improvements when using these prompts.
GPT-3 (text-davinci-on)
thttpss//pypi wttp///rjjec/yyyrrurreee ortp:///////////////////////oooo bleu_score.hte
"https://www.n www. wat.k.orgg/pi/nlttt translate.
neteor_score.htm
hhttps://gitiei www.itthuu.cco/p//e wwtwcccwwppppcccccccccccpccccc hwtp://cc//mhpppppeeeeepe// ipperemy/ Stanford-OpenIE-PPthho
"nttps://hugging: co/ co/spaces/
wvaluate-metric/berrsscom

---

Training Samples
ROUGE-1
ROUGE-2
ROUGE-L
Model
BLEU-4
METEOR
BERTScore
P
R
F1
R
F1
FI
T5
21.57
44.24
27.46
9.88
20.36
12.58
18.32
37.78
23.41
4,05
28.64
84.97
T5LARGE
22.00
45.86
28.11
10.24
21.11
13.03
18.71
38.83
23.90
4.20
29.49
85.11
BART BASE
20.10
53.16
25.65
10.15
25.02
12.61
17.35
44.93
22.00
4.11
29.66
84.71
30
BARTLARGE
19.63
52.96
25.20
9.78
24.73
12.21
16.94
44.88
21.61
3.97
29.46
84.65
FLAN-
23.16
42.87
28.14
10.62
19.60
12.91
19.92
36.79
24.26
4,43
28.30
85.35
FLAN-TS/ARGE
22.38
29.63
19.97
10.69
13.25
9.51
20.49
25.92
17.84
18.92
77.84
PEGASUS
40.28
18.29
9.58
13.38
35.84
18.34
24.71
B5BESE
21.55
44.24
27.45
20.36
12.57
18.31
37.78
23.40
28.65
84.97
AROR
45.72
BART
20.09
53.28
25.65
10.15
24.97
12.60
17.35
44.91
22.00
4,11
29.61
84.72
100
BART LARGE
19.68
53.00
25.26
9.80
24.72
12.22
16.99
44.94
21.67
3.97
29.43
84.66
FLAN-TSBASE
23.19
42.91
28.18
10.65
19.63
12.94
19.96
36.85
24.32
4,43
28.29
85,36
PLAN-T3ZAREE
22.38
29.63
19.97
10.69
13.25
9.51
20.49
25.92
17.84
3.03
18.92
77.85
PEGASUS
15.30
40.30
20.88
7.16
18.27
9.58
13.41
35.89
18.37
3.43
24.74
80.02
Table 9: Few-shot results on our dataset CLAN.
A.6 Human Evaluation Criteria
Following van der Lee et al. (2021), we define the four human evaluation measures follows:
1. Fluency: measures the linguistic proficiency exhibited by the generated responses;
2. Coherence: evaluates the intrinsic structure and the organization of the generated normal- ized claims;
3. Relevance: appraises the discerning selection of contextually appropriate content within the generated response;
4. Factual consistency: examines the intricate alignment between the factual accuracy of the generated response and the source text.

---

DIRECT
For the given post get the normaliized claim:
"Cough CPR is form of self CPR. Coughing increases intrathoracic pressure and squeezes blood out of the heart into the aorta. Theoretically, one can remain conscious if one continues to cough during cardiac arrest (not heart attack without cardiac arrest all heart attacks do not cause cardiac arrest). Normalized claim: Cough CPR can prevent cardiac arrest.
Post:
....... more in-context examples here -......>>
given post get normalized claim:
Post: <input_posttteeeeee post>
Normalized claim: rgonorated__esponse>
Q-GUIDED
What is being claimed in the given post?
Cough CPR form self CPR. Coughing increases intrathoracic pressure and squeezes blood out the heart into the aorta. Theoretically, one can remain consclous if one continues cough during cardiac arrest Inot heart attack without cardiac arrest heart attacks not cause cardiac arrest). The answer is: Cough CPR can prevent cardiac arrest.
more in-context examples here ........
What is being claimed in the given post?
dinput_posts
The answer aeenerated_responee
ZS-CoT
Identify the central claim in the given post. Let's think step-
Post: sinput_post-
Central claim: response>
Figure Our templates for in-context learning prompts used for GPT-3 (text-davinci-000)


Reviews:
[{"review_id": "LsZlx6WIgm", "reviewer": "Reviewer_GSfB", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable feedback. \n\n**[Usage of GPT Models]** We agree that GPT models are closed, meaning we lack direct access to all their training data. This opacity raises concerns about potential biases and misinformation in the model. However, these models are pre-trained on diverse datasets to reduce bias risk. Further, to address this, we're taking a multi-step approach. We begin with a pre-trained GPT model, then fine-tune it for our specific task, aligning outputs with our needs. The vast training data's advantage is two-sided: enhancing capabilities but demanding careful monitoring for issues. By meticulously crafting the prompt and monitoring, we aim to benefit from the data while reducing contamination risks. \n\n\n**[GPT Data Contamination]** Thank you for highlighting this critical concern. Since GPT-3 was trained on data available up to September 2021, we meticulously sampled 20 recent fact-checked social media posts (July-August 2023) and leveraged our model to generate normalized claims. Impressively, these recent claims demonstrated comparable effectiveness, reaffirming the relevance of our approach. This highlights the method's adaptability and high resistance to data leakage concerns, strengthening our commitment to real-world applicability and credibility. Further, the slightly amplified scores are attributed to the smaller sample size in our in-the-wild analysis.\n\n| Models           | ROUGE-1       | | | ROUGE-2 |       |       | ROUGE-L |       |       | BLEU  | METEOR | BERTScore |\n|------------------|---------|-------|-------|---------|-------|-------|---------|-------|-------|-------|--------|-----------|\n|                  | P       | R     | F     | P       | R     | F     | P       | R     | F     |       |        |           |\n| CACN (Test set)    |   37.54 |  46.1 | 38.64 |   18.85 | 23.08 | 19.32 |   33.14 | 40.92 |  34.3 |  9.66 |   35.1 |        89 |\n| CACN (In-the-wild) |   38.74 | 51.59 | 42.07 |    24.6 | 32.55 |  26.7 |   36.82 | 49.59 | 40.29 | 16.25 |   38.9 |     87.05 |\n\n**[Extended Human Evaluation]** We appreciate your feedback and have undertaken an extended human evaluation on more test samples under the same setup reported previously. The results we gathered comprise 50 samples and consistently align with the findings initially presented.\n\n|        | Fluency | Coherency | Relevance |   Factual Consistency |   Self-Contextualised |\n|--------|--------:|----------:|----------:|-----:|-----:|\n|   BART |    3.44 |      3.74 |      3.66 | 3.82 | 3.77 |\n| DIRECT |    4.48 |      4.58 |      4.03 | 4.26 | 4.38 |\n|   CACN |    4.59 |      4.63 |      4.17 | 4.34 | 4.39 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed approach behind CACN is well-justified and novel. Much thought has been put into given prior research to better the subcomponents of CACN.\n\nThe procedure for creating CLAN is clear to understand and replicate. Furthermore, the performance of the usage of CACN is evaluated in depth using multiple metrics and models. There has been good effort put to ensure robustness.\n\nThe paper overall is written with great clarity, flow, and cohesion. Nearly every section was easy to understand.", "reasons_to_reject": "My main concerns surround the usage of GPT models. There is reason to believe a contamination issue may exist. As a closed model, we do not know what data the model has trained on. As such, when using the Chain-of-thought Prompting, the model may conceivably be using the information it is aware of regarding the given assertion to influence the step-by-step procedure which then identifies the main claim. This may not seem as much of a problem, but let's use one of the sample posts given as an example: \n\nSocial Media Post: Turmeric and lemon are two simple, cheap and handy things you can use regularly to fight #CoronaVirus. Homemade rasam is also very useful.\n\nNormalized Claim: Turmeric and lemon help fight against coronavirus.\n\nLet’s consider why the GPT model chose to focus on “Turmeric and Lemon” as the primary assertion and not “Homemade rasam”. While it's logical to see rasam as secondary, the model might lean towards \"Turmeric and Lemon\" due to its training exposure to these terms in health contexts. This familiarity, combined with the \"Step-by-Step\" approach of Chain-of-thought Prompting, might skew prioritization. Such biases could overshadow lesser-known or context-specific details in the original post, affecting the claim's genuineness. More commonly discussed or popularized topics may be given more precedence over lesser-known or more specific aspects of the original post, regardless of the actual validity or relevance. \n\nAs another example, take another one of the sample posts: \n\nSocial Media Posts: Certain meats, like beef and chicken, contain an amino acid called leucine. Leucine turns on the chain reaction that stimulates the skin oil’s glands and makes the acne breakouts more likely.\n\nNormalized Claim: Acne breakouts are more common in non-vegetarians. \n\nThe original doesn't mention vegetarianism, yet the normalized claim infers it. Quite reasonably, a claim could be drawn regarding “Certain meats, like beef and chicken, contain an amino acid called leucine” or that “Leucine turns on the chain reaction that stimulates the skin oil’s glands and makes the acne breakouts more likely” instead. Clearly, the model used pre-existing knowledge regarding acne in the context of vegetarianism to draw a connection to phrase it as such. Different interpretations might be factually accurate, but they aren't the same claims.\n\nThese examples illustrate potential biases in using LLMs for claim normalization. Although Table 3 may show CACN's efficiency, the provided examples from the sample data suggest that the normalized claim, while valid, might deviate in meaning from the original. As per my understanding, since there is no way presented to prevent such occurrences, it may be problematic.  \n\nAlthough human evaluation is done to evaluate fluency, coherence, relevance, and factual consistency, it is only evaluated on 20 out of the 6338 posts (.3%). This seems too low of a sample size to draw claims of “qualitative superiority”.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors address noise within social media posts that may hinder proper identification of the main claim or assertion of post. As such, this paper proposes a prompt-tuning method (CACN) which aims to eliminate noise within a post, resulting in a \"normalized claim” more suitable for the verification of an assertion. This paper offers two primary contributions: firstly, the foundational methodology of CACN, and secondly, the development of a novel dataset (CLAN) that includes fact-checked social media posts paired with their respective normalized claims, which can be used for downstream tasks.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LVkpYPdtpy", "reviewer": "Reviewer_uYfo", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your input; we value the opportunity to address your concerns.\n\n**[Quality of the GFC API Output]** We appreciate the recommendation for quality validation of the GFC API. We would like to mention that the GFC API primarily functions as a platform for listing fact-checked posts; it compiles claims that have already undergone fact-checking by credible fact-checkers and journalists. Given its role, the GFC API's reliability is inherently high as it aggregates content produced by trusted sources. Furthermore, we included fact-checking sites with recognition from the International Fact-Checking Network (IFCN) to enhance credibility.\n\n**[Data Leakage]** We appreciate your concern and acknowledge the significance of discussing this matter explicitly within our work. Acknowledging the significant concern of data leakage in models like GPT, we have conducted a small experiment to assess its impact on our approach. We have evaluated our proposed model's performance using recent posts from the past two months (given GPT-3's training up to September 2021). Encouragingly, the results highlight that our model excelled even with new posts. This reaffirms the robustness of our approach amidst potential data leakage challenges. \n\n|           | ROUGE-1 |       |       | ROUGE-2 |       |       | ROUGE-L |       |       | BLEU  | METEOR | BERTScore |\n|------------------|---------|-------|-------|---------|-------|-------|---------|-------|-------|-------|--------|-----------|\n| Models           | P       | R     | F     | P       | R     | F     | P       | R     | F     |       |        |           |\n| CACN (Test set)    |   37.54 |  46.1 | 38.64 |   18.85 | 23.08 | 19.32 |   33.14 | 40.92 |  34.3 |  9.66 |   35.1 |        89 |\n| CACN (In-the-wild) |   38.74 | 51.59 | 42.07 |    24.6 | 32.55 |  26.7 |   36.82 | 49.59 | 40.29 | 16.25 |   38.9 |     87.05 |\n\nIn our revised version, we will include a comprehensive section addressing the data leakage problem and its potential impact.\n\n**[Novellty of the Task]** The claim normalization task shares similarities with text augmentation or summarization. However, it is important to note that the purpose of claim normalization goes beyond simple augmentation. It involves identifying relevant information, reducing ambiguity, and ensuring factual consistency, ultimately improving the credibility assessment process. This task's application within the broader context of fact-checking contributes to easing the segregation of claims (misinformation) within noisy social media posts. In claim normalization, the focus is on extracting the core claim or statement within the text. In contrast, text summarization condenses the entire content while retaining essential details. \n\n**[Baselines]** We have fine-tuned the model weights for our downstream task using our task-specific dataset (CLAN). The objective is to optimize the model's performance for the particular task while retaining the general language understanding and generation capabilities it gained during pre-training. With the inclusion of the extra page in the final version, we will be adding these details.\n\n**[System Comparison]** To perform unbiased results, we conducted experiments under consistent conditions to ensure an unbiased analysis. We employed identical prompts and in-context examples across all models, allowing for direct and meaningful comparisons. Furthermore, we maintained consistency by employing identical examples for fine-tuning across various models, specifically in the context of diverse few-shot setups.\n\n**[Inclusion of results]** We appreciate your suggestion and will ensure its incorporation.\n\n**[Degradation in Few-shot Setting]** We acknowledge the significance of addressing this thoroughly. The task is new and challenging; the models haven't encountered such a situation before. The shift from zero-shot to few-shot learning might perplex the models, resulting in a performance dip. This also sheds light on the intricacies inherent in the task, highlighting the complex relationships between the data points and the model's understanding. This insight is a valuable stepping stone in comprehending the dynamics of few-shot learning and its application in novel scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ claim normalization is an interesting task\n+ understanding and processing social media texts is an important research question", "reasons_to_reject": "1. I strongly recommend that the authors validate the quality of the GFC API output to ensure the accuracy of the ground truth. This step is crucial in ensuring the reliability of the findings.\n2. Since the ground truth output can be assess on the Internet, which may lead to test data leakage problem. I encourage authors to discuss the data leakage problem and its implications in this work.\n3. The novelty of the proposed CACN method appears limited, as it seems to be a direct combination of CoT and Reverse check worthiness. Additionally, the claim normalization task could be seen as a form of text augmentation or summarization.\n4. The description of the baseline is not sufficiently clear. For instance, it is unclear whether the finetuning refers to full-parameter or parameter-efficient prompt tuning. Clarifying this aspect would enhance the understanding of the experimental setup.\n5. It is essential to ensure fairness in the comparisons made. For example, the same method should be applied to different models, and the same base model should be evaluated with different prompt paradigms. This would provide a more comprehensive and unbiased analysis.\n6. Table 4 lacks results for the proposed CACN method. It is important to include these results to provide a complete evaluation of the proposed approach under 0 and few shot setting.\n7. The performance degradation in few-shot learning compared to 0-shot learning requires further explanation. I recommend that the authors examine the few-shot learning exemplars and provide additional insights in this aspect.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes claim normalization, a new task that aims to decompose complex social media posts into normalized claims. This paper then proposes CACN, an LLM-based approach towards this new task which has great performance on the proposed dataset.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IyGJgbqs3K", "reviewer": "Reviewer_H3iM", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Task Motivation]** Thanks for your valuable feedback. In essence, claim normalization is a valuable tool that aims to empower fact-checkers by providing them with a refined and consistent framework to navigate the intricacies of misinformation, ultimately contributing to more effective and informed credibility assessments. \n\nWe also conducted a simple experiment using the Google API to showcase the effectiveness of claim normalization in assisting fact-checkers. Here's a basic outline of the experiment:\n\n- _Data Collection_: We sampled 35 instances from our dataset comprising social media posts and corresponding normalized claims. \n\n- _Evidence Retrieval_: We leveraged Google API to retrieve the top 5 relevant articles for the post and the normalized claim. \n\n- _Evaluation_: All the retrieved articles were scored 0 or 1 based on their relevance to the given input (post or normalized claim). Three annotators independently labeled every retrieved article, and a majority voting approach was utilized to determine the final relevance score. (With the inclusion of the extra page in the final version, we will be adding this in detail.)\n\n- _Conclusion_: The consistent trend observed across the three setups, where normalized claims excelled in top-k precision evaluations for various k values, demonstrates the utility of claim normalization. This observation suggests that ClaimNorm demonstrates enhanced proficiency in evidence retrieval, making it a more concise and advantageous tool for aiding fact-checking procedures. Furthermore, the discernible performance drop in post-based retrieval, mainly as k increased from 1 to 3, highlights the robustness of ClaimNorm's approach in maintaining factual precision and minimizing complications. \n\n| Input query      | Precision@1  | Precision@3  | Precision@5  |\n|------------------|------|------|------|\n| Post             | 0.82 | 0.64 | 0.58 |\n| Normalized Claim | **0.88** | **0.73** | **0.69** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors curated a dataset CLAN on claim normalization, which if publicly available, could benefit future researchers significantly. \n\nAn LLM based approach incorporating chain-of-thought prompting and reverse check-worthiness is also proposed to tackle this research problem. The experiment is thorough and results are reasonable and well support the conclusions. \n\nI really like section 7, both the error analysis section and the human evaluation. In such a new task, I consider quality human evaluation a necessity. ", "reasons_to_reject": "The task itself is interesting, but I am a little concerned about whether this task can benefit future combat of misinformation. That means whether the task of claim normalization can help conquer misinformation or not. The paper mentioned this point in a subjective manner and I am curious if this is indeed true, or an educated hypothesis. \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "In the era of LLMs, we have lots and lots of new information coming on to the Internet. Some of them are valuable, while others are misinformation. Misinformation can be very harmful and lead to seriously bad outcomes. This paper tackles one angle of this problem, which is claim normalization. This paper tries to extract and simplify a statement in a long noisy social media post, to help readers understand better, and potentially help the downstream classifiers check facts. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introducing CACN offers a well-justified and innovative approach that builds upon prior research. The method for developing CLAN is transparent and easily replicable. There is commendable depth in evaluating CACN's performance, ensuring robustness using various metrics and models. The manuscript is praised for its clarity, flow, and cohesion. The topic of claim normalization, especially in the context of social media texts, is seen as both intriguing and significant. A notable highlight is the curated dataset, CLAN, which has potential value for future research if made public. The paper's LLM-based approach, with chain-of-thought prompting and reverse check-worthiness, is robust with well-supported results. Section 7, especially the error analysis and human evaluation, stands out for its quality. It is recommended that the authors consider feedback to further refine their contributions.
Decision: Accept-Findings