$\textit{Lost in Translation, Found in Spans}$: Identifying Claims in Multilingual Social Media
Authors: Shubham Mittal
Megha Sundriyal
Preslav Nakov
Venue: EMNLP 2023 Main (2023)
Keywords: Claim Span Identification
Multilinguality
Social Media
Claims
Low-resource Languages

Abstract: Claim span identification (CSI) is an important step in fact-checking pipelines, aiming to identify text segments that contain a check-worthy claim or assertion in a social media post. 
Despite its importance to journalists and human fact-checkers, it remains a severely understudied problem, and the scarce research on this topic so far has only focused on English. 
Here we aim to bridge this gap by creating a novel dataset, X-CLAIM, consisting of 7K real-world claims collected from numerous social media platforms in five Indian languages and English.
We report strong baselines with state-of-the-art encoder-only language models (e.g., XLM-R) and we demonstrate the benefits of training on multiple languages over alternative cross-lingual transfer methods such as zero-shot transfer, or training on translated data, from a high-resource language such as English.
We evaluate generative large language models from the GPT series using prompting methods on the X-CLAIM dataset and we find that they underperform the smaller encoder-only language models for low-resource languages.

Lost in Translation, Found in Spans: Identifying Claims in Multilingual Social Media
Shubham Mittal Megha Sundriyal
Preslav Nakova
Mohammed Bin Zayed University of Artificial Intelligence
3 Indraprastha Institute of Information Technology Delhi
Shnbhamitd&&@@gaall.ccoo meghas@iitd mensla..maorrmbbzuii
Abstract
Karnataka HC declared, police should not beat people who violate the lockdown. police took Aarthi let them go.
Claim span identification (CSI) is an impor- tant step in fact-checking pipelines, aiming to identify text segments that contain a check- worthy claim or assertion in social media post. Despite its importance to journalists and human fact-checkers, it remains severely un- derstudied problem, and the scarce research this topic so far has only focused on English. Here we aim to bridge this gap by creating novel dataset, X- X-CLAIM, consisting 7K real- world claims collected from numerous social media platforms in five Indian languages and English. We report strong baselines with state- of-the-art encoder-only language models (e.g.. XLMM-R) and demonstrate the benefits of training on multiple languages over alternative cross-lingual transfer methods such as zemo- shot transfer, training translated data, from high-resource language such as English. We evaluate generative large language models from the GPT series using prompting meth- ods on the X-CLAIM dataset and we find that they underperform the smaller encoder-only language models for low-resource languages.
qsái eme srew ure wich fesceator fittt aqper fा पााा 35t Fiा 81 qसel aww Tee aty #iè) UN A (82) अhह ă Ih सामाा #egel 81
The difference is clear. On one hand, Rahul Gandhhi is raising questions Hindi longunge in India. other hand, PM Norendro Medi increases the respect of the UN.)
(English translation:
country speaking
O
EEE रेड Wree (aा
(Engilsh transiation: The growing coravan Bharat Jado Yatro.)
Figure 1: Social media posts from X-CLAIM dataset. The English translation is shown in parenthe- for the Hindi tweet (middle) and for the Punjabi In- stagram post (bottom). The claim spans are in bold.
Claims play an integral role in propagating fake news and misinformation, serving as the build- ing blocks upon which these deceptive narratives are formed. In their Argumentation Theory, Toul- min (2003) described a claim as "a statement that asserts something as true or valid, often with- out providing sufficient evidence for verification." Such intentional or unintentional claims quickly gain traction over social media platforms, result- ing in rapid dissemination of misinformation was seen during recent events such as the COVID- 19 pandemic (van Der Linden et al., 2020) and Brexit (Bastos and Mercea, 2019). To mitigate the detrimental impact of false claims, numerous fact- checking initiatives, such as Politilaat and Snopes, dedicate substantial efforts to fact-checking claims made by public figures, organizations, and social media users. However, due to the iime-intessive nature of this process, many misleading claims dodge verification and remain unaddressed. To address this, computational linguistic approaches have been developed that can assist human fact- checkers (Vlachos and Riedel, 2014; Nakov et al., 2018; Shaar et al., 2020; Gupta and Srikumar, 2021; Nakov et al., 2021a; la; Shaar et al., 2022).
1 Introduction
Social media platforms have become a prominent hub for connecting people worldwide. Along with the myriad benefits of this connectivity, e.g., the ability to share information instantaneously with large audience, the spread of inaccurate and mis- leading information has emerged major prob- lem (Allcott and Gentzkow, 2017). Misinforma- tion spread via social media has far-reaching con- sequences, including the potential to sow chaos, to foster hatred, to manipulate public opinion, and to disturb societal stability (Wasserman and Madrid- Morales, 2019; Dewatana and Adillah, 2021).
Major part of this work was done during a research in- ternship MBZUAI 'We release our X-CLAIM dataset and code at http:///ghhh..commmmzzzzullppp

---

Recently, Sundriyal et al. (2022a) introduced the task of claim span identification (CSI), where the goal is to identify textual segments that contain claims or assertions made within the social media posts. The CST task serves as a precursor to var- ious downstream tasks such as claim verification and check-worthines, estimation.
2 Related Work
Efforts to combat misinformation and fake news have focused on claims in various sources. The ex- isting body of work in this area can be broadly cat- egorized into the following major groups: claim detection (Chakrabarty et al., 2019; Gupta et al., 2021; Wührl and Klinger, 2021; Gangi Reddy et al., 2022a,b), claim check worthiness (Jaradat et al., 2018; Wright and Augenstein, 2020), claim span identification (Sundriyal et al., 2022a), and claim verification (Ma ct al., 2019; Solcimani et al., 2020). Being the precursor of several other down- stream tasks, claim detection has garnered signif- icant attention. Various methods have been pro- posed to tackle claim detection, aiming identify statements that may contain claims (Lippi and Tor- roni, 2015; Levy et al., 2017: Gangi Reddy et al., 2022b). In response to the escalating issue of false claims on social media, there has been surge in the development of claim detection systems specif- ically designed to handle text from social media platforms (Chakrabarty et al., 2019; Gupta et al., 2021; Sundriyal et al., 2021). Recently, Sundriyal et al. (2022a) introduced the task of claim span identification where the system needs to label the textual segments from social me- dia posts, making claim detection systems more ex- plainable through this task.
While efforts have been made in combating mis- information in different languages (Jaradat al., 2018; Barrón-Cedeño al., 2023), research in identifying the claim spans has so far been limited to English. Previously, Sundriyal et al. (2022a) have manually extracted COVID-19 claim spans from Twitter in English. However, the landscape of fraudulent claims goes beyond COVID-19 and Twitter. In this work, we aim to bridge these gaps by studying the task of multilingual claim span identification (mCSI) across numerous social me- dia platforms and multiple languages. To the best of our knowledge, this is the first attempt towards identifying the claim spans in a language different from English.
We design the first data curation pipeline for the task of mCSL, which, unlike Sundriyal et al. (2022a), does not require manual annotation to cre- ate the training data. We collect data from various fact-checking sites and we automatically annotate the claim spans within the post. Using the pipeline, we create a novel dataset, named X-CLAIM, con- taining 7K media platforms in six languages: Eigglihh, Hindi, Punjabi, Tamil, Telugu, and Bengali. Figure showcases few examples from our dataset.
claims from numerous social
While most existing methods to combat fake news are primarily tailored for English (Levy al., 2014; Lippi and Torroni, 2015; Sundriyal et al., 2021, 2022b), in recent times, there has been a surge in interest regardiig the advancement fact- checking techniques for various other languages. ClaimRank (Jaradat et al., 2018) introduced an on- line system to identify sentences containing check- worthy claims in Arabic and English. The Check- That! Lab has organized several multilingual claim tasks over the past five years, progressively expand- ing language support and garnering an increas- ing number of submissions (Nakov et al., 2018; Elsayed et al., 2019; Shaar et al., 2020; Nakov et al., 2021b, 2022). In their latest edition, Barrón- Cedeño et al. (2023) featured factuality tasks in seven languages: English, German, Arabic, Ital- ian, Spanish, Dutch, and Turkish. Gupta and Sriku- mar (2021) introduced X-FACT, comprehensive multilingual dataset for factual verification of real- world claims in 25 languages. Unlike that work, here we focus on extracting the claim from a social media post, rather than fact-checking a claim.
We report strong baselines for the mCSI task with state-of-the-art multilingual models. We find that joint training across languages improves the model performance when compared to alternative cross-lingual transfer methods like zero-shot trans- fer, or training on translated data, from a high- resource language like English. In this work, we make the following contributions:
We introduce the first automated data annota- tion and curation pipeline for the mCSI task.
We create novel dataset, named X-CLAIM, for the mCSI task in six languages.
We experiment with multiple state-of- -ft-----tthttte encoder-only language models and the gener- ative large language models to achieve high performance on the proposed task.

---

Paittott:
Vite ch
PS1 Wiii aSt to Eo "oerter arre arean FURE 3.
6मं पM phahhs Choril
pe
isoo steel atee it you spmn ane sertin star aroa or the
Soogelay MMrrt
Data ooliciion
"Se Mlagent Gooor (
Rormeluxedhhem
8 ditte aire un sye gealin it go shill wt aug arrreet Mee atca ih usag yPH
veal spar
fiz Rei
Englnn (ito)
asseoited with SCOW bostee up poliee deowbstiny
SCPTScon/ d.6
SCFTTconn pal, ndi 0.82
Gusmapen:
soow sved strpe str yfter z0тm "th my armee et aris any
Nermalize daie clai spon Algnnng werth Jetwecn put welie
ScoN Is and at aee yfu zarn mit FUSTE ihe it arcal रर 4tt mar 1990
Sentonee salection
AnteT secteac (od)
MOOT chickr sormalbed clsie ind
Our two-step methodology create the X-CLAIM dataset for the multilingual claim span identification task. The top row shows the data collection (Section 3.1) from a fact-checking website. The bottom row illustrates the automated annotation step (Section 3.2) in which, first, the most similar post sentence (ps) is selected, and then, the claim span is created with the help of normalized claim (nc). We BERTScore-Ress (Zhang et al., 2020) for sentence selection and awcsomo-aliget (Dou and Neubig, 2021) for word alignment between nc and ps2.
Figure 2:
The task of claim span identification remains unexplored due to the lack of datasets in other languages. Sundriyal et al. (2022a) developed a dataset of 7.5K manually annotated claim spans in tweets, named CURT; all the tweets and claim spans in that dataset are in English. Additionally, while there has been interest in claims in other lan- guages, there is notable lack of progress on Indian languages. Here, we aim to bridge this gap.
We highlight that we deal with low-resource languages since we found only a couple of fact- checking websites that analyze social media posts in languages other than English. For each website. we scrape all the fact-checked posts with the help of a web scraping API.
Then, we collect the text of the social media post text and the normalized claim from the web page of each fact-checked post with the help of regu- lar expressions based on the structure of the fact- checking website. Finally, we use various filtering rules to remove posts that are about videos, Insta- gram reels, or when their text is too short or ex- cessively long. These rules help us to collect only the social media posts with a text modality. We provide more details about the process of data col- lection in Appendix A.
3 Dataset
We follow a two-step pipeline to develop our dataset: (i) data collection and (ii) automated an- notation. We present a high-level overview of our proposed data creation methodology in Figure 2. Below, we explain these steps in detail.
3.1 Data Collection
3.2 Automated Annotation
We observe in various fact-checking websites that professional fact-checkers, while investigating a given social media post or news article, first find the claim made in the post, which we call a normal- ized claim, and then they verify whether that claim is true, misleading, false. This is the motivation for the CSI task as a precursor to fact-checking is a step in the fact-checking process as performed by humans. Thus, we leverage the efforts of fact- checkers and we collect data from numerous fact- checking websites that are recognized by the Inter- national Fact- ChettChheking Network (IFCN). We aim to create dataset comprising claims made in so- cial media and in multiple languages, with focus Indian languages. We scrape data from fact- checked posts in six languages: English, Hindi, Punjabi, Tamil, Telugu, and Bengali.
We label the claim- textual segment within the social media post using the human- written normalized claim as guidance from the previous step. The normalized claim can be relied on to be extremely trustworthy since it was man- ually written by professional fact-checkers. How- ever, it does not have to be literally spelled out as part of the social media post. Having this normal- ized claim gives us a good guidance about where to look for the claim span, and we try to do this mapping automatically.
As shown the bottom row in Figure 2, this step includes two substeps: sentence selection and con- version the normalized claim to the claim span. Both substeps use modules that support multiple languages and do not require human intervention.
'The data was scraped May 2023. 4 "https:/ http://wwwwooo www.ooctoparse.com.
Phttps:/ http://wwwwwoot.. poynter.org//ffen.

---

Second, using awcsomc-align (Dou and Neubig, 2021), we find the word tokens in the post sen- tence that align with the word tokens in the nor- malized claim. We then obtain the claim span as English (En)
Hindi (Hi)
Punjabi (Pa)
Tamil (Ta)
Telugu (Te)
Bengali (Bn)
# train
3891
1193
346
100
# dev
400
100
100
30
# test
371
100
100
100
107
102
text len (t)
37.58 #3.59
28.59
29.00=a1s2
26,40=22.00
24.4
29. 29.48
claim len (1)
17.67 11233
17.79 111.2
17.10
14.12
13.5410673
15.00..om.
text len (c)
229.34=200 2015.0
a43.05
145.03
10075
229.63
19..333xm 1:74333
186.63 113.9
186.95 186.95a122.1
claim len (c)
108.95_s1a _1.88
85.25 186.54
85.25
122.23 1okos
104.50..0.s
97.42_1671
Table I: Statistics about the X-CLAIM dataset. The number of samples in the train, the development, and the test splits are reported in first three rows, respectively. Text len and claim len are the average (#: standard deviation) length social media post text and claim span, respectively, number tokens (t) and characters (c), respectively.
First, we look for the most relevant sentence that encapsulates the claim made in the post. We do this by computing similarity score between the nor- malized claim and each of the post's s sentences, and we select the sentence with the maximum score.
Note that we empirically chose the most appro- priate sentence similarity measure for sentence se- lection, after trying variety of similarity mea- sures. Tasks such as machine translation (Dong et al.. 2015) and text summarization (Liu and La- pata, 2019) require evaluation measures that take paraphrasing and synonyms into account while comparing the model's generated text to the gold reference text. We leverage these evaluation mea- sures for sentence similarity. To evaluate the commonly used measures such ROUGE (Lin, 2004), METEOR (Banerjee and Lavie, 2005) and BERTScore, manually annotated the claim spans for 300 randomly sampled posts in the six languages. Then, we evaluated the automatically annotated claim spans when using different sim- ilarity measures against the manually annotated claim spans. The results are shown in Table 2: we can see that BERTScore-Ree Recall yields consistently better performance for finding the annotated spans. For Punjabi and Bengali, we only used awesome- align due to the lack of sentence segmentation module and we observed high-quality F1 scores of 81 11.3%% and 78.6%, respectively.
the sequence of word tokens, starting with the first aligned word token and ending with the last aligned word token in the sentence.
We use Stanza (Qi et al., 2020) to perform sen- tence segmentation for English, Hindi, Tamil, and Telugu. For Punjabi and Bengali, consider the complete post text single sentence since did not find any publicly available sentence seg- mentation tools for these languages. While using awcsomc-align conversion from the normalized claim to the claim span, we used the official repos- itory of Dou and Neubig (2021). Recent works (Yarmohammadi et al., 2021; Kolluru et al., 2022) have used word-alignment to produce silver labels in the target language (like Hindi) using gold la- bels available in the source language (like English). Mittal et al. (2023) used word alignments from awesome-an awesome-align. align, and then considered the longest contiguous sequence of aligned tokens the trans- lated text as the final projected gold labels. Tak- ing the longest contiguous sequence is suitable for tasks where the target text, the gold labels, or both, are relatively short. However, in our mCSI task, the normalized claims and the post texts are quite long (see Table 1). Thus, we took the sequence of words from the first the last aligned word. We found that this yielded better performance than taking the longest contiguous sequence aligned words in the social media post.
Overall, our two-step data creation methodol- ogy yields robust, scalable, and high-quality auto- matically annotated data for our multilingual claim span identification task.
Approach
En
Hi
Ta
Tc
awesome-aligg
70.48
77.27
82.24
82.61
+ROUGE-FI
74.19
80.52
82.62
78.85
+METFOR
78.71
79.50
82,64
81.08
+ -BERTScore-F1
79.60
80.52
82.64
82.73
-BHRTScone-Recall
83.91
80.52
82,64
83.34
Table 2: score (in %6) of the automated annotation for our data curation pipeline when using different sentence similarity measures during sentence selection.

---

3.3 Evaluation Sets and Dataset Analysis
R3. Can large language models (LLMs) such as GPT-4 identify the claims made in multilin- gual social media? (Section 5.3)
We created the evaluation sets with the help of linguistic experts in the six languages. We pro- vided them with nearly 100 samples from the cu- rated data in each language (400 in English) along with detailed annotation guidelines for the CSI task from Sundriyal et al. (2022a). We asked them to annotate the claim spans in the social media posts under the guidance of claims authored by profes- sional fact-checkers. We created training and de- velopment splits in a ratio of 80:20 on the remain- ing curated data. For Telugu and Bengali, we only formed test sets as there were less examples avail- able for these languages. Table 1 shows statistics about the dataset and the splits, and Figure I shows few examples from our X-CLAIM dataset.
R4. How does the automatically annotated X- CLAIM dataset compare prior manually an- notated datasets like CURT? (Section 5.4)
5.1 Training on Multilingual Social Media
We train and compare two kinds of models: MoNO- LINGUAL and MULTILINGUAL models. In a MoNo- LINGUAL setup, we train one model for each language using the available training data in X- CLAIM dataset, whereas in MULTILINGUAI. setup, we train single model on the training data for all languages combined. We note that there is no MONOLINGUAL model for Telugu and Bengali due to the lack of training data for these languages. However, we evaluate the MULTILINGUAL model on them as that model was trained in multiple lan- guages. The performance of these models with dif- ferent pretrained encoders is shown in Table 3.
Table further reports the length of the post text and the claim span. As the claim spans are gener- ally concise and do not contain extra neighboring words, we observe that the claim spans are nearly half of the text of the post for all languages.
4 Experiments
We can see that the MULTILINGUAL models out- perform the MONOLINGUAL models by 1.15% pre- cision and 0.93% F1, averaged over all languages (except for Telugu and Bengali). Even though the recall gets hurt by 0.45%, the improvement in Fl suggests that the model does benefit from joint training. We posit that the drop in recall and the gain in precision indicate that the model has be- come more careful when identifying the claims.
Evaluation Measures:
Following Sundriyal al. (2022a), we address mCSI sequence tag- ging task. For evaluation, we use three measures, computed at the span level (Da San Martino et al., 2019): Precision (P), Recall (R), and Fl-score.
Models: We use state-of-the-art transformer- based (Vaswani et al., 2017) multilingual pre- trained encoder-only language models such as mBERT (Devlin et al., 2019), mDeBERTa (He al., 2023), and XLM- XLM-RoBERTa (XLM-R) (Con- neau et al., 2020). We encode each post's token with 10 -uiidd--tttiee tags to mark the claim spans. Other encodings such as BIO, BEO and BEIO performed worse (see Appendix C for de- tailed comparison of encodings). More details about the training given in Appendix B.
5.2 Cross-lingual Transfer from English
We use the English training data in two experi- mental settings and we compare them to MULTI- LINGUAL models. In the first setting, we lever- age the strong cross-lingual transfer capabilities of pretrained multilingual models (Wu and Dredze, 2019). We take MONOLINGUAL models for En- glish and test them on the remaining five languages. In this setting, monolingual-Engginn models. In the second set- ting, which we call translate-train models, we translate the English training data to the target lan- guage and we train a model only on the translated data. To perform translation of social media posts, we use Google translate," 6 and we project the claim spans (in English), or the token labels, on the trans- lated post using our automated annotation pipeline (see Section 3.2 for detail).
have zero-shot transfer from
5 Results
We carry out an exhaustive empirical investigation to answer the following research questions:
R1. Does the model benefit from joint training with multiple languages? (Section 5.1)
R2. Do we need training data in low-resource lan- guages when we have abundant data in high- resouree languages? (Section 5.2)
'We consider English to he a high-resource language.
Vntree//trannsateeggogl..o/

---

English
Hindi
Punjabi
Tamil
Telugu
Bengali
Model
R
monotingual models (arain using only training data
furger language
MAEKT
70.30
77.08
69 86
17:57
88 99.93
79.03
69:40
94.22
76.93
73,83
87.74
76.94
n/a
n/a
nia
n/a
mD:BERTa
73.79
75.94
92.30
69.55
75.78
67.68
76.29
XLMR
71.56
81.51
72.79
75.34
94.49
$1.09
68.88 85
93.58
75.62
72.68
80.42
71.82
n/a
n/
m/a
n/a
n/a
m/a
muiniingua models (train uzing training data all longuages)
MRERT
70.86
77.01
70.39
16.16
90.40
$0.04
68,
88
73.96
73.80
85.94
76.57
79.58
80.74
78.07
76.79
86.22
79.39
mDDBBERTa
72.25
80.90
73.01
75.94
92.66
80.87
70.62
90.99
76.27
78,21
88.69
80.29
82.34
87.10
82,92
77.11
85.89
79.24
XLM-R
72.45
78.61
71.93
75.30
78.63
73.65
87.95
77.03
73.23
85.29
zere-shot transfer from monotinguai-
Engligg English models
MBEKTT
n/a
n/a
74.53
83.46
76.51
66.84
79:11
69.91
76.92
72.77
70.85
79.58
68.68
70.31
72.74
80.91
74.21
mDcBERTaa
75.71
80
73.18
76.78
75.44
84.42
79.49
75.88
XLM-R
D/a
n/a
73:42
88.2
77.42
70.88
92. 24.44
76.68
76,88
78.89
74.33
80.44
79.28
77,81
73.67
80.37
75.04
transiate-troii models (train on transfated training data from English to torget longuage)
MBIET
naa
sáa
n/a
78.60
86.52
79.55
67.33
92. 22.80
74.70
75.93
81.82
76.04
75.56
72.07
71,76
70.16
83.97
74.36
mDcBERTH
mna
n/a
76.
87.43
78.77
68 84
91.75
75.36
77,46
89.06
80.15 13
82. 18
73.63
75.16
72.41
88.78
77.57
XLM-R
83.37
76.11
68.97
94.43
76.53
77.97
77.50
77.07
Table 3: Precision (P), recall (R) and Fl performance (in %) of pretrained encoder-only language models in different settings on X-CLAIM dataset. "The monolingual models for Telugu and Bengali are not available (n/a) due to the lack of training data for these languages. *n/a: we do not evaluate on the English test set since we focus on the cross-lingual transfer from English to the target language. The reported numbers are the median of three runs with different seeds as high variance was observed across the fine-tuning runs. The best scores are bold.
Both the zero-shot transfer and the translate- train models are almost consistently worse than the MULTILINGUAL models (in terms of F1) for all five languages. The translate-train models show a drop of 1.19% FI, whereas zero-shot transfer mod- els are 2.13% Fl behind MULTILINGUAL. This of- fers strong evidence that the training data in low- resource languages helps over the training data in a high-resource language.
The generated response may contain words that are either not present in the post synonyms of words from the posts. Thus, we treated the re- sponse like normalized claim (Section 3.2) and we passed it through our automated annotation step (Section 3.2) to create the corresponding claim span. We evaluated the predicted claim spans with respect to the gold claim spans. More details about this setup are given in Appendix D.
Interestingly, we notice that zero-shot transfer models are consistently worse than translate-train ones when using mBERT and mDeBERTa, for all five languages. For instance, with mBERT, zero- shot transfer models are worse by 2.92% F1. How- ever, with XLM-R, zero-shot transfer models are better than translate-train models by 1.15% preci- sion and 0.64% F1. We believe that this is because XLM-R has stronger cross- cross-lingual transfer capa- bilities, stemming from its larger pretraining data compared to mBERT and mDeBERTa.
Zero-shot Prompting. We experiment with four prompts that use no examples: IDENTIFY, Ex- TRACT, SPAN, and LANGUAGE. The exact prompt structure is given in Figure 5 in the Appendix. Ta- ble 4 shows their performance when used with dif- ferent LLMs on X-CLAIM dataset.
We noticed that the LLMs mostly responded in English even when asked to analyze post in another language. One reason could be that the prompts do not explicitly specify the language the LLM should respond in. Since our automated an- notation step is language-agnostic the correspond- ing claim span is in the target language. To over- come this, we asked the LLM to respond in the tar- get language with the LANGUAGE prompt. Interest- ingly, and unlike GPT-3.5 and GPT-4, the perfor- mance of T-DV3 with LANGUAGE prompt signifi- cantly dropped by 12-37% Fl (averaged over all languages except English) when compared to the other three prompts. This suggests that T-DV3 is weaker in a multilingual setup.
5.3 Evaluating the GPT Series LLMs
We experiment with several large language mod- els (LLMs):7 text-davinci-003 (T-DV3), gpt-3.5- turbo (GPT-3.5) and gpt-4-0314 (GPT-4) on the mCSI task using the OpenAl API. We prompted each LLM with each social post from the test sets in our X-CLAIM dataset and we asked the LLM to respond with the claim span.
http:/ppaatoroooopaa..ioo hom.aa.coom/meeeepemmmmmmmmooo www.ccooooocccccccccccco.oo.ooc/ fhoossmmdels "https://platforrmoomna wtw.:/foooommmmea..coooo docs /a///////aai-rfeeeennc refereeee
"https://platformmoor

---

English R
Hindi
Punjabi
Tamil
Telugu
Bengali
Model
F1
F1
R
F1
Inmmrry pronpe Identify centrat clain
T-DV3
70.07
60.64
61.83
70.96
63.95
60.55
67.99
92.85
72.67
71,93
48.15
48.90
73.66
42.28
46
66.43
84.49
67.37
GPT-3.5
69.76
74.68
69.28
73.82
83,72
75.09
62.98
97.28
72.72
72,06
76.53
71.08
79.90
69.53
72.09
64.47
98.26
74.00
71.89
76.72
78.80
74.64
71.96
66.95
EXTRACT prompt:
Extract the central claim
T-DV3
70.00
61.70
63.01
77.97
36.96
41.62
65.07
75.29
58.79
76.47
35.70
38.28
74.75
31.91
36.57
72.04
47.82
48,93
74,02
68.75
74.70
76.59
80.72
69.85
72.63
97.88
73.59
GPT-4
74.53
75.05
71.96
76.92
78.62
74,70
64.83
92.32
71.79
73.04
74.07
70.31
82.21
71.93
74.63
68
89.
72.97
Shad prompt:
Extract the central claim spon
T-DV3
67.35
54.78
56.99
73.33
28.52
33.78
67.46
42.67
41.15
72.76
25.41
29.57
70.92
26.71
32.90
62.45
33.97
33.40
3.5
69.05
71.48
67.07
75.44
70.91
69.23
66.81
82.25
69:03
68.53
70.53
66.39
79.35
66.60
68.96
71.47
73.40
67.78
GPT-
80,79
74.19
74.46
84.99
69.32
72.92
77.39
69.87
68.45
82,58
62.71
68 24
85.56
64.49
70.84
77.18
69.07
68.60
LANGUNGE prompt:
Extract the central cloim <Longuoge>
T-DV3
70.60
62.37
63.35
80.90
20.59
28.
73.63
26.05
77.28
09.65
81.31
08:35
14.73
72.93
23.44
GPT-3.5
70.70
76.65
70.49
73.61
79.61
72.97
64.04
83,52
67.77
77.81
71,24
70.46
82.44
67.72
71.43
68.90
79.64
69.94
GPT-
74.76
75.66
72.41
79.51
80.87
77.27
64.21
92.31
71.23
81.47
75.38
75.32
84.96
72.75
76.15
68.95
83.50
71.14
Table 4: Performance (in %) of the LLMs: text-davinci-003 (T-DV3), gpt-3.5-turbo (GPT-3.5) and gpt-4-0314
(GPT-4) the X-CLAIM dataset using zero-shot prompting. GPT-4 nearly always shows the best performance whereas GPT-3.5 shows consistent and sigmificant gains as compared to T-DV3. Huge performance drops are observed in T-DV3 with LANGUAGE prompt when compared to the remaining three language- loose prompts.
We further find that GPT-4 is nearly always bet- ter than GPT-3.5 by an average of 4.23% preci- sion and 1.5% F1 over the four prompts. GPT-3.5 consistently outperformed T-DV3 by an average of 35.96% recall and 27.6 27.63% F1, but it lags behind by 0.5% in terms of precision.
In the case of GPT-4, the best setting uses the LangUaGE prompt with 10-shot in-context learn- ing for the six languages. Figure 3 compares the two models in terms of FI scores; we further of- fer comparison in terms of precision and recall in Table 11 of the Appendix.
We can see in Figure 3 that MULTILINGUAL mDeBERTa outperforms GPT-4 by 2.07% FI, av- eraged over the six languages. GPT-4 shows com- petitive performance with mDeBERTa in English, Hindi and Punjabi. On the remaining three lan- guages, mDeBERTa outperforms GPT-4 by a large margin 2-7% F1. This suggests that LLMs show strong performance on high-resource languages like English, but still lag behind smaller fine-tuned LMs on low-resource languages such as Bengali.
In-Context Learning. Here, we give the model a few labeled examples part of the prompt as shown in Figure 6 of the Appendix. Since GPT-4 outperformed the other two LLMs and showed the best performance with LANGUAGE (Table 4), we experimented with in-context learning with GPT- 4 and LANGUAGE prompt. For Telugu and Bengali, we use examples from translated data (Section 5.2) due to the lack of training data in these languages. The results are shown in Table 5.
We see that in-context learning consistently im- proves Fl score over the zero-shot prompting in all six languages. With more examples shown, the performance increased in English, Hindi and Pun- jabi at the cost of more computation time. We find that 10-shot in-context learning improved the per- formance by average of 2.78% F1 for the six languages in comparison to zero-shot prompting.
2.37
Comparing mDeBERTa and GPT-4. We com- pared the berf-performing fine-tuned encoder-only language model the best-performing generative LLM. The MULTH.NNGUAL mDeBERTa model and GPT-4 yielded the best results for most lan- guages as reported in Table 3, Table 4, and Table 5.
-1.22
-1.31
Crolish
11rd
Punjati
Teril
Dergal
Inggunge
Figure Performance comparison (in %o of the mDe- BERTa model in MULTH.NNGUAL setup and the GPT-4 model on our X- X-CLAIM dataset.

---

English
Hiadi
Punjabi
Tamil
Telugu
Bengali
Examples
74.76
75.66
72.41
79.51
80.57
77.27
64,21
92.31
71.23
81.47
75.38
75.32
84:96
72.75
76.15
68.95
83.50
71.14
1 75.35
76.45
73.23
77.41
79.43
76.22
67.50
87.95
72.15
81.52
82.59
79.21
85.22
74.38
77.49
70.87
84.00
73.18
4 74.76
76.49
72.74
79.09
85.19
79.09
73.90
88.64
76.32
81.30
81.25
78.25
85.08
74.28
77.18
70.29
85.72
73.37
7 74.28
76.31
72.29
78.05
86.40
79.59
71.11
90.30
76.06
78.44
80.61
76.76
85.01
73.06
76.46
70,42
80.54
71.18
10
73.49
91.20
77.58
79.12
81.99
77.92
71.31
Table 5: Performance (in %) of the GPT-4 model using in-context learning with LANGUAGE prompt. The first row contains zero-shot prompting (i.e., no examples) results from Table 4. The best scores are in bold.
5.4 Comparing X-CLAIM and CURT
We analyzed the predictions on the test examples in English and Hindi, and report the kinds of errors made by the two language models in Table 6. Below, we discuss the results of the analysis.
We trained mDeBERTa on the CURT dataset (Sun- driyal et al., 2022a), containing tweets English, and we compared it to the English MONOLINGUAL model (trained with mDeBERTT on English data in X-CLAIM) on the test sets for the six languages in the X-CLAIM dataset. We show the Fl scores for both models in Figure 4 and report the precision and the recall scores in Table 12 in the Appendix.
English. In the first post in Table 6, both models
deviate from the gold claim span. GPT-4 model correctly identifies the presence of the claim but inadvertently veers away from the central check- worthy assertion and focuses the secondary claim. On the other hand, the mDeBERTa model includes information about moisture and bacteria in the mask, contains several grammatical er- rors and lacks clarity. In particular, the phrase 'ev- ery day day legionnaires disease' is confusing and doesn't convey a clear message.
DDwDERTA EUET
CC X-LLUM
90
00-
70.78
75.3578.44
(3
76.25
73.04
20.32
F3 1000
Both models provide similar claim spans for the second social media post, capturing the central as- sertion accurately. However, mDeBERTa contains the extra words 'pregnancy your' at the beginning that are not present in the gold span. These extra words introduce confusion and do not accurately represent the claim made in the social media post.
62-
kngiih 1id
Purgati
Iolagu
Dengali
Larguages
Figure 4: score (in 1%) of mDeBERTa trained on the CURT dataset (Sundriyal et al., 2022a) vs. mDeBERTa trained on the English data X-CLAIM dataset.
Hindi. Claim span identification in other lan- guages is more complicated than in English due the lack of proper guidelines pertaining to their lin- guistic characteristics. In the first example, GPT-4 almost accurately predicted the span, missing the word 'शीमयती" in the beginning. While mDeBERTa predicted both the claim and the premise. defying the very purpose of the task, which is to extract pre- cise claim phrases from the post.
The mDeBERTa model fine-tuned on the X- CLAIM English data performs competitively in English with the CURT trained model and shows 3.52% F1 average gain over the remaining five lan- guages. Note that CURT is manually annotated and is twice larger than the English part of the X- CLAIM dataset. This offers empirical evidence of better model generalization when training on the X-CLAIM dataset compared to the CURT dataset.
In the second post, both models performed well overall. However, we observe a similar issue as for English: the inclusion of additional phrases along- side the claim spans, which can potentially detract from the clarity and precision of the claim. This in- dicates that these models struggle to make precise decisions about claim boundaries.
6 Error Analysis
In this section, qualitatively analyze the er- rors made by the best-performing MULTILINGUAL mDeBERTa model. To provide insights on how LLMs can be improved for this task, we also discuss the errors made by GPT-4 its best- performing setting of 10-shot in-context learning.
We can conclude that for both languages, the models can identify the claim but might propose wider boundaries, including extra words.

---

Test Instances
GPT-4
mDeBERTa
Post: Mask wearers beware. caller radio talk show re- cently shured that his wife was hospitalized and told she COVID and only a couple days left live. doctor friend suggested she be tested for legionnaires disease because she wore the same mask every day all day long. Turns out it WAS legionnaires discase from the moisture and bacteria in her mask. She given antibiotics and within two days was better.
A doctor friend suggested be tested for legionnaires discase because she wore the same mask every day all day long.
suggested she tested legionnaires disease because she wore the same musk every day day legionnaires disease from the moisture bacteria
English
Claim span: Turns out it WAS legionnaires discase from the moisture und bacteria in her mask.
Post: If you are experiencing pregnancy complications your doctor tells you that abortion is necessary, find new doc- tor. You und your buby deserve better. Abortion is never medi- cally necessary to save the life of the mother.
Abortion is never medically necessary to save the life of the mother.
pregnancy your Abortion is never medically necessary to save the life of the mother.
Claim span: Abortion never medically necessary to save the life of the mother.
Post: डोमयरी सायतl žेz iहरगeeीीर नें "बाद yut रारि रे aाeदeाार' sit 2äz iasurette a) रिपासनक Re Rean अ-हो got A Read पयी से भाशाप ğाe रिn कााी की साम हाया शायाार अтयता qus hh पिकरा (Mrs, Amrita Devendra Fadnavis shows sorry state of affairs to 'Architect of Flooded Pune City' Mr. Devendra Fadnavis. He brought out the work done by BJP in Pune in the last 5 years. Thanks Amrita. A common Punekar.)
arria ेे instreftrr à 'ærı gui शहहर के बािमकर' aft ेæे iagordete बो करपपरररा łिीी रिाारट (Amrita Devendra Fudnavis showed the sorry affairs the "architect of flood-filled Pune city' Mr. Devendra Fadnavis.)
arraat ेदक iasvreer A भकाद got बाहर as aासमकर" elt ेहă ingoreftt aग बायनरा fिाीी दिराडट उनशी got से सिpर 5 यपी सेj साकापा दाश रिर ादद काारी aो बामरे बाबा (Amrita Devendra Fadnavis showed the sorry state of affairs to the "architect of flood-filed Pine city:. He broughu out the work done by BIP in the last 5 in Punc.)
Hindi
Claim span: अपयती अtाal dăg uagorete a "anı gel पाडर & बायं sी ेसă पंपगरीी को अेयपमन fिीारà ßिœार (Smi. Am- rita Devendra Fadnavis shows sorry state of affoirs to 'Architect of Flooded Pune City' Mr. Devendra Fadnavis. )
Post: Indonesia का याकार मिााी ds साय बात पास बरा #Alham- dulillah अहराा का qen (Indonesia's 's plane narrowly escaped with passengers, # #Alhandalillak, dhanks to Allah.)
Indonesia का लाहार अदाापीर साय बाह बास बााा
Indonesia का उहाा अहारीरर साय बास बाह बसा #Alhamdulillah (Indonesia's plane narrowly excaped with passengers. #Alhamdulilahh
# Aalhanduiillah ateenle यह ags & (Iudonesia's 's plane narrowly escaped with passengers. #Alhamdulillahh Thanks to Allah.)
Indonesiu का अहाा पमापीर के साप बाल बार कमा (Iudonesia's 's plane narrowly escuped with passengers.)
Claim span:
Table 6: Error analysis of the GPT-4 model and the MUlTILINGUAL mDeBERTa model on English and Hindi test instances from the X-CLAIM dataset. The social media post and the gold claim span are shown the second column. The predicted claim spans for both models provided in the third and fourth columns, respectively. The English translations for the Hindi examples are given inside parenthesis in italics.
7 Conclusion and Future Work
We observed lower performance for GPT-style generative LLMs when compared to smaller fine- tuned encoder-only language models and we dis- cussed their error analysis in the spirit of improv- ing the LLMs on this task.
We proposed a novel automated data annotation methodology for multilingual claim span identifi- cation. Using it, we created and released a new dataset called X-CLAIM, which consists of real- world claim spans, and social media posts contain- ing them, collected from numerous social media platforms in six languages: English, Hindi, Pun- jabi, Tamil, Telugu, and Bengali. Using state-of- the-art multilingual models, we established strong basclines based encoder-only and generative language models. Our experiments demonstrated the benefits of multilingual training when com- pared to other linss-lingual transfer methods such as zero-shot transfer, or training on the translated data, from a high-resource language like English.
Our work opens many important research ques- tions: (1) How to obtain real-world claims with- out relying on fact-checkers analysis? (2) How to improve the understanding of LLMs about claims and social media in low-resource lan- guages? (3) How to automatically curate multiple worthyy claims made in the post? (4) How improve the evaluation metric for the mCSI task? and (5) How to expand the CSI task to other low- resource languages? We plan to address these re- search questions in future work.

---

Limitations
Tuhin Chakrabarty, Christopher Hidey, and Kathy McKeown, 2019. IMHO fine-tuning improves claim detection. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 558-563, Minneapolis, Minnesota. Association for Computational Linguistics.
Our X-CLAIM dataset for the mCSI task is lim- ited to six languages. We do not know how well the developed systems will perform in languages that are not considered in this work. Moreover, the proposed dataset handles only the primary claim in the given social media post and ignores any other potentially worth-worthy claims that the post might contain. In practice, the post may contain multiple worthy claims.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary. Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-linguat representation learning at scale. In ACL Conference, pages 8440-8451.
Ethics
Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news article. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-UCCNL"), pages 5636-56446 Hong Kong, China. Association for Computational Linguistics.
Broader Impact: Our models and data will help fact-checkers filter out extraneous informa- tion, thus saving them significant amounts of time and resources.
Data: We place the utmost importance on user privacy. As a result, we have no intention of dis- closing any information about the users. The data we curated is solely for research purposes, ensuring that user confidentiality and privacy are protected.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Assaci- ation for Computational Linguisties.
Environmental Impact: It critical to acknowl- edge the environmental consequences of training large language models. In our case, we mitigate this concern to some extent by focusing primarily fine-tuning pretrained models rather than train- ing them from scratch.
Hernawan Dewatana and Siti Ummu Adillah. 2021. The effectiveness of criminal eradication on hoax in- formation and fake news. Law Development Journal, 3(3):513 133-5533-200.
References
Hunt Allcott and Matthew Gentzkow. 2017. Social me- dia and fake news in the 2016 election. Journal of economic perspectives, 31(2):211-23)
Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for mul- tiple language translation.
Proceedings of the 53rd Anmual Meeting of the Association for Compu- tational Linguistics and the 7th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 1723-1732, Beijing, China. Association for Computational Linguistics.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR:
An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACL Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Transla- tion and/or Summarization, pages 65-72, Ann Ar- bor, Michigan. Association for Computational Lin- guistics.
Zi-Yi Dou and Graham Neubig. 2021. Word alignment by fine-tuning embeddings parallel corpora. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 2112 2122-2128.
the
Alberto Barrón-Cedeñño, Firoj Alam, Tommaso Caselli, Giovanni Da San Martino, Tamer Elsayed, An- drea Galassi, Fatima Haouari, Federico Ruggeri, Ju- lia Maria StruB, Rabindra Nath Nandi, et al. 2023. The clef-2023 checkthat! lab: Checkworthiness, subjectivity, political bias, factuality, and authority. In European Conference on Information Reirieval, pages 506-517. Springer.
Tamer Elsayed, Preslav Nakov, Alberto Barrón- Cedeno, Maram Hasanain, Reem Suwaileh, Gio- vanni San Martino, and Pepa Atanasova. 2019. Overview of the clef-2019 checkthat! lab: auto- matic identification and verification of claims. In Ex- perimental IR Meets Multilinguaiity, Multimodality, and Interaction: 10th International Conference of the CLEF Association, CLEF 2019, Lugano, Switzer- land, September 9-12, 2019, Proceedings 10, pages 301-321. Springer.
Marco T Bastos and Dan Mercea. 2019. The brexit bot- net and user- user-generse generated hyperpartisan news. Social science computer review, 37(1 -5.5544.

---

Revanth Gangi Reddy, Sai Chetan Chinthakindi, Yi Fung, Kevin Small, and Heng Ji. 2022a. A zero-shot claim detection framework using question answer- ing. In Proceedings of the 29th International Con- ference on Computational Linguistics, pages 6927- 6933, Gyeongju, Republic of Korea. International Committee on Computational Linguisties.
for Computational Linguisties (Volume 1: Long Pa- pers), pages 2502-2517, Dublin, Ireland. Associa- tion for Computational Linguistics.
Ran Levy, Yonatan Bilu, Daniel Hershcovich, Ehud Aharoni, and Noam Slonim. 2014. Context depen- dent claim detection. In Proceedings of COLING 2014, the 25th International Conference on Compu- tational Linguistics: Technical Papers, pages 1489- 1500.
Revanth Gangi Reddy, Sai Chetan Chinthakindi, Zhen- hailong Wang. Yi Fung, Kathryn Conger, Ahmed ELsayed, Martha Palmer, Preslav Nakov, Eduard Hovy, Kevin Small, and Heng Ji. 2022b. News- Claims: A new benchmark for claim detection from
Ran Levy, Shai Gretz, Benjamin Sznajder, Shay Hum mel, Ranit Aharonov, and Noam Slonim. 2017. Un- supervised corpus-wide claim detection. In Pro- ceedings of the 4th Workshop Argument Mining, pages 79-84, Copenhagen. Denmark. Association for Computational Linguistics.
news with attribute knowledge. In Proceedings of the 2022 Conference on Empirical Methods Nat- ural Language Processing, pages 6002-6018, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Ashim Gupta and Vivek Srikumar. 2021. X-fact: A new benchmark dataset for multilingual fact check- ing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 2: Short Pupers), pages 675-682, Online. Association for Computa- tional Linguistics.
Marco Lippi and Paolo Torroni. 2015. Context-
independent claim detection for argument mining. In Twenty-Foud Fourth International Join! Conference on Ar- rificial Intelligence, pages 185-191.
Shreya Gupta, Parantak Singh, Megha Sundriyal, Md. Shad Akhtar, and Tanmoy Chakraborty. 2021. I. LESA: Linguistic encapsulation and semantic amal- gamation based generalised claim detection from on- line content. In Proceedings of the 16th Conference of the European Chapter of the Association for Com- putational Linguistics: Main Volume, pages 3178- 3188, Online. Association for Computational Lin- guistics.
Yang Liu and Mirella Lapata. 2019. Text summariza- tion with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730-3740, Hong Kong, China. Association for Computational Linguisties.
Jing Ma, Wei Gao, Shafiq Joty, and Kam-Fai Wong- 2019. Sentence-leve evidence embedding for claim verification with hierarchical attention networks. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2561- 2571, Florence, Italy, Association for Computational Linguistics.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. DeBERTav3: Improving deBERTa us- ing ELECTRA-style pre-training with gradient- disentangled embedding sharing. In The Eleventh International Conference Learning Representa-
tions.
Shubham Mittal, Keshav Kolluru, Soumen Chakrabarti, and Mausam 2023. mOKB6: A Multilingual Open Knowledge Base Completion Benchmark. In Pro- ceedings of the 61 Annual Meeting of the Associa- tion for Computational Linguistics (Volume 2: Short Papers), pages 201-214, Toronto, Canada. Associa- tion for Computational Linguistics.
Israa Jaradat, Pepa Gencheva, Alberto Barrón-Cedeñoo Lluís Mårquez, and Preslav Nakov. 2018 Claim- Rank: Detecting check- worthy claims in Arabic and English. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 26-30, New Orleans, Louisiana. Association for Computational Linguistics.
Shubham Mittal and Preslav Nakov. 2022. IITD at WANLP 2022 shared task: Multilingual multi- granularity network for propaganda detection. In Proceedings of the The Seventh Arabic Natural Lan- guage Processing Workshop (WANLP), pages 529- 533, Abu Dhabi, United Arab Emirates (Hybrid), As- sociation for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Keshav Kolluru, Mugeeth Mohammed, Shubham Mit- tal, Soumen Chakrabarti, and Mausam 2022. Alignment-augmenneee consiseent translation for mul- tilingual open information extraction. Proceed- ings of the 60th Annual Meeting of the Association
Preslav Nakov, Alberto Barrón-Cedeñan Giovanni da San Martino, Firoj Alam, Julia Maria StruB, Thomas Mandl, Rubén Míguez, Tommaso Caselli, Mucahid Kutlu, Wajdi Zaghouani, et al. 2022. Overview of the 2022 checkthat! lab on fighting

---

the covid-19 infodemic and fake news detection. In Experimental IR Meets Mulrilingaalit, Multimodal- ity, and Interaction: 13th International Conference of the CLEF Association, CLEF 2022, Bologna, Italy, September 5-8, 2022, Proceedings, pages 495- 520. Springer.
Amir Soleimani. Christof Monz, and Marcel Worring. 2020. Bert for evidence retrieval and claim verifi- cation. In Advances in Information Retrieval, pages 359-366, Cham. Springer International Publishing.
Megha Sundriyal, Atharva Kulkarni, Vaibhav Pulastya, Md. Shad Akhtar, and Tanmoy Chakraborty. 2022a. Empowering the fact-checkers! automatic identifica- tion of claim spans on Twitter. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Lunguage Processing, pages 7701-7715, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Preslav Nakov, Alberto Barrón-Cedemn Tamer Elsayed. Reem Suwaileh, Lluís Mårquez, Wajdi Zaghouani, Pepa Atanasova, Spas Kyuchukov, and Giovanni Da San Martino. 2018. Overview of the clef-2018 checkthat! lab on automatic identification and ver- ification of political claims. In Experimental IR Meets Multilinuuality, Multimodality, and Interac- tion: 9th International Conference of the CLEF Asso- ciation, CLEF 2018, Avignon, France, September 10- 14, 2018, Proceedings 9, pages 372-387. Springer.
Megha Sundriyal, Ganeshan Malhotra, Md Shad Akhtar, Shubbashis Sengupta, Andrew Fano, and Tanmoy Chakraborty. 2022b. Document retrieval and claim verification to mitigate COVID-19 misin- formation. In Proceedings of the Workshop on Com- bating Online Hostile Posts in Regional Languages during Emergency Situations, pages 66-74, Dublin, Ireland. Association for Computational Linguistics.
Preslav Nakov, David Corney, Maram Hasanain, Firoj Alam, Tamer Elsayed, Alberto Barrón- Cedeño, Paolo Papotti, Shaden Shaar, and Giovanni Da San Martino. 2021a. Automated fact-checking for assisting human fact-checkers. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. IJCAI-2I, pages 4551- 4558, International Joint Conferences on Artificial Intelligence Organization. Survey Track.
Megha Sundriyal, Parantak Singh, Md. Shad Akhtar, Shubhashis Sengupta, and Tanmoy Chakraborty. 2021. Desyr: Definition and syntactic representa- tion based claim detection on the web. In Proceed- ings of the 30th ACM International Conference Information Knowledge Management, CIKM '21, page 1764-1773, New York, NY, USA. Association for Computing Machinery.
Preslav Nakov, Giovanni Da San Martino, Tamer Elsayed, Alberto Barrón-Cedeño, Rubén Miguez. Shaden Shaar, Firoj Alam, Fatima Haouari, Maram Hasamain, Watheq Mansour, al. 2021 lb. Overview of the clef-2021 checkthat! lab on detecting check- worthy claims, previously fact-checked claims, and fake news. In Experimental IR Meets Multilinguality. Multimodality, and Interaction: 12th International Conference of the CLEF Association, CLEF 2021. Virtual Event, September 21-24, 2021, Proceedings 12. pages 264-291. Springer.
Stephen E Toullmn. 2003. The uses of argument. Cam- bridge university press.
Sander van Der Linden, Jon Roozenbeek, and Josh Compton. 2020. Inoculating against fake news about covid-19. Frontiers in psychology, 11:566790.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Peng Qi, Yuhao Zhang. Yuhui Zhang, Jason Bolton, and Christopher D. Manning, 2020. Stanza: A Python natural language processing toolkit for many human languages. Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics: System Demonstrations.
Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. In Proceedings of the ACL 2014 Workshop on Lan- guage Technologies and Computational Social Sci- ence, pages 18-22, Baltimore, MD, USA. Associa- tion for Computational Linguistics.
Shaden Shaar, Nikola Georgiev, Firoj Alam, Gio- vanni Da San Martino, Aisha Mohamed, and Preslav Nakow. 2022. Assisting the human fact-checkers: Detecting all previously fact-checked claims in doc- ument. In Findings of the Association for Computa- tional Linguistics: EMNLP 2022, pages 2069-20800 Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Herman Wasserman and Dani Madrid- Morales 2019.
An exploratory study of "fake news" and media trust in kenya, nigeria and south africa. African Journal- ism Studies, 40(1):107-123:
Dustin Wright and Isabelle Augenstein. 2020. Claim check- worthiness detection positive unlabelled learning. In Findings of the Association for Compu- tational Linguistics: EMNLP 2020, pages 476-488, Online. Association for Computational Linguistics.
Shaden Shaar, Alex Nikolov, Nikolay Babulkov, Firoj Alam, Alberto Barrón-Cedeno, Tamer Elsayed, Maram Hasanain, Reem Suwaileh, Fatima Haouari. Giovanni Da San Martino, et al. 2020. Overview of checkthat! 2020 english: Automatic identification and verification of claims in social media. CLEF (Working Notes), 2696.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, be- cas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on

---

Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IICNLPP, pages 833-844, Hong Kong, China. Association for Com- putational Linguistics.
Amelie Wührl and Roman Klinger. 2021. Claim detec- tion in biomedical Twitter posts. In Proceedings of the 20th Workshop on Biomedical Language Process- ing, pages 131-142, Online. Association for Compu- tational Linguistics.
Mahsa Yarmohammadi, Shijic Wu, Marc Marone, Haoran Xu, Seth Ebner, Guanghui Qin, Yunmo Chen, Jialiang Guo, Craig Harman, Kenton Murray, Aaron Steven White, Mark Dredze, and Benjamin Van Durme. 2021. Everything all it takes: multipronged strategy for zero-shot cross-lingual in- formation extraction. In Proceedings of the 2021 Conference on Empirical Methods Natural Lan- guage Processing, pages 1950-1967, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evalu- ating text generation with bert. In International Con- ference on Learning Representations.

---

Lost in Translation, Found in Spans: Identifying Claims in Multilingual Social Media (Appendix)
A Data Collection
Language
# Posts
English
17.337
Various fact-checking websites analyze social me- dia posts, news articles, and other information sources that may spread misleading information. We confine our data collection those websites that meet the following requirements. First, the website should have fact-checked numerous so- cial media posts, at least 100, so that we can have a reasonable-sized dataset. Second, it should have investigated posts containing text. We find that many social media posts investigated by fact- checkers have their claim encapsulated in another modality, such as image or video, than text. The fact-checkers manually find the claims made in the posts, which we call as normalized claim. Our last requirement is that the fact-checking website should provide the normalized claim on the web- page of the fact-checked post.
Hindi
2,378
Punjabi
1,262
Tamil
319
Telugu
261
Bengali
167
Table 7: Number of fact-checked social media posts collected from numerous fact-checking initiatives six languages with the help of web-scraping API.
when the post text the claim contains words like video, कोटी, reel, etc. We find that this rule is almost always correct. Further, we remove the data points when the length of the post or claim is less than 3 words, omitting the erroncously scraped text, or more than 700 words, more like news arti- cles. These filtering steps remove only 2.5% of the total data collected, averaged across six languages.
We find that there are only couple of fact- checking websites that have investigated social me- dia posts in low-resource languages and that meet the requirements discussed above. The website names, along with the number of fact-checked posts scraped from them, are reported in Table 7. For English. we collect data from ThipMedia, FullFact, 10 Snopes," PolitiFact, 12 Factly," 13 and Vishvasnews. 14 We use Vishvasnews for the re- maining languages along with Aajtak for Hindi alone. We find that there are relatively fewer posts in Telugu and Bengali than in other languages, highlighting the difficulty creating data for these extremely low-resource languages.
B Model Training Details
We train our models using the Adam optimizer (Kingma and Ba, 2015) with weight decay of O, 81 = 0.9 and 822 = 0.999. All experiments are carried out on a single A A100 (40 GB) GPU. We use and adapt the code of Mittal and Nakov (2022) for our task. The models are trained with three differ- ent random seeds and we report the median of three evaluation runs since we observed a high variation of scores across the runs.
We do hyperparameter tuning for the learning rate and the batch size over the English data and use the same hyperparameters over the data of the remaining five languages. Driven by the motiva- tion that the base transformer model is pretrained on a large corpus of text and requires less train- ing, we use a smaller learning rate of le-5 for it, but a slightly bigger learning rate of 3e-4 for the token-classifier network. We use a batch size of 32 for training mBERT and mDeBERTa whereas smaller batch size of 16 for the larger model, XLM-R. The maximum sequence length for the three encoder-only language models is set 512 to avoid initializing and training new positional em- beddings. We use early stopping with a patience of 7 epochs to find the best model checkpoint as per the best Fl score over the development set.
We recognize the structure of the webpage for each fact-checking website and write rules (e.g.. regular expressions) to collect the post text and the normalized claim. Once the post text and the normalized claim are collected, we pass the pair through various noise removal filters so that the noisy instances (like the ones that do not meet our requirements but dodged the previous steps) are removed from the data. These include removing
"https:/ www.thhip.m.. media.
10 hhttps: /fullfact.org/
11 "https: wwww.wwwwp.. com/
hhtpps: www.polivifact..oomm /oaiiiiccccccc
"https:/ /factly.nn in/
14 "https:/ www..wtt wwww.wwwwww...... vishvasnews.com.
15 htttps: /ww..ajjaakinn/

---

The development set differently in various training methodologies. For MONOLINGUAL mod- els, we use the development data in the target lan- guage, whereas, for MULTILINGUAL models, we combine and utilize the development sets of all lan- guages. The pranslate-trate models use the devel- opment data in the target language when available (Hindi, Punjabi, and Tamil) and use the translated English development set for Telugu and Bengali.
D Prompting the Large Language Models (LLMs) in GPT series
We use OpenAl API and evaluate text-davinci- 003 (T-DV3), gpt-3.5-turbo (GPT-3.5) and gpt- 1-0314 (GPT-4) models on multilingual claim span identification task through prompting. The prompts used in zero-shot prompting are provided in Figure 5. The decoding temperature is set to 0 and we use the default maximum response length. All the GPT series LLMs were prompted from Oct 16, 2023 to Oct 22, 2023.
We provide the number of trainable parameters of the pretrained encoder-only language models in Table 8. For training on English data, XLM-R consumes nearly hour of GPU runtime whereas mBERT and mDeBERTa take nearly 0.5 hours.
Identify the central claim in the given text. Post: <social media post text> Claim:
Model
# Trainable Parameters
mBERT
177.854,978
the central claim the given Post: <social media post text>
mDeBERTa
278,220,290
XI.M-R
559.892,482
Claim:
Table 8: Number of trainable parameters in the pre- trained encoder-type language models.
Extract the central claim span the given Post: <social media post text>
Cla Claim:
Modelling Details
Extract the central claim given text <language>, Post: <social media post text>
The encoder-only language models are trained in the frame of sequence tagging task where the model needs to predict the correct label for each token in the post text. A randomly initialized feed- forward neural network is placed top of the pre- trained encoder as a token- classifier network. takes as input the contextualized token embeddings (output by the pretrained encoder) and results the probability distribution over the label space. The cardinality of the label space depends on how the tokens are encoded.
Claim:
Figure 5: Zero-shot prompting with four prompts: IDENTIFY (first row), EXTRACT (second row), SPAN (third row), and LANGUAGE (last row). For given cial media post, its text and its language (in LANGUAGE prompt) are placed inside the prompt as shown by >.
Extract the central claim in the given text in <language>, Post: <training example #1: social media post text>
Encoding
Precision
Recall
F1
Claim: <training example #1: claim span>
70.79
84,00
73.61
Extract the central claim in the given text in <language>,
BIO
72.22
82.00
73.52
Post: <training example #2: social media post texts <training example claim span>
BEO
69.23
68.83
61.56
BEIO
72.28
80.38
72.30
Extract central claim the given text <language>, Post: <training example #3: social media post text>
Table 9: Performance comparison (in %) of encoding schemes on the English test set X-CLAIM dataset.
Claim:
<training example #3: claim span>
Extract central claim the given text <languago>, Post: <social media post text>
We experiment with token-level encoding schemes: 10, BIO, BEO and BETO. We train four XLM-R models, one with each encoding, the English training data in X-CLAIM dataset and compare their performance on the English test set in X-CLAIM. The scores are reported Table I0 encoding shows the best Fl performance among different encoding schemes.
Claim:
Figure 6: In-context learning using LANGUAGE prompt with three training examples. For each training exam- ple, the social media post's text, the claim span and its language are placed inside the prompt as shown by < >.

---

English
Hindi
Tamil
Telugu
Approach
P
R
P
R
awesome-align
64.08
95.27
73.62
89.00
74.37
99.85
77.01
96.65
ROUGE-FI
75.39
78.20
81.61
84.55
79:46
94.90
84.25
80,99
METEOR
80.27
80.90
80.91
83.53
79.91
94.70
86.48
83.21
BEReScore-Fi
80.45
81.61
84.55
79.91
94.70
83,44
-BRRTScor- Recall
84.62
86.87
81.61
84.55
79.91
94.70
87.71
84.33
Table 10: Precision (P) and recall (R) scores (in %) of the automated annotation (Section 3.2) when using different sentence similarity measures during sentence selection. use awcsomc-align alone for Punjabi and Bengali: 77.57% precision and 92.92% recall in Punjabi whereas 70.78% precision and 97.68% recall in Bengali.
English
Hindi
Punjabi
Telugu
Bengali
Model
R
F11
P
F1
P
F1
P
R
F1
GPT4
75.28
79.42
06
73.49
77.58
86.11
71.31
75.54
81.79
mDDBERTS
72.25
80.90
73.01
75.94
92.66
8087
7042
90.99
76.27
78.21
88.69
80.29
82 34
87.10
82.92
77.11
85.99
79.24
1,303
15.10
[-44
"C3A6""
F1.60
123
16.50"
12.37
17.36
13.33
"6607
Table 11: Performance comparison (in %) of GPT-4 and MULTILINGUAL mDeBERTa model on X-CLAIM dataset. A denotes the difference between the performance of MULTILINGUAL mDeBERTa and GPT-4.
English
Hindi
Punjabi
Tamil
Telugu
Bengali
Model
F1
F1
F1
F1
F1
F1
CURT
74.96
83.96
75.38
76.55
75
73.60
74.13
76.69
80.91
75.35
82.96
82.63
79.68
74.81
74.76
69.53
X-CLAIM
74.28
80.72
73.79
75.71
91.18
80008
73.18
N8.87
76.78
77.68
80.97
75.44
84.42
74.91
76.25
76.42
79.49
75.88
Table Performance comparison (in %) of the two models: mDeBERTa model trained on CURT dataset (first row) and English MoNoliNguAl mDeBERTa model trained the English data in X-CLAIM dataset (second row).


Reviews:
[{"review_id": "JL4Mz2iYYA", "reviewer": "Reviewer_oZSC", "summary": "", "questions": "- How did the authors collect post texts? (data source and methodology)", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating the contributions of our paper.\n\n[Dataset quality assessment] Based on your feedback, we evaluated the quality of training (and validation) data in Hindi, Punjabi, and Tamil with the help of linguistic experts and fixed guidelines. We provide the experts with 100 randomly sampled post texts (from train+val set) in each language and ask them to manually annotate the claim spans with the help of claims authored by fact-checkers. Then, we compare our automatically-annotated claim spans with their manually-annotated ones, and find a high data quality of 81.3% F1 in Hindi, 81.04% F1 in Punjabi, and 89.73% F1 in Tamil. For the remaining two languages, Telugu and Bengali, we only have test sets, and based on the feedback from R2, we manually annotate the test sets in all six languages to ensure the least noise in them. Our annotation budget only allowed for the size of test sets, and we provided quality assessment numbers of our automatically-annotated train (and validation) sets above.\n\nFor manual annotation of the test sets, we ask the linguistic experts in the six languages to manually annotate the claim spans in the test sets with the help of claims authored by fact-checkers. We re-evaluated our models on these manually-annotated test sets and found that the empirical findings hold, as shown in the paper. \n\n[Question] The post text is provided in fact-checking websites, and we write regular expressions based on the structure of each website to scrape the post text (lines 228-232, in the main manuscript). Example of a regex: \\((post said:?))\\s*(,|:|-)?\\s*((\\'|\")|(“|‘)))(.*(\\s.*)?)(\\.(\\5|”|’))\\. It is worth noting that the regex for each fact-checking website (Snopes, Politifact, etc) is different.\n\nIf accepted, we will utilize the 9th page and add the clarifications and the data quality assessment numbers in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed methodology of creating dataset is interesting and seems effective\n- The multilingual corpus of claim span identification is variable for researchers of fact-checking\n- The authors give thorough experiments and analyses", "reasons_to_reject": "- The quality assessment of the annotation is limited to English", "questions_to_authors": "- How did the authors collect post texts? (data source and methodology)", "paper_topic_and_main_contributions": "This paper proposes a claim span identification dataset for English and lower resource languages (Hindi, Tamil, etc.).\nThe dataset is automatically created using fact check websites and the dataset quality is assessed for English.\nExperiments are conducted in different settings: monolingual, multilingual training, and zero-shot cross-lingual transfer from English.\nThe authors also report performance of LLMs.\nResults suggest joint multilingual training is effective.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1GVou9gs9k", "reviewer": "Reviewer_bWC1", "summary": "", "questions": "Question A: If I understand correctly, the metric (BERTScore) was chosen after a small experiment (manually annotating 45 claims, and checking all metrics' performance on this set). Since this experiment was done for English wouldn't it be possible, that BERTScore's performance for other languages is actually quite poor?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments.\n\n[Dataset size] Our dataset is relatively small in languages except English due to the limited availability of fact-checked social media posts in other languages. We provide the statistics of available fact-checked posts in Table 6 (Appendix) and collect all these posts while creating our dataset. Our work is the first attempt towards claim span identification in these truly low-resource languages (due to lack of annotated data), and we believe that it will spur interest in the community to address these research challenges and build even larger datasets.\n\n[Manual verification of dataset] Based on your feedback, we ask linguistic experts in the six languages to manually annotate the claim spans in the test sets with the help of claims authored by fact-checkers. We re-evaluated our models on these manually-annotated test sets and found that the empirical findings hold, as shown in the paper. Since our annotation budget only allowed for the size of test sets, we evaluated the quality of our automatically collected train (and validation) sets. We provide the experts with 100 randomly sampled post texts (from train+val set) in each language and ask them to manually annotate the claim spans with the help of claims authored by fact-checkers. Then, we compare our automatically-annotated claim spans with their manually-annotated ones, and find a high data quality of 81.3% F1 in Hindi, 81.04% F1 in Punjabi and 89.73% F1 in Tamil. Note that for Telugu and Bengali, we only have the manually annotated test sets.\n\n[Question] We empirically answer this question. Above, we discussed the evaluation of the quality of data created using BERTScore. We use the same 100 randomly sampled post texts (from train+val set) from above and create the automatically-annotated claim spans with the other sentence-selection metrics, just like in English (Table 3). Then, we evaluate these metrics for the other languages using the 100 manually-annotated claims and find that BERTScore performs the best and shows a high average performance of 83.99% F1 over all the languages.\n\nWe will revise our paper in the final version, if accepted, with the answers provided, the performance of models on the manually-annotated test sets, and the performance of all the sentence-selection metrics for the other languages.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- the proposed dataset includes data in six different languages\n- the experiments with monolingual and multilingual models are compelling and seem to be well executed\n- the authors include an error analysis of the performance of their multilingual model", "reasons_to_reject": "- the collected dataset is relatively small (excluding English) with the total number of datapoints ranging from 81 to 1457. Since the posts/claims are not long it would be very feasible to manually verify all claims to ensure the data is of a good quality (compensating for the small size). I might have missed it, but I think at least the test sets should be manually verified.", "questions_to_authors": "Question A: If I understand correctly, the metric (BERTScore) was chosen after a small experiment (manually annotating 45 claims, and checking all metrics' performance on this set). Since this experiment was done for English wouldn't it be possible, that BERTScore's performance for other languages is actually quite poor?", "paper_topic_and_main_contributions": "This paper investigates the claim span identification task in six languages (Tamil, Hindi, Punjabi, Telugu, Bengali, and English). The authors propose a pipeline to automatically collect and annotated data (claim spans) in different languages. The authors then experiment with the dataset training claim span identification systems in monolingual and multilingual setups. The authors experiment also with cross-lingual transfer from English.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fYuN9js2wW", "reviewer": "Reviewer_G3UE", "summary": "", "questions": "1) In table 2, I assume the bolded numbers are the best performing models.  However, in Hindi F1 and Tamil F1, there are better performing models not bolded (mono mDeBERTa and multi mDeBERTa respectively).  Also, do the results consist of multiple runs?\n2) On page 7, line 461, the authors make a claim about why GPT performs worse than their transformer based models.  However, in the error analysis section, the authors seem to point to different aspects that each model gets worse or better than the other.  Therefore, I'm not sure the earlier statement is justified?  The lower GPT performance could be due to a number of factors (e.g. less in-language training data, the need for more prompt engineering) that go beyond the models inability to reason.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your critical feedback.\n\n[Multiple claims] We agree that there may be multiple claims present in the social media post and we collected only the primary or main claim, which is authored by professional fact-checkers for veracity estimation. The remaining claims in the social media post are likely to be less important and maybe easy to find. We acknowledge this as a limitation of our work (see lines 546-549, in main manuscript). We will provide more discussion on this in our revised version. \n\n[Multilingual and monolingual training] Multilingual training has been shown to perform better than monolingual training in different NLP tasks, e.g., open knowledge base completion (Mittal et al., ACL 2023) and semantic parsing (Li et al., EACL 2021). Since claim span identification has not been studied before in multilingual setting (see lines 90-92, in main manuscript), the related works on claim span identification have only been restricted to English due to lack of datasets in other languages. Thus, this empirical finding is not shown in any of the related works, and we discuss the same in Section 5.1 of our paper. \n\n[Multilingual and English-trained model] We see large gains achieved by multilingual models over the English-trained models in Table 2. Example, mBERT-multilingual performs better than mBERT-on-En by a large margin of 10.28% Recall and 5.98% F1 (averaged over all languages except English). We believe that the high-quality training data in other languages of our dataset, when added with English training data in multilingual training, helps the model generalize better and enhance performance.\n\nWe justify automatically annotating the data with the help of our discussion in Section 5.3. We compare two models, one trained on automatically-annotated English data (MOSM), and the other trained on manually-annotated data (CURT), in Figure 3. We find that the model trained on automatically-annotated data outperforms the other model by a huge margin of 7% F1 (averaged over six languages).\n\n[Question-1] Thank you for pointing out the mistake. We will correct it in the revised draft. The results consist of a single run with a fixed random seed for reproducibility (see lines 916-919).\n\n[Question-2] Thank you for highlighting the claim made in line 436. While in the error analysis section, we discussed the kind of errors made by the two models, in line 436, we wanted to mention the possible reasons. We will rephrase the claim into possible reasons and factors why the GPT model may be performing worse, including the ones you mentioned.\n\nThank you for the suggestions to improve our paper writing. We will revise the abstract for easier understanding and update the paper to focus on non-English languages.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The task is important to improving access to fact claim verification on the internet; this is made more important by the fact they include a variety of non-English languages.\n2) The paper is generally easy to understand (despite a few confusing sentences), and the authors propose to release the dataset.\n3) They compare a wide variety of baselines, including models that they train and pretrained approaches.\n4) Their annotation approach is generalizable; potentially for other data sources to be constructed in different languages.", "reasons_to_reject": "1) While the automated annotation approach is understandable, I wonder if the authors deal with the ways this might shape the dataset.  Given that the claims that are used to find the original claim are rewritten, I wonder if this does not skew the dataset to more easily found claims.  Perhaps claims that are more subtle are not included?  I don't think any of this is disqualifying, but I'd like to see more of a discussion of it.\n2) From an NLP modeling perspective (especially the span identification task), the results are in line with what previous work has shown -- multilingual training performs better than monolingual training, with zero-shot modeling often performing slightly worse (such as in the cited Wu and Dredze 2019).  However, this doesn't appear to have been shown on this specific task (looking through the related work).  But it is a finding that is of more narrow interest within the community.\n3) Following with the above point, I think it would be helpful if the authors showed what concrete performance improvements can be gained by using the multilingual model over an English-trained model.  From what I can tell, the difference is usually pretty small.  Is there a concrete performance gain on a specific type of claim that justifies automatically or manually annotating data?", "questions_to_authors": "1) In table 2, I assume the bolded numbers are the best performing models.  However, in Hindi F1 and Tamil F1, there are better performing models not bolded (mono mDeBERTa and multi mDeBERTa respectively).  Also, do the results consist of multiple runs?\n2) On page 7, line 461, the authors make a claim about why GPT performs worse than their transformer based models.  However, in the error analysis section, the authors seem to point to different aspects that each model gets worse or better than the other.  Therefore, I'm not sure the earlier statement is justified?  The lower GPT performance could be due to a number of factors (e.g. less in-language training data, the need for more prompt engineering) that go beyond the models inability to reason.", "paper_topic_and_main_contributions": "The authors focus on the task of claim identification, seeks to label spans of text that contain a claim. They build automatically construct a multilingual corpus of claims by matching manually written claims to the sentence they came from in the linked document.   They do so for English and a set of Indian languages.\n\nThey train several transformer-based sequence labelling models, all using one of mBERT, mDeBERTa, and XLM-R.  They also include several training approaches, including monolingual training, multilingual training, English-only training and training translated from English.  They results do not show a consistent pattern across languages, although the multilingual models tend to work better than the multilingual ones do.\n\nThe authors also explore the performance of GPT models, and find that the best performing model (GPT-3.5-turbo) still performs worse than the best performing translation baseline.  The authors then do a small qualitative analysis of the paper.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I found the two sentences in the abstract beginning at line 16 hard to follow, and I'd suggest a rephrasing.\n\nClearly focusing on non-English languages is very important!  However, I'm not sure that many of the included languages (e.g. Hindi) can be considered low-resource...it is in relative to English certainly.  But compared to languages with much fewer speakers (e.g. Manipuri), this is a high-resource language.  I'd suggest the authors flesh out this point more in the paper."}}]

Author rebuttal:

Meta-review: The paper introduces a multilingual resource (6 languages including English) for claim span identification, created by their proposed methodology. The reviewers appreciate the importance of the task, the dataset creation methodology, and the experiments conducted. Most of their concerns are addressed by the authors' responses.
Decision: Accept-Main