Detecting Propaganda Techniques in Code-Switched Social Media Text
Authors: Muhammad Umar Salman
Asif Hanif
Shady Shehata
Preslav Nakov
Venue: EMNLP 2023 Main (2023)
Keywords: propaganda detection
code-switching
low-resource languages
multilinguality
roman-urdu
natural language processing
TL;DR: We propose a novel task of detecting propaganda techniques in code-switched data, create an annotated corpus for it, and confirm through analysis that directly modeling multilinguality yields better results than translation of code-switched text.

Abstract: Propaganda is a form of communication intended to influence the opinions and the mindset of the public to promote a particular agenda. With the rise of social media, propaganda has spread rapidly, leading to the need for automatic propaganda detection systems. Most work on propaganda detection has focused on high-resource languages, such as English, and little effort has been made to detect propaganda for low-resource languages. Yet, it is common to find a mix of multiple languages in social media communication, a phenomenon known as code-switching. Code-switching combines different languages within the same text, which poses a challenge for automatic systems. Considering this premise, we propose a novel task of detecting propaganda techniques in code-switched text. To support this task, we create a corpus of 1,030 texts code-switching between English and Roman Urdu, annotated with 20 propaganda techniques at fragment-level. We perform a number of experiments contrasting different experimental setups, and we find that it is important to model the multilinguality directly rather than using translation as well as to use the right fine-tuning strategy. We plan to publicly release our code and dataset.

Detecting Propaganda Techniques in Code-Switched Social Media Text
Muhammad Umar Salman, Asif Hanif, Shady Shehata, Preslav Nakov Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI) asif. hanif, shady. shehata, preslav.
ac. ae
Abstract
Unfortunately, this democratic channel also used by malicious users for harmful purposes, e.g.. to spread disinformation and hate speech. An im- portant tool to make such content believable and viral is the use of propaganda techniques. Propa- ganda is the dissemination of biased or misleading information in order manipulate people's be- liefs and opinions towards particular objective. Propaganda has always existed, but social media platforms have made it casier for individuals and or- ganizations to promote their agenda and narratives quickly across large audiences. Propaganda and the spread of misleading claims and opinions neg- atively affects many people and may cause harm. This was particularly seen during the COVID-19 pandemic when incorrect details about the virus and the effectiveness of vaccines spread social media, leading to confusion and mistrust, Propa- ganda is also used as tool manipulate people's emotions and behavior, often for political or ide- ological purposes. This can result in a decline critical thinking abilities, impairing the capacity to make well-informed decisions grounded in factual information and evidence. Thus, there is need to detect the use of propaganda and to combat the negative impact it poses well as to promote more accurate and truthful exchange of information in social media.
Propaganda is a form of communication in- tended to influence the opinions and the mind- set of the public to promote particular agenda. With the rise of social media, propaganda has spread rapidly, leading to the need for auto- matic propaganda detection systems. Most work on propaganda detection has focused on high-resouree languages, such as English, and little effort has been made to detect propaganda for low-resource languages. Yet, it is common find mix of multiple languages in social media communication, phenomenon known as code-switching. Code-switching combines different languages within the text, which poses challenge for automatic systems. Con- sidering this premise, we propose novel task of detecting propaganda techniques code- switched text. To support this task, we cre- ate corpus of 1,030 texts code-switching be- tween English and Roman Urdu, annotated with 20 propaganda techniques at the fragment level. We perform number of experiments contrasting different experimental setups, and we find that it is important to model the multi- linguality directly rather than using translation as well as to use the right fine-tuning strategy. code and the dataset are publicly avail- able at https: thub. com/mbzuai-nlp pr propaganda-codee codeswi tched- text
Social media platforms such as Facebook. Twit- ter, YouTube, and Instagram are used by millions of people around world, and many of these users are bilingual and even multilingual. Due to the increase of multilingual users and informal inter- actions on social media, a large number of people have begun to resort to mixing multiple languages various social media platforms in the form of posts, tweets, comments, and especially chats. This is where the code-swicchin comes into play. Code-switching primarily refers to switching or alternating between two or more languages the same context (Sitaram et al., 2019).
1 Introduction
The rise of social media has brought significant change in the way people consume and share in- formation. With the advent of the Internet and the widespread use of social media platforms, it has become easier for anyone to spread information to wide audience. Social media platforms in the digital era have become the primary source of news consumption and used on daily basis.
WARNING: This paper contains examples and words that are offensive in nature.

---

Name calling/Labelling Loaded Language
ExaggerationnMnniizzatoon
There are only two types of fans. Wo toxic log jo martay waqt tak iss bakwas team ka side nai chorain ge and people like me who realize this team is gobar
Black and White Fallacy/Dictatorship
Name calling/Labeling
Smears
Figure 1: Example of code-switched text annotated at the fragment level. Here, the entire text is labeled as Smears and Black and White Fallacy/Dictatorsiin Translation of the text: There are only two types of fans. Those toxic people who till death will not leave this rubbish team's 's side and people like me who realize this team is dung.
This switching may occur within the same sen- lence or between neighboring sentences. This phe- nomenon of code-switching or mixing multiple lan- guages has now become prevalent on social media platforms among bilingual and multilingual com- munities. It often compensates for lack of proper expression. This helps individuals to express their ideas and thoughts more accurately and convinc- ingly and achieve better impact the readers and the listeners (Nilep, 2006; Tay, 1989; Scotton, 1982). In this work, we aim detect propaganda techniques code-switched text: we use English and Roman Urdu (i.e., Urdu language written using Latin script) as our high-resource and low-resource languages. Although Urdu is typically written in its native Arabic script, most online users prefer to use Roman Urdu in their online textual communication due to the convenience of typing. Figure shows an example of a code-switched text (English and Roman Urdu), where the different fragments the text are labeled as propaganda techniques.
However, face several challenges in this task, including lack of specialized corpora, high- quality and reliable annotations, and fine-tuned models. Our approach aims to overcome these challenges by focusing on propaganda detection on social media platforms. We can summarize the contributions of our work as follows:
1. We formulate the novel NLP task of detecting propaganda techniques in code-switched text.
2. We construct and annotate new corpus specifically for this task, comprising 1,030 code-switched texts in English and Roman Urdu. These texts are annotated the frag- ment level with 20 propaganda techniques.
We experiment with various model classes, including monolingual, multilingual, cross- lingual models, and Large Language Models (LLMs), for this task and dataset, and we pro- vide detailed comparative performance anal- ysis, which can inform future research.
Significant research efforts have been dedicated to various codc-switched tasks in the field of Natu- ral Language Processing (NLP), such as Language Identification, Named Entity Recognition (NER),
Related Work
Propaganda originated in the 17th century in the form of agendas that were propagated during gath- erings and events such as carnivals, fairs, religious festivals, and theatres (Margolin, 1979). From there, propaganda took the form of text and pictures in newspapers and posters. During the American Revolution in the 18th century, newspapers and the printing press in various colonies propagated their views to promote patriotism (Cole, 1975). During the First World War, the British government orga- nized large-scale propaganda campaign, which was claimed by military officials to have played key role in the defeat of the Germans in 1914 (Yourman, 1939). Later, in the early 20th century, after the advent of motion pictures, this medium became a new propaganda tool used to promote military and political agendas.
POS Tagging, Sentiment Analysis, Question An- swering, and Natural Language Inference (NLI) (Khanuja et al., 2020; Jose et al., 2020; Chen et al., 2022: Rizwan al., 2020). However, there has been limited exploration in the domain of propa- ganda detection, particularly for low-resource lan- guages. Previous work in this area has focused mainly on high-resource languages such as English, with propaganda detection at the locument-leved as both binary-class and multi-class classification task Cerréno et al., 2019; Rashkin et al., 2017), well the fragment-level for journalis- tic news articles (Da San Martino al., 2019). Our research represents the first attempt to address pro- paganda detection in code-switched low-resource languages on social media.

---

The Artificial Intelligence (AI) community has primarily focused on textual propaganda. With the widespread accessibility of the Internet in the past two decades, propaganda has proliferated at an enormous scale, compelling research communi- ties to devote efforts towards detecting propaganda in its various forms. Initially, research on propa- ganda in Al started at the document level. Barrón- Cedeño al. (2019) performed binary-class pro- puganda detection the news article level. Their model assessed the level of propaganda based on the presence of keywords, specific writing style, and readability level. Rashkin et al. (2017) focused on distinguishing between four types of documents in their annotated corpus (TSHP-17): satire, hoax, trusted, and propaganda. They took an analytical approach to the language of the news media in the context of political fact-checking and fake news detection.
Moreover, Piskorski et al. (2023a) (SemlVval- 2023 Task 3) worked on detecting the category. the framing, and the persuasion techniques in a multilingual setup. This work makes use of news articles in nine different languages during training and evaluates on additional three languages in the testing phase.
(Dimitrov et al., 2021 a) focused on multimodal propaganda detection, as a multi-label multimodal task that focuses on propaganda detection in memes. They released a corpus of 950 memes, labeled with 22 propaganda techniques. Yu et al. (2021) developed an interpretable model for pro- paganda detection. More recently, Baleato Ro- dríguez et al. (2023) performed multi-task learning with propaganda identification as the main task and metaphor detection an auxiliary task. Their work makes use of the close relationship between propa- ganda and metaphors. They leveraged metaphors which turn improves the model performance, es- pecially while identifying name calling and loaded language. Piskorski et al. (2023b) presented a new multilingual multi-facet dataset comprising 1,612 news articles used for understanding the news (par- ticularly "fake news"). The dataset includes an- notated documents categorized by genre, framing. and rhetoric (persuasion). The annotation of per- suasion techniques is conducted at the span level. using a taxonomy that consists of 23 techniques organized into six categories.
Da San Martino al. (2019) took more fine- grained approach to propaganda detection. To counter the lack of interpretability caused by bi- nary labeling of the entire article, they proposed to detect propaganda techniques in spans/fragments of the text. The annotation in their corpus was at the fragment level, and featured 18 propaganda techniques. This allows for two tasks: (i) binary classification (whether propaganda is present in sentence or not), and (ii) multi-label classification (where a span of text in a sentence can be labeled as one or more classes from a total of 18 tech- niques). The list of the 18 propaganda techniques was selected due to their occurrence in journalis- tic articles. This task led to the creation of the PTC corpus (Da San Martino et al., 2019), which contains 451 1 news articles from 48 news outlets.
Dataset
3.1 Data Collection
We collected data from publicly available sources, including Twitter, Facebook, Instagram, and YouTube. Twitter contributes 60% of the dataset, Facebook 25% Instagram 10%, and the remaining 5% are from YouTube. Four data collectors were selected based on the quality of their submission of 20 examples of code-switched text. They were then instructed to collect a total of 500 texts each. We ensured diversity among the collectors by including two male and two female collectors. The collectors were also advised to gather texts at different times of the day and from different time periods to in- crease the diversity of trending topics. After some filtering, we ended up with 1,030 examples. Some examples were removed as they required exten- sive prior knowledge to understand the annotations, while others were heavily dependent on the source post/tweet to make sense.
There were also some related shared tasks. SemEval-2020 Task 11 (Da San Martino et al., 2020) focused on detecting persuasion techniques in news articles. The task involved identifying specific text spans and predicting their correspond- ing types (14 different techniques). Dimitrov et al. (2021b) extended the scope of fragment-level pro- paganda identification in SemEval-200 Task 6: the task includes the detection of propaganda tech- niques present in both textual and image data. In WANLP'2022 shared task, Alam et al. (2022) fo- cused on detecting propaganda techniques in Ara- bic, specifically using Arabic tweets. A follow up WANLP'2025 task, focused on propaganda in tweets and news articles (Hasanain et al., 2023).

---

3.2 Annotation Process
D
Exampl
Comple
Annotator Training After the data collected, it was annotated by two annotators, who underwent training to prepare themselves for the task. We con- sidered 20 different propaganda techniques whose definitions can be found in Appendix A. Both anno- tators were experienced Al researchers with Mas- ter's degrees, fluent in English, Urdu, and Roman Urdu. Annotating propaganda techniques presents greater challenge compared to tasks such as im- age classification or sentiment analysis. This is due to the annotators' requirement to comprehend each propaganda technique, recall and identify suit- able techniques for specific texts, and accurately label the text spans associated with each propa- ganda technique. The training process consists of three stages. For Stages and 2, both annotators were provided with English text examples contain- ing propaganda. In Stage 1, both annotators were given subset of these examples (Set A) and were asked to annotate them independently. The anno- tators had to identify and list all possible labels. Once they had completed their annotations, they compared their results and cross-questioned each other to resolve any conflicts that had arisen. After reconciling any discrepancies, they prepared a final document. Stage 2 is repetition of Stage 1, but with different set of examples. The annotators notated Set independently and prepared the doc- ument the same way they did for Set A. In Stage the annotators discussed their annotated documents with domain expert who provided valuable feed- back to refine their understanding of propaganda techniques. After discussion, both annotators up- dated their knowledge state and repeated the entire process, this time focusing annotating the text spans and corresponding propaganda labels. This thorough training program ensured that the anno- tators had comprehensive understanding of the propaganda techniques and were well ceeeapre to annotate the code-switched dataset.
Labels Spans Example
Labels Spana
Example
Compare
Annotations
Consensus
Conflict
Resolve Conflict
Resolved
Unresolved
Discard
Figure 2: Our methodology for annotating the code-
switched dataset with propaganda techniques.
Once the annotation was complete, the two anno- tators compared and consolidated their annotations. The consolidation process was done to ensure that there were no discrepancies in the labeling of tech- niques and spans. This step was crucial for en- suring that each example went through the same standard process. If both annotators agreed the annotations, the example was approved and the labeled example added to the dataset.
In the event of conflict, where the two anno- tators had different labels or spans, they worked together to resolve that conflict. The conflict resolution process involved reviewing the propa- ganda class definitions that were under conflict, discussing the relevance of the propaganda class for that particular example, and cross-questioning each other to determine whether to include exclude the propaganda class in question. con- sensus was reached, the example was approved and added to dataset. However, the conflict could not be resolved, the example was discarded. To ensure accuracy and to prevent fundamental errors, regular discussions between annotators and domain experts were conducted. These discussions were crucial to maintain the quality of the dataset, and they helped identify potential errors during the annotation process. The main reason why two an- notations of the same example would differ was primarily because one of the annotators missed a technique instance.
Our approach is highly methodological, to en- sure consistency across all examples in the dataset and to prevent any discrepancies that may impact the quality of our data. Figure visually outlines the systematic process followed while annotat- ing the dataset. The annotation process begins with providing two annotators with an example from the dataset, which they must independently label by identifying the possible techniques used and their corresponding spans within the text.

---

Statistic
Value
Propaganda Techniques
Number of
Avg Span Length
Instances
= Std Dev.
# of Labeled Examples
923
Name calling/Labeling
563
16,60 10.23
of Unlabeled Examples
107
Repetition
15
146.06 81.02
Average Example Length
147.56 + 53.79
Doubt
39
124.87 76.35
Maximum Example Length
400
Reductio hitlerum
8
119.00 19.51
Appeal fean/prejudice
87
144.13 50.20
Minimum Example Length
42
Straw
18
141.38 53.94
Average Span Length
63.64 + 67.81
Loaded language
693
8.78 = 8.22
Maximum Span Length
400
Bandwagon
4
102,00 19.45
Minimum Span Length
2
Smears
382
144,25 60.03
Ohfuscation, Ini. vagueness...
12
139.33 63.04
Total of Words
28452
Glittering generalities (Virtuc)
44
91.97 55.74
Vocabulary Size
7154
Causal oversimplification
86
90,96 + 42.44
Appeal to authority
12
136.08 63.70
Table I: Statistics about our dataset. Here the unlabeled examples are those with no propaganda class assigned.
Red herring
61
142.40 51.66
clichó
43
27.13 1 23.03
Black-and-whit fallacy
34
76.38 38.95
Slogans
24.51 12.11
Whatahoutism
38
163.15
There were rarely disagreements between the two annotators after an example with a conflict has been discussed. This process was repeated for all the examples in the dataset. Then, a quick double- checking over all examples was conducted. The purpose of this iteration was ensure consistency in labeling the spans and the propaganda classes. In this iteration, the annotators were allowed to the labels they made in the first iteration ensure the uniformity of the decision decision-making process while labeling the spans and the propaganda classes.
Exaggeratoon/imimissn
366
85.40 45.23
Flag-waving
27
140.14
Overall
2,577
63.64
Table Total number of instances and average span lengths (number of characters) for each of the propa- ganda classes.
3,4 Statistics about the Dataset
Table shows statistics about the dataset: it has a total of 1,030 examples with 2,577 labeled spans annotated with the use of 20 propaganda techniques the span level.
To annotate code-switched text, we built web- based platform. Snapshots of the interface are shown in Appendix B. Other work, like (Dimitrov al., 2021a), used PyBossa' for annotation. Al- Py
ternatives PyBossa include Label Studio? and INCEpTION. Although these annotation tools are powerful, we found some potential issues with them, such as limited customizability, insufficient documentation, steep learning curve, or need for paid subscription.
In Table we see statistics about the 20 classes with their total number of instances as well as the average span length = the standard deviation. From the number of instances, can be noted that there is class imbalance, where the top-4 class instances comprise 77.6% of the total number of instances. These include Loaded language, Ex- aggeraation/Miniiiisttino Smears, and Name call- ing/Labeling. The other 16 classes contribute less than 5% each. Table 2 shows the varying span lengths different techniques.
3.3 Quality the Annotations
To measure the quality of the annotations, com- puted the Krippendorff's a, reliability co-cfficient developed measure agreement, which takes into account multi-label annotations (Artstein and Poe- sio, 2008). The Krippendorff's s a score between the two annotators for our annotations is 0.76. As per Krippendorff (2011), I), a score above 0.8 indi- cates high level of agreement. Hence, our results demonstrate fairly reasonable agreement between the annotators, particularly considering that the an- notation involved multiple labels.
Figure shows a histogram of number of techniques per example, i.e., the distribution of the number of distinct techniques with respect to the number of examples. For example, in the distribu- tion, we can see y=263. which shows that there are 263 examples that contain only two dis- tinct techniques. Another example (x=0,y=107 shows that for 107 examples there are tech- niques labeled. The maximum number of distinct techniques in an example is 9, which occurs only once in our entire dataset.
'https://yybosaa "https://labelstud. io/ ihttps://inception--rooo io/

---

Nauber of Deseed rolloreo a brande Numbo Exarsloy
Fine- Tuning Strategy Type
Model
Model Name
203
Mi
BERT
Out of Domain
M;
mBERT
Meme Dataset (Teat-Ouly)
Ms
XLM ROBERTa
M4
BERT
Translated
M;
mBEKT
(Code Switched English)
Ms
XLM RoBERTa
Mr
BERT
M:
mBEKT
Code Switched
Mr
RUBERT
M1u
XLM RoBERTa
Mn
RoBERTa (Roman Undu)
My
DeBERTaV3
No fine-tuning
M1a
GPT-3.5. Turba @:20shot
Table 3: List of models and fine-tuning strategies (on particular datasets). Here M. refers to specific model fine-tuned using a specific fine-tuning strategy type.
Figure 3: Histogram of the number of propaganda tech- niques per example.
XLM RoBERTa (Roman Urdu) (Aimlab, 2022) is fine-tuned on a Roman Urdu Twitter dataset (Khalid al., 2021) without making any additions to the vocabulary, which effective as the model attempts to Iearn a better contextual rep- resentation of the existing vocabulary.
4 Experiments
4.1 Models
In experiments, we fine-tuned several state- of-the-art monolingual, multilingual, and cross- lingual pre-trained models on our labeled propa- ganda dataset. We use different fine-tuning strate- gies to adapt these models to our specific task. We also use Large Language Models (LLMs), specif- ically GPT-3.5 Turbo, by providing a few exam- ples as prompts in different few-shot settings (see Appendix C for detail). Below are the pre-trained models used.
DeBERTaVE V3 (He et al., 2021) is an enhanced multilingual model, building upon the improve- ments made in DeBERTa (He et al., 2020), by placing the MLM approach with Replaced Token Detection (RTD).
4.2 Fine-Tuning Strategies
We use three fine-tuning strategies for evaluating the performance of models dataset, as shown in Table 3. Models MM to M12 are validated on our code-switched validation split, and their performances are evaluated on our code- switched test split. In the models Mj to MM are fine-tuned on a meme propa- ganda dataset (Dimitrov et al., 2021a) using the text data extracted via OCR from the memes, without considering the image. For the Translated (Code- Switched + English) strategy, models M4 Ms are fine-tuned on the English translated version of our code-switched training split. The transla- tion is performed using GCP's Cloud Translation API, 4 which translates the text from Roman Urdu- English to English. Lastly, in the Code-Switched strategy, models M7 to M12 are fine-tuned on our code-switched training set. LLM M13 is used in a few-shot setting, where limited number of code-switched training examples, along with their corresponding gold labels, are incorporated into the prompt.
code-switched
BERT (Devlin et al., 2019) is a language model based on the Transformer architecture, trained on a large corpus of text using Masked Language Mod- eling (MLM) and Next Sentence Prediction (NSP).
mBERT (Devlin al., 2019) is multilingual pre-trained language model based on the BERT architecture that can represent words and phrases consistently across different languages by training on a multilingual corpus. It is trained on an exten- sive dataset of text in 104 languages.
strategy,
XLM RoBERTa (Conneau et al., 2020) is ofate-of-the-art pre-trained cross-lingual language model which is an extension of RoBERTa (Zhuang et al., 2021), pre-trained on large amounts of text data from over 100 different languages, using MLM and NSP pre-training objectives.
RUBERT (Khalid et al., 2021) is bilingual Roman Urdu model created by continuing the pre- training of BERT on their own dataset as well as on other publicly available Roman Urdu datasets (Amjad et al., 2020; Azam et al., 2020; Khana et al., 2016).
"https://cloud. http://ccooodoooggleeeeeeo wwwgge..oo//taa.....oooooooooooe com/ttranslate/docs/

---

Fine-Tuning Strategy Type
Exact Match
Hamming
Model
Avg. Precision
Avg. Recall
Avg. Fl-Score
Accuracy
Ratio
Score
Micro
Macro
Micro
Macro
Micro
Macro
M1
.57
.16
.18
.05
.27
.07
.898
.083
.185
Out Domuin
M2
.45
.06
.35
.886
.071
.239
Meme Dataset (Text-nlyy
.33
.889
.083
.261
M2
.12
.844
.038
.288
Trunslated
Ms
.49
.10
.37
.11
.42
.10
.891
.064
.267
(Code-Switched + English)
Ms
.54
.26
.40
.14
.46
.16
.900
.103
.320
.55
.21
.37
.12
.44
.14
.900
.096
.308
.893
.083
.263
Code-Switched
.083
.280
M1o
.901
.354
Mn
.135
.375
M12
.51
.43
.15
.17
.895
090
.307
No fine-tuning
M13
.39
.31
.53
.42
.45
.28
.862
.051
.306
Table 4: Results on the nine evaluation measures listed in subsection 4.4 for the different models Mi to M13- Green highlights show the highest score for each of the evaluation measures.
4.3 Experimental Settings
Results
We prepared our dataset by annotating propaganda labels at the fragment level, but we trained and evaluated our models in a multi-class multi-label setting without considering the span of the text. We used the Py ToTch framework for our implementa- tion and Huggingl'ace to load pre-trained state-of- the-art models. We took other pre-trained models from their source repositories. We ran each periment on a single NVIDIA Quadro RTX 6000 24GB GPU. Due to GPU RAM limitation, we used a training and validation batch size of 12. Here are the remaining hyperparameter values: 10 epochs, maximum sequence of length 256, AdamW as the optimizer with epsilon le-8 and weight decay of 0.1, learning rate of 3e-5 and Linear Warmup as the learning rate scheduler. We used the Binary Cross Entropy (BCE) as our loss function.
Table shows the performance of different models on our code-switched test split. We can see that models M1 and M13 show relatively higher per- formance. We can further observe that M11. which was fine-tuned on Roman Urdu data using XLM- RoBERTa, demonstrates superior performance in terms of micro-average F1-Score, Accuracy, EMR, and Hamming Score. On the other hand, M13. GPT-3.5 Turbo, which is provided with 20 code- switched examples in the prompt, exhibits higher performance in terms of micro- and macro-average recall. Additional information regarding the perfor- mance of M13 other few-shot settings be found in Table C.I in the Appendix C. Model M1 achieves 5% higher micro-average F1-Score com- pared to the second-best model, M13- Moreover, M11 demonstrates a higher micro-average preci- sion than its micro-average recall, implying that the model tends to produce fewer false positives. Essentially, the model is more conservative in its predictions and leans towards making fewer pos- itive predictions overall. Additionally, M11 out- performs the other models in terms of Hamming score and EMR, which are two of the most com- monly used evaluation measures for multi-label classification.
We used iterative_ Setrratve_errin____ trrin_test__se test_split from scikit-muliin multilarnn,,, which splits our multi-label data in stratified manner, so that there is a balance of classes in our training, validation, and test splits. The training, validation, and test splits are 786 (76%), 89 (9%), and 155 (15%). respectively.
4.4 Evaluation Measures
We evaluated the performance of our models us- ing micro/macro-averrge precision/recall/FI -ccore, hamming score, Exact Match Ratio (EMR), and ac- curacy, As have multi-label problem, i.e.. each example can be assigned 0, 1, or multiple labels, with class imbalance, we use the micro-ave averrge Fl -core as our primary evaluation measure.
Table 5 shows the F1-Score for each model and class. The column Percentage of Instances indi- cates the percentage of instances for each propa- ganda technique in our test split, and we can ob- serve that only four classes have higher percent- age than 5%.

---

Models -
Percentage Instances (%)
M:
MM.
Ma
M.
MMs
Mr.
MM
Mr
Ma
Man
Mr
Ma
Propaganda Techniques
Louded Language
26,9
.61
.50
.57
.56
.64
.53
.61
.74
.70
.70
Obfuscation, Intemional vagueness, Confasion
0.50
.00
.00
.00
.00
.00
.00
.00
Appeal to feurfprejudice
3.40
.00
000
.12
.00
.30
.20
.30
.00
.35
.30
-32
.33
Appcal authority
.00
.00
Whtttaboutism
1.50
00
.00
000
000
14
.40
.00
255
000
00
.00
000
.40
Slogans
.00
.00
Baageeration/ Minimiitiiin
14.2
00
.00
.00
.25
.17
29
.40
.31
.44
.47
.56
24
.37
Black and-anddddd white Fallacy/Distatoshhip
.00
.29
Smears
14.8
22
.09
27
.59
.57
.57
.47
.47
.48
.49
.53
48
.40
Doubt
1.50
29
.00
.00
.00
.00
.00
.29
.00
.00
-40
.50
22
.31
Bandwagon
0.20
.00
.00
.00
.00
.00
.00
.00
.00
.00
.50
Name LallinggLabeling
21.8
.22
.46
-22
.51
.57
.52
.52
.32
.45
.51
.63
.56
.69
Reductio ad hitlonam
.00
.00
(00
Presenting Inrelevant Data (Red Herring)
2.40
(X)
.00
(00
.00
(00
.20
.00
.00
.00
100
.00
000
000
Repetttion
.00
.00
Straw Man
0.70
.00
.00
.00
.00
.00
.00
.00
.00
.00
.00
.00
.00
.00
Thought terminating cliché
.00
Glinering generalittes (Virtue)
1:70
.00
.00
.00
.00
.00
.29
.00
.00
.00
.25
.20
22
.20
Flag-waving
1.00
.00
.00
.00
100
00
Causal Oversimplilictionn
3.30
.00
.00
.00
.00
.22
.50
.12
.24
Table 5: Comparison of class-level performance (FI-Score) on 13 different models. The naming convention of the models can be found in Table 3. Green highlights indicate the highest F1-Score each propaganda technique.
Comparing LLM M13 to the other fine-tuned models in Tables and 5, we observe that although M13 does not achieve the highest micro-Fi score, it demonstrates significantly higher macro-average recall and macro-average Fl score. Moreover, it performs remarkably well on underrepresented classes, i.e., such with instances below 5%. This can be attributed to two main factors. First, LLMs have the advantage of leveraging their extensive pre-training and contextual understanding, which enables them better comprehend and predict un- derrepresented classes, such as propaganda tech- niques in our case. This advantage sets them apart from smaller pre-trained models. Second, LLMs are trained on vast corpora containing diverse lan- guages and code-switched text. This exposure en- ables them to capture general language patterns and knowledge, even on smaller low-resource code- switched datasets. Consequently, the model ex- hibits a high macro-Fi score, indicating the LLM's ability to identify relevant instances for each class, regardless of the overall distribution of classes in the dataset.
The potential reason for the lower performance that we observed in translating code-switched data to high-resouree languages could be attributed to the translation service's inability to capture and preserve the inherent propagandistic characteristics present in code-switched text from low-resource languages. We also observe that the cross-lingun model's ability to transfer knowledge and capabil- ities from one language another one allows it to exhibit superior performance compared to both monolingual and multilingual models when han- dling code-switched text.
Conclusion and Future Work
We proposed the novel task of detecting propa- ganda techniques in code-switched social media test. We created a corpus of 1,030 code-switched English and Roman Urdu texts annotated with the use of 20 propaganda techmiques. We performed a number of experiments using different fine-tuning strategies and we found that modeling the multilin- guality directly rather than using translation of the code-switched text yields better results.
In future work, we plan experiments with detect- ing propaganda techniques in code-switching text for other resource-poor languages. We further plan to expand our annotated corpus with many more ex- amples. We also want to fine-tune pre-trained mul- tilingual and cross-lingual language models specif- ically aiming at detecting fine-grained propaganda at the fragment level. Last but not least, we want to understand the best strategy for handling code- switched text with LLMs.
Through various experiments, our study has re- vealed that training models on code-switched text yields superior performance for our propaganda detection task compared to training on translated versions of the code-switched text in high- languages. These findings emphasize the signif- icance of fine-tuning the models specifically on code-switched text, as opposed to relying on the translated versions of the text in higu-resouree lan- guages.

---

Limitations
Annotation
Annotating propaganda techniques at the fragment level differs from most other annotation tasks, as it is very time-consuming and mentally demand- ing. We assigned two annotators who underwent comprehensive training accurately annotate pro- paganda within the text. By engaging in extensive training sessions and analyzing pre-labeled cxam- ples from domain professionals, the annotators be- came proficient in identifying propaganda in mixed Roman Urdu and English code-switched text. How- ever, it is essential to acknowledge the limitation regarding the number of volunteers willing to un- dergo such training. Additionally, among those who are willing to participate the training, there is a scarcity of individuals who possess fluency in both Urdu and English languages, along with a solid understanding of Roman Urdu (i.e., Urdu written using the Latin script).
Language
Our first limitation is related to the language dif- ferences between Roman Urdu and English, which we faced while annotating code-switched propa- ganda text. This limitation stemmed from the fact that the original annotations for fine-gri ann-ggained pro- paganda techniques started with English, and thus the annotation guidelines reflected the English lan- guage and grammar. However, due to the differ- ent linguistic typologies of Urdu and English, we found it difficult to choose text spans that mix Ro- man Urdu and English. English follows sub- ject-verb-object (SVO) sentence structure, where the subject comes first, the verb second, and the object is third. Urdu, on the other hand, fol- lows a subject-object object- verb (SOV) sentence struc- ture where the subject comes first, the object is sec- ond, and the verb is third. Another constraint posed by the languages is that Roman Urdu is written in Latin/Roman script. Therefore, when individuals write Urdu in Roman Urdu, there is variability in the spellings for some words. This causes prob- lems as two different spelling will identify as two unique words in our model dictionary, where in fact they might actually correspond to the same word. Moreover, we use code-switched transla- tions to English, but translating propaganda text between languages presents a challenge because propaganda techniques may be language-specific and thus difficult preserve in translation. This results in texts losing their propagandistic nature in translation, which makes difficult to compare propaganda detection across languages.
Definition and Number of Propaganda Techniques
Another limitation is differing views on the exact number of propaganda techniques. While experts generally agree on the definition of propaganda. there are noticeable discrepancies in the techniques listed by various scholars (Torok, 2015). These dif- ferences occur due to one of two reasons. A scholar may ignore a set of techniques while compiling their list, or may use broader definitions that cover more fine fine-grainne grained definitions used by other schol- ars. For instance, Miller (1939) considers a total of seven techniques, whereas Weston (2018) proposes 24 propaganda techniques, while Wikipedia lists over 70 techniques.
Dataset
Ethics and Broader Impact
We faced several constraints related the small size of the dataset and the class imbalance of the propaganda labels. These limitations result in vari- ous types of noise in the dataset, including random noise and systematic bias. In small datasets, the model may seem to perform well on the training and on the testing splits, but this may not accu- rately reflect its performance is because the small dataset may not represent the broader range of data that the model may encounter, resulting in unreliable evaluation results. Class im- balance can also have significant impact on evalu- ation and may result in biased performance results, which makes the evaluation measures noisy.
User Privacy
The dataset we have only consisted of code- switched social media text and the social media platform was taken from. It does not contain any user information. The data collectors refrained from recording any user names or links, to maintain complete anonymization. The data collection was limited to public accounts, pages, and groups, and by adhering to this restriction, the collected data aligned with the platforms' public data policies, allowing it to be used for public purposes.
new data. This
htt //ep:.://ee../ httss:///ewwiiiiiee www.pedia.i.... aariiii..rg//////o.. org/wiki /Propaganda_ techniques

---

Biases
We acknowledge that the process of annotating pro- paganda techniques can involve subjectivity, which inevitably leads to biases in our gold labels or the distribution of the labels. To address these con- cerns, we first collect examples from multiple di- users. We also ensure that rigorous annota- tion training is conducted for the annotators. Fi- nally, we use well-defined annotation process with a clear methodology. The high agreement score between our annotators gives us confidence that the labels assigned to the data are accurate in the majority of cases.
Misuse Potential
We kindly request researchers to be aware of the potential for malicious misuse of our dataset. Con- sequently, urge them exercise caution.
Data Collector and Annotator Rights
The involvement in the data collection and annota- tion was entirely voluntary. An informed consent was obtained from all participants before data col- lection, and their identities were protected through anonymization. They had the option to withdraw their consent at any time without any penalty. Re- searchers were transparent about the purpose of the annotation and the intended use of the annotated data with the participants. The risks to the partici- pants were very low and comparable to what one encounters their daily life. Last, all participants received fair monctary compensation for their par- ticipation, which was well above the minimal wage in their country.
Intended Use
The aim of our dataset is to encourage the detec- tion of social media content with propagandistic characteristics, particularly in code-switched text. We firmly believe that this resource holds signifi- cant value for the research community, particularly when used appropriately and effectively.

---

References
Nakov. 2020. SemEval-2022 task 11: Detection of propaganda techniques in news articles. In Proceed- ings of the Fourteenth Workshop on Semantic Evalu- ation, pages 1377-1414, Barcelona (online).
Aimlab. 2022. XLM RoBERTa fine-tuned on Ro- man Urdu. https::/huugiingfen wwtpa/cce....ommiaeeieee co /iimabb -uttrrrbbbrrar-rr------------eee
Giovanni Da San Martino, Seunghak Yu, Alberto Berrón-Cedefoo, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news article. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP)) pages 5636-5646, Hong Kong, China. Association for Com- putational Linguistics.
Firoj Alam, Hamdy Mubarak, Wajdi 7aghouani, Gio- vanni Da San Martino, and Preslav Nakov. 2022.
Overview of the WANLP 2022 shared task pro- paganda detection in Arabic. In Pruceedings of the Seventh Arabic Natural Language Processing Work- shop (WANLP), pages 108-118, 118, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computa- tional Linguistics.
Maaz Amjad, Grigori Sidorov, Alisa Zhila, Helena Gómez-Adorno, Ilia Voronkov, and Alexander Gel- bukh. 2020. "Bend the truth"" Benchmark dataset for fake news detection in Urdu language and its evaluation. Journal of Intelligent & Fuzzy Systems, 39(2):2457-2469)
Lavinia Dan. 2013, Techniques for the translation of ad- vertising slogans. In Proceedings of the International Conference Literature, Discourse and Multicultural Dialogue (LDMD), volume 15, pages 13-23.
Jacob Devlin, Ming- Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computa- tional linguistics, 34(4):555-596
Nazish Azam, Bilal Tahir, and Muhammad Amir Mehmood. 2020. Sentiment and emotion analysis of text: A survey on approaches and resources. Lan- guage & Technology. 87.
Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021a. Detect- ing propaganda techniques in memes. In Proceed- ings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 1 1ith International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6603 6617, Online. Association for Computational Linguistics.
Daniel Baleato Rodríguez, Verna Dankers, Preslav Nakov, and Ekaterina Shutova. 2023. Paper bullets: Modeling propaganda with the help of metaphor. In Findings of the Association for Computational Lin- guistics: EACL 2023, pages 472-489, Dubrovnik, Croatia. Association for Computational Linguisties.
Alberto Barrón-Cedenna Israa Jaradat, Giovanni Da San Martino, and Preslav Nakov. 2019. Proppy: Organizing the news based on their propagandistic content. Information Processing & Management, 56(5):1849-185
Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021b. SemEval-2021 task 6: Detection of persuasion tech- niques in texts and images. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-20211 I) pages 70-98, Online. Association for Computational Linguistics.
Shuguang Chen, Gustavo Aguilar, Anirudh Srinivasan, Mona Diab, and Thamar Solorio. 2022. CALCS 2021 shared task: Machine translation for code-switched data. arxiv preprint arXiv:2202.n 222.005....
Richard Cole. 1975. The reformation in print:
Jean Goodwin and Raymie McKerrow. 2011. Account- ing for the force of the appeal to authority. OSSA Conference Archive 10.
German pamphlets and propaganda. Archiv für Refformationngescciiheee-tte for Reformation His- tory, 66(j))933110) 102.
Maram Hasanain, Firoj Alam, Hamdy Muharak, Samir Abdaljalil, Wajdi Zaghouani, Preslav Nakov, Gio- vanni Da San Martino. and Abed Alhakim Freihat. 2023. ArAlEval shared task: Persuasion techniques and disinformation detection in Arabic text. In Pro- ceedings of the The Eighth Arabic Natural Language Processing Workshop (WANLP), Singapore.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradient- disentangled embedding sharing. ArXiv. abs/2111.09543.
Giovanni Da San Martino, Alberto Barrón- -edeeño, Henning Wachsmuth, Rostislav Petrov, and Preslav

---

Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. DeBERTa: Decoding- enhanced BERT with disentangled attention. ArXiv, 005520066.0364..
online news terms of genre, framing, and persua- sion techniques. pages 3001-3022, Toronto, Canada. Association for Computational Linguisties.
Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlama Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and po- litical fact-checking. In Proceedings of the 2017 Conference Empirical Methods in Natural Lan- guage Processing, pages 2931-2937, Copenhagen, Denmark.
Renee Hobbs and Sandra McGee. 2014. Teaching about propaganda: An examination of the historical roots of media literacy. Journal of Media Literacy Education, 6(2):56 66.
John Hunter. 2013.
Brainwashing large group awareness training?: The classical conditioning hy- pothesis of brainwashing. Ph.D. thesis.
Monika L. Richter. 2017. The Kremlin's platform for 'useful idiots' in the West: An overview of RT's edi- torial strategy and evidence of impact. European Val- ues. URL: http://www. europeanvalues. net/ri/(31.0) 2018).
Navya Jose, Bharathi Raja Chakravarthi, Shardul Suryawanshi, Elizabeth Sherly, and John P. Mc- Crae. 2020. A survey of current datasets for code- switching research. In Proceedings of the 6th Inter- national Conference on Advanced Computing and Commumication Systems (ICACCS), pages 136-141.
Hammad Rizwan, Muhammad Haroon Shakeel, and Asim Karim. 2020. Hate-speech and offensive lan- guage detection in Roman Urdu. In Proceedings of the 2020 Conference on Empirical Methods in Natural Longuage Processing (EMNLP), pages 2522, Online, Association for Computational Lin- guistics.
Usama Khalid, Mirza Omer Beg. and Muham- mad Umair Arshad. 2021. RUBERT: A bilingual Roman Urdu BERT using cross lingual transfer learn- ing. arXiv preprint arXiv:2102.110
pagess 2512-
Wahab Khana, Ali Daudb, Jamal A Nasira, and Tehmina Amjada. 2016. Named entity dataset for Urdu named entity recognition task. LANGUAGE & TECHNOL- OGY, 51.
Carol Myers Scotton. 1982. The possibility of code- switching: motivation for maintaining multilingual- ism. Anthropological linguistics, pages 432 444.
Sunayana Sitaram, Khyathi Raghavi Chandu, Sai Kr- ishna Rallabandi, and Alan W Black. 2019. A survey of code speech and language processing. arXiv preprint arXiv: 1904.00744
Simran Khanuja, Sandipan Dandapat, Anirudh Srini- vasan, Sunayana Sitaram, and Monojit Choudhury. 2020. GLUECoS: An evaluation benchmark for code-switched NLP. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 3575-3585, Online. Association for Computational I Linguistics.
Mary WJ Tay. 1989. Code switching and code mixing communicative strategy in multilingual discourse. World Englishes, 8(3):407-417.
Gabriel H Teninbaum. 2009.
Reductio ad Hitlerum: Trumping the judicial Nazi card. Mich. St. I. Rev., page 541.
Klaus Krippendorff. 2011. Agreement and information the reliability of coding. Communication Methods and Measures, 5:112 111--95 93.
Robyn Torok. 2015. Symbiotic radicalisation strategies: Propaganda tools and neuro linguistic programming.
Victor Margolin. 1979. The visual rhetoric of propa- ganda. Information design journal, 1(2):107-122
Anthony Weston. 2018. Rulebook for Arguments. Hackett Publishing.
Clyde R. Miller. 1939. The techniques of https: http///ssffllsssseeeeee parcattorpages.oom htt//usrriils/jcarr/ Srgpagandax20Techmiioe pdf.
propaganda.
Julius Yourman. 1939, Propaganda techniques within nazi germany. The Journal of Educational Sociology. 13(3): 148-:144-163.
Chad Nilep. 2006. "Code switching"" in sociocultural linguistics. Colorado research in linguistics.
Seunghak Yu, Giovanni Da San Martino, Mitra Mo- htarami, James Glass, and Preslav Nakov. 2021. In- terpretable propaganda detection in news articles. In Proceedings of the International Conference Recent Advances in Natural Language Processing (RANLP 2021), pages 1597-1605, Varna, Bulgaria.
Jakub Piskorski, Nicolas Stefanovitch, Giovanni Da San Martino, and Preslav Nakov. 2023a. SemEval-20233 task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multi- lingual setup. In Proceedings of the 17th Interna- tional Workshop on Semantic Evaluation (SemEval- 2023), pages tion for Computational Linguistics.
Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. 2021. A robustly optimized BERT pre-training approach with post-training. In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 221-1227 Huhhot, China. Chinese Informa- tion Processing Society of China.
Toronto, Canada. Associa-
Jakub Piskorski, Nicolas Stefanovitch, Nikolaos Niko- laidis, Giovanni Da San Martino, and Preslav Nakov. 2023b. Multilingual multifaceted understanding of

---

Appendix
3. Repetition
Propaganda Techniques
Repetition refers to repeating the same message multiple times that it is eventually accepted among the public. Repetition makes a fact seem more true, regardless of whether it is or not. English example: "I promise, promise, promise will deliver."
In this section, we go over the formal definitions for each of the propaganda techniques. These defi- nitions are sourced from Dimitrov et al. (Dimitrov et al., 2021a), with some modifications made for clarity. The previous work used 22 propaganda techniques, but we only use 20 for our dataset an- notation. We decided to exclude Transfer and Ap- peal to Emotions as they were only applicable to the images in the memes and were not relevant to our code-switched text. Additionally, we provide code-switched examples for each technique. The underlined text in the English translation shows the fragments that is associated with the propaganda technique.
Code-switched example: "If they were previously politically motivated then in saroon ka court mar- shall kuro. Aur saza dou saza dou saza dou saza dou"
English translation: "If they were previously po- litically motivated then have them all court mar- tialled. And punish them punish them punish them punish them" "
4. Exaggeation/iinnniztion
1. Loaded Language
Exaggeration refers to overstating an idea, prod- uct's value or someone's characteristics than what they actually are to propagate their own narrative. This technique involves "hyping" up by using im- pressive sounding words that are nonetheless mean- ingless and vague. Minimization on the other hand refers to understating an idea, product's value or someone's 's characteristics than what they actually are. This is particularly done to make something seem less important or smaller than is.
Loaded language refers to the use of words or phrases with strong emotional indications (posi- tive or negative). It is used to influence the public's s opinion or mindset by using these words or phrases with strong connotations.
English example: "how stupid and petty things have become in Washinglon"
Code-switched example: "I hope women get over their bad boy syndrome soon. Pehle ameer baap ki aulaad dekh ke shaadi krti hain phir domestic abuse ke randi rone, you girls know what you'l re getting yourself into"
English example: "I am so hungry could eat a horse."
Another English example: "I did not hit her; it was a slight pat"
English translation: "I hope women get over their bad boy syndrome soon. First they see rich dads kids and marry them and then cry foul of domestic abuse, you girls know what you'l ve get- ting yourself into"
Code-switched example:
"You are the top crypto scammer of Pakistan. Tumhain pata hai tun ne in masoom logon ke sath kya kya kiya hai kabhi ponzi scheme, kabhi courses aur kabhi paid sub- scriptions"
2. Name Calling/Labeling
"You are the top crypto sca- mer in Pakistan. You know what you have done these innocent people sometimes running Ponzi schemes, other times offering courses or paid sub- scriptions."
English translation:
Name Calling/Labeling refers to labeling the object of propaganda campaign (individual or group) as something that the public lovers, praises or admires or something that they find undesirable, fear and hate. Primarily they are in the form of an argument where labels are insulting or demeaning.
5. Doubt
English example: "Republican congressweasels", "Bush the Lesser"
Doubt refers to questioning the credibility of some- thing or someone. Some people use it to cast doubt on their opponents, controversial issues or the cred- ibility of some media organizations.
Code-switched example: "Patwaari aur youthiye larte rehein ge, meanwhile Nawaz and Niyazi will loot the country and leave for England"
English example: "A candidate talks about his opponent and says: he ready to be the Mayor?" Code-switched example: "Death penalty is ridicu- lous. Aik rapist ko latka dene se baqion ko kia faraq
English translation: Patwaari and youthiye will keep on fighting, meanwhile Nawaz and Niyazi will loot the country and leave for England"

---

pare ga?"
"Expel Afghanis from Pak- istan, these morally corrupt people benefit from our land and then curse us. Also remember how they caused destruction through Kalashnikov culture in our beloved country"
English translation:
English translation: "Death penalty is ridiculous. What difference will make to others if one rapist is hanged?"
6. Appeal to Fear/Prejudice
Appeal to Fear/Prejudice refers to secking to create support for idea by instilling anxiety and/or at- tempting to increase fear in the population toward an alternative. In some cases, the support is built based on preconceived judgements and acts as a tactic commonly used in marketing, politics, and media.
9. Causal Oversimplification
Causal Oversimplification assumes a single cause or reason when there are actually multiple causes for an issue. This includes transferring blame to one person group of people without investigating the complexities of the issue.
English example: "The reason New Orleans was hit hard with the hurricane was because of all the immoral people who live there."
English example: "We must stop these refugees; they terrorists"
Code-switched example: "Pakistan ne bankrupt hojana soon. Pack your bags aur nikal lo Canada before it's too late"
Code-switched example: "Women divorce when- ever they need money. Muft mein alimony bhi lo bachon ki custody bhi"
English translation: "Pakistan is about to get bankrupt soon. Pack your bags and leave for Canada before it's too late"
English translation: "Women divorce whenever they need money. Get the alimony for free and the children's 's custody"
7. Appeal to Authority
10. Slogans
Appeal to Authority refers stating that claim must be true simply because a valid authority or expert on the issue said was true, without any other supporting evidence(Goodwin and McKer- row, 1).
Slogans refer to a brief, striking phrase that may include labeling and stereotyping which acts an emotional appeal. This technique attempts arouse prejudices in an audience by labeling the object as something the target audience likes or dislikes (Dan, 2015).
English example: "Richard Dawkins, an evolu- tionary biologist and perhaps the foremost expert in the field says evolution is true, therefore must be true."
English example: "Make America great again!" Code-switched example: "Utho Pakistanio! Save your country!"
Code-switched example: "Imran Khan keh raha hai isko follow nai karna chahye tou isko unfollow karna is the best option"
English translation:
"Wake up Pakistanis! Save
your country!"
English translation: "Imran Khan is saying that we shouldn's i follow him so unfollowing him is the best option"
11. Black-and- Fallacy/Dictatorship
Black-and- Wnitt Fallacy presents two opposite/al- ternative options as the only possibilities, when in fact more possibilities exist (Torok, 2015). Dicta- torship is an extreme case where the audience is told exactly what actions to take, eliminating any other possible choices.
8. Flag- Waving
MaggWWving refers to the technique which plays on strong national feeling to justify an action based on the undue connection to nationalism or patriotism or benefit for an idea, group or country.
English example: "Either we go to war or we will perish"
English example: "Entering this war will make us have a better future for our country."
"Either you are a Mus- lim woman or a feminist, koi beech ka raasta nhi, faisla kr lo and stick it"
Code-switched example:
Code-switched example: "Afghanion ko Pakistan nikalo, haraamkhor humara khaa kr humein hi gaali dete hain. Also remember how they caused destruction through Kalashnikov culture in our beloved country"
English translation:
"Fither you are a Muslim
woman or a feminist, there no middle way, de- cide now and stick to it"

---

12. Thought-terminating cliche
movie they should stop it's spread."
"
Words or phrases that discourage critical thought and meaningful discussion about given topic. They are typically short, generic sentences that offer seemingly simple answers to complex ques- tions or that distract the attention away from other lines of thought (Hunter, 2015).
15. Presenting Irrelevant Data (Red Herring)
Red Herring refers to introducing irrelevant mate- rial to the issue being discussed so that everyone's 's attention is diverted away from the points made. This form of misleading or distraction may either be logical fallacy or literary device. Those sub- jected to red herring argument are led away from the issue that had been the focus of the discussion and urged follow an observation or claim that may be associated with the original claim, but is not highly relevant to the issue in dispute (Tenin- baum, 2009).
English example: "It is what it is", "Its common sense". "It doesn't I matter", "You can't I change hu- man nature"
Code-switched example: "No matter how much you try, mard aur orat kabhi barabar nhi ho sakte, It is what it is"
English translation: "No matter how much you try, man and woman can never be equal. It is what it is"
English example: "You may claim that the death penalty is an ineffective deterrent against crime but what about the victims of crime? How do you think surviving family members feel when they see the man who murdered their son kept in prison their expense? right that they should pay for their son's 's murderer to be fed and housed?"
13. Whataboutsim
Whataboutism refers to when critical question argument is not answered or discussed, but deflected with a critical counter-question that ex- presses counter-accusatioon. Here an opponent's position is discredited by charging them with hypocrisy without directly disproving their argu- ment (Richter, 2017).
Code-switched example: "Joyland banned Punjab but is playing in cinemas across Sindh. Achi tarah pta chal gaya kon si hakumat ghulamon ki hai"
English translation: "Joyland is banned in Pun- jab but is playing in cinemas across Sindh. We now know very well whose government is being by slaves"
English example: "A nation deflects criticism of its recent human rights violations by pointing to the history of slavery in the United States"
Code-switched example: "Jahez ke naam pe itni takleef kyu holi hai aurton ko, what about their demands of big house and big car from the hus- band?"
16. Bandwagon
Bandwagon refers to the fallacious argument which is based on claiming a truth or affirming something is good because the majority thinks so. It is used to persuade the target audience to join in and take the course of action because "everyone else is taking the same action" (Hobbs and McGee, 2014).
English translation: "Why do women feel agitated in the name of dowry, what about their demands of big house and big car from the husband?"
14. Reductio ad Hitlerum
English example: "Would you vote for Clinton as president? 57% say yes"
Reductio ad Hitlerum can be thought of play- ing the Nazi card. It is an attempt to persuade the audience to disprove an action or idea of someone by invalidating their position on the basis that their view is popular among hated groups. It can refer to any person or concept with negative connotation (Teninbaum, 2009).
Code-switched example: "We hate Babar cause all of India does. Apnay liay nai tou issiliay karlo ke hum sab bhi kartay hain"
"We hate Babar cause all of India does. If not for yourself you should hate him because we all do." "
English translation:
English example: "Such a thought would come in the mind of Hitler"
17. Obfuscation, Intentional vagueness, Confusion
Code-switched example: "This movie has been directed by a gay guy iss movie ko phailainay ke bajai isko rokna chahye"
Obfuscation, Intentional vagueness, and Confusion can be termed as deliberately making use of un- clear words or phrases so that the target public create their own interpretation (Weston, 2018). For
English translation: "This movie has been di- rected by a gay guy rather than spreading this

---

instance, when an unclear phrase with multiple pos- sible meanings used within the argument, and, therefore, does not really support the conclusion. English example: "It is a good idea to listen to victims of theft. Therefore, if the victims say to have shot, shot, then you should do it."
makeup)", "I'm Lovin' It (McDonald's fast food)", "The Best a Man Can Get (Gilette razors)" Code-switched example: "Taliban are the true mujahids defending their country like lions" " En- glish translation: "Taliban are the true warriors defending their country like lions"
the
Here the word "listen" can either mean to listen to the personal account of the experience of being could
20. Smears
vietim of theft and empathize with them mean to earry out the punishment of their choice. Code-switched example: "As we all know in big guns circles a term used "Get a promotion with wife swap". Baqi aap samajhdar ho keh keya mat- lab hey es ka"
A smear refers to an effort to damage or call into question someone's reputation, by propounding negative propaganda through false accusations and slander, etc. It can be applied individuals or groups.
English example: "I'm not pregnant, I haven't 't been tested. But the President says if it's not tested it's 's not real. I think it's gas..."
English translation: "As we all know in big guns circles a term used "Gel promotion with wife swap". The rest of you are wise enough to under- stand the meaning of this"
Code-switched example: "Malik Riaz is cult. PPP ke saath mil kar paani ke daamon zamin kharredi hai to make Bahria Town Karachi" English translation: "Malik Riaz is a cult. After colluding with PPP, land property was bought ex- tremely low prices to make Bahria Town Karachi"
18. Misrepresentation of Someone's Position (Strawman)
Strawman refers to refuting of an argument where the real subject of the argument was not addressed or refuted but instead substituted with a weaker similar one in place of the original so that it can be easily confuted. One who engages in this fallacy is said to be "attacking a straw man".
B Annotation Web Platform
In this section, we present our web-based annota- tion platform, which we developed from scratch to annotate our code-switched dataset. The basic layout of our website is presented in Figure B. The right side of the page displays a list of pro- paganda techniques, each with a corresponding colored box. On the left side, there is a text box where code-switched text is annotated. The text is imported from a comma separated values (CSV) file with columns names: ID, TEXT, LABELS, INCLUDED, and TRANSLATED. In Figure B.2, we can see the annotation feature which is clearly shown as a 2-step process. Here, we can see that the text example is already labeled as Black-and- White Fallacy/Dictatorhhip in the JSON output. To label another span of text, we follow two steps. In Step 1, we select the span of text that needs to be labeled. In Step 2, we click the colored boxes corresponding to the label we wish to assign to that span. In Figure B.2, the first step illustrates the se- lection of the span "Aurat Raj na manzoor", while the second step displays the updated annotation in ISON format below, where the span is labeled as a Slogan.
English example: "Claiming that all vegans are opposed to all forms of animal captivity, including pet ownership."
Code-switched example: "Aurat march trash as rather than helping women who actually needs helps all they talk about is nanga honei do, lesbian honei dou"
English translation: "Aurat march is trash as rather than helping women who actually needs helps all they talk about is let us naked, let us be lesbians"
19. Glittering generalities (Virtue)
Glittering generalities are vague "feel good" state- ments that people are predisposed to want to iden- tify with. The technique makes use of emotionally appealing phrases or words that are closely associ- ated with highly valued concepts and beliefs such that they carry conviction without supporting in- formation reason. Such highly valued concepts attract general approval when attached to a person issue. For example: Peace, hope, happiness, security, leadership, freedom, family values, and patriotism.
English example: "Because I I'm Worth (L Oreal

---

Annotate Text
Example ID
Completed:
Appoal to fear/prejudice
Name calingl-abeing
Search
Excludee (Red)
Appeal to authority
Reductio ad hiterum
Lebel Annolation Seritence
Bendwagon
Sentence
Slogans
Repetition
Exegirattoeniiinnsttti
Doubt
Loaded Language
Flag-waving
Smeers
Causal Oversimpiiiicatin
Undo Amnotation
Black-and-whiie FallecyDiettttrshhi
Thought-terminating cliché
Update Text
Presenting irrelevant (Red Herring)
Gittteingg generalitios (Virtuc)
Misrepresentation
vagueness, Contusion
Someone's Poeltion (Straw Man)
Figure B.1: Interface design of our web-based annotation tool.
Label Annotation Sentence
Label Annotation Sentence
Aurat Raj ne menzoor. not permitted in Islam, acha lge ya bura.
Aurat Raj manzoor, is permitted lss ya bura.
Prev
Next
"id": "I1",
Prev
Next
"text": "Aurat Raj na manzoor. It is
"id": "II", "text": "Aurat Raj nа nanzoor. It is "labels": [
"start_index"" "8", "eed_index"";
"text_fragnent"" "Aurat Raj na
"start indexx"
"technique": "Black-and-whitee fal.
"end_index": "69",
"taxt_"ragrent": "Aurat Raj
"start_inde" sttar_iindxx:: "8", "end_index":
"technique": "Black-and-white Fal
"text fragnent"; "Aurst Raj na
"technique": "Stogens"
Figure B.2: The two-step annotation process of a text. Step 1: Select the span of text that needs to be annotated. Step Click on the boxes next the corresponding propaganda label shown Figure B.I. The image on the right shows the updated JSON output after completing the two steps.

---

C Prompts for Propaganda Detection using GPT-3.5-Turbo
this section, we leverage the power of Large Language Models (LLMs), specifically GPT-3.5 Turbo, to enhance our language processing capabilities. In particular, we use LLMs by providing them with set of examples within the prompt and run them in four different few-shot settings. The prompt incorporates small set of code-switched training examples, along with their respective gold labels, to facilitate the LLM in generating accurate predictions for the code-switched test set. Table C.I shows their results on several different evaluation measures.
An example of 10-shot prompt shown below:
Can you help me analyze code-swi text in Roman Urdu and English to identify one or multiple propaganda techniques? There can also be no propaganda techniques in a text.
The list of 20 propaganda technique ques is given below:
Loaded Language
Obfuscation, Intentional vagueness, Confusion Appeal to fear/prejudice
Appeal to authority
Whataboutism
Slogans
Black-and FalltcyyDictatorshhi
Smears
Doubt
Bandwagon
Name calling/Labeling
Reductio ad hitlerum
Presenting Irrelevant Data (Red Herring)
Repetition
Misrepresentation of Sameone's Position (Straw Man)
Thought-terminatinn cliche
Glittering generalities (Virtue)
Flag-waving
Causal Oversimplification
Here are a few examples different code-switched texts, along their output in the form of comma-separen propaganda techniques, provided for your guidance.
Examples:
Text: This movie has been directed by a gay guy iss movie ko phailainay ke bajai isko rokna chahye. They are just manipulating the whole situation. Also it won the queer award.
Output: Reductio ad hitlerum, Smears, Loaded Language, Exaggeration/ Minimisation, Appeal to feau/prejudice
Text: Joyland is banned in Punjab but is playing in cinemas across Sindh. Achi tarah pta chal gaya kon si hakumat ghulamon ki hai. Output: Presenting Irrelevant Data (Red Herring), Loaded Language
Text: We hate Babar cause all of India does. Apnay liay nai tou issiliay karlo ke hum sab bhi kartay hain Output: Bandwagon, kaaggeration/Miiiiisttteee
Loaded Language
Text: Aurat march is trash as rather than helping women who actually needs helps all they talk about is nanga honei do. lesbian honei dou Output: Misrepresentation of Someone's Position (Straw Man), Smears, Loaded Language, Name calling/Labeling
Listing I: Example of 10-shot ChatGPT (GPT-3.5 Turbo) Prompt (1/2).

---

seperated if multiple propaganda Text: Either you are Muslim woman or feminist, koi beech raasta nhi, Fallacy/Dietatorship
faisla kr lo and stick
Output: Black-and-we
Text: matter how much you try, mard aur orat kabhi barabar nhi ho sakte.
It is what it is
Output: Thought Briigt-terriing terminating cliche
Text: I hope women get over their bad boy syndrome soon. Pehle ameer baap ki aulaad dekh ke shaadi krti hain phir domestic abuse ke randi rone, you girls know what you'le re getting yourself into.
Output: Appeal to fear saar/prejudice Name calling/Labelinn Loaded Language, Causal Oversimplification
Patwaari aur youthiye larte rehein meanwhile Nawaz and Niyazi will loot the country and leave for England. Output: Name calling/Labeling, Loaded Language, Smears
Text:
Text: Women divorce whenever they need money. Muft mein alimony bhi lo bachon ki custody bhi
Output: Loaded Language, Causal Oversimplification Exaggeration/ Minimisation
Text: Taliban the true mujahids defending their country like lions Output: Glittering generalities (Virtue), Name calling/Labeling
The Text analyze is: () Please make sure the techniques given.
Listing 2: Example of 10-shot ChatGPT (GPT-3.5 Turbo) Prompt (2/2).
Model GPT-3.5 Turbo
Exact Match Ratio
Hamming Score
Avg. Precision
Avg. Recall
Avg. F1- F1-Score
Accuracy
Micro
Macro
Miero
Macro
Micro
Macro
0-shot
.39
.25
.884
.083
.214
5-shot
.35
.33
.53
.35
.42
.22
.846
.032
.279
10-shot
.37
.30
.54
.41
.44
.25
.853
.032
.299
20-shot
.39
.31
.53
.862
.051
.306
Table C.I: Performance evaluation metries for different shot settings using the GPT-3.5 Turbo model. The highest scores are in bold and highlighted in green


Reviews:
[{"review_id": "9nfnFtEgj5", "reviewer": "Reviewer_X7cv", "summary": "", "questions": "A. What instructions were given to the collectors for collecting this data? Did they search for certain keywords or keep an eye on certain online conversations? How did you ensure balance across different topics?\nB. Did you analyze the distribution of topics across the dataset and look at which topics are more prone to certain types of propaganda? There is a mix of social and political examples among the ones provided in the appendix and it would be interesting to see which techniques are more prevalent in each.\nC. Did you explore normalizing the spelling variations of Roman Urdu words to improve the performance of the classification models since they’re operating on a rather small dataset?\nD. Have you considered the issue of people being able to search the examples to trace their origins on social media and what impact that might have on the original posters?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Questions for the Authors**\n\n---\n---\n\n\n**Question-A:** What instructions were given to the collectors for collecting this data? Did they search for certain keywords or keep an eye on certain online conversations? How did you ensure balance across different topics?\n\n**Response:** We introduced data collectors to propaganda techniques by presenting them with various examples from existing datasets in English *(Dimitrov, Dimitar, et al. \"Detecting propaganda techniques in memes.\")*, as well as from posts that combined English and Roman Urdu. These code-switched posts/texts were gathered by the authors from social media platforms. The purpose was to guide the collectors on the types of posts they should look for. The data collectors were already proficient in both languages i.e, native/Roman Urdu and English.\n\nAfterwards, the collectors were tasked with finding posts on specific topics that exhibited signs of propaganda. These topics included, but were not limited to, politics, cricket, women's march (Aurat march), feminism, religious views, and gender equality. The data collectors were instructed to collect a roughly equal number of posts from each topic. This approach aimed to prevent potential biases during the fine-tuning of pre-trained models.\n\nThroughout the process, the collected data was available to the authors in real-time. Regular feedback was provided to the data collectors to maintain the quality of the examples/posts. This feedback focused on the presence of propaganda techniques and the percentage of code-switched words, ensuring that the examples/posts met the established standards.\n\n---\n\n**Question-B:** Did you analyze the distribution of topics across the dataset and look at which topics are more prone to certain types of propaganda? There is a mix of social and political examples among the ones provided in the appendix and it would be interesting to see which techniques are more prevalent in each?\n\n**Response:** We didn't instruct the data collectors to include the topics for the posts or examples during the data collection process. As a result, we lack information about the distribution of topics and propaganda techniques across these topics. However, we have shown the distribution of propaganda techniques in the overall annotated dataset.\n\n ---\n\n**Question-C:** Did you explore normalizing the spelling variations of Roman Urdu words to improve the performance of the classification models since they’re operating on a rather small dataset?\n\n**Response:** While this technique could enhance the quality of embeddings for most Roman Urdu words and minimize noise and inconsistencies in our text data, we made the decision not to proceed with its implementation. This is because real-world textual data often contains errors, typos, and irregularities, particularly these inconsistencies are more pronounced for languages written in non-native scripts like Roman Urdu. \nTo ensure our models remain resilient to variations in spelling, we chose not to alter any spellings within the text. However, in the interest of model cleanliness, we did remove unnecessary punctuation, which tends to be overused in social media posts.\nFurthermore, we did not label the words as \"English\" or \"Roman Urdu\". Our data was solely annotated with \"propaganda technique\" on a fragment-level basis. Consequently, we were unable to assess the impact of \"spelling variations of Roman Urdu words\" on model performance.\n\n ---\n\n \n**Question-D:** Have you considered the issue of people being able to search the examples to trace their origins on social media and what impact that might have on the original posters?\n\n**Response:** While there's a chance that the source user could be identified by searching for the post or example on social media, it's important to note that we have taken precautions to ensure privacy. We intentionally refrained from gathering source links, user IDs/names, and incorporating them into our dataset. We have also withheld information about which social media platform each post/example originates from.\nFurthermore, our data collection focused on publicly accessible posts that fall within the scope of \"Public Information,\" as defined by the data policies of platforms (Twitter, Facebook, Instagram and YouTube). Users of these social media platforms have consented to their terms and conditions regarding the information they publicly share. It's worth highlighting that we did not collect data from any non-public groups or sources.\n\n \n ---\n \n\n**Typos Grammar Style And Presentation Improvements**\n\n**Comment** The aspect ratio of the screenshots in figure B.2 doesn’t match\n\n**Response** We appreciate your suggestion and will work on enhancing the visual presentation of the figure to achieve a more professional look in the revised manuscript. Thank you for your feedback.\n\n&nbsp;\n\n**Missing Reference Response** We will ensure that we include the following reference in our revised manuscript, as it is indeed pertinent to the work we have conducted. Thank you for your suggestion.\n\n*Hammad Rizwan, Muhammad Haroon Shakeel, and Asim Karim. 2020. Hate-Speech and Offensive Language Detection in Roman Urdu. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2512–2522, Online. Association for Computational Linguistics*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Propaganda on social media is a major problem but little attention has been paid to it in the setting of low-resource languages from the Global South, especially when the way the language is commonly written on the internet differs from its original style and script. This paper presents an important study of the phenomenon of online propaganda in Roman Urdu, a major language in South Asia. The dataset itself is a very useful resource and has been annotated in detail with a good analysis of the different propaganda techniques covered in it. The models trained can prove useful for extracting more examples of propaganda from the internet to better understand how it manifests itself in different socio-political conversations. Finally, the paper is very well-written and easy to understand, and should prove very useful for future researchers interested in studying this topic.", "reasons_to_reject": "None.", "questions_to_authors": "A. What instructions were given to the collectors for collecting this data? Did they search for certain keywords or keep an eye on certain online conversations? How did you ensure balance across different topics?\nB. Did you analyze the distribution of topics across the dataset and look at which topics are more prone to certain types of propaganda? There is a mix of social and political examples among the ones provided in the appendix and it would be interesting to see which techniques are more prevalent in each.\nC. Did you explore normalizing the spelling variations of Roman Urdu words to improve the performance of the classification models since they’re operating on a rather small dataset?\nD. Have you considered the issue of people being able to search the examples to trace their origins on social media and what impact that might have on the original posters?", "paper_topic_and_main_contributions": "This paper studies how propaganda is communicated on social media in a code-switched setting. The authors develop a novel corpus of social media posts which contain code-switched content in Roman Urdu and English and annotate spans of text in each example for many different kinds of propaganda techniques. Finally, they perform experiments to detect this propaganda using a variety of different models from different model families.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "A. The aspect ratio of the screenshots in figure B.2 doesn’t match.\n"}, "extra_text": {"Missing_References": "Hammad Rizwan, Muhammad Haroon Shakeel, and Asim Karim. 2020. Hate-Speech and Offensive Language Detection in Roman Urdu. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2512–2522, Online. Association for Computational Linguistics."}}, {"review_id": "fdx0dRJIkH", "reviewer": "Reviewer_o5WW", "summary": "", "questions": "Why does the annotation training work?\nWhat if the dataset includes actual Urdu written in native script, instead of Roman?\nHow did you pick and choose the 'propaganda related texts' from social media?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reasons to Reject**\n\n---\n---\n\n**Comment** The dataset is very small. Most datasets in the same domain have significantly more amount of data. \n\n**Response** We acknowledge that the dataset is indeed small, but it is important to emphasize that the emphasis here is on ensuring the highest quality rather than sheer quantity. While a larger dataset might offer more data points, a smaller dataset curated with meticulous attention can provide a more valuable foundation for analysis and learning. By prioritizing quality over quantity, the data collected and included in the dataset has undergone rigorous scrutiny, filtering, and validation processes.\n\nIt is important to note that within the specific domain of identifying propaganda techniques at the span-level, the task is inherently challenging. It demands comprehensive training, which our annotators underwent. This training encompassed reviewing over 250 existing texts labeled with the 20 propaganda techniques in English from a previous dataset. Subsequently, the actual annotation process commenced, which required intense focus and diligence for each span of text. This complete endeavor of annotation training and the annotation on our dataset spanned approximately two and a half months, during which two annotators underwent rigorous training and became eligible to annotate. The annotation training task itself extended over a period of two weeks and the the actual annotation process on our dataset took approximately 2 months (two and a half months in total). This duration was influenced by the limited resources dedicated to this meticulous task. There is a scarcity of individuals who are fluent in both Urdu + English and possess the ability to critically analyze and identify propaganda in the text (unless they undergo proper training).\n\n---\n\n**Comment** The annotator training procedure is a completely new approach and not directly motivated from any previous study. It's unclear why this training would make an annotator eligible enough to annotate the data.\n\n**Response** The annotator training method ensures that both annotators thoroughly review the definitions and examples from a previously labeled dataset. The method involves a training sequence where annotators first work through one set of labeled texts. Following this, they receive feedback from a domain expert (an individual experienced in this task who has annotated texts before). Subsequently, they proceed to train on another set, leveraging the training and feedback from the first set. We found this approach to be remarkably effective. The shared learning experience during the annotation of both sets significantly enhanced the consensus between the annotators. This positive impact on consensus became particularly evident as they trained on the second set.\n\n---\n\n**Comment** The motivation behind the experimental analysis on the out-of-domain meme dataset is unclear.\n\n**Response** Existing literature shows that training on an out-of-domain dataset can enhance the model's ability to generalize and perform well on different types of data, including one's own dataset. This can help the model capture broader patterns and nuances in propaganda techniques. The comparison between out-of-domain and in-domain performance can provide insights into the model's adaptability towards new datasets. If the model performs well on both datasets, it indicates that it has successfully learned transferable features. Following these observations from existing literature, we used a model fine-tuned on an out-of-domain dataset for inference on our own code-switched dataset. Results indicate that fine-tuning an out-of-domain dataset does not help improve the performance of propaganda technique identification on our code-switched dataset. This can be attributed to the nature of the out-of-domain dataset, which consists of text captions from meme images. Without the accompanying images, the contextual information within the text may degrade which leads to poor performance. We will add motivation behind experimental analysis on an out-of-domain dataset in the manuscript.\n\n\n&nbsp;\n&nbsp;\n\n**Questions for the Authors**\n\n---\n--- \n\n**Question-A** Why does the annotation training work?\n\n**Response** The annotator training method ensures that both annotators thoroughly review the definitions and examples provided by a previously labeled dataset. Both annotators undergo training using more than 250 labeled texts which include examples for each of the 20 propaganda techniques. These 250 texts are split into two sets where they work through one set of labeled texts and receive feedback from a domain expert (an individual who has previously trained for this task and annotated texts). Following this, they proceed to label another set, benefiting from the training and feedback received for the first set. We have found this method to be highly effective. The collaborative learning experience while annotating both sets significantly enhanced the consensus between the annotators. This became particularly evident as they worked through the second set. Furthermore, during the actual annotation of the dataset, conflicts between the annotators were rare and easily resolved. The Krippendorff's alpha score was notably favorable, especially considering that the annotation involved span-level precision on which Krippendorff's alpha usually doesn't score too well.\n\n\n---\n\n**Question-B** What if the dataset includes actual Urdu written in native script, instead of Roman?\n\n**Response** In our dataset, we collected only posts/texts/examples written in Roman Urdu + English and did not include those posts/texts/examples that were written in the native Urdu script. However, the annotators were well-versed in native Urdu, Roman Urdu, and English. If needed, annotating native Urdu texts wouldn't be an issue. The reason for the choice of Roman Urdu was its abundance on social media compared to the native script, as many people prefer to type in Latin script rather than Urdu's native script because of its convenience in typing.\n\n--- \n\n**Question-C** How did you pick and choose the 'propaganda related texts' from social media?\n\n**Response** The social media texts were manually hand-picked from social media accounts and public groups and pages on different propagandistic topics, such as politics, cricket, woman’s march (Aurat march), feminism, religious views, gender equality and more whose texts were full of propagandistic posts, examples, and content. Furthermore, the data collectors prioritized posts/texts that displayed a higher degree of richness in propaganda techniques. This approach aimed to encompass a wider array of propaganda techniques within our limited dataset. Nonetheless, we intentionally also included posts/texts/examples devoid of propaganda labels. This inclusion prevents the model from consistently attempting to identify a propaganda technique when it is not inherently present.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Novelty of the task and dataset. ", "reasons_to_reject": "The dataset is very small. Most datasets in the same domain have significantly more amount of data.\nThe annotator training procedure is a completely new approach and not directly motivated from any previous study. It's unclear why this training would make an annotator eligible enough to annotate the data.\nThe motivation behind the experimental analysis on the out-of-domain meme dataset is unclear.", "questions_to_authors": "Why does the annotation training work?\nWhat if the dataset includes actual Urdu written in native script, instead of Roman?\nHow did you pick and choose the 'propaganda related texts' from social media?\n", "paper_topic_and_main_contributions": "The authors introduced a novel task - \"Detecting propaganda in a Code-switching Scenario\", which was not previously explored before. The dataset seems novel and the experimental analysis is well presented. However, the two languages are English and 'Transliterated' Urdu, hence labelling it as a Code-switching phenomena, might not be factually right.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OyWEtOtBwl", "reviewer": "Reviewer_s1qS", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Paper Topic And Main Contributions**\n\n---\n---\n\n**Comment** A reference article defining “code-switching” can be added, when introducing the term in the Introduction\n\n**Response** Thank you for your feedback and suggestions. Following your recommendations, we intend to revise our manuscript by incorporating the paper mentioned below. This paper defines the concept of Code-Switching and provides a comprehensive overview of various collected datasets, along with the diverse tasks conducted on these datasets.\n\n\n*Sitaram, Sunayana, et al. \"A survey of code-switched speech and language processing.\"*\n\n---\n\n**Comment** It would be good to add a very short discussion of what topics the texts are on.\n\n**Response** We will revise the manuscript by including additional content about the topics from which these code-switched texts were derived from. Some of the topics covered in these texts include politics, sports (cricket), women's march (Aurat march), feminism, religious views, and gender equality.\n\n\n&nbsp;\n&nbsp;\n&nbsp;\n\n\n**Reproducibility**\n\n---\n---\n\n\n**Comment** Could reproduce the results with some difficulty. The settings of parameters are underspecified or subjectively determined; the training/evaluation data are not widely available. \n\n**Response** We assure the reviewer that the reproducibility of our work will be a straightforward process. Once the anonymity period concludes, we publicly release the code and dataset. The dataset will be accessible in both JSON and CSV formats. Furthermore, we will provide the train and test splits that we use along with the seed value. We will also provide comprehensive instructions and Python commands for executing each model for both training and inference. Specific hyperparameters will be presented in the Python commands which are the same as those available in Section 4.3, \"Experimental Settings.\" Moreover, we have included the code for generating propaganda techniques using GPT-3.5 Turbo, complete with the corresponding prompts. The prompt for GPT-3.5 can also be located in the Appendix C.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. New task proposed\n2. New dataset proposed, while small, it contains a rare combination \n3. Meaningful and useful results\n4. Very well written article", "reasons_to_reject": "none", "questions_to_authors": "", "paper_topic_and_main_contributions": "This article proposes the task of detecting propaganda in code-switched text, featuring a low-resourced language. The article presents a corpus of 1030 texts, annotated on fragment level with propaganda techniques, and experiments with different types of models. The paper provides useful outcomes regarding which approach is better.\n\nContributions:  \n1) a new task proposed\n2) a new English - Roman Urdu corpus, annotated with propaganda techniques at the fragment level is presented\n3) useful conclusions about which modelling approach is better, are provided\n\nThe article is very clearly written, but some details would make it better:\n\n1.\tA reference article defining “code-switching” can be added, when introducing the term in the Introduction.\n2.\tIt would be good to add a very short discussion of what topics the texts are on.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7LeGOIv6br", "reviewer": "Reviewer_KqAR", "summary": "", "questions": "A: Why was 20-shot chosen for the GPT model? This seems a high number to use as the only experiment.\n\nB: BERT, a model which should have minimal knowledge of roman Urdu, appears to in some cases outperform models that should have bilingual understanding (e.g. the F1 of model 7 is higher than 9). Was anything done to understand why this happens, does this performance come from solely the English parts of the sentence?\n\nC: The annotation process is rigorous and it says \"there were rarely disagreements after the conflict was discussed\". Yet, the Krippendorff alpha is still (slightly) below the 0.8 threshold. What caused this, as I would expect if conflicts were discussed and annotations amended, the score would be higher?\n\nD: How will the dataset be distributed in the final version of the paper, given that 60% of the dataset is Twitter who require datasets only contain the ID and end users must retrieve text, which is now difficult given Twitter API changes?\n\nE: Is GCP Cloud Translate capable of actually translating roman Urdu? I tried the first example from Appendix A in Google Translate and it seems unable to - is GCP Cloud Translate better? The limitations discussion for translation only discuss that propaganda techniques may be affected when translated.\n\n---\n\n_Following the rebuttal, I am satisfied with the answers to these questions_", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reasons to Reject**\n\n---\n---\n\n**Comment** The results section lacks discussion on the majority of the models, almost exclusively discussing the highest scoring (11,12 and 13), so the results for the earlier strategies are not very well covered. \n\n**Response** In our experiments, our primary focus is to present a comparison among the four different fine-tuning strategies depicted in Table 3. The comparison we conduct pertains to the fine-tuning of pretrained models using out-of-domain, translated, and code-switched datasets - our three fine-tuning strategies. These comparisons are meaningful only when carried out using monolingual, multilingual (including bilingual), and cross-lingual language models which we do in models $M_{1}$ to $M_{10}$ using BERT, mBERT, RUBERT and XLM-RoBERTa. Thus, rather than engaging in an in-depth model-to-model comparison for $M_{1}$ to $M_{10}$, we make a comparison between the different fine-tuning strategies. In addition, we introduce the usage of a LLM (GPT-3.5) without fine-tuning as an approach, and its results are analyzed under $M_{13}$. We then delve into a more detailed discussion of models $M_{11}$, $M_{12}$, and $M_{13}$ which achieve a much better performance in comparison to models  $M_{1}$ to $M_{10}$. These models are highlighted due to their superior performance according to our primary evaluation metric (micro-average f1-score).\n\n---\n\n**Comment** The out-of-domain dataset chosen (Dimitrov et al.'s meme propaganda dataset) seems an odd choice, given it's only in English. Although I see the value of training with an out-of-domain dataset and an English-only dataset, I think combining them makes it difficult to interpret the results. I appreciate that this is where the label schema used comes from, but I Comment the dataset itself's usage.\n\n**Response** Given the scarcity of labeled data in the propaganda detection domain, particularly with the specific 20 propaganda labels that we use, there was an absence of out-of-domain datasets using code-switched languages, including Roman Urdu and English. Therefore, there are no code-switched out-of-domain datasets available for use. Consequently, for any form of out-of-domain fine-tuning comparison, we must rely on the few existing options from which we choose: the meme dataset (Dimitrov et al.'s meme propaganda dataset) which was in English. In our comparison, we contrast this out-of-domain fine-tuning approach with the code-switched fine-tuning approach on our dataset.\n\n---\n\n**Comment** There is no discussion of the ethics of the annotation tasks. No information is given on how they were recruited, pay, whether they gave proper informed consent etc \n\n**Response** We make sure to uphold the highest ethical standards in all our research endeavors and confirm that all necessary annotations and procedures were meticulously carried out internally to ensure the validity and integrity of the study. The 4 data collectors, selected by the authors based on the quality of their submissions from a pool of 10 candidates, consented to the collection task and were fairly compensated for gathering 2000 code-switched texts (500 each). We will add this information in the ethics section of revised manuscript.\n\n\n\n&nbsp;\n&nbsp;\n&nbsp;\n&nbsp;\n&nbsp;\n\n**Questions for the Authors**\n\n**Question-A** Why was 20-shot chosen for the GPT model? This seems a high number to use as the only experiment.\n\n**Response** Firstly, we use the GPT model, providing a set of examples within the prompt for four distinct few-shot settings (Shown in Appendix C). We then assess performance across these four settings and observe that the 20-shot configuration yields the highest micro-average F1-Score which is our primary evaluation metric, hence one of the factors motivating its selection. Secondly, another rationale behind opting for the 20-shot setting in the GPT model is that the prompt contains 20 examples. This is the only prompt using representations of all our 20 propaganda techniques. As a result, each propaganda technique is covered by at least one example, further underscoring the suitability of the 20-shot setup.\n\n---\n\n**Question-B** BERT, a model which should have minimal knowledge of roman Urdu, appears to in some cases outperform models that should have bilingual understanding (e.g. the F1 of model 7 is higher than 9). Was anything done to understand why this happens, does this performance come from solely the English parts of the sentence?\n\n**Response**\nAlthough we did not specifically delve into this aspect. This performance discrepancy could arise from a combination of factors, potentially including the efficacy of BERT in comprehending the English parts of the sentence and its ability to glean insights from the context that extends beyond individual languages. Further analysis could shed light on the nuanced interactions contributing to BERT's performance in future work.\n\n---\n\n**Question-C** The annotation process is rigorous and it says \"there were rarely disagreements after the conflict was discussed\". Yet, the Krippendorff alpha is still (slightly) below the 0.8 threshold. What caused this, as I would expect if conflicts were discussed and annotations amended, the score would be higher?  \n\n**Response** The reviewer raises a valuable question, and we apologize for any lack of clarity. The computation of Krippendorff's alpha was conducted when both annotators compared labels and spans within the text/example, prior to conflict resolution. Our remark \"there were rarely disagreements after the conflict was discussed\" refers to the case during conflict resolution where there was a conflict on a certain propaganda label, and both annotators discussed and cross-questioned each other. After this discussion, both annotators would almost always reach a consensus on the propaganda label in question and thus there was no conflict left. Additionally, the relatively lower threshold (the Krippendorff's alpha before conflict resolution) can be attributed to the incorporation of both labels and spans in the calculation. The intricacies of handling spans posed a substantial challenge, significantly impacting the overall score. Notably, if the calculation were solely based on labels, the Krippendorff's alpha would surpass 0.9\n\n---\n\n**Question-D** How will the dataset be distributed in the final version of the paper, given that 60\\% of the dataset is Twitter who require datasets only contain the ID and end users must retrieve text, which is now difficult given Twitter API changes?  \n\n**Response** The dataset is now in its finalized version, with 1030 code-switched texts, which might be expanded in future research. To prioritize anonymity, we intentionally abstain from collecting ID/user information associated with the collected text/examples. The data collectors (totaling 4)  manually collected the data/posts/texts from public accounts without using the Twitter API. Therefore, any potential modifications to the Twitter API in the future will not impact our forthcoming research or data gathering procedures. It's also worth noting that the collectors received appropriate compensation for their diligent efforts in manually collecting the code-switched texts.\n\n---\n\n\n**Question-E**\nIs GCP Cloud Translate capable of actually translating roman Urdu? I tried the first example from Appendix A in Google Translate and it seems unable to - is GCP Cloud Translate better? The limitations discussion for translation only discuss that propaganda techniques may be affected when translated. \n\n**Response** Yes, the GCP Cloud Translate effectively translates Roman Urdu to English, which is why we opted for the paid service. We observed that Google Translate did not seem to give proper translations, leading us to choose the paid GCP Cloud Translate API. The quality of the translation decision could be attributed to the paid service's capability to facilitate the translation of Roman Urdu + English to English. While the quality of translation provided by the GCP Cloud Translate API is not a limitation, we acknowledge that the challenges due to the stylistic nuances inherent in the code-switched language are not transferable to the different languages (In our case English). Furthermore, specific propaganda techniques exhibit language-specific attributes, which can result in the loss of the text's propagandistic essence even with a proficient translation from code-switched to English. Here is an example from our dataset of a translated code-switched text (Roman-Urdu) to English using GCP Cloud Translate API to show the quality of translations (Which seem to be of a high quality):\n\n**Code-switched Text** *\"See this group of foreign funded women dancing on the roads, phir kekte hain iss mulk mein zalzale kyun aate hain. Ye Khuda ka azaab nhi hai toh aur kya hai?\"*\n\n**English Translation** *\"Watch this group of foreign-funded women dance in the streets, then see why earthquakes happen in this country. If this is not God's punishment, what else is?\"*\n\n\n\n&nbsp;\n&nbsp;\n&nbsp;\n\n\n**Typos Grammar Style And Presentation Improvements**\n\n---\n---\n\n**Comment** Line 228: Citation should not be in parentheses Table 4: Missing dividing line between models 12 and 13, as in table 3 \n\n**Response** Thank you for your feedback on the improvements. We will address the two corrections in our final manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Annotation methodology is excellent. Previous similar datasets (e.g. semeval 2023 task 3) have suffered from significant annotation noise due to the complexities of the problem, but this methodology would appear to improve on this\n- A wide selection of models are trialled, including general-purpose and Roman Urdu-specific.\n- It considers and rules out translation for this task, a widespread method used in multilingual analysis\n- Much propaganda research is for English or other European languages. This work not only considers an uncommon language for work in this field, but also considers it in code switched form, which widens its usefulness for dealing with social media where code switching is common.\n- The presentation of results is clear", "reasons_to_reject": "- The results section lacks discussion on the majority of the models, almost exclusively discussing the highest scoring (11,12 and 13), so the results for the earlier strategies are not very well covered. - _Given the rebuttal, I now understand why and no longer consider this an issue_\n- The out-of-domain dataset chosen (Dimitrov et al.'s meme propaganda dataset) seems an odd choice, given it's only in English. Although I see the value of training with an out-of-domain dataset and an English-only dataset, I think combining them makes it difficult to interpret the results. I appreciate that this is where the label schema used comes from, but I question the dataset itself's usage. _I understand the rebuttal argument that data is scarce, and that this is very possibly the best choice of data to use, but I still think the fact this dataset is both English-only and collected from memes makes it problematic to compare_\n- There is no discussion of the ethics of the annotation tasks. No information is given on how they were recruited, pay, whether they gave proper informed consent etc. _The authors have promised to include this information in the next revision_", "questions_to_authors": "A: Why was 20-shot chosen for the GPT model? This seems a high number to use as the only experiment.\n\nB: BERT, a model which should have minimal knowledge of roman Urdu, appears to in some cases outperform models that should have bilingual understanding (e.g. the F1 of model 7 is higher than 9). Was anything done to understand why this happens, does this performance come from solely the English parts of the sentence?\n\nC: The annotation process is rigorous and it says \"there were rarely disagreements after the conflict was discussed\". Yet, the Krippendorff alpha is still (slightly) below the 0.8 threshold. What caused this, as I would expect if conflicts were discussed and annotations amended, the score would be higher?\n\nD: How will the dataset be distributed in the final version of the paper, given that 60% of the dataset is Twitter who require datasets only contain the ID and end users must retrieve text, which is now difficult given Twitter API changes?\n\nE: Is GCP Cloud Translate capable of actually translating roman Urdu? I tried the first example from Appendix A in Google Translate and it seems unable to - is GCP Cloud Translate better? The limitations discussion for translation only discuss that propaganda techniques may be affected when translated.\n\n---\n\n_Following the rebuttal, I am satisfied with the answers to these questions_", "paper_topic_and_main_contributions": "Social media users who speak multiple languages may code switch, i.e. post in a mixture of two languages. This paper investigates the detection of propaganda in code-switched English and romanised Urdu.\n\nThe main contributions are:\n- A dataset propaganda techniques in English/Roman Urdu tweets, annotated at span level via a rigorous process\n- Results for training a variety of model configurations (including specific roman Urdu models) in different scenarios (out of domain training, multilingual/monolingual and few-shot prompting)\n- Overall findings that translating romanised Urdu is insufficient and the multilingualism should be used directly", "excitement": 4, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 228: Citation should not be in parentheses\n\nTable 4: Missing dividing line between models 12 and 13, as in table 3"}, "extra_text": {"Justification_for_Ethical_Concerns": "The paper uses a set of workers to collect social media post and two annotators. However, it lacks discussion how these annotators were selected, pay, and consent. It does not state whether the annotation process passed an institutional ethics board or policy.\n\n_In the rebuttal, the author has promised to include this information in the paper - without having read this it is difficult to say whether it will now meet the ethics policy_"}}, {"review_id": "OHlTICLVHV", "reviewer": "Reviewer_ut2p", "summary": "", "questions": "- Do the code-switched words contribute to the propaganda classification with monolingual language models or supervised classification models?\n- Do you plan to release the corpus?", "limitations": "", "rating": 1, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reasons to Reject**\n\n---\n---\n\n**Comment** It is not motivated the relevance of the new dataset. \n\n**Response** Significant efforts have been made in investigating the identification of propaganda techniques in high-resource languages like English; however, low-resource languages have not received sufficient attention in this domain. A considerable number of social media users from low-resource languages often engage in code-switching while expressing their opinions online. Given the presence of these users on social media platforms, it is important to have a dataset for effectively studying the identification of propaganda techniques on code-switched text. The introduction of a new code-switched dataset marks a step in this direction.\n\n---\n\n**Comment** The experimental setup does not show the contribution of code-switched words. \n\n**Response** We agree that we do not show the influence of code-switched words as we did not annotate the words with classes \"English\" and \"Roman Urdu\". The data was only annotated with \"propaganda techniques\" on a fragment-level. Due to this reason, we could not observe the distinct contributions of Roman Urdu words versus English words within our models.\n\n---\n\n**Comment** The unique evidence is that multilingual language models do a good job with code-switched words, which is known by the community. \n\n**Response** While the community acknowledges the capabilities of multilingual language models in handling code-switched words, it's important to clarify that detecting propaganda techniques using multilingual models on code-switched text is more intricate than a simple comparison with monolingual models on translated content. Multilingual models have limitations, especially with low-resource languages, which can complicate the task of identifying subtle nuances inherent in propaganda, making it a challenging task. Therefore, we give attention to this comparison in our concluding remarks. It's also worth noting that these discussions aren't the primary focus of our highlighted contributions in the paper.\n\n\n\n&nbsp;\n&nbsp;\n&nbsp;\n&nbsp;\n&nbsp;\n\n**Questions for the Author**\n\n---\n---\n\n**Question-A** Do the code-switched words contribute to the propaganda classification with monolingual language models or supervised classification models? \n\n**Response** Please note that all of the models used in this work fall under the category of supervised classification (i.e. detecting propaganda techniques in multi-class multi-label task settings on code-switched examples/texts). Secondly, we did not annotate the data (individual words of examples) with classes such as \"English\" and \"Code-switched (Roman-Urdu)\", but rather with propaganda techniques. Due to the non-availability of labels of individual words (i.e. \"English\" and \"Code-switched (Roman-Urdu)\"), we have not conducted a study to see the impact of only \"code-switched\" words on the identification of propaganda techniques in monolingual, multilingual and cross-lingual language models.\n\n---\n\n**Question-B** Do you plan to release the corpus? \n\n**Response** Yes, as mentioned in the abstract, we will make the corpus publicly available once the EMNLP anonymity period concludes. Additionally, we will release the code for our custom-developed annotation platform, used for fragment-level annotation of texts. Furthermore, we will make the code for our models and comprehensive instructions accessible to ensure complete reproducibility.\n\n\n&nbsp;\n&nbsp;\n&nbsp;\n&nbsp;\n&nbsp;\n\n**Reproducibility**\n\n---\n---\n\n**Comment** Would be hard pressed to reproduce the results. The contribution depends on data that are simply not available outside the author's institution or consortium; not enough details are provided. \n\n**Response** We assure the reviewer that the reproducibility of our work will be a straightforward process. Once the anonymity period concludes, we publicly release the code and dataset. The dataset will be accessible in both JSON and CSV formats. Furthermore, we will provide the train and test splits that we use along with the seed value. We will also provide comprehensive instructions and Python commands for executing each model for both training and inference. Specific hyperparameters will be presented in the Python commands which are the same as those available in Section 4.3, \"Experimental Settings.\" Moreover, we have included the code for generating propaganda techniques using GPT-3.5 Turbo, complete with the corresponding prompts. The prompt for GPT-3.5 can also be located in the Appendix C.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The creation of a new dataset in a non-English language.", "reasons_to_reject": "- It is not motivated the relevance of the new dataset.\n- The experimental setup does not show the contribution of code-switched words.\n- The unique evidence is that multilingual language models do a good job with code-switched words, which is known by the community.", "questions_to_authors": "- Do the code-switched words contribute to the propaganda classification with monolingual language models or supervised classification models?\n- Do you plan to release the corpus?", "paper_topic_and_main_contributions": "The pape proposes a new dataset for propaganda in Latin-Urdu detection with the particularity of including code-switching words in English.\n\nTo sum up it is the common new dataset paper, which describes the way of taking the documents, the annotation and some experiments to validate the new corpus. I have to say that the use of code-switching words is not novel, and it is known that it helps in multilingual scenarios, and in particular in social media genre [1]. For this reason, the multilingual LM are the ones with higher performance.\n\nThe differences among using the code-switched words and not using them (translation approach) are not large, when the same learning models (LM) are used. I did not considered the results with different models because they are not comparable. Therefore, in this case, I do not see the contribution of code-switched words. This may be motivated by the low presence of code-switched words. Therefore, I do not see the real contribution of code-switched words for the classification of propaganda. At least, I see that multlingual LM can work with code-switched words, which is known [1].\n\nTo sum up,  I do not see this dataset paper in EMNLP, as well as it lacks of some relevant details to assess the contribution of this new dataset for the research community.\n\nReferences\n\n[1] Camacho-Collados, J., Doval, Y., Martínez-Cámara, E., Espinosa-Anke, L., Barbieri, F., & Schockaert, S. (2020, May). Learning cross-lingual word embeddings from twitter via distant supervision. In Proceedings of the international AAAI conference on web and social media (Vol. 14, pp. 72-82).", "excitement": 1, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "c6rmctCFEC", "reviewer": "Reviewer_YUyd", "summary": "", "questions": "Question A: Authors claim that \"one main method of circulating falsified information is through propaganda\". This is not true. Propaganda does not necessarily involve false information. \n\nQuestion B: How did the authors get permission to publish data coming from Facebook and Instagram?\n\nQuestion C: Section 3.1. What are the criteria to select a post? What type of post/from what sort of account is considered?\n\nQuestion D: line 292. What is a domain expert here? Why asking AI experts to annotate instead of domain experts?\n\nQuestion E: \"if the conflict cannot be resolved, the example is discarded.\" This is unclear. Is a whole post discarded in the presence of one single disagreement?\n\nQuestion F: the authors developed an annotation interface because they claim that existing alternatives run short. Are they going to release it?\n\nQuestion G: Table 1. The length is measured in terms of tokens or characters? If the largest span contains 400 tokens (or even characters), it sounds huge. What is such a long span?\n\nQuestion H: Table 3. How is this translated? What is the quality of such translation?\n\nQuestion I: \"i.e., each example can be assigned zero, one, or multiple labels\" What is an example? The whole post?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reasons to Reject**\n\n---\n---\n\n**Comment** The authors claim this is a new task, and even mention it as one of the main contributions, but it is not. It is still propaganda technique identification, it is just that the text is not monolingual.\n\n**Response** We appreciate the reviewer's feedback and would like to clarify the distinction between our proposed task and existing propaganda technique identification. While it's true that both tasks involve identifying propaganda techniques, our task of detecting propaganda techniques in code-switched text introduces novel challenges. In existing propaganda detection, the focus is primarily on the monolingual text which is in a high-resource language i.e., English. In contrast, our task deals with code-switched text, which combines two languages (English and Roman-Urdu) including a low-resource language where no work has been done. This is a different domain and requires a different dataset and approach to tackle problems such as complexities related to language switching and cultural nuances that are typically not present in monolingual text.\n\n---\n\n**Comment** The authors annotate spans and multi-label, but their experiments neglect the spans. This is counterintuitive and unrealistic\n\n**Response** We recognize that our data is annotated at the fragment/span level, while the models were trained on example-level classes, specifically propaganda techniques. Our primary objective in this work was to introduce a code-switched (English + Roman Urdu) dataset annotated with propaganda techniques at the fragment level. Nevertheless, we have presented preliminary results of the detection of propaganda techniques in the entire example, without predicting the spans of the techniques. Our plan is to enhance this research by extensively analyzing span/fragment-level propaganda techniques using pre-trained language models. \n\nA key factor affecting our decision to avoid fragment-level detection initially was our preliminary analysis, which mainly involved comparing our outcomes with those from out-of-domain and code-switched texts translated to English. Translating our code-switched data into English would inevitably alter the propaganda technique spans due to differing sentence structures—Urdu (Roman Urdu) follows a subject-object-verb (SOV) structure, while English follows subject-verb-object (SVO). Hence, this structural distinction would require us to do manual annotation of the translated text. Furthermore, the out-of-domain text is also annotated as a multi-label multi-class challenge and would also require manual annotation for that dataset.\n\n\n&nbsp;\n&nbsp;\n&nbsp;\n&nbsp;\n&nbsp;\n\n**Questions for the Authors**\n\n---\n---\n\n\n**Question-A** Authors claim that \"one main method of circulating falsified information is through propaganda\". This is not true. Propaganda does not necessarily involve false information. \n\n**Response** We agree with the reviewer that propaganda is not exclusively tied to false information. However, our assertion doesn't claim that all propaganda hinges on false data; rather, we emphasize it as one method among various others. Our definition, as stated in lines 41 to 43, clarifies: \"Propaganda is the dissemination of biased or misleading information in order to manipulate people's beliefs and opinions towards a particular objective.\" In this context, \"biased\" signifies an inclination toward a specific perspective, resulting in an unfair representation of information. This information is frequently true but skewed. \"Misleading\" refers to information, statements, or actions that create a false impression or deceive, even when the underlying statement is accurate.\nMoreover, among the propaganda techniques we examine is \"Loaded Language,\" which employs emotive language to evoke strong feelings. This technique doesn't necessarily involve false information. Many other techniques, like this one, also don't disseminate false information.\n\n---\n\n**Question-B** How did the authors get permission to publish data coming from Facebook and Instagram?\n\n**Response** We collected public data by following Facebook and Instagram’s policy on “Public Information”. While collecting data, we ensured that data privacy related concerns are addressed. We have not included source links or user IDs in the data for each of the code-switched examples/texts. We collected the data from posts that fall under the category of “Public Information” according to Facebook and Instagram data policies. All users of these social media sites agree to the terms and conditions regarding the information they post on them publicly. We did not collect data from any groups which were not in public view. \n\n---\n\n**Question-C** Section 3.1. What are the criteria to select a post? What type of post/from what sort of account is considered?\n\n**Response** We chose those posts/texts that we found were richer in the propaganda techniques so that we could capture more propaganda techniques in our limited data. We also ensure we select posts/texts based on the percentage of Roman Urdu and English words making sure that there are enough words from each language. Having said that, we also added posts/texts/examples which had no labels present. The social media texts are collected from the public Facebook accounts of the 4 data collectors, and we also followed various public Facebook groups/pages on different topics, such as politics, cricket, feminism, women’s march (Aurat march), religious views, gender equality, and more to collect text from diverse topics. \n\n\n---\n\n**Question-D** Line 292. What is a domain expert here? Why asking AI experts to annotate instead of domain experts?\n\n**Response** When referring to a \"Domain Expert,\" we mean an individual who possesses prior experience supervising the labeling of text into various propaganda techniques, including those employed in our study. We want to clarify that the term \"AI experts\" denotes that both annotators were actively engaged in AI research and were not intended to favor them over domain experts. Both annotators were proficient in English, Urdu, and Roman-Urdu languages. They underwent training under the guidance of the main \"Domain Expert\" to annotate the data. Following training, both annotators became qualified and well-prepared to annotate code-switched data. Finding existing domain experts in this field proved challenging, as few individuals were well-acquainted with identifying the labels and spans of the propaganda techniques we employed. This challenge was further compounded when narrowing down the pool to those who are familiar with native Urdu, Roman Urdu, and English languages.\n\n\n---\n\n**Question-E** \"if the conflict cannot be resolved, the example is discarded.\" This is unclear. Is a whole post discarded in the presence of one single disagreement? \n\n**Response** We apologize for any confusion caused by our statement. To clarify, if a conflict arises regarding any of the fragments or spans within an example, and both authors are in disagreement over the identified propaganda, the entire example or post is discarded.\n\n---\n\n**Question-F** The authors developed an annotation interface because they claim that existing alternatives run short. Are they going to release it? \n\n**Response** We plan to make the code of our newly developed annotation platform, which annotates propaganda at the fragment level, publicly available. Additionally, we intend to deploy our annotation platform as a demo project on a web service.\n\n---\n\n**Question-G** Table 1. The length is measured in terms of tokens or characters? If the largest span contains 400 tokens (or even characters), it sounds huge. What is such a long span? \n\n**Response** When referring to \"length,\" we are specifically addressing the \"number of characters\" in a post. In an exceptional instance, the longest post, comprising 400 characters, was identified as a \"smears\" campaign targeting an individual. This post aimed to inflict reputational harm throughout its entirety, leading to its extended length.\n\n---\n\n**Question-H** Table 3. How is this translated? What is the quality of such translation?\n\n**Response** The process of translating the code-switched text (comprising both English and Roman Urdu) into English involves utilizing Google Cloud Platform's GCP Cloud Translation API. This API is capable of translating text across more than 100 language pairs. It relies on a Google pre-trained Neural Machine Translation (NMT) model and can automatically detect the presence of Roman Urdu words within the English code-switched text. Although the API does not explicitly mention a qualitative measure for translation, such as the Bleu score, we performed a manual assessment of 20 posts, encompassing both short and long sentences. Based on our observations from these evaluations, we found that the model provided reasonably accurate translations. Consequently, we chose to proceed with utilizing its translation services.\n\n---\n\n**Question-I** \"i.e., each example can be assigned zero, one, or multiple labels\" What is an example? The whole post?\n\n**Response** We apologize for the confusion. An example is the entire post/text (This post/text/example can have zero, one, or multiple labels)\n\n\n&nbsp;\n&nbsp;\n&nbsp;\n&nbsp;\n&nbsp;\n\n---\n\n**Typos Grammar Style And Presentation Improvements**\n\n**Response** We express our gratitude to the reviewers for their diligent reading and the considerable effort they invested in their review. We will address all of the issues and revise the manuscript. We will also review the problematic references and rectify the mistakes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This seems to be the first effort to produce resources for propaganda techniques identification in code-switched Urdu-English text.", "reasons_to_reject": "The authors claim this is a new task, and even mention it as one of the main contributions, but it is not. It is still propaganda technique identification, it is just that the text is not monolingual. \n\nThe authors annotate spans and multi-label, but their experiments neglect the spans. This is counterintuitive and unrealistic.", "questions_to_authors": "Question A: Authors claim that \"one main method of circulating falsified information is through propaganda\". This is not true. Propaganda does not necessarily involve false information. \n\nQuestion B: How did the authors get permission to publish data coming from Facebook and Instagram?\n\nQuestion C: Section 3.1. What are the criteria to select a post? What type of post/from what sort of account is considered?\n\nQuestion D: line 292. What is a domain expert here? Why asking AI experts to annotate instead of domain experts?\n\nQuestion E: \"if the conflict cannot be resolved, the example is discarded.\" This is unclear. Is a whole post discarded in the presence of one single disagreement?\n\nQuestion F: the authors developed an annotation interface because they claim that existing alternatives run short. Are they going to release it?\n\nQuestion G: Table 1. The length is measured in terms of tokens or characters? If the largest span contains 400 tokens (or even characters), it sounds huge. What is such a long span?\n\nQuestion H: Table 3. How is this translated? What is the quality of such translation?\n\nQuestion I: \"i.e., each example can be assigned zero, one, or multiple labels\" What is an example? The whole post?", "paper_topic_and_main_contributions": "The manuscript introduces a new corpus for propaganda identification at the span level for code-switched text Urdu-English. Then experiments are performed, at a different granularity level.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Rather than missing, some references include wrong information. For instance:\n\nBarrón-Cedeno -> Barrón-Cedeño\nG Martino -> Giovanni Da San Martino\nGoodwin and Raymie McKerrow seems incomplete."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "\"During the American Revolution in the 18th century, newspapers and the printing press in various American colonies propagated their views to promote patriotism\"\n\nThere were many American colonies, from Canada to Argentina. Most of them turned independent in the 19th century. I guess what the authors mean by American Revolution is the War of Independence of the 13 colonies against the UK.\n\n\"Rashkin et al. (2017) 170 focused on detecting multi-class propaganda at a 171 document-level on their annotated corpus\". This is not true. It is multi-class, but not multi-class propaganda. Propaganda is just one of the four classes.\n\nTable 2 does not demonstrate. It shows. The same applies to M11 and M13, Roberta. They do not demonstrate. They show.\n\nThe fontsize of the number sin Fig. 3 should be larger."}}]

Author rebuttal:

Meta-review: The reviews for this paper contain rather divided opinions, some arguing the importance of this contribution, others that it is insufficient.   
Reviewers X7cv and  s1qS praise the paper for addressing a significant problem, offering a well-annotated dataset, and presenting a clear and accessible write-up. They find no reasons to reject the paper. I tend to agree with the importance of the proposed resource. 
Reviewer KqAR had concerns about discussion of certain choices as well as ethics concerns, however most of their questions were answered through the rebuttal and turned to a positive outlook. 
Overall, authors provided a thorough and convincing  responses in the rebuttal period to all of the reviewers. 
Reviewer YUyd provided important clarification that while the dataset is novel and challenging, the task of detecting propaganda itself is existing, and finds the contribution mediocre.  Reviewer o5WW is concerned with the small size of the dataset and explanations behind some experimental choices. 
Overall, I would recommend the authors to include the reviewers suggestions, answers to their questions, clarifications into the text of the paper.
Decision: Accept-Main