ReDASPersuasion at SemEval-2023 Task 3: Persuasion Detection using Multilingual Transformers and Language Agnostic Features
Year: 2023

Abstract:
This paper describes a multilingual persuasion detection system that incorporates persuasion technique attributes for a multi-label classification task. The proposed method has two advantages. First, it combines persuasion features with a sequence classification transformer to classify persuasion techniques. Second, it is a language agnostic approach that supports a total of 100 languages, guaranteed by the multilingual transformer module and the Google translator interface. Our persuasion system outperforms the SemEval baseline in all languages, except zero shot prediction languages, which was not the main focus of our research. With the highest F1-Micro score of 0.45 for Italian, it achieved the eighth position on the leaderboard.

heading: 1 Introduction
text: The goal of the SemEval-2023 Task 3 (Piskorski et al., 2023) Subtask 3 is to detect the presence of one or many persuasion techniques in text data. It is a multilingual task and consisted initially of six languages: English, French, Polish, Italian, Russian and German. In the testing phase, organizers added three surprise languages (Spanish, Greek, and Georgian) for zero-shot classification tasks.
In this work, we aim to build a cross-lingual predictive model that detects persuasion techniques in a given text with the help of language transformers and linguistic features. Our system uses XLMRoBERTa (Conneau et al., 2019) as a backbone model, incorporating language agnostic features provided using Google translation over 100 languages for persuasion detection. XLM-RoBERTa has been shown to be a powerful multilingual pretrained language model compared against other models like Multilingual BERT (M-BERT) (Devlin et al., 2018) and DistilBERT (Sanh et al., 2019). Also, this language transformer can process all the languages existing in the SemEval-2023 Task3.
Through this task, we realized the importance of data quality in Multi-label classification tasks with
cross-lingual settings. We were also able to build a new system that incorporates language agnostic features to capture persuasion in text semantics, accompanied by the powerful capacity of language transformers based on attention mechanism.

heading: 2 Background

heading: 2.1 Task Description
text: The SemEval-2023 Task 3 (Piskorski et al., 2023) had three subtasks: (i) Subtask 1 is news genre categorization, (ii) Subtask 2 is framing detection, and (iii) Subtask 3 is persuasion techniques detection. We focus solely on the multi-label classification problem Subtask 3 that aims to detect 23 persuasion techniques. All subtasks are multilingual and contain zero-shot learning.
In Subtask 3, each article in the dataset is labeled with multiple classes. One article can contain different persuasion techniques (persuasion data statistics are listed in Table 6, Appendix B).

heading: 2.2 Task Evaluation
text: Subtask 3 is evaluated using the Micro F1 score between the gold labels and model predictions defined in Equation (1).
F1_Micro = TP
TP + FP+FN2 (1)
F1-Micro is based on True Positives (TP), False Positives (FP) and False Negatives (FN), instead of individually for each class, which makes it well suited for multi-label classification.

heading: 3 System Overview
text: In this section, we describe our model and the steps in our approach. Figure 1 depicts the architecture of the entire ReDASPersuasion system.
Our System consists of three modules: (i) The first module is a multilingual transformer model, (ii) The second feature engineering module focuses
2124
on building language agnostic features for cross-lingual classification of persuasion techniques, and (iii) The third final module is a multi-label classification head where the [CLS] embeddings from transformer classification are combined with persuasion features to feed this information to a dropout layer, followed by a dense linear layer, and finally a sigmoid activation function in order to perform multi-label classifications.
Initially, we concatenate the [CLS] embeddings with a total of six persuasion features that we will introduce in the next section. We also use a dropout of 0.3 to prevent overfitting. The dropout layer has the convenient property of speeding up training since fewer weights are required in each forward pass.
In our approach, the final layer consists of k independent sigmoid activation outputs (k = 19 for
English and k = 23 for all other languages). The persuasion label prediction is based on a threshold value of the logit outputs, fine-tuned using the development set.

heading: 3.1 Persuasion Feature Engineering
text: We created task-specific and language agnostic features to detect most of the persuasion techniques within the scope of 100 languages. We will explain below the process we followed to engineer each feature individually.

heading: 3.1.1 Appeal to Fear
text: For this persuasion technique, we lean towards sentiment analysis to find if fear and anger are represented in input text. Polyglot (Al-Rfou et al., 2013) has polarity lexicons for over 136 languages to detect positive, negative and neutral text (Chen and
Skiena, 2014). We extract polarity as the first persuasion feature because fear has a strong negative emotional inclination. In fact, polarity lies between [-1,1], where -1 defines a negative sentiment and 1 defines a positive sentiment.
Moreover, the “Slogans” persuasion technique also tends to act as emotional appeals. Therefore, using sentiment analysis is very important, and combines the scope of multiple persuasion techniques (Fear, Loaded Language, Slogans, Exaggeration).

heading: 3.1.2 Exaggeration and Minimization
text: We automatically analyze the text to find any Extreme Case Formulations (ECF) defined by (Mora, 2009) that invoke extreme descriptions of events or objects based on POS Tags, adverb wordlists and hyperboles. A simple example of an ECF is a sentence that contains an extreme description via an adjective or intensified using an adverb expressing exaggeration. Biddle et al. (2021) assembled a synthetic dataset called HyperProbe 1 including ECFs, qualitative and quantitative hyperboles.
We extract indefinite pronouns that express exaggeration and minimization (e.g., everybody, nobody), quantifiers (e.g., all, none), adjectives using the following POS Tags JJ (adjective), JJR (adjective, comparative) or JJS (adjective, superlative), and adverbs using RB (adverb), RBR (adverb, comparative) or RBS (adverb, superlative). Besides adverb wordlist, another source of descriptive exaggeration terms is distilled from the HyperProbe dataset. In Appendix A, we provide in Table 4 detailed resources of the wordlists and dictionaries used to implement our feature extraction methods.
With the help of the TextBloB python package 2, we applied POS tagging on text and we extracted subjectivity and polarity. Persuasion features also include exaggeration, minimization average proportion in text, ECF identifier counts, boolean categorical variable of presence of exaggeration or minimization. (Examples of adverbs used to capture persuasion techniques are listed in the appendix)
Troiano et al. (2018) also considered sentiment analysis and polarity as features to detect exaggeration. Similarly, we extracted subjectivity and emo-
1https://github.com/biddle-r/ HyperProbe
2https://textblob.readthedocs.io/en/ dev/
tional intensity as additional persuasion features. Furthermore, subjectivity quantifies the amount of personal opinion and factual information contained in the text. Subjectivity lies within [0,1] where 0 is very objective and 1 is very subjective. The higher subjectivity means that the text contains personal opinion rather than factual information.

heading: 3.1.3 Loaded Language
text: We simply use the “Profanity-filter” package 3 that detects use of profane and offensive terms in text. The function is_profane returns a boolean value if this text contains loaded language or not. Profanityfilter only supports English and Russian, we further extend the filter to include other languages and maintain the multilingual system design through implementing dictionaries following the Hunspell 4 format provided by LibreOffice 5.
We use this boolean categorical variable as another persuasion feature in our approach to detect loaded and offensive language. We use the Profanity-filter with deep analysis module which not only matches the exact profane word with custom wordlists but also finds derivative and distorted loaded words in text using the Levenshtein automata (Schulz and Mihov, 2002) initially applied to find fast corrections in string variables.

heading: 4 Experimental Results
text: We ran all classification experiments on a high performing cluster machine with an Intel® Xeon® Gold 6252 (3.70GHz) processor with 24 cores and 48 threads running Linux Red Hat Enterprise Server 8.6 (with Nvidia® Volta V100 GPU for our Pytorch-Lightning 6 implemented system).
On average, it took approximately 31 minutes and 29 seconds to train, validate and predict data across all six languages (More details in Table 5).
In this section, we will provide all experimental results starting by SemEval Task-3 baseline. In accordance with the task organizers’ request, all fine-tuning and enhancements made to our system after the deadline have been included in Section 4.3
3https://pypi.org/project/ profanity-filter/
4https://github.com/hunspell/hunspell 5https://cgit.freedesktop.org/
libreoffice/dictionaries/plain/ 6https://lightning.ai/docs/pytorch/ stable/

heading: 4.1 SemEval Baseline
text: As a baseline for Sub-Task3, SemEval used the Support Vector Machine (SVM) as a base estimator for the multi-output classifier, and it achieved a F1-Micro of 0.16 on the development set and 0.20 on the test set for English. From the provided code, They vectorize the raw input text using unigram and bigram count vectorizer. As preliminary experiments, we tuned other baselines like Random Forest and XGBoost using GridSearch to find a better estimator, but SVM achieved the best results.

heading: 4.2 Pre-deadline Results
text: Our system was developed using the results of XLM-RoBERTA on the development set, so we use it as the foundation model. However, it is easy to switch the first module of our system by simply changing the model name. We tested during development phase different multilingual pre-trained transformers distilled from literature including MBERT, DistilBERT, XLM-RoBERTA.
In order to be able to make a fair comparison, we further evaluate our initial system on the test set using a range of transformers and the same hyperparameters as before (no tuning was applied at this stage). In Table 1, DistilBERT comes in second place, while XLM-RoBERTA achieves the best results.

heading: 4.3 Post-deadline Results
text: Across all languages, the ReDASPersuasion system has been in constant progress by the help of persuasion features, tuned hyperparameter for model optimization, augmented data and the larger XLMRoBERTa model. Italian still remains the best with an F1-score of 0.56098 which got us currently ranked second in Italian as well as French.
As illustrated in Figure 2 above, the largest increase percentage in performance on the test set goes to the Polish language, with a 66.4% increase compared to the initial Pre-Deadline model.
We compute F1-Micro percentage increase following Equation (2):
Increase = (PostF1Micro− PreF1Micro)
|PreF1Micro| ∗100 (2)
Detailed information regarding the reasons for this increase in performance will be provided in the subsections that follow.

heading: 4.3.1 Fine-tuning and Optimization
text: For every language, there is a different dataset size, so we fine-tune the hyperparameters following Equation (3) and Equation (4) below:
The number of steps for the warm-up phase:
WarmupSteps = stepsepoch
2
= nbrtrain_samples 2 ∗ batch_size
(3)
The number of total steps:
TotalSteps = (stepsepoch ∗ nbrepochs) −WarmupSteps (4)
We also changed the XLM-RoBERTa tokenizer and model from the base to the large version, which contains 559 million trainable parameters. This change needs a smaller batch size to run the model on restricted memory. We tuned the hyperparameters in considerations to the model and data size. (see Table 2 reporting the best hyperparameters we obtained for our system optimization)
On the test set, our system works best for Italian language. However, when we use data augmentation and other features, all languages significantly enhance their classification.

heading: 4.3.2 Additional Features
text: Appeal to Time We observed that Appeal to time persuasion technique had few samples both on train and dev sets (Table 6 in Appendix B reports that Appeal to time has 121 training samples in all 5 languages combined), which impacts the overall
classification of the system. Therefore, we created a new time feature that use adverbs of time with a searching method from the python package “dateparser” 7 that extracts time and date from text.
Repetition We simply use MoreThanSentiments python package (Jiang and Srinivasan, 2023) to extract redundancy and specificity features in text to catch repeated words and phrases. In fact, redundancy is represented as the percentage of n-grams (n=2) that occur more than once in each document. To detect repetitions in long texts, Cazier and Pfeiffer (2016) used 10-k fillings, showing that the choice of n is directly related to its length. While specificity is the number of specific entity names and quantitative values scaled by the total number of words in a document.

heading: 4.3.3 Persuasion Data Augmentation
text: Static Text Augmenter A common approach for text augmentation is to replace words with their synonyms selected from WordNet (Kober et al., 2020; Li et al., 2022). We use the PPDB database (Ganitkevitch and Callison-Burch, 2014) as source of synonym replacement function. We download paraphrase multilingual packages and use the largest package size XXXL. The PPDB corpora contains a million paraphrases in 16 different languages. These packages include lexical, phrasal and syntactic paraphrasing types.
Dynamic Text Augmenter Rather than using only static word embeddings, we also use contextualized word embeddings to substitute words. In this approach, the length of the sentence is the same, but some words are replaced. Therefore, the augmentation won’t impact the max length of the input data, opposite to random insertion that might impact both text length and meaning.
Also, only the most probable tokens with probabilities that add up to top_p = 0.95 or higher are
7https://dateparser.readthedocs.io/en/ latest/introduction.html
kept for substitution. The top_p sampling method works as a control variable for the diversity of the textual augmentation.
To obtain both static and dynamic transformations, we use a sequential flow pipeline provided by the python package “nlpaug” 8 to include multiple text augmenters (static and dynamic). We have generated an alteration for each sample to produce double the amount of training and dev examples.

heading: 5 Conclusions and Future Work
text: The model’s performance has continually improved since the leaderboard reopened. From recent results, we found that performance improvement is evident as the model size grows (from XLM-R base to large) but also that tuning the model can immensely enhance the model’s training and ability to generalize data correctly.
A future direction would be to create an instance of our method to include zero-shot classification tasks using transfer learning. In fact, the transformer has the ability to be trained on zero-shot data, using additional features which can be persuasion attributes in our system. We would like to further explore the persuasive text interpretability per class to better understand the language indicators of persuasion in text.

heading: Acknowledgments
text: We thank the reviewers for their constructive comments. This research was partly supported by NSF grants 1433817, 1950297 and 2210198, ARO grant W911NF-20-1-0254, and ONR award N00014-19S-F009. Verma is the founder of Everest Cyber Security and Analytics, Inc.

heading: B Data Analysis and Statistics
text: B.1 Persuasion Techniques Table 6 below represents the total number of persuasion techniques present in the training and development sets. It is important to emphasize the imbalanced nature of these classes, as this has a detrimental effect on classification.
In comparison with confusion matrices on the development sets, the system failed to identify persuasion techniques such as obfuscation, straw man, and whataboutism. They represent only a few examples of both training and development sets.
Future work includes building new features and focusing on qualitative data augmentation to target these particular persuasion techniques that lead to weaker model detection.
B.2 Persuasion Adverbs Table 7 presents some examples of our exhaustive wordlist that contains a list of adverbs expressing time, doubt, exaggeration degree and minimization.
B.3 Sequence Length We created a dataset loader module that encodes the textual data using a transformer tokenizer and takes a different max_length which depends on the data set type (training, testing or development sets).
Figure 3 and Figure 4 describe the sequence length distribution of the tokenized samples in train, dev, and test sets. When optimizing the model, we have observed that max_length argument impacts the transformer classification, and it increases time of execution as well.
We would like to explore in the future using models like BigBird (Zaheer et al., 2021) and Longformer (Beltagy et al., 2020) in a cross-lingual environment, to expand the restricted max_length=512 tokens for XLM-R to max_length= 4096 tokens.
This would guarantee the input text tokenization to the maximum token length based on each language. These models are currently also available on the HuggingFace repository 9 10.
9https://huggingface.co/google/ bigbird-roberta-base
10https://huggingface.co/allenai/ longformer-base-4096

heading: Future Work
text: Future work includes building new features and focusing on qualitative data augmentation to target these particular persuasion techniques that lead to weaker model detection. B. 2 Persuasion Adverbs Table 7 presents some examples of our exhaustive wordlist that contains a list of adverbs expressing time, doubt, exaggeration degree and minimization. B. 3 Sequence Length We created a dataset loader module that encodes the textual data using a transformer tokenizer and takes a different max_length which depends on the data set type (training, testing or development sets). Figure 3 and Figure 4 describe the sequence length distribution of the tokenized samples in train, dev, and test sets. When optimizing the model, we have observed that max_length argument impacts the transformer classification, and it increases time of execution as well. We would like to explore in the future using models like BigBird (Zaheer et al. , 2021) and Longformer (Beltagy et al. , 2020) in a cross-lingual environment, to expand the restricted max_length=512 tokens for XLM-R to max_length= 4096 tokens. This would guarantee the input text tokenization to the maximum token length based on each language. These models are currently also available on the HuggingFace repository 9 10. 9https://huggingface. co/google/ bigbird-roberta-base
10https://huggingface. co/allenai/ longformer-base-4096