NL4IA at SemEval-2023 Task 3: A Comparison of Sequence Classification and Token Classification to Detect Persuasive Techniques
Year: 2023

Abstract:
The following system description presents our approach to the detection of persuasion techniques in online news. The given task has been framed as a multi-label classification problem. In a multi-label classification problem, each input chunk—in this case paragraph—is assigned one of several class labels. Span level annotations were also provided. In order to assign class labels to the given documents, we opted for RoBERTa (A Robustly Optimized BERT Pretraining Approach) for both approaches—sequence and token classification. Starting off with a pre-trained model for language representation, we fine-tuned this model on the given classification task with the provided annotated data in supervised training steps.

heading: 1 Introduction
text: Political rhetoric, propaganda, and advertising are all examples of persuasive discourse. As defined by Lakoff (1982), persuasive discourse is the nonreciprocal "attempt or intention of one party to change the behavior, feelings, intentions, or viewpoint of another by communicative means". Thus, in addition to the purely content-related features of communication, the discursive context of utterances plays a central role. SemEval-2023 Task 3 (Piskorski et al., 2023) considers persuasion as a communication phenomenon. With this approach, it is assumed that communication depends not only on the meaning of words in an utterance, but also on what speakers intend to communicate with a particular utterance. This concept is from the linguistic subfield of pragmatics. It is not always possible to derive the function of an utterance from its form and additional contextual information is often needed. Recent research like (Tenney et al., 2019) (Jawahar et al., 2019) indicates the possibility that transformer-based networks capture structural information about language ranging from syntactic
up to semantic features. Beyond these features, these architectures remain almost entirely unexplored. This task poses an attempt to explore the limits of the prevailing approach, in particular, investigating Transformers ability to capture pragmatic features.

heading: 2 Background
text: The central focus of this assignment is manipulative persuasion. The related task is to identify and evaluate propaganda and persuasion as found in social media. To identify and characterize manipulative persuasion, the context can be stretched arbitrarily far across aspects of epistemology, logic, intent estimates, psychological biases, knowledge of pre-existing narratives, and even physical context. However, to potentially solve this problem in an automated fashion, the prevailing method is to frame the given task as a classification problem. The different propaganda methods are understood as distinguishing criteria. Documents or even subsections of documents are annotated based on these features and thus form the input for training machine learning models, which should then be able to automatically recognize and classify corresponding sections. The undeniable success of this approach for many applications (for example, in NER) is due to the fact that the required features arise directly from the data or are already captured in the data representation used (word embeddings). In fact, current word embeddings already contain representations of a wide range of syntactic and morphological features that can be used to solve many problems. In the following pages, we discuss whether and to what extent the required characteristics are reflected in the training data. In particular, we consider whether and to what extent linguistic structures can be used as a decision criterion. In explaining our findings, a pragmatic perspective is adopted. In general, descriptive, analytical, and linguistic approaches such as speech act theory and
794
rhetoric (or the use of specific rhetorical devices) are used to characterize public (political) discourse. Referring to the speech act theory (Austin, 1975), linguistic features, also in their form as rhetorical features, are assumed to be identified in the locutionary act. The illocutionary and perlocutionary acts, meanwhile, involve more complex information that might be used in feature engineering, thus incorporating the dimension of discourse.
Exploratory data analysis The training data as input of this task was provided by 446 news and web articles in plain text format with 7201 annotations. Labels were given on paragraph level as one or more of those categories depicted in Figure 1. Imbalance in data can exert a major impact on the value and meaning of accuracy and other well-known performance metrics of an analytical model. Figure 1 depicts a clear skew towards three categories that account for twothirds of the total annotations: Loaded_Language, Name_Calling-Labeling, and Repetition. Coincidentally, these categories are characterized by a very short annotation length, with a median of 20 characters (cf. Figure 2). Figure 3 on POS tag distribution shows the relevance of syntactic features in the annotated data, e.g. nouns and verbs seem to be most prominently represented within the annotation spans. Figure 4 further describes the relevance of certain POS tags for a specific annotation. For example, while the distribution for Loaded_Language and Repetition is largely uniform across the identified POS tags, Name_CallingLabeling is characterized by the absence of verbs.

heading: 3 System overview
text: In this study, we evaluate and compare two different approaches: sequence classification and token classification. The comparison is performed at the
level of trained models on the same set of data. The different scoring paradigms arise from applying token classifier and sequence classier heads, respectively, on a pre-trained model as the base model. We suggest that contextual information is captured and processed differently in both approaches, leading to a qualitative difference in the scores. Our results show that sequence classification is superior.

heading: 3.1 Pre-trained language representation
text: At the core of each solution of the given task lies a pre-trained language model derived from BERT (Devlin et al., 2018). BERT stands for Bidirectional Encoder Representations from Transformers. It is based on the Transformer model architectures introduced by Vaswani et al. (2017). The general approach consists of two stages. First, BERT is pretrained on vast amounts of text, with an unsupervised objective of masked language modeling and next-sentence prediction. Second, this pre-trained network is then fine-tuned on task-specific, labeled data. The Transformer architecture is composed of two parts, an encoder and a decoder, for each of the two stages. The encoder used in BERT is an attention-based architecture for NLP. It works by performing a small, constant number of steps. In each step, it applies an attention mechanism to understand relationships between all words in a sentence, regardless of their respective position. By pre-training language representations, the encoder yields models that can either be used to extract high quality language features from text data, or fine-tune these models on specific NLP tasks (classification, entity recognition, question answering, etc.). We rely on RoBERTa (Liu et al., 2019), a pretrained encoder model, which builds on BERT’s language masking strategy. However, it modifies key hyper-parameters in BERT such as removing BERT’s next-sentence pre-training objective, and training with much larger mini-batches and learning rates. Furthermore, RoBERTa was also trained on an order of magnitude more data than BERT, for a longer amount of time. This allows RoBERTa representations to generalize even better to downstream tasks compared to BERT.

heading: 3.2 Multi-Label Sequence Classification Problem
text: Model Architecture The task is given as a multilabel classification problem. The models for the experimental setup were based on RoBERTa. For
the classification task, fine-tuning is initially performed using RobertaForSequenceClassification (Wolf et al., 2020)—RoBERTaLARGE—as the pre-trained model. RobertaForSequenceClassification optimizes for a regression loss (Binary CrossEntropy Loss) using an AdamW optimizer with an initial learning rate set to 2e-5.

heading: 3.3 Multi-Label Token Classification Problem
text: Tagging format We transformed the initial span markup into the IOB tagging format (Inside, Outside, Begin). As we have 19 possible entity classes, each token can be assigned one of the 39 tags given by an O-tag, and the I-tag and B-tag of the various techniques, respectively.
Model Architecture We fine-tuned a RoBERTa model to predict the above IOB tags for each token in the input sentence. In the default configuration each token is classified independently of the surrounding tokens. Although the surrounding tokens are taken into account in the contextualized embeddings, there is no modeling of the dependency between the predicted labels: for example, an I tag logically cannot follow an O tag. Since RoBERTa does not model the dependencies between the predicted tokens, we further added a linear-chain Conditional Random Field (CRF) model (Lafferty et al., 2001) as an additional layer, in order to model the dependency between the predicted labels of individual tokens. Since the sequence of an O-tag following an I-tag does not appear in the training set, it assigns by observation a very low probability to the transition from an O-tag to an I-tag. The CRF re-
ceives the logits for each input token, and makes a prediction for the entire input sequence, taking into account the dependencies between the labels, similarly to Lample et al. (2016). Note that RoBERTa works with byte pair encoding (BPE) units, while for the CRF itis necessary to work with complete words. Thus, only head tokens were used as input to the CRF, and any word continuation tokens were omitted.

heading: 4 Experimental setup
text: In both cases, fine-tuning was done with an NVIDIA TESLA V100 GPU using the Pytorch (Paszke et al., 2019) framework with a vocabulary size of 50265 and an input size of 512. The model was trained to optimize the objective for 50 epochs.

heading: 5 Results
text: We participated in the persuasion techniques detection task and focused on the English dataset. Official evaluation results on the test set are presented in Table 1. During the training phase, we focused on finding the best combinations of deep learning methods and optimized the corresponding hyperparameter settings. Finetuning pre-trained language models like RoBERTa on downstream tasks has become ubiquitous in NLP research and applied NLP. Even without extensive pre-processing of the training data, we already achieve competitive results. The resulting models serve as strong baselines, which, when fine-tuned, significantly outperform models trained from scratch.
Our submission is based on the trained model af-
ter 100 training epochs in the case of the multi-label sequence classification. We were able to improve the official F1-micro score of 0.30761 to 0.34280 by evaluating the intermediate checkpoints. Based on this difference, we can assume a significant overfitting in our resulting model. With a maximum F1-micro score of 0.31325 in the case of the token classification approach, results remained in all cases below those of the sequence classification approach. Compared to the sequence-level assignment, it seems that token-level assignment requires an increased discriminatory power between the individual categories, which is clearly not sufficiently satisfied in this case.
When improving on the pretrained baseline models, class imbalance appears to be a primary challenge. With a highest ranked F1-micro score of 0.37562, it is necessary to discuss other causes for the low discriminatory power.
Possible challenges related to neural architectures arise either from under-specification of the objective function or from general difficulties of feature engineering. Difficulties with the objective function arise when the target variables, in our case the individual persuasion techniques, conceptually cannot be well separated. Issues with feature engineering are to be expected when required features
cannot be captured from the training data. Tenney et al. (2019) suggest that transformer-based networks are able to glean structural information–both syntactic and semantic–from language. If this is so, we expect that further important features may be hidden in the broader context, especially when it comes to manipulative communication. Since these features do not emerge from the training data, they must be made available to the training process in some other way. Features of interest may be derived from research in pragmatics.

heading: 6 Conclusion
text: The use of neural architectures in the field of pragmatics remains largely unexplored. The results of the given task demonstrate the limitations of this method. In the future, we would like to extend the current approach to features of the extended communicative context. Our research concerns the specification of a consistent objective function aligned with the discursive context of manipulative communication. We hypothesize that the target variables of this function in the form of different discourse elements will respond to different features of the given communicative context. If the required features cannot be derived from the linguistic structure of the utterances, they have to be obtained from the extended context of the communication. We are investigating ways to make external features available to the training process. In order to identify pragmatic features and how to exploit them, XAI methods might come to help.

heading: A Classification reports

heading: Future Work
text: In the future, we would like to extend the current approach to features of the extended communicative context. Our research concerns the specification of a consistent objective function aligned with the discursive context of manipulative communication. We hypothesize that the target variables of this function in the form of different discourse elements will respond to different features of the given communicative context. If the required features cannot be derived from the linguistic structure of the utterances, they have to be obtained from the extended context of the communication. We are investigating ways to make external features available to the training process. In order to identify pragmatic features and how to exploit them, XAI methods might come to help