Kb at SemEval-2023 Task 3: On Multitask Hierarchical BERT Base Neural Network for Multi-label Persuasion Techniques Detection
Year: 2023

Abstract:
This paper presents a solution for Semeval 2023 subtask3 of task3: persuasion techniques in paragraphs detection. The aim of this task is to identify all persuasion techniques in each paragraph of given news articles. We use hierarchical multitask neural networks combined with transformers. Span detection is an auxiliary task that helps in the main task: identify propaganda techniques. Our experiments show that if we change the index of BERT embedding from the first token of the whole input to the first token of the identified span it can improve the performance. Span and label detection can be performed using one network so we save data and, when data is limited, we can use more of it for training.

heading: 1 Introduction
text: News articles are expected to present reliable information, however quite often contain some kind of manipulation. Unconscious reader can be unable to spot all kind of persuasion that he is exposed to. Reading such news can influence a reader’s point of view especially if the reader has a low level of political knowledge (Eberl et al., 2017). Automatic tools for detection of persuasion can help in media analysis and creating more objective news.
The task prepared by the organisers (Piskorski et al., 2023) covers three topics, as follows: category of news article identification, framing classification and paragraph-level persuasion techniques recognition. Our team focused on the third topic and created solution tested on all languages that have training data available: English, French, German, Italian, Polish and Russian and for additional languages with only test data: Spanish, Greek and Georgian.
Our system combines multitask learning and hierarchical neural networks. The system identifies span where the persuasion techniques appear and then passes this information to the next module that
classifies it to one or more categories of persuasion techniques. We used pre-trained and fine-tuned Bert embedding as the shared input layer.
We discover that adding an auxiliary task such as span identification as the first step of training neural network may give better results than simply classifying the whole paragraph. Moreover, using one neural network for that approach makes solving the task possible even for limited data.
The code of our solution is available at https://github.com/Katarzynaa/ persuasion_detection

heading: 2 Background
text: The persuasion technique identification problem has been addressed before as a problem of span and technique classification. There exist several works on solving the persuasion techniques detection problem, for example: as a problem of multimodal image and text input (Dimitrov et al., 2021) or as problem of span and technique classification (Da San Martino et al., 2020). In our case it is the multi-label classification task, the authors of the task are expected to classify all persuasion techniques that appear in every paragraph of a news article. The input data is a single paragraph, the output is a list of detected techniques. The data is provided in three folders: “train”, “dev” and “test”. Each folder contains files with articles. Each article is divided into paragraphs separated by an additional empty line. For each paragraph of each article in “train” and “dev” sets organisers provide the lists of labels of persuasion techniques in separate files. Labels of test data are unknown. As an additional information authors provide for each paragraph the list of spans, start and end character index, and its class.
There are 23 classes with a distribution that is not balanced (far from the uniform one). These classes are presented in table 1 One can find a detailed description in the task description paper.
1395

heading: 2.1 Multitaskand and hierarchical Networks for persuasion techniques identification
text: Our solution is based on multitask networks. This kind of network shares the same part or full architecture to solve several tasks being trained at once. Much research evidence demonstrates that proper choosing of auxiliary task may help to get better results at the main one (e.g. (Bjerva, 2017)).
Multi-task learning is mostly used for emotion and sentiment analysis (Zhang et al., 2022), aspect based sentiment analysis (He et al., 2019) or named entity recognition, etc.
Hierarchical networks are formed as an acyclic graph, which means that the tasks are learned by the networks’ modules in some order. The results of previous modules influence the next modules. Hierarchical multitask approach has many forms and applications in NLP, for example: embedding learning (Sanh et al., 2019) or aspect-based sentiment analysis (Wang et al., 2021), etc. In our solution we created a network that solves two tasks: span identification and persuasion techniques identification. Both share the same input of Bert embedding but have separate layers for the classification part. The second one uses the results of the first one. Span identification can be treated as sequence tag classification. We predict the span where the persuasion technique is present. For the second task we used the predicted span to identify the technique used in that span. If more than one span is identified we predicted techniques based on the first index. One span can represent many persuasion techniques.
In contrast to the network presented in (Ju-
rkiewicz et al., 2020) our main aim is only persuasion technique detection, we do not use special tokens and we use one multitask network.
We also experimented with other modifications or variants of network architectures. For example, we tried to add auxiliary task of POS tagging, adding general sentiment based on “Vader” approach (Hutto and Gilbert, 2014), add entity-level sentiment trained on the SEN benchmark dataset (Baraniak and Sydow, 2021) however no improvement was observed so details are not included.
In our preliminary approach we tried to translate articles between languages to get more data but no improvement was observed. That can be caused by the fact that specific persuasion techniques for a given language are not easy to translate to other language or the fact that a basic translator omits persuasion techniques and translates it in more neutral words.

heading: 3 System overview
text: Our system proceeds in the following general steps: 1. Use pre-trained model to continue pretraining using masked language model task on provided news articles data from all subtasks
2. Train the multitask hierarchical model on specific tasks: span detection and multilabel classification
3. Evaluate model on the devset 4. Train model on joint trainset and devset For the step 1 we used standard code provided by huggingface1. Step 2 uses our implementation of a multitask hierarchical BERT based network that is described in the next subsection.
One approach is used for all languages that contain train and dev set but models are trained separately.
For languages that do not have train and dev set we translate the articles from Polish to the required language. For these languages we use simple BERT model for sequence classification. We do not use our model as we believe it is hard to find exact spans after translation.

heading: 3.1 Model architecture
text: General schema of the model architecture is presented in Figure 1.
The first layer of a model is a BERT layer. It takes tokenized input paragraph and calculates embeddings. Next we add dropout layer. Then it is
1https://huggingface.co/
followed by the first linear layer responsible for tag classification for span identification. We use softmax as activation function of this layer. It classifies each token as I or O.
We take the index of the first token of a span and we use it to find the BERT embedding from this token from the previous layer. Then this embedding is passed to the second linear layer. If no span is identified the layer takes the first token of BERT embedding. The second linear layer is followed by sigmoidal activation function. It has as many outputs as the number of classes and determines whether the sample belongs to each class or not (it can activate for any class).
Loss function for our model is a sum of two:
Loss = 0.5 ∗ Loss1 + Loss2 (1) The Loss1 is cross entropy that calculates the loss for span identification and is used without any modification.
Loss1n = −[ynlogxn + (1− yn) ∗ log(1− xn)] (2) where yn is the correct answer and xn is the prediction for the nth batch. The Loss2 is used for a multi-label classification and is binary cross entropy calculated for each label with added weight pc:
Loss2n,c = −[pcynlogxn+(1−yn)∗log(1−xn)] (3)
We add weight pc because of unbalanced number of classes. Weight of a class c is a proportion
of samples from other classes to the number of samples from class c calculated as follows:
pc = nall − nc
nc (4)
where nall is the number of all samples in the dataset and nc of the current class.

heading: 4 Experimental Setup
text: For final evaluation we use train file for training and test file for evaluation. After choosing the right parameters we train the whole model on both datasets.
We used random search for hyper-parameter tuning. For final model we used hyper-parameters as follows. Maximum length of a sequence is 256, as it speeds up the computation and we did not see much improvement when using longer sequences. Train batch size is 8 as it is mostly limited by the machines that were available. Maximum number of epochs is 30 and we saved the best model on dev set according to the micro-f1 metric. Learning rate was set to 1e-05 and max gradient norm was set to 10. Dropout before both linear layers is set to 0.1. The network is trained using AdamW optimization algorithm with weight decay equal to 0.01.

heading: 4.1 Data preprocessing
text: Spans are provided as a list of starting and ending indexes of characters. We converted them to list of I and O tags where each tag corresponds to one word. Tag O means that the word does not contain persuasion and I means that word belongs to a persuasion span. First we identify which parts are between start and end index and then we split into words using the “spacy” library2.
When preparing the input data to the BERT model, transformers have their own tokenizer and it may split the words into smaller tokens. In such case each token has the same tag as the original word.
All labels were provided as a list. We convert them to multi-hot encoding vectors. Pre-trained models from table 2 are used in our retraining . All of them come from the huggingface3. We use all sets from all the three tasks to continue pre-training our BERT model for a particular language. We use masked language model for that.
Models were evaluated by micro− F1 in a first place and then macro− F1.
2https://spacy.io/ 3https://huggingface.co/

heading: 5 Results
text: The results on the official test set are presented in table 4. Our system for all languages performs better than the baseline. Interestingly, models that have no train and dev set such as Spanish, Greek or Georgian still perform better than baseline but much worse than other languages. That means we can achieve better performance using translation but it is never good enough. The reason could be that translation tries to preserve the meaning but may skip or change the type of persuasion. Also the techniques for different languages may vary.
All models on the dev set perform much better than the baseline (Figure 3). According to micro-f1, the results for 3 random runs of model are stable, the standard deviation is rather low. According to macro-f1, the standard deviation is slightly higher, that means one can observe differences between dev set results and test set results. The reason may be that the data for test set may come from different topics and time. The system learns the techniques that are more specific for train and dev set.

heading: 5.1 Error analysis
text: We analysed which classes in the English devset are the easiest/hardest to be recognized (Table 5). The most frequent class ("loaded language") gets a high f-measure value, but the highest one is achieved by the “Guilt by Association” class which has only 4 and 59 observations in the test and train sets, respectively, which means it is probably the easiest class to detect. Classes "Loaded Language" and "Name-Calling Labeling" get high recall but lower precision, which means they are often wrongly detected. "False Dillema-No Choice" gets much higher precision than recall which means it is precisely recognized. There are a few classes that are not recognized at all and most of them have the low number of samples.
Some classes may require another approach, like adding broader context. For example "Red Herring" is when someone introduces irrelevant information, what may be hard to detect based on a single sentence like He died there. what was classified wrongly as "Repetition" or "Melania paired the mid-length half price frock with Christian Loubotin heels" what was classified as "Loaded language" and "Name-Calling-Labelling". Both cases are hard to be recognized without the context. For example, the second is suited to an article about fashion but not about politics.
The system was wrong also about the "Conversation Killer" technique which is often a short and rather obvious statement: "Everybody knows it." or hidden in some long paragraph "How about sorting that stuff out instead of politicizing something that should be fun for everyone? How many times does it have to be said".
We noticed that sometimes a broader context not only from the article but also from the world of politics is necessary to correctly recognise a technique. For example, the paragraph containing "Appeal to Hypocrisy" But he didn’t mention Mueller for the rest of the day. and Of course, Sir Kim would have had plenty of targets had he decided to pass judgement on the present incumbent of the White House. are not easy to be classified based only on what is given in the sentence.
Only 129 paragraphs were predicted correctly
from the English devset (all true labels are correctly recognised and no other labels). Most of them have one or two labels.

heading: 6 Conclusion
text: We discovered that simple change of the index in Bert embedding may help to improve the persuasion classification. Moreover, we are able to identify spans and perform classification on limited data using the described networks. Our system works better than classic BERT for sequence classification but still needs some improvements.
The next step would be to check another language models, that may be more powerful for this task. One could improve so that all the detected spans are included in classification. If more data is available, we can use separate network for spans and for classification. As a future work we should also extend the error analysis and find an approach
for finding classes that are poorly recognized. We believe that persuasion techniques are strongly diverse and hard to be detected by one model. We think that trying ensemble models could improve the results.

heading: Future Work
text: As a future work we should also extend the error analysis and find an approach
for finding classes that are poorly recognized. We believe that persuasion techniques are strongly diverse and hard to be detected by one model. We think that trying ensemble models could improve the results