Hierarchical Multi-Instance Multi-Label Learning for Detecting Propaganda Techniques
Year: 2023

Abstract:
Since the introduction of the SemEval 2020 Task 11 (Martino et al., 2020a), several approaches have been proposed in the literature for classifying propaganda based on the rhetorical techniques used to influence readers. These methods, however, classify one span at a time, ignoring dependencies from the labels of other spans within the same context. In this paper, we approach propaganda technique classification as a Multi-Instance Multi-Label (MIML) learning problem (Zhou et al., 2012) and propose a simple RoBERTa-based model (Zhuang et al., 2021) for classifying all spans in an article simultaneously. Further, we note that, due to the annotation process where annotators classified the spans by following a decision tree, there is an inherent hierarchical relationship among the different techniques, which existing approaches ignore. We incorporate these hierarchical label dependencies by adding an auxiliary classifier for each node in the decision tree to the training objective and ensembling the predictions from the original and auxiliary classifiers at test time. Overall, our model leads to an absolute improvement of 2.47% micro-F1 over the model from the shared task winning team in a cross-validation setup and is the best performing non-ensemble model on the shared task leaderboard.

heading: 1 Introduction
text: The development of the Web and social media has amplified the scale and effectiveness of propaganda (Barrón-Cedeno et al., 2019). Automatic propaganda detection through text analysis enables social science researchers to analyze its spread at scale (Glowacki et al., 2018; Martino et al., 2020b). Within text analysis, two broad approaches include the identification of rhetorical techniques used to influence readers and document-level propagandistic article classification. The former is more promising because propaganda techniques are easy to identify
and are the very building blocks (Martino et al., 2020b). Hence, it is the focus of this paper.
Recent research on fine-grained propaganda detection has been spurred by the NLP4IF-2019 Shared Task (Da San Martino et al., 2019) and its follow-up SemEval 2020 Task 11 (Martino et al., 2020a). In this paper, we focus on its Technique Classification subtask where, given a news article text and spans identified as propagandistic, systems need to classify each of the spans into one or more of 14 different common propaganda techniques.
All top systems submitted to the task (Jurkiewicz et al., 2020; Chernyavskiy et al., 2020; Morio et al., 2020; Raj et al., 2020) employed a pretrained RoBERTa-based model (Zhuang et al., 2021) which was trained to classify one span at a time. However, labels of different spans within the same article clearly depend on each other. Thus, we approach the task within a Multi-Instance Multi-Label framework (Zhou et al., 2012), where we model each article as an object with multiple instances (spans), each with its own labels. This allows us to model the dependencies between different labels within the same article. We show that this MIMLRoBERTa observes a 1.98% micro-F1 improvement over the replicated ApplicaAI system (referred to as baseline) (Jurkiewicz et al., 2020).
Besides, as a decision tree was used to guide annotations (Martino et al., 2020a), we explore incorporating this hierarchical relationship among the labels into classifiers. To do so, we add 7 more auxiliary classifiers on top of the span representations from RoBERTa, one for each intermediate node in the tree, and train these classifiers to predict the path to a leaf node and hence the corresponding label (see Figure 1). We show that incorporating the label hierarchy in this manner improves both the single-instance and the MIML versions of the RoBERTa model (referred to as hierarchical baseline and hierarchical MIMLRoBERTa respectively).
155

heading: 2 Related Work
text: The MIML framework was first introduced by Zhou et al. (2012) aiming for better representing complicated objects composed of multiple instances, e.g., an image with several bounding boxes each with its own label. Since then, it has been applied to many tasks, such as relation extraction (Surdeanu et al., 2012) and aspect-category sentiment analysis (Li et al., 2020). The latter work uses a Bi-LSTM architecture which aggregates over the words in a sentence (instances) to classify the sentiments of different aspects (labels). MIML has also been applied to BERT-based models in the biomedical text analysis (Tian and Zhang, 2021). To our best knowledge, our work is the first one to apply it for propaganda technique classification.
Given the decision tree used for annotations (reproduced in Figure 4 in Appendix), this task can also be viewed as a hierarchical text classification problem, with mandatory leaf node prediction and a tree-structured hierarchy (Silla and Freitas, 2011). Exploiting hierarchical information has been useful in significantly enhancing the performance of the system for medical image annotations (Dimitrovski et al., 2011) and presents us with an opportunity to apply to the propaganda method detection. Similar to (Dumais and Chen, 2000; Weigend et al., 1999), we use the multiplicative rule to combine probabilities along the different paths in the hierarchy, leading to a distribution across the leaf nodes which can be combined with the distribution predicted by the non-hierarchical baseline. Different approaches to hierarchical classification across multiple domains can be found in Silla and Freitas (2011).
3 Methods1
Task Description. Given an article d with propagandistic spans s1, ..., sm identified by their start and end indices, the task is to identify the sets of techniques y1, ..., ym where yi ⊆ Y ∀i ∈ {1, ...,m} and Y represents the set of 14 techniques. Following the shared task, we assume that the number of labels |yi| for each span is known.

heading: 3.1 Single-Instance Baseline
text: The baseline system from the winning ApplicaAI team (Jurkiewicz et al., 2020) uses a RoBERTabased classifier and applies to each span separately. The span is padded by special tokens <bop> and
1Code is available at https://github.com/ Dranoxgithub/propaganda-nlp-new.
<eop> on either side. A total context of 256 tokens on both sides of the target span spanning multiple sentences are included in the input, unlike other systems (Chernyavskiy et al., 2020; Morio et al., 2020; Raj et al., 2020) on the leaderboard which limit to the target sentence. For classification, we use the <bop> representation output from RoBERTa and pass it through a linear layer followed by softmax. Jurkiewicz et al. (2020) further show marginal improvements by re-weighting the loss for underrepresented classes, self-training over unlabeled data and a Span-CLS approach which adds a second transformer network on top of RoBERTa only over the tokens in the target span. In our experiments, we did not use any of the improvements.

heading: 3.2 MIMLRoBERTa
text: During the initial inspection of data, we observed high absolute pointwise mutual information between certain techniques. For example, in an article, if the slogans technique appears, the flagwaving technique usually follows. This observation motivates us to predict the labels of all spans in a text simultaneously.
During preprocessing, we pad the spans with pairs of <bop0>, <eop0>, <bop1>, <eop1>... to indicate the start and the end of a text fragment, even in cases where some spans are nested inside or overlapping with others. After padding, every article is split into windows of size 512 tokens with a stride of 256 tokens until there are no more labeled spans. Whenever there are spans that need to be truncated as we create a window, its corresponding <eop> is appended near the end of the window to ensure that the number of <bop>s and <eop>s match. In the case where there is a specific nesting structure between any two text fragments, its nesting structure is respected in appending the <eop> to represent the differences in text fragments even after truncation. Also, since only around 1.8% of the total annotations are spans with multiple labels (Martino et al., 2020a), for ease of implementation, we use the rarer label for every text span during preprocessing and the later training.
Let hs denote the RoBERTa output at the <bop> for a span s, then we compute the logits for each of the labels as pflat(c) ∝ exp(hs · wc), for c = 0, . . . , 13. The model is trained using the cross-
entropy loss:
lflat =
C∑
c=1
−1y=c log pflat(c), (1)
where wc is a weight vector and y is the ground truth label of the span. We compute lflat simultaneously for all spans in the text and take an average.
During inference, we predict the number of labels requested for each span by selecting the classes with the highest in pflat. Since some spans can appear in multiple windows, the set of predictions from the window where the span is least truncated and has the most surrounding context is chosen.

heading: 3.3 Hierarchical MIMLRoBERTa
text: Due to the complex and subjective nature of the task, span annotations by Martino et al. (2020a) were guided by a decision tree. For example, annotators first consider whether the argument is rational, followed by whether emotional words are used, and so on. This process induces a hierarchy among the labels, and to model such relationships, we use a simplified version of the decision tree whose leaf nodes are the 14 propaganda techniques (Figure 1, right). We add an auxiliary loss to the training objective based on hierarchical text classification which trains local classifiers at each intermediate node (Silla and Freitas, 2011).
Let K be the number of intermediate nodes and hence the number of auxiliary classifiers, and let C1, C2, . . . , CK denote the number of outgoing edges or the number of labels for each classifier.
Then, given the RoBERTa representation for a span hs, we compute the probability of following an edge i from an intermediate node k as:
pk(i) ∝ exp(hs · wk,i), ∀i = 0, . . . , Ck (2)
where wk,i is a weight vector and the probabilities are normalized across the Ck labels for each classifier. Given a leaf node label c, we denote the path of classifiers and edges from the root to it as Ic = {(k1, i1), (k2, i2), . . .}, where each tuple in the set denotes a pair of classifier and its corresponding label along the path. Then we can compute the overall probability of selecting as:
paux(c) = ∏
(k,i)∈Ic pk(i). (3)
Note that paux forms a valid distribution over the leaf nodes since each probability along the path is normalized.
During training, we compute an auxiliary loss for each span which minimizes the negative loglikelihood of selecting the edges along the path to its correct label y:
laux = 1 |Iy| ∑
(k,i)∈Iy
Ck∑
c=1
−1i=c log pk(i). (4)
λtraining and λeval are both hyperparameters. The overall training loss is a combination of the flat loss described in the previous section and the auxiliary loss above: lovr = (1−λtraining)·lflat+λtraining ·laux.
During inference, we combine the predicted probabilities over the labels from both the flat classifier and auxiliary classifiers: povr(c) = (1 − λeval) · pflat(c) + λeval · paux(c).

heading: 4 Experiments

heading: 4.1 Dataset
text: The original training and development set respectively contains 357, 74 articles, and thus correspondingly 6128, 1063 data points. Following Jurkiewicz et al. (2020), we also evaluate using sixfold cross-validation where the folds are created by mixing the original training and development sets. As a result each fold roughly consists of 6000 training and 1200 evaluation data points. Below we report the average and standard deviation of the metrics across six folds. We also submit our best models, trained on the original training set, to the leaderboard for evaluation on the official test set.

heading: 4.2 Training & Evaluation
text: We tune both λtraining between 0 and 1 with a step of 0.1. For every value of λtraining, we also tune λeval at 0, 1 and λtraining.The model’s performance peaks at λtraining = λeval = 0.5 (Figure 2) and all the results reported use this set of values. Hyperparameter details are included in Appendix B.
The scorer script provided by the shared task organizers evaluates a micro-averaged F1 score, taking the best match between the predictions and ground truth labels when a span has multiple labels (Martino et al., 2020a). We use this script for evaluation in all our experiments. Additionally, we also use a Tree-F1 score from the hierarchical text classification literature which measures the overlap between the paths from the root node to the
ground truth and predicted nodes (Kosmopoulos et al., 2015) (details in Appendix C). In particular, we are interested in seeing if models with hierarchical information make mistakes closer to the ground truth in the tree.

heading: 5 Results and Discussion
text: Table 1 shows the micro-averaged F1 scores of the different models discussed in Section 3. We see a significant improvement when using the MIML framework: in cross validation, MIMLRoBERTa has a micro-F1 of 72.10%, an absolute improvement of 1.98% compared to the baseline single-instance model (70.12%). This improvement holds whether we use the hierarchical loss or not – the hierarchical MIMLRoBERTa has a 2.1% improvement over the hierarchical baseline (70.49%).
We also observe small but consistent improvements when training and predicting with hierarchical information in the form of auxiliary classifiers: the baseline model improves by 0.37%, while MIMLRoBERTa improves by 0.49%. Table 2 shows the Tree-F1 scores over the full validation splits, as well as only for the incorrect predictions from the various models. Again, we observe a small improvement due to incorporating the hierarchical information – specifically, the incorrect predictions
are now closer to the ground truth labels, as evidenced by the higher Tree-F1 over the mistakes.
To further confirm that these improvements are due to learning hierarchical information rather than any regularization effect from the additional auxiliary loss term, we also run the experiments with the labels on the decision tree randomly shuffled with the same lambda values. We obtain a micro-F1 of 69.54 with λtraining = λeval = 0.5, which is significantly lower. Particularly, when λtraining = 0.5 and λeval = 1, i.e. inference is done using only the auxiliary classifiers, the micro-F1 in cross validation is an extremely low 5.25%, in contrast to a 72.33% when not shuffled.
https://propaganda.qcri.org/ptc/ leaderboard.php
The micro-F1 results on the official test set are in Table 3, where we see that the hierarchical MIMLRoBERTa is the best performing single model. Other than improving accuracy, MIMLRoBERTa also reduces the training and evaluation time, since it predicts multiple spans in a single forward pass through the RoBERTa model. One epoch of MIMLRoBERTa (one evaluation every 25 steps) takes 5.04 minutes compared to 15.50 minutes by the single-instance baseline, a 68% reduction.

heading: 6 Conclusion
text: We propose two simple extensions to a RoBERTabased model for propaganda technique classification, which lead to notable improvements. Our approach to incorporating hierarchical information about the labels into training could also be useful for other tasks where the annotation procedure involves making a series of decisions about instances. Future work can also explore other methods to incorporate the hierarchical information, e.g., via regularizing the label embeddings.
Limitations
The use of auxiliary classifiers at every node of the decision tree is not feasible when the hierarchical tree is huge, such as the large hierarchical terminologies for medical literature indexing (Gasco et al., 2021).
Besides, in Table 1, even though the integration of the hierarchical information shows a consistent improvement in both the baseline and MIMLRoBERTa models, these improvements are still within one standard deviation of micro-F1.
Lastly, it is worth noting that we do not focus on large language models since our approach is to explore improvements on a published state-of-theart model. While they might improve accuracy, a careful exploration of those on a new task is beyond the scope.
Ethics Statement
Propaganda detection is a sensitive topic and any practical application of the model needs to be carefully orchestrated. Both false positives and false negatives of the model can have harmful impacts. Moreover, there might be certain biases in the training data, and consequently this leads to systematic issues in the model, such as a higher tendency to mislabel certain kinds of text.
Furthermore, this paper follows the same definition as SemEval 2020 Task 11 (Martino et al., 2020a) whereas there might a broader debate on the definition of propaganda.
Lastly, we also acknowledge the concern that a perfect classifier for propaganda text can be used to train a language model that generates propaganda which in turn evades the classifier’s detection.

heading: A Techniques
text: The hierarchical diagram to guide the annotation of propaganda techniques can be found in Figure 4.

heading: B Hyperparameters
text: We use HuggingFace’s library (Wolf et al., 2019) and a single Nvidia RTX A6000 GPU. Both the baseline and the hierarchical baseline methods use a learning rate of 2e − 5, a dropout of 0 and a batch size of 16, while the MIMLRoBERTa and its hierarchical version use a learning rate of 1e− 5, a dropout of 0.1 and a batch size of 8. All the models are trained for 20 epochs. More details can be found in Table 4.

heading: C Tree-F1 Metric
text: Let Y be the ground truth label and Y ′ be the prediction, and let K be the lowest common ancestor between the two. Then the Tree-F1 is given by:
PrecisionTree = LK/LY ′
RecallTree = LK/LY
Tree-F1 = 2 ∗ PrecisionTree ∗ RecallTree
PrecisionTree + RecallTree where LM refers to the number of nodes tracing
from the root node down to node M .

heading: Future Work
text: Future work can also explore other methods to incorporate the hierarchical information, e. g. , via regularizing the label embeddings. Limitations
The use of auxiliary classifiers at every node of the decision tree is not feasible when the hierarchical tree is huge, such as the large hierarchical terminologies for medical literature indexing (Gasco et al. , 2021). Besides, in Table 1, even though the integration of the hierarchical information shows a consistent improvement in both the baseline and MIMLRoBERTa models, these improvements are still within one standard deviation of micro-F1. Lastly, it is worth noting that we do not focus on large language models since our approach is to explore improvements on a published state-of-theart model. While they might improve accuracy, a careful exploration of those on a new task is beyond the scope