SlowGuess/NAACL_2022_OCR huggingface dataset:
Prashanth Vijayaraghavan  
IBM Research  
Almaden Lab  
San Jose, CA  
prashanthv@ibm.com
Soroush Vosoughi  
Dartmouth College  
Department of Computer Science  
Hanover, NH  
soroush@dartmouth.edu
Recently, several studies on propaganda detection have involved document and fragment-level analyses of news articles. However, there are significant data and modeling challenges dealing with fine-grained detection of propaganda on social media. In this work, we present TWEETSPIN, a dataset containing tweets that are weakly annotated with different fine-grained propaganda techniques, and propose a neural approach to detect and categorize propaganda tweets across those fine-grained categories. These categories include specific rhetorical and psychological techniques, ranging from leveraging emotions to using logical fallacies. Our model relies on multi-view representations of the input tweet data to (a) extract different aspects of the input text including the context, entities, their relationships, and external knowledge; (b) model their mutual interplay; and (c) effectively speed up the learning process by requiring fewer training examples. Our method allows for representation enrichment leading to better detection and categorization of propaganda on social media. We verify the effectiveness of our proposed method on TWEETSPIN and further probe how the implicit relations between the views impact the performance. Our experiments show that our model is able to outperform several benchmark methods and transfer the knowledge to relatively low-resource news domains.
Propaganda refers to any idea or information, that is often false or exaggerated, and is used to promote or publicize a particular cause or point of view. In recent years, there has been a surge in research and development of methods to detect propaganda from text. For example, some of the earlier works like (Rashkin et al., 2017a) and (Barrón-Cedeno et al., 2019) released a corpus of news articles containing coarse-grained document-level annotation of propaganda. Da San Martino et al. (2019b) described
a corpus of news articles containing annotations of 18 fine-grained propaganda techniques. Following this work, two subtasks were presented as a part of NLP4IF workshop (Da San Martino et al., 2019a) that focused on the identification of propagandist text units at fragment and sentence level.
We believe that the challenge for this task lies in the varied nature of propaganda techniques, including cognitive and information distortion and logical fallacies. The challenges are further exacerbated in the social media setting, which has become a key battleground in the spread of propaganda. Research on propaganda detection in social media platforms like Twitter has been limited by the: (a) lack of sufficiently annotated social media propaganda data, (b) idiosyncratic nature of the content on social media, (c) difficulty in modeling the social context in which propaganda is disseminated, and (d) varying propaganda techniques that require factual knowledge, structural relationships, and reasoning abilities.
In this work, we address a subset of both the data and modeling challenges. First, we introduce TWEETSPIN, a corpus of tweets containing weak labels of fine-grained propaganda techniques. We accomplish this through a data collection pipeline that incorporates keyword-based search and users calling out propaganda techniques publicly on Twitter (see Figure 1 for examples). Next, we present a transformer-based multi-view propaganda detection model, Mv-PROP, that identifies varied aspects of the textual data using multi-view contextual embeddings and captures their interaction via pairwise cross-view transformers. The main contributions of this work are described as follows:
(1) Creation of the TweETSPIN corpus containing weak annotations of fine-grained propaganda techniques for tweets.
(2) An end-to-end Transformer-based MV-PROP model augmented with multiple views that infuse context, relational information and external knowl
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3433 - 3448 July 10-15, 2022 ©2022 Association for Computational Linguistics
This is what's called the bandwagon fallacy. Just because a lot of people believe something, that doesn't make it true.
Whataboutism again?"ABC" is a fraud.No citizen of any party should vote for him for anything again. That's all you had to say.
Figure 1: Data from TWEETSPIN where users are being called out for using specific propaganda techniques.
edge into the representation and capture their pairwise interactions through cross-view transformers. To the best of our knowledge, this is the first study that incorporates such multi-view representations for the propaganda detection task.
(3) We conduct experiments using TwEETSPIN to demonstrate the capability of our model in detecting fine-grained propaganda techniques for tweets. We also demonstrate the transferability of our model to data from the news domain.
This work is closely related to a broad spectrum of topics, including offensive language detection, computational argumentation, and fake news detection. Some of the propaganda techniques might overlap with the categories of argumentation techniques (e.g., strawman argument) or offensive language (e.g., name-calling). However, there are considerable variations in propaganda techniques across different social contexts. Thus, methods used for the detection of offensive language or argumentation techniques may not be directly transferable to fine-grained propaganda detection. In this section, we review the prior researches in two main areas that are relevant to our work - (a) propaganda analyses and detection and (b) multi-view representation learning.
Propaganda Analyses and Detection Research in propaganda detection has primarily focused on document-level analysis. A work by Rashkin et al. (Rashkin et al., 2017b) constructed a corpus of news stories from eight different sources and labeled them with four broad categories – propaganda, hoax, trusted, and satire. Barron et al. (Barrón-Cedeno et al., 2019) addressed some of the limitations of the previous work by obtaining more data using distant supervision. They explored a range of features such as keywords, rich representations, writing styles, and readability level to discern propaganda text from other forms of news
stories by casting it as a binary classification problem. More recently, there has been a dedicated line of research aimed at identifying fine-grained propaganda methods at the fragment level (Da San Martino et al., 2019b,a). This involved a manually annotated corpus flagging specific text spans in news articles as containing one of 18 propaganda techniques beyond the binarized setting used earlier. Various text classification models (Da San Martino et al., 2019a; Alhindi et al., 2019) incorporating TF-IDF features (Li et al., 2019) and contextual representations (e.g. BERT (Zellers et al., 2019), RoBERTa and ELMo (Cruz et al., 2019)) have been proposed to handle this fragment-level task. To detect propaganda on social media, many studies (Williamson III and Scrofani, 2019; Caldarelli et al., 2020) utilize datasets that contain propaganda content spread on social media by Russian-based IRA (Farkas and Bastos, 2018; Miller, 2019) or extremists (Johnston and Weiss, 2017; Nizzoli et al., 2019). Subsequent works investigated the influence of propaganda on public opinion (Caldarelli et al., 2020) and the techniques applied to disseminate targeted political agenda (Gorrell et al., 2019). Another recent work (Wang et al., 2020b) leveraged cross-domain learning approach to label propagandistic content. Wang et al. (2020b) implement different classifiers using different informative features and constraints based on labeled documents and sentences from news and tweets to detect propaganda within and across domains. Most of these works either apply bot or troll detection techniques or apply feature engineering methods to conduct binary classification of propagandistic content.
Multi-View Representation Learning Multi-view representation learning has numerous applications involving images, texts, graphs or videos. A line of work in computer vision has extensively studied the benefits of multi-view representation in embedding social images (Gong et al., 2014), object detection (Chen et al., 2017), viewpoint classification (Su et al., 2009), shape/face recognition (Chen et al., 2017; Su et al., 2015; Li et al., 2016), to list a few. However, there has been a limited exploration of multi-view representation explicitly for texts. For instance, the widely used Seq2Seq with attention module (Bahdanau et al., 2014), used in several state-of-the-art NLP tasks, can be seen from the perspective of multi-view fusion where information from different time-steps are fused and encoded together into a semantic representation.
More recently, a work by (Bian et al., 2020) focused on learning to match a resume with a relevant job using multi-view representation learning approach.
Unlike previous studies that either focused on news articles or individual social media posts, our work performs a fine-grained analysis of propaganda for tweets considering the importance of the discussion context. Furthermore, we collect a weakly-annotated corpus of tweets associated with different propaganda techniques and develop a multi-view learning approach that has not yet to the best of our knowledge been explored for propaganda detection.
Given an input tweet text along with a discussion context, our goal is to predict if the input text exhibits any propaganda techniques. Determining the propaganda technique of the input tweet can be formulated as a multi-class classification problem. In this work, we denote the input tweet text as  $T_{i} = [w_{1}, w_{2}, \dots, w_{N_{i}}]$ , where  $N_{i}$  is the sequence length of the input text. Each tweet text may be accompanied by a context  $C_{i} = [T_{1}, T_{2}, \dots, T_{i-1}]$ , referring to  $M$  prior tweets in the discussion thread related to the input tweet  $T_{i}$ . Every input text might not necessarily contain a context (in such a case,  $M = 0$ ). In this task, the target is to develop a model that can learn a mapping function  $f: T \mapsto p_{k}$ , where  $k \in \{1, 2, \dots, L\}$ ,  $p_{k} \in P$  indicates one of the  $L$  propaganda labels and  $p_{1}$  denotes the special case of non-propaganda category or absence of any propaganda technique.
4 TWEETSPIN
We construct a dataset of English-language tweets, referred to as the TweetSPIN corpus, containing weak annotations of 18 propaganda techniques same as in (Da San Martino et al., 2019b). Table B1 shows the full statistics of our dataset.
Our data collection pipeline consists of three components: (a) propaganda keyword expansion, (b) keyword-based tweet retrieval, (c) tweet filtration and (d) data augmentation.
Propaganda Keyword Expansion: For each propaganda technique, we select the name of the technique as the initial keyword. This works well for techniques that cannot be easily characterized by specific lexical patterns. Typical examples of such techniques include red herring, obfuscation, causal
oversimplification, and strawmen. Additionally, we expand the list of keywords related to some of the propaganda techniques by combining information from publicly available resources including technique-specific phrases, idioms, and examples listed in Table B2.
Keyword-based Tweet Retrieval: We use Twitter's standard search API to ingest tweets based on the keywords identified from the previous step. We enclose the keywords or phrases within quotation marks. We observed that many Twitter users explicitly call out any usage of specific propaganda techniques in their discussions on the platform. Therefore, we select quoted tweets and replies containing keywords related to the propaganda techniques. Additionally, we search for tweets containing both the word "you" along with the identified keywords to capture instances where the users call out the usage of propaganda techniques.
Tweet Filtration: Given the tweets retrieved based on the keywords, we remove tweets that are extremely short (less than 5 words) and those that are replies to tweets from deleted or protected accounts. For the remaining tweets, we collect the discussion thread for each tweet which provide the context for the tweet. Though the tweet threads may involve complex tree structures with different reply branches, we are only interested in the specific branch of the tree that contains the tweet being called out for using specific propaganda techniques. In tweet threads that are long (>50 tweets in a discussion thread), not all tweets in the discussion context might be important for the classification task. Thus, we apply temporal filtering on such discussion threads, where we only retain the source tweet and discussion context that falls in a 7-day window before a particular tweet was being called out as propaganda.
Data Augmentation: We adopt data augmentation strategies to handle potential data scarcity and also address the problem of overfitting. We employ linguistically informed transformations of text to prevent meaning distortion leading to new misclassification errors (Li et al., 2020). Hence, we randomly select  $10\%$  of examples from each class from our training set and perform linguistically informed augmentations using the code from Li et al. (2020).
Manual Validation of TweETSPIN We randomly sampled 1,000 samples from the TweETSPIN corpus containing the tweet to be classified, the prior
context, and anonymized user information indicating if the text in the discussion context is from the same user or a different user. Three MTurk workers annotated each of these samples with one of the 19 propaganda techniques (18 techniques + 1 for nonpropaganda). The definitions of the propaganda techniques were made available to the MTurk workers for reference. The inter-rater reliability as calculated using Fleiss'  $\kappa$  was 0.85, indicating a substantial agreement between the annotators. The agreement between the labels in our corpus and the labels provided by the annotators (through majority agreement) was  $89.3\%$ , indicating the relative high fidelity of our corpus, especially given its reliance on a weak-annotation scheme.
5 MV-PROP: Multi-View Propaganda Detection
Here, we describe our proposed model for fine-grained propaganda detection on social media. Unlike the fragment-level classification task used for propaganda detection in news articles, we formulate this problem as a text classification task where we aim to map the input tweet text to one of the several propaganda labels conditioning on the discussion thread context if it exists (as explained in Section 3). Figure 2 illustrates the overview of our model architecture. Inspired from the literature in multi-modal learning (Tsai et al., 2019), we propose a transformer-based multi-view propaganda detection model, Mv-PROP, that integrates multiview contextual embeddings via pairwise cross-view transformers. The main motivation behind such a modeling choice comes from the fact that different propaganda techniques require focus on varying aspects of the data. For example, propaganda techniques like loaded language can be identified from the usage of specific words or phrases, while repetition or red herring necessitates a contextual understanding of the tweet. Similarly, most propaganda techniques can benefit from word sense disambiguation and entity information that can be accumulated from external knowledge sources. Thus, the fine-grained differences between these propaganda techniques call for multi-view representations that can unravel such variations in the data. Our MV-PROP model comprises the following components:
Multi-View Encoding Layer, which computes multi-view representation from input tweet  $T_{i}$  and context  $C_{i}$ .
Cross-View Transformer, that reinforces representations obtained from a specific view with those computed from another view. We compute this for all pairs of such cross-view transformers.
Classification Layer, which fuses the embeddings from the previous step and computes the likelihood of the input tweet with the given context belonging to a particular propaganda label.
5.1 Multi-View Encoding Layer
In this work, we compute three different views: (a) context-aware semantic view, which derives semantic representation from text and context by leveraging a pre-trained language model, (b) relationship structure view, that calculates a relational representation by applying relation-based graph neural network on dependency graph and speaker-dependent context graph, and (c) knowledge-enriched view, which enriches the input tweet embedding with different kinds of knowledge. We discuss them in detail in subsequent sections.
5.1.1 Context-Aware Semantic View
The accompanying discussion context of a tweet can significantly shift how the tweet is perceived and hence plays a critical role in determining the propaganda technique used in the given tweet. Therefore, we employ a hierarchical incremental transformer encoder to obtain a context-aware semantic representation of the tweet text. Inspired from some of the existing hierarchical approaches (Zhang et al., 2019; Liu and Lapata, 2019), we implement a two-level transformer encoding process: (a) a tweet encoder  $f_{T}$  that operates at the word-level to transform each tweet into an embedding including those in the discussion thread and (b) a context encoder  $f_{C}$  that enriches the input tweet representation by capturing the influence of the previous tweets in the discussion thread relevant to our classification task. Thus, we learn a context-aware semantic view of the given input tweet.
We utilize a BERT-based pretrained language model (Devlin et al., 2018) as our tweet encoder,  $f_{T}$ . Each tweet  $T_{j} = [w_{1}, w_{2}, \dots, w_{N_{j}}]$  is fed to the wordpiece tokenization algorithm. Here  $N_{j}$  is the number of words in the tweet text. We add special tokens [CLS] and [SEP] at the start and end of the tokenized tweet token list. We feed each tweet text into the BERT model and produce contextual word embedding as:
$$
\hat {H} _ {j} = \left[ \hat {h} _ {j} ^ {1}, \hat {h} _ {j} ^ {2}, \dots , \hat {h} _ {j} ^ {N _ {j}} \right] = f _ {T} (T _ {j}) = B E R T (T _ {j}) \tag {1}
$$
Figure 2: Illustration of our MV-PROP model.
We obtain an overall tweet embedding by performing a maxpool operation on the contextual word embeddings intended primarily to retain the important information in each dimension:
$$
h _ {j} ^ {s} = \operatorname {m a x p o o l} \left(\hat {H} _ {j}\right) \tag {2}
$$
The choice of maxpool was made after considering other options like average pooling and  $[CLS]$  token representation. We found the maxpool worked best for our classification task.
Given an input tweet  $T_{i}$  and its prior context  $\{T_k\}_{k = 1}^{i - 1}$ , we account for the sequence information in the discussion context by performing an elementwise summation of the tweet embedding  $h_j^s$  with the positional embedding  $p_k$ . We compute the position incorporated tweet context embeddings as:
$$
C ^ {(<   i)} = \left[ h _ {1} ^ {s}, h _ {2} ^ {s}, \dots , h _ {i - 1} ^ {s} \right] \odot \left[ p _ {1}, p _ {2} \dots , p _ {i - 1} \right] \tag {3}
$$
Since our goal is to detect the propaganda technique used in the input tweet by conditioning on the prior context, we enrich the input tweet embedding using the information from the computed context embeddings  $C^{(<i)}$ . This is done by feeding the input tweet and the context representations to a context encoder,  $f_{C}$ , comprising  $N_{C}$  transformer encoding layers. However, we introduce an additional context-attention sub-layer in the transformer layer that integrates discussion context into the encoder. This is implemented as:
$$
U ^ {(l)} = \mathrm {M H A T T} (\hat {H} _ {i} ^ {(l - 1)}, \hat {H} _ {i} ^ {(l - 1)}, \hat {H} _ {i} ^ {(l - 1)})
$$
$$
V ^ {(l)} = \underset {(1)} {\mathrm {M H A T T}} \left(U ^ {(l)}, C ^ {(<   i)}, C ^ {(<   i)}\right) \tag {4}
$$
$$
{ } ^ { ( l ) } = F F N ( V ^ { ( l ) } ) \tag {4}
$$
$$
H _ {c a s} = H _ {i} ^ {(N _ {C})}
$$
where  $l$  refers to the  $l^{th}$  context encoding layer,  $l \in 1,2,\dots,N_{C}$ , MHATT and FFN refer to the multi-head attention step and feed-forward network in each transformer encoding layer and  $H_{i}^{1} = \hat{H}_{i}$
$C^{(< t)}$  is the prior discussion context embedding as computed in Equation 3 and  $H_{i}^{(l)}$  is the embedding of the input tweet at the  $l^{th}$  layer. The output from the  $N_{C}$ -th layer is the final context-aware semantic view  $H_{cas}$  of the input tweet  $T_{i}$ .
5.1.2 Relationship Structure View
The goal of this view is to compute a hierarchical relational embedding that captures two main aspects: (a) dependency graph-based structural information from individual tweets and (c) speaker-dependent structural relationships from the discussion context. First, we intuit that a representation that encapsulates the syntactic structures explicitly and learns relationships between specific words and phrases can better guide the propaganda detection model. We explain the reason as follows. Certain words can express an attitude or sentiment towards specific key terms or entities in the sentence. Despite the advantages of using flat attention-based models, the limitations of assigning higher attention scores to irrelevant words or wrong associations can lead to performance degradation. Additionally, we differentiate between the input tweet user's previous tweets and the other users' tweets in the discussion context. We believe that self-dependency (relationship between input tweet user's previous tweets in the context) and interspeaker dependency (relationship between input tweet with other users' tweets in the context) can be critical to understanding the speaker motivation (or intention) or attitudinal/sentiment shifts in the conversation. Therefore, both these aspects require the extraction of some form of structural relationship. Recently, graph neural networks (Scarselli et al., 2008; Schlichtkrull et al., 2018; Kipf and Welling, 2016; Velicković et al., 2017) have been applied to tackle challenges in effective representation of nodes from graph-structured data and have proven effective in a number of NLP applications
such as aspect-level sentiment analysis (Huang and Carley, 2019; Wang et al., 2020a), reading comprehension (Tu et al., 2019; Zhang, 2020; Song et al., 2018) and relation extraction (Fu et al., 2019; Zhang et al., 2018). However, most of the models like graph-convolution networks (GCN) (Kipf and Welling, 2016) or graph attention networks (GAT) (Veličković et al., 2017) operate on homogeneous links or edges. In our work, we introduce a hierarchical relation-based graph neural network that can aggregate incoming information from neighbors depending on the edge type.