SlowGuess/NAACL_2022_OCR huggingface dataset:
First, we perform coreference resolution<sup>1</sup> given the discussion context tweets and replace the mentions with proper entity information. Next, We apply a dependency parser from (Kong et al., 2014; Liu et al., 2018) to transform a tweet text into a dependency parse graph  $\mathcal{G}$ . This graph is denoted as  $\mathcal{G} = (\mathcal{V}, \mathcal{E}, \mathcal{R})$  where a node refers to a word in the tweet text  $(v_k \in V)$  and a labeled edge indicating a dependency relation between two words  $(v_k, r, v_m) \in \mathcal{E}$ , where  $r \in \mathcal{R}$  is a relation type. Each tweet  $T_j$  is converted into a graph  $\mathcal{G}_j$ . Since the traditional GAT model fails to consider different relation types into consideration, there is a significant loss of crucial dependency information. Following some of the prior work on relation-based graphical propagation of information (Busbridge et al., 2019; Velicković et al., 2017; Ishiwatari et al., 2020), we aggregate embeddings of the  $k^{th}$  node,  $\bar{h}_k^{(l)}$ , using varied relation-specific influences of the relation-specific neighborhood nodes  $\mathcal{N}_k^r$  computed using an attention mechanism. Stacking  $N_S$  layers allows information from nodes  $N_S$ -hops away to propagate to a particular node. Therefore, we implement a relational graph attention network (R-GAT) that intuitively aggregates the incoming information from different relations with varying influences. These steps are defined by:
$$
\bar {h} _ {k} ^ {(l)} = \sum_ {r = 1} ^ {R} \bar {h} _ {k r} ^ {(l - 1)}
$$
$$
\bar {h} _ {k r} ^ {(l - 1)} = \sum_ {m \in \mathcal {N} _ {k} ^ {r}} \beta_ {k j} ^ {r (l - 1)} W _ {r} ^ {(l - 1)} \bar {h} _ {m} ^ {(l - 1)} \tag {5}
$$
$$
\beta_ {k m} ^ {r (l - 1)} = \text {a t t e n t i o n} \left(\bar {h} _ {k} ^ {(l - 1)}, \bar {h} _ {m} ^ {(l - 1)}\right)
$$
$$
\bar {H} _ {j} = [ \bar {h} _ {1} ^ {N _ {S}}, \bar {h} _ {2} ^ {N _ {S}}, \dots , \bar {h} _ {N _ {j}} ^ {N _ {S}} ]
$$
where  $\beta_{kj}^{r(l-1)}$  denotes the normalized attention coefficient calculated using dot-product mechanism for the node  $k$  based on its neighborhood node  $m$  under relation type  $r$ . Additionally, we compute multi-head attention and concatenate its outputs.
We also incorporate relational position embeddings as in (Ishiwatari et al., 2020). At  $l = 1$ ,  $\bar{h}_j^1$  is assigned to the contextual word embeddings obtained in Equation 1. Similar to Equation 2, we obtain a structural information enriched tweet embedding  $h_j^{rel}$  for a tweet  $T_j$  using maxpool operation on  $\bar{H}_j$ .
For the input tweet  $T_{i}$ , we differentiate discussion context tweets into two types: (i)  $T^{A}$ : tweets that are produced by the same user as the input tweet and (ii)  $T^{B}$ : tweets that are produced by all users other than the input tweet user. Using the computed tweet-level structural embedding  $h_{j}^{re1}$ , we construct a graph to account for speaker-dependent structural relationships (An example is shown in Figure B1). This includes two labeled edges indicating how the input tweet  $T_{i}$  is influenced by  $T^{A}$  (self-dependency) and  $T^{B}$  (inter-speaker dependency). Once the graph is constructed, we run R-GAT as in equations 5. Finally, we obtain relationship structure view of the input tweet  $T_{i}$  as:
$$
H _ {r e l} = \bar {H} _ {i} ^ {(N _ {R})} \tag {6}
$$
where  $N_{R}$  refers to the number of layers in the speaker-dependent relationship extraction layer.
5.1.3 Knowledge-Enriched View
Some of the propaganda techniques involve distortion of facts and data to promote their cause or point of view. Thus, we intuit that models which infuse external knowledge could potentially improve the overall performance in our task. Therefore, the primary aim of the knowledge-enriched view is to compute a text representation by enriching them with different kinds of knowledge. We leverage K-ADAPTER $(F + L)$  (Wang et al., 2020c) that combines both factual knowledge and linguistic knowledge to derive the knowledge-enriched view. K-ADAPTER acquires factual knowledge from the relationships among entities in text by training on a large scale alignment dataset between Wikipedia abstracts and Wikipedia triples. We modify the input by concatenating the context  $C_i$  and input tweet  $T_i$ : “<SEP> context</SEP> input tweet</SEP>” and use the embedding of the first token to get the knowledge-enriched view. This is given as:
$$
\widetilde {H} _ {\text {k n o w}} = \mathrm {K - A D A P T E R} \left(T _ {i}, C _ {i}\right) \tag {7}
$$
5.2 Cross-View Transformer
First, we fuse the cross-view information between any two views  $A$  and  $B$  using an additional multi-attention sub-layer as in Equation 4. We denote the three views as  $S,R,K$ . Next, we introduce a
1 https://spacy.io/universe/project/neuralcoref
transformer encoder layer on top of the cross-view transformer layer. Finally, we perform a maxpool operation on the output from the previous transformer layer. These steps are defined by:
$$
H _ {R \mapsto S} = \mathrm {M H - A T T} (Q u e r y _ {S}, K e y _ {R}, V a l u e _ {R})
$$
$$
H _ {K \mapsto S} = \mathrm {M H - A T T} (Q u e r y _ {S}, K e y _ {K}, V a l u e _ {K}) \tag {8}
$$
$$
z _ {S} = \operatorname {m a x p o o l} \left(\operatorname {T r a n s f o r m e r} \left(\left[ H _ {R \mapsto S}; H _ {K \mapsto S} \right]\right)\right)
$$
Similarly, we compute  $z_{R}, z_{K}$  and feed the concatenated outputs to the final classification layer.
5.3 Classification Layer
We calculate the probability that the input tweet uses a particular propaganda technique using a softmax layer, where  $f_{p}$  is a fully-connected layer, and  $z$  is the embedding produced by concatenating the outputs from the previous layer as:
$$
z = z _ {S} \left\| z _ {R} \right\| z _ {K} q = \operatorname {s o f t m a x} \left(f _ {p} (z)\right) \tag {9}
$$
6 Training & Implementation Details
We optimize the categorical cross-entropy loss between the predicted and true propaganda labels as in Equation 10, where  $L$  is the total number of propaganda labels,  $q_{j}$  is the predicted distribution that the input tweet falls under propaganda technique  $j$ , and  $p_{j} \in \{0,1\}$  denotes the ground-truth of whether the input tweet can be categorized under the  $j^{th}$  propaganda label.
$$
L _ {C E} = \sum_ {j = 1} ^ {L} - p _ {j} \log \left(q _ {j}\right) \tag {10}
$$
We use the publicly released default pre-trained model parameters for the BERT variants used. We perform a grid-search and optimize the hyperparameters using the validation set:  $N_{C} = 3$ ,  $N_{S} = N_{R} = 2$ . We used Adam with a learning rate of  $\alpha = 2\mathrm{e - }5$  and a warmup proportion of 0.1 for optimization. To account for randomness, we report the numbers which are the mean of five experimental runs with different random seeds. To alleviate the problem of unbalanced datasets, we utilize class weights in categorical cross-entropy loss based on the training and validation sets. See Appendix A for details on the hardware.
Our experiments are designed to investigate the following research questions:
RQ1: How well does our MV-PROP model perform compared to the other baselines in the propaganda detection task on social media data?
RQ2: What are the influences of different views and their interactions on the overall performance?
RQ3: Can our model be applied to detect propaganda on in-domain and cross-domain datasets?
7.1 Dataset
We run experiments using TweETSPIN dataset containing 210,392 tweets labeled with 19 propaganda types (referring to 18 propaganda techniques and 1 non-propaganda label). Due to the imbalance of the TweETSPIN dataset, we divide our TweETSPIN dataset into training  $(70\%)$ , validation  $(10\%)$ , and test  $(20\%)$  sets using a stratified shuffle split<sup>2</sup>.
7.2 Baselines
We use the following baselines in our experiments: BERT FT (Devlin et al., 2018) is a fine-tuned version of BERTbase model on the input tweets with and without considering the discussion context.
ROBERTA FT (Liu et al., 2019) is a fine-tuned version of ROBERTA<sub>base</sub> model on the input tweets conditioning on the discussion context tweets.
LATEXPRO (Wang et al., 2020d) leverages the declarative knowledge expressed in both first-order logic and text. We reimplement a variant of this model without the token-level loss to suit the sentence-level classification task. We further investigate the importance of the discussion context.
7.3 Model Variants
We investigate the importance of different modeling components by introducing variants to our proposed model and evaluating their performance on the TWEETSPIN validation set. These variants assess the influence of critical aspects: (a) discussion context, (b) different views, and (c) different fusion techniques. Depending on the fusion technique, we replace the cross-view transformer with simple late fusion techniques involving concatenation, mean, and sum of embeddings obtained from multiple views. View-specific variants include:
MV-PROP, which refers to our full model comprising all the three views as shown in Section 5.
MV-PROP-K, which integrates semantic and relational views while removing the knowledge-enriched view from our model.
MV-PROP-R, which combines semantic and knowledge-enriched views while removing the relationship structure view from our model.
2In this paper, we performed the stratified shuffle split using Python's Scikit-learn module
Models	P	R	F1	Std.
BERT FT	28.18	27.52	27.85	5.23
BERT FT w/ ctx	34.86	31.19	32.92	4.79
ROBERTA FT w/ ctx	35.16	32.20	33.72	3.27
LATEXPRO	32.27	29.40	30.77	3.96
LATEXPRO w/ ctx	43.76	34.56	38.62	3.24
MV-PROP	68.48	59.62	63.74	1.06
Table 1: Evaluation results on the TWEETSPIN test set. Std. refers to the standard deviation of the F1-scores across five runs.
Model	P	R	F1
Views
MV-PROP - K	59.18	50.71	54.62
MV-PROP - R	62.95	49.09	55.16
MV-PROP - S	61.54	52.38	56.60
Fusion
Concat	63.86	54.80	59.56
Mean	64.02	53.34	58.19
Sum	53.91	53.09	57.99
MV-PROP	70.17	59.63	64.47
Table 2: Ablation study on the TWEETSPIN validation set. We observe that the performance degrades when a specific view is removed or the cross-view transformer is replaced with other fusion techniques.
MV-PROP-S, which computes knowledge and relational views while removing the context-aware semantic view from our model.
We report the precision, recall, micro-averaged F1 scores and standard deviation of the computed F1 scores across five runs. In Table 1. Notably, the context plays a critical role in determining the propaganda label. This is evident from an average  $\sim 17.8\%$  drop in F1 without the context information in the baseline methods. We also find that the context information reduces the sensitivity of the models as indicated by a diminished standard deviation value whenever context comes into play. The results in Table 1 also demonstrate the ability of our multi-view representations to surpass the other baseline models by a large margin.
7.4.1 Effect of Multi-view Representations
In addition to experiments that emphasize the importance of the context in Table 1, we study the necessity of each view by discarding one view at a time and reporting the relative impact on the performance. It is clear from Table 2 that removal of each view leads to a significant drop in performance with
Figure 3: Analysis of cross-view transformer on TwEEET-SPIN. Left: Effect of number of cross-view transformer layers on performance. Right: Impact of cross-view transformers over simple late fusion method (concatenation) by varying the proportion of training samples.
knowledge-enriched view diminishing the performance by  $\sim 14\%$ . Since K-Adapter model injects additional knowledge into a pre-trained language model, it retains the benefits of the semantic representation obtained from the language model and also additionally incorporates both factual knowledge and entity information into the embedding. Therefore, discarding this component leads to a significant loss of information. This is the same reason why the drop in performance is relatively smaller when the context-aware semantic view is removed. The embedding from the knowledge-enriched view partially compensates for the information loss when the semantic view is discarded. We also highlight that the relationship structure view has a noticeable effect on the recall. We show sample tweets that were misclassified by our variants compared to our full model in Table B3 to further illustrate the importance of our views.
7.4.2 Effect of Cross-View Transformer
To study the effect of the cross-view transformer, we analyze the following: (a) effect of number of cross-view transformer layers, (b) impact of cross-view transformers over simple late fusion methods such as concat, mean, and sum of embeddings obtained from different views. Figure 3 (left) shows model performance with varying number of layers. The performance improves initially with the increase in number of layers and then drops beyond a point. The optimal number of layers on the validation and test datasets is 3. Moreover, Table 2 reports the performance for model variants involving simple late fusion techniques instead of the cross-view transformers. Visibly, the best performing late fusion technique (concat) lags behind the full MvPROP model containing cross-view transformer layers. With a performance drop of  $\sim 8\%$ , it is evident that the cross-view transformer efficiently
Models	F1
LR*	81.7
SVM*	79.5
LSTM*	80.7
LSTMR*	78.9
MV-PROP	84.36
computes the interaction between multi-view representations. Further, we vary the proportion of the training data and evaluate the potential impact of the cross-view transformer. We observe that our model with the cross-view transformer allows for quicker learning than the simple concatenation-based late fusion strategy. Figure 3 (right) shows that our full MV-PROP model plateaus closer to the best F1 score with  $\sim 60\%$  of the training data.
7.5 Performance on Unseen In-Domain dataset
Wang et al. (2020b) constructed a TWE dataset by combining two pre-existing datasets, the Twitter IRA corpus (Edgett, 2017) and the "twitter7" data from SNAP (Yang and Leskovec, 2011) as propagandistic and non-propagandistic data respectively. However, this dataset doesn't contain the discussion context that is critical to exploit the full potential of our MV-PROP model. Table 3 shows evaluations on the TWE dataset. Though certain views like the relationship structure view may not be utilized to their capabilities due to the lack of the discussion context, our MV-PROP model is able to significantly outperform the baselines used in the original work. Notably, our trained MV-PROP model performs well on the unseen in-domain dataset.
7.6 Performance on Cross-Domain Dataset
We hypothesize that our full model trained on the TWEETSPIN dataset is transferable to a cross-domain dataset like news articles. To verify this, we conduct an experiment on the Propaganda Techniques Corpus (PTC) (Da San Martino et al., 2019b), which is a manually annotated dataset for propaganda detection. Given that our model detects propaganda at the tweet level, we perform the sentence level propaganda detection (SLC) task from Da San Martino et al. (2019b) that determines whether a given sentence from a news article is pro
Table 3: Evaluations results on the TWE dataset. * indicates the scores reported in the original paper (Wang et al., 2020b).
Models	P	R	F1
Random	30.48	51.04	38.16
All-Propaganda	30.54	100.00	46.80
Fine-tuned BERT*1	63.20	53.16	57.74
BERT-Joint*1	62.84	55.46	58.91
MGN*1	60.41	61.58	60.98
Proper Gander*2	56.50	70.10	62.56
LatexPRO*3 (L)	56.53	73.17	63.79
LatexPRO*3 (L+T)	59.04	71.66	64.74
Mv-PROP (ZS)	54.08	62.75	58.09
Mv-PROP (FT)	64.35	84.58	73.09
Table 4: Evaluation results on the test set of PTC dataset for the sentence level propaganda classification (SLC) task.* refers to the scores reported from their original work. 1 refers to (Da San Martino et al., 2019b), 2 refers to (Madabushi et al., 2020), 3 refers to (Wang et al., 2020d). ZS refers to zero-shot and FS to finetuned variants of our model. All-Propaganda model always classifies the input text as propaganda.
pagandistic. We train a zero-shot (Mv-PROP (ZS)) variant of our model that directly takes the input from the PTC dataset and outputs the likelihood of it being propagandistic. With a threshold of 0.6, our model performs comparably to the fine-tuned BERT model, showing that our model demonstrates transfer capability to a similar task in the news domain. Further, we fine-tune our MV-PROP model (Mv-PROP (FT)) using the PTC training set and observe that we outperform other benchmarks for the SLC task. Results are shown in Table 4.
We introduced TweetSpin, a corpus of tweets containing weak labels of fine-grained propaganda techniques. Next, we presented a transformer-based multi-view propaganda detection model, MvPROP, that integrates multi-view contextual embeddings via pairwise cross-view transformers. We demonstrate how the semantic, relational, and knowledge view enrichment of the input tweet text leads to significant performance improvement over other baseline methods. Our experiments also demonstrated the transferability of our trained model to propaganda detection for news articles. The main limitation of our work is the reliance on weak annotations of Twitter data, which is unavoidable given the scale of our dataset. Future work could investigate leveraging the multi-view representations for span-level detection of fine-grained propaganda techniques.
Social media has become a battleground for propaganda and influence campaigns. This paper is an attempt to provide a dataset and models for detecting various propaganda techniques on Twitter to aid with the fight against this scourge on society. We release TweETSPIN, a Twitter corpus containing weak-annotations of fine-grained propaganda. Consistent with Twitter TOS, TweETSPIN contains only tweet IDs (with code provided to hydrate them) and no identifying information. Given the nature of the task the dataset contains potentially offensive and hateful language which should be taken into consideration. Additionally it is possible that our models, analyses, and dataset can potentially be used to create more advanced and harder to detect propaganda techniques. Though we should be aware of this possibility it is imperative that we in the research community stay ahead of miscreants by actively pushing this field forward. Finally, our models can lead to false positives where a user is falsely accused of spreading propaganda. Thus, it is important that the techniques presented here be used as a part of a larger effort to combat propaganda with humans in the loop for checks and balances.
TWEETSPIN was validated using MTurk. The annotators were paid 0.08 USD per task which took on average 30 seconds, for an hourly rate of 9.6 USD, above the federal and our state's minimum wage.
Tariq Alhindi, Jonas Pfeiffer, and Smaranda Muresan. 2019. Fine-tuned neural models for propaganda detection at the sentence and fragment levels. arXiv preprint arXiv:1910.09702.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Alberto Barrón-Cedeno, Israa Jaradat, Giovanni Da San Martino, and Preslav Nakov. 2019. Proppy: Organizing the news based on their propagandistic content. Information Processing & Management, 56(5):1849-1864.
Shuqing Bian, Xu Chen, Wayne Xin Zhao, Kun Zhou, Yupeng Hou, Yang Song, Tao Zhang, and Ji-Rong Wen. 2020. Learning to match jobs with resumes from sparse interaction data using multi-view co-teaching network. In Proceedings of the 29th ACM
International Conference on Information & Knowledge Management, pages 65-74.
Dan Busbridge, Dane Sherburn, Pietro Cavallo, and Nils Y Hammerla. 2019. Relational graph attention networks. arXiv preprint arXiv:1904.05811.
Guido Caldarelli, Rocco De Nicola, Fabio Del Vigna, Marinella Petrocchi, and Fabio Saracco. 2020. The role of bot squads in the political propaganda on twitter. Communications Physics, 3(1):1-15.
Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. 2017. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1907-1915.
Andre Ferreira Cruz, Gil Rocha, and Henrique Lopes Cardoso. 2019. On sentence representations for propaganda detection: From handcrafted features to word embeddings. In Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 107-112.
Giovanni Da San Martino, Alberto Barron-Cedeno, and Preslav Nakov. 2019a. Findings of the nlp4if-2019 shared task on fine-grained propaganda detection. In Proceedings of the second workshop on natural language processing for internet freedom: censorship, disinformation, and propaganda, pages 162-170.
Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeno, Rostislav Petrov, and Preslav Nakov. 2019b. Fine-grained analysis of propaganda in news article. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 5636-5646.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Sean Edgett. 2017. Testimony of sean j. edgett. United States Senate Committee on the Judiciary, Subcommittee on Crime and Terrorism.
Johan Farkas and Marco Bastos. 2018. Ira propaganda on twitter: Stoking antagonism and tweeting local news. In Proceedings of the 9th International Conference on social media and society, pages 281-285.
Tsu-Jui Fu, Peng-Hsuan Li, and Wei-Yun Ma. 2019. Graphrel: Modeling text as relational graphs for joint entity and relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1409-1418.
Yunchao Gong, Qifa Ke, Michael Isard, and Svetlana Lazebnik. 2014. A multi-view embedding space for modeling internet images, tags, and their semantics. International journal of computer vision, 106(2):210-233.