A Multi-Faceted NLP Analysis of Misinformation Spreaders in Twitter (2024)

Abstract: Social media is an integral part of the daily life of an increasingly large number of people worldwide. Used for entertainment, communication and news updates, it constitutes a source of information that has been extensively used to study human behaviour. Unfortunately, the open nature of social media platforms along with the difficult task of supervising their content has led to a proliferation of misinformation posts. In this paper, we aim to identify the textual differences between the profiles of user that share misinformation from questionable sources and those that do not. Our goal is to better understand user behaviour in order to be better equipped to combat this issue. To this end, we identify Twitter (X) accounts of potential misinformation spreaders and apply transformer models specialised in social media to extract characteristics such as sentiment, emotion, topic and presence of hate speech. Our results indicate that, while there may be some differences between the behaviour of users that share misinformation and those that do not, there are no large differences when it comes to the type of content shared.

heading: 1 Introduction
text: The emerging popularity of social platforms such as Facebook, Twitter, and WhatsApp has revolutionised the way information is disseminated and consumed (Fac, 2023; Murthy, 2018; Deshmukh, 2015). People are able to express their sentiments, share their opinions on multiple topics, and to discuss and influence each other with ease and at a speed that has transformed not only how we communicate but also how we perceive the world around us. Unfortunately, the capacity to reach a vast audience within seconds, along with the challenges that arise with verifying an ever-expanding volume of content, has created fertile ground within social media for malicious actors, or unaware users, to spread misinformation. The recent examples
of fake news related to the COVID-19 pandemic (Evanega et al., 2020), and the ongoing war in Ukraine (Pierri et al., 2023) demonstrate that misinformation in social media is a complex problem with far-reaching implications for society, democracy, and information integrity.
Combating misinformation in social media is a topic that is studied extensively in academia (Vosoughi et al., 2018; Pennycook et al., 2020) and in the natural language processing (NLP) community (Su et al., 2020) specifically, among others. Common approaches of dealing with misinformation include defining the problem as a classification task (Serrano et al., 2020; Hamid et al., 2020) and classifying a post as fake or not; with factchecking (Thorne and Vlachos, 2018) often defined as an information retrieval task (Lazarski et al., 2021). However, research regarding the agents that share misinformation is rather limited in comparison (Shu et al., 2020; Rangel et al., 2020; Dou et al., 2021) particularly when it comes to analysing language-specific features.
In this paper, we focus on misinformation in Twitter and perform an analytical comparison between different types of user based on their content shared online and the reliability of their sources. To this end, we first compiled three diverse datasets in which spreaders of misinformation are categorised using different techniques. Then, we perform an exhaustive analysis of the content of these users by leveraging transformer-based language models specialised on social media tasks such as sentiment analysis, emotion recognition, topic categorisation and hate speech detection. The main contributions of this paper are the following: (1) we gather and consolidate existing and new Twitter datasets related to misinformation spreaders; and (2) we extract insights for the behaviour of such users in comparison with users sharing content from reliable sources.
71

heading: 2 Related Work
text: The study of identifying misinformation has been a prominent area of research in recent years. Initially, efforts focused on addressing the problem through classification, either in a binary or multiclass context. Some studies delved into examining the spread of true and false information online work on the topic of information dissemination (Vosoughi et al., 2018). Meanwhile, others opted for a data mining approach in the realm of fake news detection on social media, utilising various features and machine learning algorithms to classify news articles as true or false (Shu et al., 2017).
Moreover, beyond binary classification, researchers explored multiclass classification methods. For instance, Castillo et al. (2011) investigated the credibility of information on Twitter and proposed a framework categorizing tweets into four groups: true, false, unverified, and non-informative. Zubiaga et al. (2016) delved into the analysis of conversational threads on social media to gain insights into how rumors propagate and how individuals respond to them, shedding light on the dynamics of misinformation propagation.
These approaches evolved to better serve journalists and fact-checkers. The focus shifted from classification to fact-checking and information retrieval, aiming to assist journalists in source verification. This transition led to the development of tools to meet their specific needs (Schlichtkrull et al., 2023). The availability of datasets like FEVER (Thorne et al., 2018), MultiFC (Augenstein et al., 2019), and X-Fact (Gupta and Srikumar, 2021) has been instrumental in enabling researchers to experiment with and develop novel methods for evidence retrieval and rumour verification (Nasir et al., 2021; Lee et al., 2020; Lewis et al., 2020).
While there have been notable studies in the broader field of misinformation and fact verification, there’s a notable gap when it comes to a systematic analysis of the textual content of fake news spreaders. Much of the existing research has predominantly focused on the detection of misinformation sources, fact-checking, or the development of classification algorithms to distinguish true from false content. However, there is limited in-depth work that methodically dissects the text generated by those actively involved in spreading fake news that utilises state-of-the-art models (Ghanem et al., 2020; Rangel et al., 2020) and where the language
analysis is not supplementary to the network and graph analysis (Aswani et al., 2019).
In this work we seek to methodically analyse the textual content generated by those responsible for spreading fake news. The primary objective is to gain a deeper understanding of the characteristics, strategies, and linguistic patterns employed by these actors in disseminating misleading or false information. Unlike traditional fact-checking, our work does not intend to verify or debunk specific claims but rather aims at understanding the textual content shared by individuals or groups behind the spread of fake news, thereby providing further insights into their content dissemination strategies.

heading: 3 Data
text: For our analysis, we exclusively focus on Twitter users, particularly tweets in the English language. Our goal was to extract a diverse tweet corpus for both users regularly spreading fake news or news from questionable sources, and users sharing content from verified sources. In the following we describe our data collection methodology stemming from various sources.

heading: 3.1 Data Collection
text: In total, we draw upon three diverse data sources to extract relevant tweets from user account sharing trusted and untrusted sources. Moreover, we extract tweets from legacy-verified Twitter accounts as a control group.

heading: 3.1.1 Media Bias Fact Check (MBFC)
text: Our first corpus is extracted from a list of known conspiracy sites provided by "Media Bias Fact Check" (MBFC). This source is commonly used in the study of fake news (Nakov and Da San Martino, 2020; Cinelli et al., 2020). For this dataset, we extracted tweets that share URLs from known untrusted sites1 and then sample users based on the frequency of sharing these links. In particular, we considered only those users in the 75 percentile in terms of number of links shared. In order to gather enough information, all user accounts that were not older than 30 days were excluded from the analysis. Subsequently, all posts made by the sampled users during September 2021 were collected, which aligns with the date when the MBFC lists were last updated prior to conducting this experiment. User accounts were then further filtered based on their activity, only keeping those users posting more
1https://mediabiasfactcheck.com/conspiracy/
frequently than the median daily posts. Finally, to ensure a diverse representation, users were sampled based on their number of followers by maintaining the original distribution and thus encompassing both popular and less popular accounts. This final sample represents the MBFC-untrusted subset.
The above methodology is mirrored to collect users that share links form trusted news-sources according to MBFC2 resulting in the MBFC-trusted subset.

heading: 3.1.2 FakeNewsNet (FNN)
text: The FakeNewsNet dataset, referred to as FNN (Shu et al., 2018), contains two subsets: (1) tweets related to news content, e.g. tweets revolving around US politics and tweets; and (2) tweets related to social context, e.g. tweets talking about celebrities. Tweets in each groups are further classified as either untrusted or trusted. For the purpose of this study, we concentrate solely on the politics-related subset, as it exhibits a closer alignment with the majority of the links found within the MBFC lists. To extract relevant users, we initially scrape all tweets in the dataset and randomly sample users. Finally, all tweets posted by the selected users from September 2021 are retrieved. Only the accounts that have at least 100 posts were considered to create the FNN-untrusted and FNN-trusted subsets.

heading: 3.1.3 Profiling Fake News Spreaders (PAN)
text: The English subset of the PAN 2020: Profiling Fake News Spreaders task (PAN) (Rangel et al., 2020) is dataset that comprises a total of 50,000 English tweets obtained from 500 users, with each user contributing 100 tweets. These users are categorized as either trusted news spreaders (PAN-trusted) or untrusted news spreaders (PAN-untrusted). In the interest of privacy, no additional user-specific information, such as author descriptions or popularity metrics, is disclosed. Despite its relatively modest size and the limitation on the extraction of additional user details, the PAN dataset is considered robust and reliable. Its construction involved manual checks, and it underwent thorough scrutiny by multiple individuals, primarily due to its relevance in a competitive context. This rigorous validation process enhances the dataset’s trustworthiness and accuracy.
2https://mediabiasfactcheck.com/pro-science/, https://mediabiasfactcheck.com/center/

heading: 3.1.4 Control (Verified users)
text: In order to have a control group to compare in our experiments, we sampled tweets from legacyverified accounts for which the authenticity is known. This dataset was compiled by sampling verified users and collecting their tweets during the same time period as the previous datasets. Our aim was to select users whose characteristics align closely with the distribution patterns observed in the FNN and MBFC datasets.

heading: 3.2 Statistics and Descriptive Analysis
text: By considering these diverse data sources, we aim to comprehensively examine and understand the dynamics of untrusted news spreaders on the Twitter platform. Our analysis encompasses a total of 4,206,799 tweets contributed by 5,233 users, as presented in Table 1. In addition to the number of tweets and users, we also investigate the average length of tweet and average emoji usage. We did not identify a clear pattern between the trusted and untrusted subsets as far as these metrics are concerned.
Looking into the lexical characteristics of each dataset, distinctions between the untrusted and trusted subsets become more apparent. For instance, when assessing lexical diversity using the Type Token Ratio (TTR), we observe that untrusted users, with the exception of the FNN dataset, tend to employ a less diverse vocabulary which is consistent with previous research (Horne and Adali, 2017). Our analysis based on the average presence of emojis in each tweet reveals no consistent pattern, despite prior research suggesting higher emoji usage among untrusted news spreaders (Er and Yilmaz, 2023). For example, while the untrusted subset of the MBFC dataset exhibits higher emoji usage, the opposite holds true for the FNN dataset.
Lexical specificity. To gain an overall understanding of the prevalent topics within our corpora, we employ lexical specificity (Lafon, 1980). Lexical specificity is a word-level metric that indicates the importance of each word in a subcorpus. In particular, for this analysis we use the formulation outlined in Camacho-Collados et al. (2016), and extract the top terms in each dataset. Table 2 displays the top ten lemmas3 in each dataset based on their lexical specificity scores.
Notably, due to the same time period during data collection, a significant overlap exists between the MBFC and FNN datasets, particularly within their ’untrusted’ subsets. Terms such as ’biden’ and ’vaccine’ are common across both. Additionally, a discernible trend emerges, indicating that ’untrusted’ subsets across datasets often feature more controversial and divisive topics. This is evident in the presence of terms like ’covid,’ ’prohibition,’ and ’impeachment,’ in contrast to the ’trusted’ subsets, which exhibit more generic and neutral terms such as ’bts,’ ’music,’ and ’film.’ This distinction becomes even more pronounced when examining the top terms in the Verified dataset, which include terms like ’game,’ ’football,’ and ’love.’

heading: 4 Methodology
text: Our goal is to analyse various content-related features from the extracted posts in Section 3. To capture the nuanced language features present in the data, we employ a range of pre-trained language models designed for social media usage. Our primary focus encompasses sentiment and affection analysis, topic classification, and the identification of hate speech in textual content, features are frequently employed in the study of misinformation propagation (Vicario et al., 2019; Verma et al., 2020), aiming to uncover emotionally charged language and controversial topics.
All the language models used are built upon the
3Lemmatization was done using spaCy https://spacy.io/.
RoBERTa architecture (Liu et al., 2019) and trained on social media corpora, making them well-suited for analysing Twitter data. More specifically:
Sentiment Analysis. The model twitter-robertabase-sentiment-latest (Loureiro et al., 2022) is used to extract the sentiment polarity where each tweet is classified as negative, neutral, or positive. This model has been fine-tuned for sentiment analysis using the dataset provided in the Sentiment Analysis in Twitter task of Semeval 2017 (Rosenthal et al., 2019). By analysing the sentiment expressed in social media content, we can gain insights into information being shared (Baishya et al., 2021). Specifically, presence of exaggerated positive sentiment or negative sentiment in response to fake news can serve as indicators of misinformation (Alonso et al., 2021).
Emotion Analysis. We leverage twitterroberta-base-emotion-multilabel-latest (CamachoCollados et al., 2022) to assign one or more emotions to each tweet. This model is trained using data from the ’Affect in Tweets’ Semeval 2018 task (Mohammad et al., 2018), covering 11 different emotions. Similar to sentiment analysis, the presence of specific emotions has been used to analyse the spread of rumours and misinformation, with negative emotions potentially contributing to the spread of misinformation (Vosoughi et al., 2018; Weeks, 2015).
Hate Speech Detection. We use the twitterroberta-base-hate-multiclass hate speech detection model (Antypas and Camacho-Collados, 2023), which is trained on a combination of 13 different hate speech Twitter datasets and is capable of identifying hate speech from seven target groups. The inclusion of hate speech detection as a feature is motivated by previous research indicating a positive correlation between the presence of hate speech and misinformation (Inwood and Zappavigna, 2023).
Topic Classification. We use tweet-topic-21multi (Antypas et al., 2022), a multi-label classification model fine-tuned on a Twitter topic classification dataset. This model assigns one or more topics to each tweet from a list of 19 topics. Our hypothesis is that there may be a significant difference between the topics discussed by untrusted news spreaders and regular users, e.g. untrusted news spreaders potentially engaging in discussions related to sensitive topics at a higher volume.
All the specialised models described above are perform in line of the state of the art for each of the tasks in the social media context4 and they enable us to delve deeper into the complex linguistic nuances within the social media data. Nonetheless, as we describe in the Limitations section, they all have a degree of error that needs to be considered when making conclusions.

heading: 5 Analysis
text: We consider each pair of collected datasets (untrusted and trusted), along with the Verified control dataset. Our examination involves a comparison of the tweets within each dataset individually, as well as their aggregation for each user. This holistic approach enables us to explore a variety of perspectives and insights across the datasets and their combined impact.

heading: 5.1 Textual Analysis
text: Table 3 displays the aggregated results for the sentiment, emotion, hate speech and topic analysis. For each feature we consider each user independently by taking their mean value and then aggregate the results of users belonging in the same subset. Even though differences between untrusted and trusted
4Sentiment Analysis: 73.7% Recall, Emotion Analysis: 80% F1-macro, Hate Speech: 94% Accuracy, Topic Classification: 59% F1-macro – please refer to the individual references for more details.
subsets exist, it is challenging to identify trends that are consistent across the datasets. In the following sections we investigate each characteristic individually.

heading: 5.1.1 Sentiment
text: When evaluating the presence of sentiment in tweets, a noticeable trend emerges: tweets associated with untrusted news spreaders tend to exhibit a higher degree of negativity compared to those posted by other users. The distribution of sentiment across the datasets is displayed in Figure 1. In the case of the FNN dataset, however, this difference is almost not negligible. Finally, even though there is more negativity in untrusted users, the distributions among negative, neutral and positive tweets are very similar in all cases except for the verified users that tend to be more positive overall.

heading: 5.1.2 Emotion
text: Similarly to the findings in sentiment analysis, the analysis of affect reveals a consistent pattern where untrusted news spreaders tend to gravitate toward more negative emotions. Figure 2a provides insight into the distribution of the 11 emotions present across across all subsets.
A clear contrast emerges, with tweets attributed to trusted users generally displaying greater joy and featuring a lesser presence of anger and disgust, in stark contrast to the tweets originating from untrusted users. This trend remains consistent even when evaluating the per-user aggregation. Finally, similarly to the sentiment distribution patterns, there are no noticeable differences when analysing the overall emotion distribution and, in this case, it also related to that of verified users.

heading: 5.1.3 Hate Speech
text: When examining hate speech, a feature that often coexists with misinformation (Inwood and Zappavigna, 2023), such as Holocaust denial and the Great Replacement theory, it does not appear to be a prominent feature in the collected datasets. Our analysis indicates an absence of hate speech, with 99% of all tweets being devoid of it.
There does appear to be a variance in the types of hate speech across the subsets (as displayed in Figure 2b). Untrusted subsets exhibit a higher inclination towards racism, while in the trusted subset, sexism appears to be more prevalent. However, given the limited number of instances, it is prudent to exercise caution when drawing extensive conclusions based on this data.

heading: 5.1.4 Topics
text: Regarding the topics that untrusted news spreaders and regular users typically discuss, the results appear to suggest a similar distribution of topics (Figure 3). Untrusted news spreaders appear to engage more extensively in posting tweets related to news and social issues, which are those related to politics, among others. This suggests that these accounts may be more socially active, and can create the illusion of a larger representation than that of the general population.
Conversely, there is no discernible distinction in the case of the remaining popular topics, with variations existing among the datasets. For instance, the topic "celebrity_&_pop_culture" is more prevalent in the Panuntrusted dataset but less common in the other untrusted subsets. Again here we can observe more differences with respect to verified users, where sports and diaries_&_daily_life topics are much more prominent.

heading: 5.2 Spreader Detection Analysis
text: Recognising that a significant portion of our datasets relies on weak labels, with the distinction between users propagating untrusted news and those who do not being based on heuristics based on the number of posts shared from untrusted sources, we perform a robustness analysis on the Pan20 dataset which includes train and test splits. To this end, we train a classifier capable of discerning between the trusted and untrusted classes and compare the results with our approach.
The train/test split originally utilised in the competition is retained, consisting of 300 users for training and 200 users for testing. We assess the performance of two classifiers: (1) A classifier based on the best-performing models as presented in the competition (Buda and Bolonyai, 2020; Pizarro, 2020), utilising an XGBoost classifier (Chen and Guestrin, 2016). This model is trained using TFIDF features and a combination of word and character ngrams; and (2) A pre-trained Longformer (Beltagy et al., 2020), which is further fine-tuned using the PAN dataset. We leverage the implementation provided by Hugging Face (Wolf et al., 2020) for the fine-tuning of the Longformer5. Hyper-parameter tuning, including batch size, epochs number, and learning rate, is conducted using Ray Tune (Liaw et al., 2018).
The results reveal that the XGBoost model (XGB) surpasses the Longformer classifier, achieving a 74% macro F1 score compared to the Longformer’s 70%. One possible explanation for this outcome lies in the unstructured nature of Twitter text, which presents an added challenge to the language model. The Longformer, not explicitly trained on social media corpus data, may face limitations in handling this specific type of text.
When examining the results of the XGB classifier in the PAN dataset, we observe an almost identical trend when compared with our initial results. For example, when looking the sentiment distribution of user accounts using the XGB classifier
5https://huggingface.co/allenai/longformer-base-4096
and our initial distinction of trusted and untrusted users, only minimal differences can be observed (Pan-untrusted: 23% Negative and 13% Positive; XGB-untrusted: 24% and 13%). Our experiment indicates that even though developing a classifier to identify untrusted users may not be the optimal approach, it can still be used as a proxy to derive useful information and identify patterns that can be used to reveal malicious actors. Additional results for all tasks regarding the performance of the XGB model and the differences with our approach follow a similar trend and can be found in Appendix C.
As a final experiment, we attempt to enhance our XGB classifier by integrating the features already extracted. Our results reveal that while the incorporation of new features generally results in only marginal variations in the model’s performance, the addition of certain features, especially sentiment, holds the potential to notably improve its effectiveness. This suggests that careful selection and integration of specific features can yield incremental but meaningful gains in the classifier’s performance, an exploration that we leave for future work as it falls out of the scope of this work.

heading: 6 Conclusions
text: This paper’s comparative analysis aims to delve into the dynamics of misinformation dissemination in the digital age by examining the distinctions between untrusted news spreaders and other users. To this end, we have compiled a substantial sample of untrusted news spreaders and the general content shared of these users in Twitter. Using this large corpus stemming from three diverse datasets (MBFC, FNN and PAN), we have analysed the disparities in their language usage.
The initial exploration of traits associated with untrusted news spreaders, including the presence of hate speech, did not necessarily reveal the distinctions we anticipated. Other language features such as sentiment and emotional content indicate the existence of relatively small language differences between the two groups of users. These differences provide valuable insights that can inform the development of systems designed to identify and counteract malicious accounts. In particular, our results suggest that misinformation mitigation efforts should be focused on the specific content shared, rather than in profiling individual accounts.

heading: 7 Limitations
text: While we strived to derive insights from a large dataset using state-of-the-art classifiers and a robust analytical setup, we acknowledge the presence of factors that constrain the depth of our findings. For example, the focus on English-language content, potentially limiting the scope of global social media interactions and perspectives. Additionally, the exclusive use of Twitter data might not fully represent the dynamics on other social media platforms. While verified accounts are employed as a control group, it should be noted that they may not serve as a perfect control due to factors like their popularity, potential biases, or unique behaviours. Furthermore, the extraction of users relies on heuristics, introducing some degree of noise and potential inaccuracies in the data. Finally, we made use of automatic models based on transformers. While these have been tested extensively in prior work, there are inherent limitations in these models, as well as possible unwanted biases. All these limitations should be considered when interpreting our results and conclusions.

heading: 8 Ethical Statement
text: In our study involving user-generated content from social media, we ensured user privacy in several ways. First, we replaced all user mentions in the texts with placeholders and removing user IDs. Moreover, all the data utilised in our research is sourced from publicly available information or collected using the official Twitter API. Finally, all the information is provided in an aggregated fashion, without reporting sensitive information from individual users.
While our dataset and methodology have the potential for analysing individual behaviours, our primary objective is to offer researchers a valuable tool for the analysis and aggregation of social media content.

heading: A Computational Resources
text: An NVIDIA GeForce RTX 4090 GPU was utilised for the experiments conducted:
• 18 hours for the inference process of (sentiment, emotion, topic) on the MBFC, FNN, and Verified datasets.
• 6 hours for the training of the Longformer model (Section Spreader Detection Analysis.

heading: B Model Categories
text: B.1 Emotion Categories
The twitter-roberta-base-emotion-multilabel model classifies each entry in one or more of the following classes: anger, anticipation, disgust, fear, joy, love, optimism, pessimism, sadness, surprise, trust.
B.2 Hate Speech Categories. The twitter-roberta-base-hate-multiclass model classifies each entry in one of the following classes: not_hate, sexism, racism, religion, other, sexual_orientation, disability
B.3 Topic Classification Categories The tweet-topic-21-multi model assigns each tweet one or more topics from the following list: arts_&_culture, business_&_entrepreneurs, celebrity_&_pop_culture, diaries_&_daily_life, family, fashion_&_style, film_tv_&_video, fitness_&_health, food_&_dining, gaming, learning_&_educational, music, news_&_social_concern, other_hobbies, relationships, science_&_technology, sports, travel_&_adventure, youth_&_student_life
C Spreader Detection: XGB
Table 4 highlights the performance of each feature set, with the ’text and sentiment (text-s)’ combination achieving the highest F1 score of 75 and accuracy of 76, suggesting it is the most effective combination for this analysis.
Figures 6b, 4, and 6a illustrate that the discrepancies in the distribution of the examined features, sentiment, emotion and hate speech respectively6, between the XGB model’s predictions and the PAN dataset are negligible, indicating that they exhibit comparable trends.
6Results related to the topic distribution can be found in Appendix C.

heading: Future Work
text: This suggests that careful selection and integration of specific features can yield incremental but meaningful gains in the classifier’s performance, an exploration that we leave for future work as it falls out of the scope of this work