SlowGuess/ACL_2020_OCR huggingface dataset:
Peter Stefanov<sup>1</sup>, Kareem Darwish<sup>2</sup>, Atanas Atanasov<sup>3</sup>, Preslav Nakov<sup>2</sup>  
<sup>1</sup>SiteGround Hosting EOOD, Bulgaria
$^{2}$ Qatar Computing Research Institute, HBKU, Doha, Qatar
$^{3}$ Sofia University "St. Kliment Ohridski", Sofia, Bulgaria {stefanov.peter.ps,atanas.atanasov.sf}@gmail.com, {kdarwish,pnakov}@hbku.edu.qa
Discovering the stances of media outlets and influential people on current, debatable topics is important for social statisticians and policy makers. Many supervised solutions exist for determining viewpoints, but manually annotating training data is costly. In this paper, we propose a cascaded method that uses unsupervised learning to ascertain the stance of Twitter users with respect to a polarizing topic by leveraging their retweet behavior; then, it uses supervised learning based on user labels to characterize both the general political leaning of online media and of popular Twitter users, as well as their stance with respect to the target polarizing topic. We evaluate the model by comparing its predictions to gold labels from the Media Bias/Fact Check website, achieving  $82.6\%$  accuracy.
Online media and popular Twitter users, which we will collectively refer to as influencers, often express overt political leanings, which can be gleaned from their positions on a variety of political and cultural issues. Determining their leaning can be done through the analysis of their writing, which includes the identification of terms that are indicative of stance (Groseclose and Milyo, 2005; Gentzkow and Shapiro, 2011). Performing such analysis automatically can be done using supervised classification, which in turn would require manually labeled data (Groseclose and Milyo, 2005; Gentzkow and Shapiro, 2011; Mohammad et al., 2016). Alternatively, leanings can be inferred based on which people share the content (blogs, tweets, posts, etc.) on social media, as social media users are more likely to share content that originates from sources that generally agree with their positions (An et al., 2012; Morgan et al., 2013; Ribeiro et al., 2018; Wong et al., 2013).
Here, we make use of this observation to characterize influencers, based on the stances of the Twitter users that share their content. Ascertaining the stances of users, also known as stance detection, involves identifying the position of a user with respect to a topic, an entity, or a claim (Mohammad et al., 2016). For example, on the topic of abortion in USA, the stances of left- vs. right-leaning users would typically be "pro-choice" vs. "pro-life", respectively.
In this paper, we propose to apply unsupervised stance detection to automatically tag a large number of Twitter users with their positions on specific topics (Darwish et al., 2020). The tagging identifies clusters of vocal users based on the accounts that they retweet. Although the method we use may yield more than two clusters, we retain the two largest ones, which typically include the overwhelming majority of users, and we ignore the rest. Then, we train a classifier that predicts which cluster a user belongs to, in order to expand our clusters. Once we have increased the number of users in our sets, we determine which sources are most strongly associated with each group based on sharing by each group. We apply this methodology to determine the positions of influencers and of media on eight polarizing topics along with their overall leaning: left, center or right. In doing so, we can also observe the sharing behavior of right- and left-leaning users, and we can correlate their behavior with the credibility of the sources. Further, given the user stances for these eight topics, we train a supervised classifier to predict the overall bias of sources using a variety of features, including the so-called valence (Conover et al., 2011a), graph embeddings, and contextual embeddings. Using a combination of these features, our classifier is able to predict the bias of sources with  $82.6\%$  accuracy, with valence being the most effective feature. Figure 1 outlines our overall methodology.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 527-537 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: General outline of our methodology.
- We use unsupervised stance detection to automatically determine the stance of Twitter users with respect to several polarizing topics.
- We then use distant supervision based on these discovered user stances to accurately characterize the political leaning of media outlets and of popular Twitter accounts. For classification, we use a combination of source valence, graph embeddings, and contextualized text embeddings.
- We evaluate our approach by comparing its bias predictions for a number of news outlets against gold labels from Media Bias/Fact Check. We further evaluate its predictions for popular Twitter users against manual judgments. The experimental results show sizable improvements over using graph embeddings or contextualized text embeddings.
The remainder of this paper is organized as follows: Section 2 discusses related work. Section 3 describes the process of data collection. Section 4 presents our method for user stance detection. Section 5 describes how we characterize the influencers. Section 6 discusses our experiments in media bias prediction. Finally, Section 7 concludes and points to possible directions for future work.
Recent work that attempted to characterize the stance and the ideological leaning of media and Twitter users relied on the observation that users tend to retweet content that is consistent with their world view. This stems from selective exposure, which is a cognitive bias that leads people to avoid the cognitive overload from exposure to opposing views as well as the cognitive dissonance in which people are forced to reconcile between their views and opposing views (Morgan et al., 2013).
Concerning media, Ribeiro et al. (2018) used the Facebook advertising services to infer the ideological leaning of online media based on the political leaning of Facebook users who consumed them. An et al. (2012) relied on follow relationships to online media on Twitter to ascertain ideological leaning of media and users based on the similarity between them. Wong et al. (2013) studied retweet behavior to infer the ideological leanings of online media sources and popular Twitter accounts. Barbera and Sood (2015) proposed a statistical model based on the follower relationships to media sources and Twitter personalities in order to estimate their ideological leaning.
As for individual users, much recent work focused on stance detection to determine a person's position on a topic including the deduction of political preferences (Barberá, 2015; Barber and Rivero, 2015; Borge-Holthoefer et al., 2015; Cohen and Ruths, 2013; Colleoni et al., 2014; Conover et al., 2011b; Fowler et al., 2011; Hasan and Ng, 2014; Himelboim et al., 2013; Magdy et al., 2016a,b; Makazhanov et al., 2014; Trabelsi and Zairane, 2018; Weber et al., 2013). User stance classification is aided by the tendency of users to form so-called "echo chambers", where they engage with like-minded users (Himelboim et al., 2013; Magdy et al., 2016a), and the tendency of users' beliefs to be persistent over time (Borge-Holthoefer et al., 2015; Magdy et al., 2016a; Pennacchiotti and Popescu, 2011b).
Studies have examined the effectiveness of different features for stance detection, including textual features such as word  $n$ -grams and hashtags, network interactions such as retweeted accounts and mentions, and profile information such as user location (Borge-Holthoefer et al., 2015; Hasan and Ng, 2013; Magdy et al., 2016a,b; Weber et al., 2013). Network interaction features were shown to yield better results compared to using textual features (Magdy et al., 2016a; Wong et al., 2013). Sridhar et al. (2015) leveraged both user interactions and textual information when modeling stance and disagreement, using a probabilistic programming system that allows models to be specified using a declarative language.
Trabelsi and Zaiane (2018) described an unsupervised stance detection method that determines the viewpoints of comments and of their authors. It analyzes online forum discussion threads, and therefore assumes a certain structure of the posts.
It also assumes that users tend to reply to each others' comments when they are in disagreement, whereas we assume the opposite in this paper. Their model leverages the posts' contents, whereas we only use the retweet behavior of users.
Many methods involving supervised learning were proposed for stance detection. Such methods require the availability of an initial set of labeled users, and they use some of the aforementioned features for classification (Darwish et al., 2018; Magdy et al., 2016b; Pennacchiotti and Popescu, 2011a). Such classification can label users with precision typically ranging between  $70\%$  and  $90\%$  (Rao et al., 2010; Pennacchiotti and Popescu, 2011a). Label propagation is a semi-supervised method that starts with a seed list of labeled users and propagates the labels to other users who are similar based on the accounts they follow or retweet (Barberá and Sood, 2015; Borge-Holthoefer et al., 2015; Weber et al., 2013). While label propagation may label users with high precision (often above  $95\%$ ), it is biased towards users with more extreme views; moreover, careful choice of thresholds is often required, and post-checks are needed to ensure quality.
Abu-Jbara et al. (2013) and more recently Darwish et al. (2020) used unsupervised stance detection, where users are mapped into a lower dimensional space based on user-user similarity, and then clustered to find core sets of users representing different stances. This was shown to be highly effective with nearly perfect clustering accuracy for polarizing topics, and it requires no manual labeling of users. Here, we use the same idea, but we combine it with supervised classification based on retweets in order to increase the number of labeled users (Darwish, 2018). Other methods for user stance detection include collective classification (Duan et al., 2012), where users in a network are jointly labeled and classification in a low-dimensional user-space (Darwish et al., 2017).
As for predicting political leaning or sentiment, this problem was studied previously as a supervised learning problem, where a classifier learns from a set of manually labeled tweets (Pla and Hurtado, 2014; Bakliwal et al., 2013; Bermingham and Smeaton, 2011). Similarly, Volkova et al. (2014) predicted Twitter users' political affiliation (being Republican or Democratic), using their network connections and textual information, relying on user-level annotations.
We obtained data on eight topics that are considered polarizing in the USA (Darwish et al., 2020), shown in Table 1.
They include a mix of long-standing issues such as racism and gun control, temporal issues such as the nomination of Judge Brett Kavanaugh to the US Supreme Court and Representative Ilhan Omar's polarizing remarks, as well as non-political issues such as the potential dangers of vaccines. Further, though long-standing issues typically show right-left polarization, stances towards Omar's remarks are not as clear, with divisions on the left as well.
Since we are interested in US users, we filtered some tweets to retain such by users who have stated that their location was USA. We used a gazetteer that included words that indicate USA as a country (e.g., America, US), as well as state names and their abbreviations (e.g., Maryland, MD).
Other data that we used in our experiments is a collection of articles that were cited by users from the tweets collection and that originate from media, whose bias is known, i.e., is discussed on the Media Bias/Fact Check website.
4 User Stance Detection
In order to analyze the stance of influencers on a given topic, we first find the stances of Twitter users, and then we project them to the influencers that the users cite. A central (initial) assumption here is that if a user includes a link to some article in their tweet, they are more likely to agree or endorse the article's message. Similarly, when a user retweets a tweet verbatim without adding any comments, they are more likely to agree with that tweet. We label a large number of users with their stance for each topic using a two-step approach, namely projection and clustering and supervised classification.
For the projection and clustering step, we identify clusters of core vocal users using the unsupervised method described in (Darwish et al., 2020). In this step, users are mapped to a lower dimensional space based on their similarity, and then they are clustered. After performing this unsupervised learning step, we train a supervised classifier using the two largest identified clusters in order to tag many more users. For that, we use FastText, a deep neural network text classifier, that has been shown to be effective for various text classification tasks (Joulin et al., 2017).
Topic	Keywords	Date Range	No. of Tweets
Climate change	#greenddeal, #environment, #climate, #climatechange, #carbonfootprint, #climatehoax, #climategate, #globalwarming, #agw, #renewables	Feb 25-Mar 4, 2019	1,284,902
Gun control/rights	#gun, #guns, #weapon, #2a, #gunviolence, #secondamendment, #shooting, #massshooting, #gunrights, #GunReformNow, #GunControl, #NRA	Feb 25-Mar 3, 2019	1,782,384
Ilhan Omar remarks on Israel lobby	IlhanOmarIsATrojanHorse, #ISTandWithILhan, #ilhan, #Antisemitism, #IlhanOmar, #IlhanMN, #RemoveIlhanOmar, #ByeILhan, #RashidaTlaib, #AIPAC, #EverydayIslamophobia, #Islamophobia, #Ilhan	Mar 1-9, 2019	2,556,871
Illegal immigration	#border, #immigration, #immigrant, #borderwall, #migrant, #migrants, #illegal, #aliens	Feb 25-Mar 4, 2019	2,341,316
Midterm	midterm, election, elections	Oct 25-27, 2018	520,614
Racism & police brutality	#blacklivesmatter, #bluelivesmatter, #KKK, #racism, #racist, #policebrutality, #excessiveforce, #StandYourGround, #ThinBlueLine	Feb 25-Mar 3, 2019	2,564,784
Kavanaugh Nomination	Kavanaugh, Ford, Supreme, judiciary, Blasey, Grassley, Hatch, Graham, Cornyn, Lee, Cruz, Sasse, Flake, Crapo, Tillis, Kennedy, Feinstein, Leahy, Durbin, Whitehouse, Klobuchar, Coons, Blumenthal, Hirono, Booker, Harris	Sept. 28-30, 2018 & Oct. 6-9, 2018	2,322,141
Vaccination benefits & dangers	#antivax, #vaxxing, #BigPharma, #antivaxxers, #measlesoutbreak, #Antivaccine, #VaccinesWork, #vaccine, #vaccines, #Antivaccine, #vaccinestudy, #antivaxx, #provaxx, #Vaccines-SaveLives, #ProVaccine, #VaxxWoke, #mykidmychoice	Mar 1-9, 2019	301,209
Table 1: Polarizing topics used in study.
Once we have expanded our sets of labeled users, we identify influencers that are most closely associated with each group using a modified version of the so-called valence score, which varies in value between  $-1$  and 1. If an influencer is being cited evenly between the groups, then it would be assigned a valence score close to zero. Conversely, if one group disproportionately cites an influencer compared to another group, then it would be assigned a score closer to  $-1$  or 1. We perform these steps for each of the given topics, and finally we summarize the stances across all topics. Below, we explain each of these steps in more detail.
4.1 Projection and Clustering
Given the tweets for each topic, we compute the similarity between the top 1,000 most active users. To compute similarity, we construct a vector for each user containing the number of all the accounts that a user has retweeted, and then we compute the pairwise cosine similarity between them. For example, if user A has only retweeted user B 3 times, user C 5 times and user E 8 times, then user A's vector would be  $(0,3,5,0,8,0,0,\dots 0)$ . Solely using the retweeted accounts as features has been shown to be effective for stance classification (Darwish et al., 2020; Magdy et al., 2016a). Finally, we perform dimensionality reduction and we project the users using Uniform Manifold Approximation and Projection (UMAP). When performing dimensionality reduction, UMAP places users on a two-dimensional plane such that similar users are placed closer together and dissimilar users are pushed further apart. Figure 2 shows the top users for the "midterm" topic projected with UMAP onto the 2D plane. After the projection, we use Mean Shift to cluster the users as shown in Figure 2. This is the best setup described in (Darwish et al., 2020).
Figure 2: Top active users on the midterm topic clustered using UMAP + Mean Shift.
Clustering high-dimensional data often yields suboptimal results, but can be improved by projecting to a low-dimensional space (Darwish et al., 2020).
4.2 Supervised Classification
Since unsupervised stance detection is only able to classify the most vocal users, which only constitute a minority of the users, we wanted to assign stance labels to as many additional users as we can. Given the clusters of users that we obtain for each topic, we retain the two largest clusters for each topic, and we assign cluster labels to the users contained therein. Next, we use all the automatically labeled users for each topic to train a supervised classifier using the accounts that each user retweeted as features (same as the features we used to compute user similarity earlier). For classification, we train a FastText model using the default parameters, and then we classify all other users with five or more retweeted accounts, only accepting the classification if FastText was more than  $80\%$  confident  $(70 - 90\%)$  yielded nearly identical results.
Topic	No. of Users	Clustered Users	Classified Users
climate change	724,470	860	5,851
gun control	973,206	813	11,281
Ilhan Omar	563,706	723	25,484
immigration	940,840	901	22,456
midterm elections	312,954	860	12,765
police brutality & racism	1,175,081	891	18,978
Kavanaugh	809,835	891	10,100
vaccine	194,245	545	556
Table 2: Users per topic: total number of users, number of clustered users, and number of automatically labeled users.
In order to obtain a rough estimate of the accuracy of the model, we trained FastText using a random  $80\%$  subset of the clustered users for each topic and we tested on the remaining  $20\%$ . The accuracy was consistently above  $95\%$  for all topics. This does not mean that this model can predict the stance for all users that accurately — the clustered users were selected to be the most active ones. Rather, it shows that the classifier can successfully capture what the previous, unsupervised step has already learned. Table 2 lists the total number of users who authored the tweets for each topic, the number of users who were automatically clustered using the aforementioned unsupervised clustering technique, and the number of users who were automatically labeled afterwards using supervised classification. Given that we applied unsupervised stance detection to the most active 1,000 users, the majority of the users appeared in the largest two clusters (shown in Table 2).
4.3 Calculating Valence Scores
Given all the labeled users for each topic, we computed a valence score for each influencer. As mentioned earlier, the valence score ranges between  $[-1, 1]$ , where a value close to 1 implies it is strongly associated with one group of users,  $-1$  shows it is strongly associated with the other group of users, and 0 means that it is being shared or cited by both groups. The original valence score described by Conover et al. (2011a) is calculated as follows:
$$
V (u) = 2 \frac {\frac {t f (u , C _ {0})}{\text {t o t a l} (C _ {0})}}{\frac {t f (u , C _ {0})}{\text {t o t a l} (C _ {0})} + \frac {t f (u , C _ {1})}{\text {t o t a l} (C _ {1})}} - 1 \tag {1}
$$
where  $tf(u, C_0)$  is the number of times (term frequency) item  $u$  is cited by group  $C_0$ , and  $total(C_0)$  is the sum of the term frequencies of all items cited by  $C_0$ .  $tf(u, C_1)$  and  $total(C_1)$  are defined in a similar fashion.
We use the above equation to compute valence scores for the retweeted accounts, but we using a modified version for calculating the score for influencers  $(I)$ :
$$
V (I) = 2 \frac {\frac {t f (I , C _ {0})}{\text {t o t a l} (C _ {0})}}{\frac {t f (I , C _ {0})}{\text {t o t a l} (C _ {0})} + \frac {t f (I , C _ {1})}{\text {t o t a l} (C _ {1})}} - 1 \tag {2}
$$
$$
\begin{array}{l} t f (I, C _ {i}) = \sum_ {a \in I \cap C _ {i}} [ \ln (C n t (a, C _ {i})) + 1 ] \\ t o t a l (C _ {i}) = \sum_ {I} t f (I, C _ {i}) \\ \end{array}
$$
In the latter equation,  $Cnt(a,C_i)$  is the number of times article  $a$  was cited by users from cluster  $C_i$ . In essence, we are replacing term frequencies with the natural log of the term frequencies. We opted to modify the equation in order to tackle the following issue: if users from one of the clusters, say  $C_1$ , cite only one single article from some media source a large number of times (e.g., 2,000 times), while users from the other cluster  $(C_0)$  cite 10 other articles from the same media 50 times each, then using equation 1 would result in a valence score of -0.6. We would then regard the given media as having an opposing stance to the stance of users in  $C_0$ . Alternatively, using the natural log would lead to a valence score close to 0.88. Thus, dampening term frequencies using the natural log has the desired effect of balancing between the number of articles being cited by each group and the total number of citations. We bin the valence scores between -1 and 1 into five equal size bands as follows:
$$
C a t (V) = \left\{ \begin{array}{l l} - -, & \text {i f} s \in [ - 1, - 0. 6) \\ -, & \text {i f} s \in [ - 0. 6, - 0. 2) \\ 0, & \text {i f} s \in [ - 0. 2, 0. 2) \\ +, & \text {i f} s \in [ 0. 2, 0. 6) \\ + +, & \text {i f} s \in [ 0. 6, 1 ] \end{array} \right. \tag {3}
$$
5 Characterizing the Influencers
We use valence to characterize the leaning of all cited influencers for each of the topics. Table 3 shows the valence categories for the top-cited media sources across all topics. It also shows each media's factuality of reporting, i.e., trustworthiness, and bias (ranging from far-left to far-right) as determined by mediaBiasFactCheck.com. Since the choice of which cluster should be  $C_0$  and which would be  $C_1$  is arbitrary, we can multiply by  $-1$  the valence scores for any topic and the meaning of the results would stay the same.
Figure 3: Valence category vs. bias: number of media.
We resorted to doing so for some topics in order to align the extreme valence bands across all topics. Given tweet samples from users in a given cluster for a given topic, labeling that cluster manually was straightforward with almost no ambiguity. Table 4 shows the most frequently cited media source for each topic and for each valence band.
Of the 5,406 unique media sources that have been cited in tweets across all topics, 806 have known political bias from mediaBiasFactCheck. com. Figure 3 shows the confusion matrix between our valence categories and the goold labels from mediaBiasFactCheck.com.
We notice that many of the media that have a negative valence score (categories - and --) are classified on the right side of the political spectrum by mediaBiasFactCheck.com, while most media with positive scores (categories + and ++) are classified as slightly left-leaning. Although there are almost no extreme-left cases, there is a correlation between bias and our valence score. mediaBiasFactCheck.com seems to rarely categorize media sources as "extreme-left". This could be a reflection of reality or it might imply that mediaBiasFactCheck.com has an inherent bias.
We also computed the valence scores for the top-200 retweeted accounts, and we assigned each account a valence category based on the score. Independently, we asked a person who is well-versed with US politics to label all the accounts as left, center, or right. When labeling accounts, right-leaning include those expressing support for Trump, the Republican party, and gun rights, opposition to abortion, and disdain for Democrats.
Figure 4: The top-200 retweeted accounts, projected on a number line according to their average valence.