SlowGuess/ACL_2020_OCR huggingface dataset:
The results in rows 3-5 show that averaging embeddings from a fine-tuned BERT to encode articles (row 4) works better than using NELA features (row 3). They also show that using the posterior probabilities obtained from applying a softmax on top of BERT's [CLS] token (row 5) performs worse than using average embeddings (row 4). This suggests that it is better to incorporate information from the articles' word representations rather than using [CLS] as a compact representation of the articles. Also, since our BERT was fine-tuned on articles with noisy labels obtained using distant supervision, its predictions for individual articles are also noisy, and so are the vectors of posterior. Yet, this fine-tuning seems to yield improved article-level representations for our task.
The results in rows 7-10 show that captions are the most useful type of feature among those extracted from YouTube. This makes sense since captions contain the most essential information about the contents of a video. We can further see that the BERT-based features outperform the NELA ones. Overall, the YouTube features are underperforming since for half of the media we could not find a corresponding YouTube channel, and we used representations containing only zeroes.
Rows 11-16 show the results for systems that combine article, Twitter, and YouTube features, either directly or in an ensemble. We can see on rows 13-16 that the YouTube and the Twitter profile features yield loss in performance when added to the article features (rows 11-12). Note that the article features already outperform the individual feature types from rows 3-10 by a wide margin, and thus we will use them to represent the What Was Written aspect of the model in our later experiments below.
7The data and the corresponding code, both old and new, are available at https://github.com/ramybaly/News-Media-Reliability
<sup>8</sup>We ignored mixed as it does not apply to articles.
Group	#	Features	Dim.	Macro F1	Accuracy
Baselines	1	Majority class	-	19.18	40.39
2	Best model from (Baly et al., 2018a)	764	72.90	73.61
A. What Was Written	3	Articles: NELA	141	64.82	68.18
4	Articles: BERT representations	768	79.34	79.75
5	Articles: BERT probabilities	3	61.21	62.27
6	Twitter Profiles: Sentence BERT	768	59.23	60.88
7	YouTube: NELA (title, description)	260	45.78	50.46
8	YouTube: OpenSmile (LLDs)	385	46.13	50.69
9	YouTube: BERT (title, description, tags)	768	48.36	53.94
10	YouTube: BERT (captions)	768	49.14	53.94
11	Articles: ALL (c)	912	81.00	81.48
12	Articles: ALL (en)	912	81.27	81.83
13	Articles + Twitter Prof. (c)	1,691	76.59	77.20
14	Articles + Twitter Prof. (en)	1,691	80.00	80.56
15	Articles + Twitter Prof. + YouTube cap. (c)	2,315	75.73	76.39
16	Articles + Twitter Prof. + YouTube cap. (en)	2,315	79.70	80.32
B. Who Read It	17	Twitter Follower: Sentence BERT	768	62.85	65.39
18	YouTube: Metadata	5	40.05	46.53
19	Facebook: Political Leaning Estimates	6	27.87	43.87
20	Twitter Fol. + YouTube Meta. (c)	773	63.72	65.86
21	Twitter Fol. + YouTube Meta. (en)	773	65.12	66.44
22	Twitter Fol. + YouTube Meta. + Facebook Estimates (c)	779	63.63	65.74
23	Twitter Fol. + YouTube Meta. + Facebook Estimates (en)	779	64.18	66.20
C. What Was Written About the Medium	24	Wikipedia: BERT	768	64.36	66.09
Combinations	25	All features: rows 3–11; 18–20; 25 (c)	5,413	78.17	78.70
26	All features: rows 3–11; 18–20; 25 (en)	5,413	79.42	80.32
27	A+B: rows 12 & 21 (c)	1,685	84.28	84.87
28	A+B: rows 12 & 21 (en)	1,685	84.15	84.64
29	A+C: rows 12 & 24 (c)	1,680	81.53	81.98
30	A+C: rows 12 & 24 (en)	1,680	82.99	83.48
31	A+B+C: rows 12, 21 & 24 (c)	1,691	83.53	84.02
32	A+B+C: rows 12, 21 & 24 (en)	1,691	84.77	85.29
Table 2: Political bias prediction: ablation study of the proposed features. Dim refers to the number of features, whereas  $(c)$  and  $(en)$  indicate whether the features are concatenated or an ensemble was used, respectively.
We can further notice that the ensembles consistently outperform feature concatenation models, which is actually true for all feature combinations in Table 2.
Next, we compare rows 6 and 17, which show results when using Twitter information of different nature: from the target medium profile (row 6) vs. from the profiles of the followers of the target medium (row 17). We can see that the latter is much more useful, which confirms the importance of the Who Read It aspect, which we have introduced in this paper. Note that here we encode the descriptions and the self-description bio information using Sentence BERT instead of the pre-trained BERT; this is because, in our preliminary experiments (not shown in the table), we found the former to perform much better than the latter.
Next, the results in rows 20-23 show that the YouTube metadata features improve the performance when combined with the Twitter followers' features. On the other hand, the Facebook audience features' performance is deficient and hurts the overall performance, i.e., these estimates seem not to correlate well with the political leanings of news media. Also, as pointed by (Flaxman et al., 2016), social networks can help expose people to different views, and thus the polarization in news readership might not be preserved.
Row 24 shows that the Wikipedia features perform worse than most individual features above, which can be related to coverage as only  $61.2\%$  of the media in our dataset have a Wikipedia page. Nevertheless, these features are helpful when combined with features about other aspects; see below.
Group	#	Features	Dim.	Macro F1	Accuracy
Baselines	1	Majority class	-	22.93	52.43
2	Best model from (Baly et al., 2018a)	764	61.08	66.45
A. What Was Written	3	Articles: NELA	141	55.54	62.62
4	Articles: BERT representations	768	61.46	67.94
5	Articles: BERT probabilities	3	51.39	61.46
6	Twitter Profiles: Sentence BERT	768	49.96	56.71
7	YouTube: NELA (title, description)	260	32.52	51.04
8	YouTube: OpenSmile (LLDs)	385	37.17	52.08
9	YouTube: BERT (title, description, tags)	768	38.19	54.28
10	YouTube: BERT (captions)	768	38.82	55.56
11	Articles: ALL (c)	912	59.34	64.82
12	Articles: ALL (en)	912	48.27	59.95
13	Articles: BERT + Twitter Prof. (c)	1,691	61.06	66.09
14	Articles: BERT + Twitter Prof. (en)	1,691	61.50	68.63
15	Articles: BERT + Twitter Prof. + YouTube: cap. (c)	2,315	60.23	65.51
16	Articles: BERT + Twitter Prof. + YouTube: cap. (en)	2,315	58.21	66.44
B. Who Read It	17	Twitter Follower: Sentence BERT	768	42.19	58.45
18	YouTube: Metadata	5	31.92	52.78
19	Facebook: Political Leaning Estimates	6	27.24	53.70
20	Twitter Fol. + YouTube Meta. (c)	773	42.48	58.76
21	Twitter Fol. + YouTube Meta. (en)	773	39.66	57.64
22	Twitter Fol. + YouTube Meta. + Facebook Estimates (c)	779	42.28	57.76
23	Twitter Fol. + YouTube Meta. + Facebook Estimates (en)	779	39.33	57.99
C. What Was Written About the Medium	24	Wikipedia: BERT	768	45.74	55.32
Combinations	25	All features: rows 3–10; 17–19; 24 (c)	5,413	62.42	67.79
26	All features: rows 3–10; 17–19; 24 (en)	5,413	45.24	60.42
27	A+B: rows 14 & 24 (c)	1,680	65.45	70.40
28	A+B: rows 14 & 24 (en)	1,680	61.80	69.25
29	A+C: rows 14 & 20 (c)	1,685	67.25	71.52
30	A+C: rows 14 & 20 (en)	1,685	62.53	69.90
31	A+B+C: rows 14, 20 & 24 (c)	1,691	64.14	69.36
32	A+B+C: rows 14, 20 & 24 (en)	1,691	60.35	68.90
Table 3: Factuality of reporting: ablation study of the proposed features. Dim refers to the number of features, whereas  $(c)$  and  $(en)$  indicate whether the features are concatenated or an ensemble was used, respectively.
Finally, rows 25-32 in Table 3 show the evaluation results when combining all aspects. We can see that the best results are achieved when using the best features from each of the three aspects, where the combination is performed as an ensemble (row 32). This combination improves over using information from the article only (row 12) by  $+3.5$  macro- $F_{1}$  points absolute. It further yields sizeable absolute improvements over the baseline system from (Baly et al., 2018a), by  $+11.87$  macro- $F_{1}$  points absolute. While this improvement is due to a large extent to improved techniques for text representation such as using fine-tuned BERT instead of averaged GloVe word embeddings, modeling the newly-introduced media aspects further yielded a lot of additional improvements.
4.4 Factuality Prediction
Table 3 demonstrates the evaluation results when using the proposed sources/features for the task of predicting the factuality of reporting of news media.
Similarly to the results for political bias prediction, rows 3-10 suggest that the features extracted from articles are more important than those coming from YouTube or from Twitter profiles, and that using BERT to encode the articles yields the best results. Note that overall, the results in this table are not as high as those for bias prediction. This reflects the level of difficulty of this task, and the fact that, in order to predict factuality, one needs external information or a knowledge base to be able to verify the published content.
The results in rows 11-16 show that combining the Twitter profile features with the BERT-encoded articles improves the performance over using the article text only.
Comparing rows 6 and 17 in Table 3, we can see that the Twitter follower features perform worse than using Twitter profiles features; this is the opposite of what we observed in Table 2. This makes sense since our main motivation to look at the followers' profiles was to detect political bias, rather than factuality. Moreover, the metadata collected from media profiles about whether the corresponding account is verified, or its level of activity or connectivity (counts of friends and statuses) are stronger signals for this task.
Finally, rows 25-32 show the results for modeling combinations of the three aspects we are exploring in this paper. The best results are achieved using the best features selected from the What was written and the What was written about the target medium aspects, concatenated together. This combination achieves sizeable improvements compared to the baseline system from (Baly et al., 2018a): by  $+6.17$  macro- $F_{1}$  points absolute. This result indicates that looking at the audience of the medium is not as helpful for predicting factuality as it was for predicting political bias, and that looking at what was written about the medium on Wikipedia is more important for this task.
We have presented experiments in predicting the political ideology, i.e., left/center/right bias, and the factuality of reporting, i.e., high/mixed/low, of news media. We compared the textual content of what media publish vs. who read it on social media, i.e., on Twitter, Facebook, and YouTube. We further modeled what was written about the target medium in Wikipedia.
We have combined a variety of information sources, many of which were not explored for at least one of the target tasks, e.g., YouTube channels, political bias of the Facebook audience, and information from the profiles of the media followers on Twitter. We further modeled different modalities: text, metadata, and speech signal. The evaluation results have shown that while what was written matters most, the social media context is also important as it is complementary, and putting them all together yields sizable improvements over the state of the art.
In future work, we plan to perform user profiling with respect to polarizing topics such as gun control (Darwish et al., 2020), which can then be propagated from users to media (Atanasov et al., 2019; Stefanov et al., 2020). We further want to model the network structure, e.g., using graph embeddings (Darwish et al., 2020). Another research direction is to profile media based on their stance with respect to previously fact-checked claims (Mohtarami et al., 2018; Shaar et al., 2020), or by the proportion and type of propaganda techniques they use (Da San Martino et al., 2019, 2020). Finally, we plan to experiment with other languages.
This research is part of the Tanbih project<sup>9</sup>, which aims to limit the effect of "fake news," propaganda and media bias by making users aware of what they are reading. The project is developed in collaboration between the Qatar Computing Research Institute, HBKU and the MIT Computer Science and Artificial Intelligence Laboratory.
Ashutosh Adhikari, Achyudh Ram, Raphael Tang, and Jimmy Lin. 2019. DocBERT: BERT for document classification. arXiv preprint arXiv:1904.08398.
Matheus Araujo, Yelena Mejova, Ingmar Weber, and Fabricio Benevenuto. 2017. Using Facebook ads audiences for global lifestyle disease surveillance: Promises and limitations. In Proceedings of the 2017 ACM Conference on Web Science, WebSci '17, pages 253-257, Troy, NY, USA.
Atanas Atanasov, Gianmarco De Francisci Morales, and Preslav Nakov. 2019. Predicting the role of political trolls in social media. In Proceedings of the 2019 SIGNLL Conference on Computational Natural Language Learning, CoNLL '19, pages 1023-1034, Hong Kong, China.
Ramy Baly, Georgi Karadzhov, Dimitar Alexandrov, James Glass, and Preslav Nakov. 2018a. Predicting factuality of reporting and bias of news media sources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP '18, pages 3528-3539, Brussels, Belgium.
Ramy Baly, Georgi Karadzhov, Abdelhman Saleh, James Glass, and Preslav Nakov. 2019. Multi-task ordinal regression for jointly predicting the trustworthiness and the leading political ideology of news
9http://tanbih.qcri.org/
media. In Proceedings of the 17th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT '19, pages 2109-2116, Minneapolis, MN, USA.
Ramy Baly, Mitra Mohtarami, James Glass, Lluis Marquez, Alessandro Moschitti, and Preslav Nakov. 2018b. Integrating stance detection and fact checking in a unified corpus. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT '18, pages 21-27, New Orleans, LA, USA.
Ceren Budak, Sharad Goel, and Justin M Rao. 2016. Fair and balanced? Quantifying media bias through crowdsourced content analysis. *Public Opinion Quarterly*, 80(S1):250-271.
Kevin R. Canini, Bongwon Suh, and Peter L. Pirolli. 2011. Finding credible information sources in social networks based on content and social structure. In Proceedings of the IEEE International Conference on Privacy, Security, Risk, and Trust, and the IEEE International Conference on Social Computing, SocialCom/PASSAT '11, pages 1-8, Boston, MA, USA.
Carlos Castillo, Marcelo Mendoza, and Barbara Poblete. 2011. Information credibility on Twitter. In Proceedings of the 20th International Conference on World Wide Web, WWW '11, pages 675-684, Hyderabad, India.
Giovanni Da San Martino, Stefano Cresci, Alberto Barrón-Cedeno, Seunghak Yu, Roberto Di Pietro, and Preslav Nakov. 2020. A survey on computational propaganda detection. In Proceedings of the 29th International Joint Conference on Artificial Intelligence and the 17th Pacific Rim International Conference on Artificial Intelligence, IJCAI-PRICAI '20, Yokohama, Japan.
Giovanni Da San Martino, Seunghak Yu, Alberto Barron-Cedeno, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news articles. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, EMNLP '19, pages 5636-5646, Hong Kong, China.
Kareem Darwish, Michael Aupetit, Peter Stefanov, and Preslav Nakov. 2020. Unsupervised user stance detection on twitter. In Proceedings of the International AAAI Conference on Web and Social Media, ICWSM '20, Atlanta, GA, USA.
Stefano DellaVigna and Ethan Kaplan. 2007. The Fox News effect: Media bias and voting. *The Quarterly Journal of Economics*, 122(3):1187-1234.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of
the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT '19, pages 4171-4186, Minneapolis, MN, USA.
Yoan Dinkov, Ahmed Ali, Ivan Koychev, and Preslav Nakov. 2019. Predicting the leading political ideology of Youtube channels using acoustic, textual and metadata information. In Proceedings of the 20th Annual Conference of the International Speech Communication Association, INTERSPEECH '19, pages 501-505, Graz, Austria.
Erick Elejalde, Leo Ferres, and Eelco Herder. 2018. On the nature of real and perceived bias in the mainstream media. *PloS one*, 13(3):e0193765.
Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010. openSMILE - the Munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM International Conference on Multimedia, pages 1459-1462, Florence, Italy.
Masoomali Fatehkia, Ridhi Kashyap, and Ingmar Weber. 2018. Using Facebook ad data to track the global digital gender gap. World Development, 107:189-209.
Seth Flaxman, Sharad Goel, and Justin M Rao. 2016. Filter bubbles, echo chambers, and online news consumption. *Public opinion quarterly*, 80(S1):298-320.
Matthew Gentzkow and Jesse M Shapiro. 2006. Media bias and reputation. Journal of political Economy, 114(2):280-316.
Doris A Graber and Johanna Dunaway. 2017. Mass media and American politics. SAGE Publications.
Benjamin D. Horne, William Dron, Sara Khedr, and Sibel Adali. 2018a. Assessing the news landscape: A multi-module toolkit for evaluating the credibility of news. In Proceedings of the The Web Conference, WWW '18, pages 235-238, Lyon, France.
Benjamin D. Horne, Sara Khedr, and Sibel Adali. 2018b. Sampling the news producers: A large news and feature data set for the study of the complex media landscape. In Proceedings of the Twelfth International Conference on Web and Social Media, ICwsm '18, pages 518-527, Stanford, CA, USA.
Shanto Iyengar and Kyu S Hahn. 2009. Red media, blue media: Evidence of ideological selectivity in media use. Journal of communication, 59(1):19-39.
Daniel Kopev, Ahmed Ali, Ivan Koychev, and Preslav Nakov. 2019. Detecting deception in political debates using acoustic and textual features. In Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop, ASRU '19, pages 652-659, Singapore.
Vivek Kulkarni, Junting Ye, Steven Skiena, and William Yang Wang. 2018. Multi-view models for political ideology detection of news articles. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP '18, pages 3518-3527, Brussels, Belgium.
Jing Ma, Wei Gao, Prasenjit Mitra, Sejeong Kwon, Bernard J. Jansen, Kam-Fai Wong, and Meeyoung Cha. 2016. Detecting rumors from microblogs with recurrent neural networks. In Proceedings of the 25th International Joint Conference on Artificial Intelligence, IJCAI '16, pages 3818-3824, New York, NY, USA.
Jing Ma, Wei Gao, Zhongyu Wei, Yueming Lu, and Kam-Fai Wong. 2015. Detect rumors using time series of social context information on microblogging websites. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, CIKM '15, pages 1751-1754, Melbourne, Australia.
Mitra Mohtarami, Ramy Baly, James Glass, Preslav Nakov, Lluis Márquez, and Alessandro Moschitti. 2018. Automatic stance detection using end-to-end memory networks. In Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT '18, pages 767-776, New Orleans, LA, USA.
Subhabrata Mukherjee and Gerhard Weikum. 2015. Leveraging joint interactions for credibility analysis in news communities. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, CIKM '15, pages 353-362, Melbourne, Australia.
Jeppe Nørregaard, Benjamin D. Horne, and Sibel Adah. 2019. NELA-GT-2018: A large multi-labelled news dataset for the study of misinformation in news articles. In Proceedings of the International AAAAI Conference on Web and Social Media, ICWSM '19, pages 630-638, Munich, Germany.
Kashyap Popat, Subhabrata Mukherjee, Jannik Strötgen, and Gerhard Weikum. 2017. Where the truth lies: Explaining the credibility of emerging claims on the Web and social media. In Proceedings of the 26th International Conference on World Wide Web Companion, WWW '17, pages 1003-1012, Perth, Australia.
Kashyap Popat, Subhabrata Mukherjee, Jannik Strötgen, and Gerhard Weikum. 2018. CredEye: A credibility lens for analyzing and explaining misinformation. In Proceedings of The Web Conference 2018, WWW '18, pages 155-158, Lyon, France.
Martin Potthast, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, and Benno Stein. 2018. A stylistic inquiry into hyperpartisan and fake news. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL '18, pages 231-240, Melbourne, Australia.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP '19, pages 3982-3992, Hong Kong, China.
Filipe N Ribeiro, Lucas Henrique, Fabricio Benevenuto, Abhijnan Chakraborty, Juhi Kulshrestha, Mahmoudreza Babaei, and Krishna P Gummadi. 2018. Media bias monitor: Quantifying biases of social media news outlets at large-scale. In Proceedings of the Twelfth International AAAI Conference on Web and Social Media, ICWSM '18, pages 290-299, Stanford, CA, USA.
Abdelrhman Saleh, Remy Baly, Alberto Barron-Cedeno, Giovanni Da San Martino, Mitra Mohtarami, Preslav Nakov, and James Glass. 2019. Team QCRI-MIT at SemEval-2019 task 4: Propaganda analysis meets hyperpartisan news detection. In Proceedings of the 13th International Workshop on Semantic Evaluation, SemEval '19, pages 1041-1046, Minneapolis, MN, USA.
Björn Schuller, Stefan Steidl, and Anton Batliner. 2009. The INTERSPEECH 2009 emotion challenge. In Proceedings of the 10th Annual Conference of the International Speech Communication Association, IN-TERSPEECH '09, pages 312-315, Brighton, UK.
Shaden Shaar, Giovanni Da San Martino, Nikolay Babulkov, and Preslav Nakov. 2020. That is a known lie: Detecting previously fact-checked claims. In Proceedings of the Annual Conference of the Association for Computational Linguistics, ACL '20, Seattle, WA, USA.
Petar Stefanov, Kareem Darwish, Atanas Atanasov, and Preslav Nakov. 2020. Predicting the topical stance and political leaning of media using tweets. In Proceedings of the Annual Conference of the Association for Computational Linguistics, ACL '20, Seattle, WA, USA.
Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online. Science, 359(6380):1146-1151.
Felix Ming Fai Wong, Chee Wei Tan, Soumya Sen, and Mung Chiang. 2013. Quantifying political leaning from tweets and retweets. In Proceedings of the Seventh International AAAI Conference on Weblogs and Social Media, ICWSM '13, pages 640-649, Boston, MA, USA.
Tauhid Zaman, Emily B. Fox, and Eric T. Bradlow. 2014. A bayesian approach for predicting the popularity of tweets. Ann. Appl. Stat., 8(3):1583-1611.
Arkaitz Zubiaga, Maria Liakata, Rob Procter, Geraldine Wong Sak Ho, and Peter Tolmie. 2016. Analysing how people orient to and spread rumours in social media by looking at conversational threads. PLoS ONE, 11(3):1-29.