Venue: NAACL 2019

# Multi-Task Ordinal Regression for Jointly Predicting

the Trustworthiness and the Leading Political Ideology of News Media

 Ramy Baly\({}^{1}\), Georgi Karadzhov\({}^{2}\), Abdelrhman Saleh\({}^{3}\), James Glass\({}^{1}\), Preslav Nakov\({}^{4}\)

###### Abstract

In the context of fake news, bias, and propaganda, we study two important but relatively under-explored problems: (_i_) _trustworthiness estimation_ (on a 3-point scale) and (_ii_) _political ideology detection_ (left/right bias on a 7-point scale) of entire news outlets, as opposed to evaluating individual articles. In particular, we propose a multi-task ordinal regression framework that models the two problems jointly. This is motivated by the observation that hyper-partisanship is often linked to low trustworthiness, e.g., appealing to emotions rather than sticking to the facts, while center media tend to be generally more impartial and trustworthy. We further use several auxiliary tasks, modeling centrality, hyper-partisanship, as well as left-vs-right bias on a coarse-grained scale. The evaluation results show sizable performance gains by the joint models over models that target the problems in isolation.

+
Footnote †: Minneapolis, Minnesota, June 2 - November 6, 2021

## 1 Introduction

Recent years have seen the rise of social media, which has enabled people to virtually share information with a large number of users without regulation or quality control. On the bright side, this has given an opportunity for anyone to become a content creator, and has also enabled a much faster information dissemination. However, it has also opened the door for malicious users to spread disinformation and misinformation much faster, enabling them to easily reach audience at a scale that was never possible before. In some cases, this involved building sophisticated profiles for individuals based on a combination of psychological characteristics, meta-data, demographics, and location, and then micro-targeting them with personalized "fake news" with the aim of achieving some political or financial gains Lazer et al. (2018); Vosoughi et al. (2018).

A number of fact-checking initiatives have been launched so far, both manual and automatic, but the whole enterprise remains in a state of crisis: by the time a claim is finally fact-checked, it could have reached millions of users, and the harm caused could hardly be undone. An arguably more promising direction is to focus on fact-checking entire news outlets, which can be done in advance. Then, we could fact-check the news before they were even written: by checking how trustworthy the outlets that published them are. Knowing the reliability of a medium is important not only when fact-checking a claim Popat et al. (2017); Nguyen et al. (2018), but also when solving article-level tasks such as "fake news" and click-bait detection Brill (2001); Finberg et al. (2002); Hardalov et al. (2016); Karadzhov et al. (2017); De Sarkar et al. (2018); Pan et al. (2018); Perez-Rosas et al. (2018)

Political ideology (or left/right bias) is a related characteristic, e.g., extreme left/right media tend to be propagandistic, while center media are more factual, and thus generally more trustworthy. This connection can be clearly seen in Figure 1.

Figure 1: Correlation between bias and factuality for the news outlets in the Media Bias/Fact Check website.



Despite the connection between factuality and bias, previous research has addressed them as independent tasks, even when the underlying dataset had annotations for both [1]. In contrast, here we solve them jointly. Our contributions can be summarized as follows:

* We study an under-explored but arguably important problem: predicting the factuality of reporting of news media. Moreover, unlike previous work, we do this jointly with the task of predicting political bias.
* As factuality and bias are naturally defined on an ordinal scale (factuality: from _low_ to _high_, and bias: from _extreme-left_ to _extreme-right_), we address them as ordinal regression. Using multi-task ordinal regression is novel for these tasks, and it is also an under-explored direction in machine learning in general.
* We design a variety of auxiliary subtasks from the bias labels: modeling centrality, hyper-partisanship, as well as left-vs.-right bias on a coarse-grained scale.

## 2 Related Work

Factuality of ReportingPrevious work has modeled the factuality of reporting at the medium level by checking the general stance of the target medium with respect to known manually fact-checked claims, without access to gold labels about the overall medium-level factuality of reporting [35, 46, 207, 18].

The trustworthiness of Web sources has also been studied from a Data Analytics perspective, e.g., Dong et al. dongEtAl:2015 proposed that a trustworthy source is one that contains very few false claims. In social media, there has been research targeting the user, e.g., finding malicious users [9, 10, 11], _sockuppets_[10], _Internet water army_[12], and _seminar users_[1].

Unlike the above work, here we study source reliability as a task in its own right, using manual gold annotations specific for the task and assigned by independent fact-checking journalists. Moreover, we address the problem as one of ordinal regression on a three-point scale, and we solve it jointly with political ideology prediction in a multi-task learning setup, using several auxiliary tasks.

Predicting Political IdeologyIn previous work, political ideology, also known as media bias, was used as a feature for "fake news" detection [1]. It has also been the target of classification, e.g., Horne et al. horneEtAl:2018 predicted whether an article is biased (_political_ or _bias_) vs. unbiased. Similarly, Potthast et al. potthastEtAl:2018 classified the bias in a target article as (\(i\)) left vs. right vs. mainstream, or as (\(ii\)) hyper-partisan vs. mainstream. Left-vs-right bias classification at the article level was also explored by Kulkarni et al. kulkarni2018, who modeled both the textual and the URL contents of the target article. There has been also work targeting bias at the phrase or the sentence level [10], focusing on political speeches [21] or legislative documents [11], or targeting users in Twitter [14]. Another line of related work focuses on propaganda, which can be seen as a form of extreme bias [15, 16]. See also a recent position paper [22] and an overview paper on bias on the Web [1]. Unlike the above work, here we focus on predicting the political ideology of news media outlets.

In our previous work [1], we did target the political bias of entire news outlets, as opposed to working at the article level (we also modeled factuality of reporting, but as a separate task without trying multi-task learning). In addition to the text of the articles published by the target news medium, we used features extracted from its corresponding Wikipedia page and Twitter profile, as well as analysis of its URL structure and traffic information about it from Alexa rank. In the present work, we use a similar set of features, but we treat the problem as one of ordinal regression. Moreover, we model the political ideology and the factuality of reporting jointly in a multi-task learning setup, using several auxiliary tasks.

Multitask Ordinal Regression_Ordinal regression_is well-studied and is commonly used for text classification on an ordinal scale, e.g., for sentiment analysis on a 5-point scale [1, 10]. However, _multi-task ordinal regression_ remains an understudied problem.

Yu et al. yu 

Walecki et al. (2016) were interested in jointly predicting facial action units and their intensity level. They argued that, due to the high number of classes, modeling these tasks independently would be inefficient. Thus, they proposed the _copula ordinal regression_ model for multi-task learning and demonstrated that it can outperform various single-task setups. We use this model in our experiments below.

Balikas et al. (2017) used multi-task ordinal regression for the task of fine-grained sentiment analysis. In particular, they introduced an auxiliary coarse-grained task on a 3-point scale, and demonstrated that it can improve the results for sentiment analysis on the original 5-point scale. Inspired by this, below we experiment with different granularity for political bias; however, we explore a larger space of possible auxiliary tasks.

## 3 Method

Copula Ordinal RegressionWe use the _Copula Ordinal Regression_ (COR) model, which was originally proposed by Walecki et al. (2016) to estimate the intensities of facial action units (AUs). The model uses copula functions and conditional random fields (CRFs) to approximates the learning of the joint probability distribution function (PDF) of the facial AUs (random variables), using the bivariate joint distributions capturing dependencies between AU pairs. It was motivated by the fact that (\(i\)) many facial AUs co-exist with different levels of intensity, (\(ii\)) some AUs co-occur more often than others, and (\(iii\)) some AUs depend on the intensity of other units.

We can draw an analogy between modeling facial AUs and modeling news media, where each medium expresses a particular bias (political ideology) and can also be associated with a particular level of factuality. Therefore, bias and factuality can be analogous to the facial AUs in (Walecki et al., 2016), and represent two aspects of news reporting, each being modeled on a multi-point ordinal scale. In particular, we model bias on a 7-point scale (_extreme-left_, _left_, _center-left_, _center_, _center-right_, _right_, _right_, and _extreme-right_), and factuality on a 3-point scale (_low_, _mixed_, and _high_).

In our case, we train the COR model to predict the joint PDF between political bias and factuality of reporting. This could potentially work well given the inherent inter-dependency between the two tasks as we have seen on Figure 1.

Auxiliary TasksWe use a variety of auxiliary tasks, derived from the bias labels. This includes converting the 7-point scale to (\(i\)) 5-point and 3-point scales, similarly to (Balikas et al., 2017), and to (\(ii\)) a 2-point scale in two ways to model extreme partisanship, and centrality. Here is the list of the auxiliary tasks we use with precise definition of the label mappings:

* **Bias5-way:** Predict bias on a 5-pt scale; 1:_extreme-left_, 2:_left_, 3:{_center-left_, _center, center-right_}, 4:_right_, and 5:_extreme-right_.
* **Bias3-way:** Predict bias on a 3-pt scale; 1:{_extreme-left_, _left_}, 2:{_center-left_, _center, center-right_}, and 3:{_right, extreme-right_}.
* **Bias-extreme:** Predict extreme vs. non-extreme partisanship on a 2-pt scale; 1:{_extreme-left_, _extreme-right_}, 2:{_left_, _center-left_, _center, center-right_}.
* **Bias-center:** Predict center vs. non-center political ideology on a 2-pt scale, ignoring polarity: 1:{_extreme-left_, _left_, _right_, _extremeight_}, 2:{_center-left_, _center, center-right_}.

FeaturesWe used the features from (Bay et al., 2018)1. We gathered a sample of articles from the target medium, and we calculated features such as POS tags, linguistic cues, sentiment scores, complexity, morality, as well as embeddings. We also used the Wikipedia page of the medium (if any) to generate document embedding. Then, we collected metadata from the medium's Twitter account (if any), e.g., whether is is verified, number of followers, whether the URL in the Twitter page matches the one of the medium. Finally, we added Web-based features that (\(i\)) model the orthographic structure of the medium's URL address, and (\(ii\)) analyze the Web-traffic information about the medium's website, as found in Alexa rank.2

Footnote 1: [https://github.com/ramybaly/News-Media](https://github.com/ramybaly/News-Media)–Reliability

Footnote 2: [https://www.olexa.com/siteinfo](https://www.olexa.com/siteinfo)

## 4 Experiments and Evaluation

DataWe used the MBFC dataset (Baly et al., 2018) that has 1,066 news media manually annotated for factuality (3-pt scale: _high_, _mixed_, _low_) and political bias (7-pt scale: from _extreme-left_ to _extreme-right_). This dataset was annotated by volunteers using a detailed methodology3 that is designed to guarantee annotation objectivity.

 

Furthermore, readers can provide their own feedback on existing annotations, and in case of a large discrepancy, annotation is adjusted after a thorough review. Therefore, we believe the annotation quality is good enough to experiment with. We noticed that 117 media had _low_ factuality because they publish _satire_ and _pseudo-science_, neither of which has a political perspective. Since we are interested in modeling the relation between factuality and bias, we excluded those websites, thus ending up with 949 news media. Some examples from this dataset are shown in Table 1 with both factuality and bias labels, in addition to their corresponding Twitter handles and Wikipedia pages. Overall, 64% of the media in our dataset have Wikipedia pages, and 65% have Twitter accounts. Table 2 further provides detailed statistics about the label distribution in the MBFC dataset.

Experimental SetupWe used the implementation4 of the Copula Ordinal Regression (COR) model as described in Walecki et al. (2016). In our experiments, we used 5-fold cross-validation, where for each fold we split the training dataset into a training part and a validation part, and we used the latter to fine-tune the model's hyper-parameters, optimizing for Mean Absolute Error (MAE). MAE is an appropriate evaluation measure given the ordinal nature of the tasks. These hyper-parameters include the copula function (_Gumbel_ vs. _Frank_), the marginal distribution (_normal_ vs. _sigmoid_), the number of training iterations, the optimizer (_gradient descent_, _BFGS_), and the connection density of the CRFs. We report both MAE and MAE\({}^{M}\), which is a variant of MAE that is more robust to class imbalance. See Baccianella et al. (2009); Rosenthal et al. (2017b) for more details about MAE\({}^{M}\) vs. MAE. We compare the results to two baselines: (_i_) majority class, and (_ii_) single-task ordinal regression.

Footnote 4: [https://github.com/RWalecki/copula_ordinal_regression](https://github.com/RWalecki/copula_ordinal_regression)

Results and DiscussionTable 3 shows the evaluation results for the COR model when trained to jointly model the main task (_shown in the columns_) using combinations of auxiliary tasks (_shown in the rows_). We can see that the single-task ordinal regression model performs much better than the majority class baseline based on both evaluation measures. We can further see that the performance on the main task improves when jointly modeling several auxiliary tasks. This improvement depends on the auxiliary tasks in use.

For factuality prediction, it turns out that the combination of _bias-center\(+\)bias-extreme_ yields the best overall MAE of 0.481. This makes sense and aligns well with the intuition that knowing whether a medium is centric or hyper-partisan is important to predict the factuality of its reporting. For instance, a news medium without a political ideology tends to be more trustworthy compared to an extremely biased one, regardless of their polarity (left or right), as we should expect based on the data distribution shown in Figure 1 above.

For bias prediction (at a 7-point left-to-right scale), a joint model that uses political bias at different levels of granularity (5-point and 3-point) as auxiliary tasks yields the best overall MAE of 1.479. This means that jointly modeling bias with the same information at coarser levels of granularity, i.e., adding 3-point and 5-point as auxiliary tasks, reduces the number of gross mistakes.

\begin{table}
\begin{tabular}{l|l|l|l l|l} \hline \hline
**Name** & **URL** & **Bias** & **Factuality** & **Twitter Handle** & **Wikipedia page** \\ \hline London Web News Daily Mirror NRC News & londonwebnewes.com & Extreme Left Left & Low & @londonwebnews N/A \\  & www.mirror.co.uk & Left & Mixed & @DailyMirror & -/Daily\_Mirror \\  & www.nbcnewes.com & Center-Left & High & @nbcnews & -/NBBC\_Newsws \\ Associated Press & apnews.com & Center & Very High & @apnews & -/Associated\_Press \\ Gulf News & quiffnews.com & Center-Right & High & @gift.news & -/Guil\_News \\ Russia Insider & russia-insider.com & Right & Mixed & @russiansidier & -/Russia\_Insider \\ Breitbar & www.breitbart.com & Extreme Right & Low & @Breitburst News & -/Breitbart\_News \\ \hline \hline \end{tabular}
\end{table}
Table 1: Examples of media and their labels for bias and factuality of reporting derived from MBFC 

E.g., predicting _extreme-left_ instead of _extremepright_, since the model is encouraged by the auxiliary tasks to learn the correct polarity, regardless of its intensity. We can see that _factuality_ is not very useful as an auxiliary task by itself (MAE=1.584 and MAE\({}^{M}\)=1.695). In other words, a medium with low factuality could be extremely biased to either the right or to the left. Therefore, relying on _fractality_ alone to predict bias might introduce severe errors, e.g., confusing extreme-left with extreme-right, thus leading to higher MAE scores. This can be remedied by adding _factuality_ to the mix of other auxiliary tasks to model the main task (7-point bias prediction). The results of these experiments, shown in parentheses in Table 3, indicate that adding _factuality_ to any combination of auxiliary tasks consistently yields lower MAE scores. In particular, modeling the combination of _factuality_+_bias5_-_way_+_bias3-way_ yields the best results (MAE=1.475 and MAE\({}^{M}\)=1.623). This result indicates that _factuality_ provides complementary information that can help predict bias.

We ran a two-tailed t-test for statistical significance, which is suitable for an evaluation measure such as MAE, to confirm the improvements that were introduced by the multi-task setup. We found that the best models (shown in bold in Table 3) outperformed both the corresponding majority class baselines with a p-value \(\leq\) 0.001, and the corresponding single-task ordinal regression baselines with a p-value \(\leq\) 0.02.

Finally, we compared the above results to our previous work [1] by independently training a Support Vector Machine (SVM) classifier for each task, using the same features. The resulting MAE was 0.450 for actuality and 1.184 for bias prediction, which is slightly better then our results (yet, very comparable for factuality). However, our goal here is to emphasize the advantages of modeling the two tasks jointly.

## 5 Conclusion and Future Work

We have presented a multi-task ordinal regression framework for jointly predicting trustworthiness and political ideology of news media sources, using several auxiliary tasks, e.g., based on a coarser-grained scales or modeling extreme partisanship. Overall, we have observed sizable performance gains in terms of reduced MAE by the multi-task ordinal regression models over single-task models for each of the two individual tasks.

In future work, we want to try more auxiliary tasks, and to experiment with other languages. We further plan to go beyond _left vs. right_, which is not universal and can exhibit regional specificity [21], and to model other kinds of biases, e.g., _eurosceptic vs. europhile_, _nationalist_ vs. _globalist_, _islamist vs. secular_, etc.

#### Acknowledgments

This research is part of the Tanbih project,5 which aims to limit the effect of "fake news", propaganda and media bias by making users aware of what they are reading. The project is developed in collaboration between the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) and the Qatar Computing Research Institute (QCRI), HBKU.

Footnote 5: [http://tanbih.qcri.org/](http://tanbih.qcri.org/)

\begin{table}
\begin{tabular}{l|c c|c c|c c} \hline \hline  & \multicolumn{4}{c}{**Factuality**} & \multicolumn{2}{c}{**Bias**} \\ \hline
**Auxiliary Tasks** & **MAE** & **MAE\({}^{M}\)** & **MAE** & **MAE\({}^{M}\)** \\ \hline (None) majority class & 0.714 & ! & 1.000 & 1.798 & ! & 1.857 \\ (None) single-task COR & 0.514 & 0.567 & 1.582 & ! & 1.728 \\ \hline \(+\)bias & 0.526 & ! & 0.566 & & & & \\ \(+\)detraultty & & ! & & 1.584 & ! & 1.695 \\ \(+\)bias5-way & 0.495 & ! & 0.541 & 1.504 _(1.485)_ & ! & 1.627 _(1.647)_ \\ \(+\)bias3-way & 0.497 & ! & 0.548 & 1.528 _(1.498)_ & ! & 1.658 _(1.654)_ \\ \(+\)bias-center & 0.509 & ! & 0.561 & 1.594 _(1.535)_ & ! & 1.745 _(1.695)_ \\ \(+\)bias-extreme & 0.498 & ! & 0.550 & 1.584 _(1.558)_ & ! & 1.743 _(1.726)_ \\ \(+\)bias5-way\(+\)bias3-way & 0.493 & ! & 0.541 & 1.479 _(1.475)_ & ! & 1.637 _(1.623)_ \\ \(+\)bias-center\(+\)bias-extreme & **0.481** & **0.529** & 1.563 _(1.526)_ & ! & 1.714 _(1.672)_ \\ \(+\)bias5-way\(+\)bias3-way\(+\)bias-center\(+\)bias-extreme & 0.485 & ! & 0.537 & 1.513 _(1.504)_ & ! & 1.665 _(1.677)_ \\ \hline \end{tabular}
\end{table}
Table 3: Evaluating the copula ordinal regression model trained to jointly model the mask (_shown in the columns_) and different auxiliary tasks (_shown in the rows_). The results in parentheses correspond to the case when factuality is added as an additional auxiliary task (only applicable when the main task is bias prediction).

 



## References

* Baccianella et al. (2009) Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2009. Evaluation measures for ordinal regression. In _Proceedings of the 9th IEEE International Conference on Intelligent Systems Design and Applications_, ISDA '09, pages 283-287, Pisa, Italy.
* Baeza-Yates (2018) Ricardo Baeza-Yates. 2018. Bias on the web. _Commun. ACM_, 61(6):54-61.
* Balikas et al. (2017) Georgios Balikas, Simon Moura, and Massih-Reza Amini. 2017. Multitask learning for fine-grained Twitter sentiment analysis. In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR '17, pages 1005-1008, Tokyo, Japan.
* Baly et al. (2018) Ramy Baly, Georgi Karadzhov, Dimitar Alexandrov, James Glass, and Preslav Nakov. 2018. Predicting factuality of reporting and bias of news media sources. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, EMNLP '18, pages 3528-3539, Brussels, Belgium.
* Barron-Cedeno et al. (2019a) Alberto Barron-Cedeno, Giovanni Da San Martino, Israa Jaradat, and Preslav Nakov. 2019a. Proppy: A system to unmask propaganda in online news. In _Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence_, AAAI'19, Honolulu, HI, USA.
* Barron-Cedeno et al. (2019b) Alberto Barron-Cedeno, Giovanni Da San Martino, Israa Jardat, and Preslav Nakov. 2019b. Proppy: Organizing news coverage on the basis of their propagating content. _Information Processing and Management_.
* Brill (2001) Ann M Brill. 2001. Online journalists embrace new marketing function. _Newspaper Research Journal_, 22(2):28.
* Chen et al. (2013) Cheng Chen, Kui Wu, Venkatsh Srinivasan, and Xudong Zhang. 2013. Battling the Internet Water Army: detection of hidden paid posters. In _Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining_, ASONAM '13, pages 116-120, Niagara, Canada.
* Darwish et al. (2017) Kareem Darwish, Dimitar Alexandrov, Preslav Nakov, and Yelena Mejova. 2017. Seminar users in the Arabic Twitter sphere. In _Proceedings of the 9th International Conference on Social Informatics_, SocInfo '17, pages 91-108, Oxford, UK.
* De Sarkar et al. (2018) Sohan De Sarkar, Fan Yang, and Arjun Mukherjee. 2018. Attending sentences to detect satirical fake news. In _Proceedings of the 27th International Conference on Computational Linguistics_, COLING '18, pages 3371-3380, Santa Fe, NM, USA.
* Dong et al. (2015) Xin Luna Dong, Evgeniy Gabrilovich, Kevin Murphy, Van Dang, Wilko Horn, Camillo Llugaresi, Shaohua Sun, and Wei Zhang. 2015. Knowledge-based trust: Estimating the trustworthiness of web sources. _Proc. VLDB Endow._, 8(9):938-949.
* Finberg et al. (2002) Howard Finberg, Martha L Stone, and Diane Lynch. 2002. Digital journalism credibility study. _Online News Association. Retrieved November_, 3:2003.
* Gerrish and Blei (2011) Sean M. Gerrish and David M. Blei. 2011. Predicting legislative roll calls from text. In _Proceedings of the 28th International Conference on International Conference on Machine Learning_, ICML '11, pages 489-496, Bellevue, Washington, USA.
* Hardalov et al. (2016) Momchil Hardalov, Ivan Kovcychev, and Preslav Nakov. 2016. In search of credible news. In _Proceedings of the 17th International Conference on Artificial Intelligence: Methodology, Systems, and Applications_, AIMSA '16, pages 172-180, Varna, Bulgaria.
* He et al. (2016) Yunchao He, Liang-Chih Yu, Chin-Sheng Yang, K Robert Lai, and Weiyi Liu. 2016. YZU-NLP team at semeval-2016 task 4: Ordinal sentiment classification using a recurrent convolutional network. In _Proceedings of the 10th International Workshop on Semantic Evaluation_, SemEval '16, pages 251-255, San Diego, CA, USA.
* Horne et al. (2018a) Benjamin Horne, Sara Khedr, and Sibel Adali. 2018a. Sampling the news producers: A large news and feature data set for the study of the complex media landscape. In _Proceedings of the Twelfth International Conference on Web and Social Media_, ICWSM '18, pages 518-527, Stanford, CA, USA.
* Horne et al. (2018b) Benjamin D. Horne, William Dron, Sara Khedr, and Sibel Adali. 2018b. Assessing the news landscape: A multi-module toolkit for evaluating the credibility of news. In _Proceedings of the The Web Conference_, WWW '18, pages 235-238, Lyon, France.
* Iyyer et al. (2014) Mohit Iyyer, Peter Enns, Jordan Boyd-Graber, and Philip Resnik. 2014. Political ideology detection using recursive neural networks. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics_, pages 1113-1122, Baltimore, MD, USA.
* Karadzhov et al. (2017) Georgi Karadzhov, Pepa Gencheva, Preslav Nakov, and Ivan Koychev. 2017. We built a fake news & clickbait filter: What happened next will blow your mind! In _Proceedings of the International Conference on Recent Advances in Natural Language Processing_, RANLP '17, pages 334-343, Varna, Bulgaria.
* Kulkarni et al. (2018) Vivek Kulkarni, Junting Ye, Steven Skiena, and William Yang Wang. 2018. Multi-view models for political ideology detection of news articles. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, EMNLP '18, pages 3518-3527, Brussels, Belgium.
* Lazer et al. (2018) David M.J. Lazer, Matthew A. Baum, Yoshai Benkler, Adam J. Berinsky, Kelly M. Greenhill, Filippo Menczer, Miriam J. Metzger, Brendan Nyhan, Gordon Pennycok, David Rothschild, Michael Schudson, Steven A. Sloman, Cass R. Sunstein, Emily A. Thorson, Duncan J. Watts, and Jonathan L. Zittrain. 2018. The science of fake news. _Science_, 359(6380):1094-1096

* Maity et al. (2017) Suman Kalyan Maity, Aishik Chakraborty, Pawan Goyal, and Animesh Mukherjee. 2017. Detection of sockoppupets in social media. In _Proceedings of the ACM Conference on Computer Supported Cooperative Work and Social Computing_, CSCW '17, pages 243-246, Portland, OR, USA.
* Mihaylov et al. (2018) Todor Mihaylov, Tsvetomila Mihaylova, Preslav Nakov, Lluis Marquez, Georgi Georgiev, and Ivan Koychev. 2018. The dark side of news community forums: Opinion manipulation trolls. _Internet Research_, 28(5):1292-1312.
* Mihaylov and Nakov (2016) Todor Mihaylov and Preslav Nakov. 2016. Hunting for tell comments in news community forums. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_, ACL '16, pages 399-405, Berlin, Germany.
* Mihaylova et al. (2018) Tsvetomila Mihaylova, Preslav Nakov, Lluis Marquez, Alberto Barron-Cedeno, Mitra Mohtarami, Georgi Karadjov, and James Glass. 2018. Fact checking in community forums. In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence_, AAAI '18, pages 879-886, New Orleans, LA, USA.
* Mukherjee and Weikum (2015) Subhabrata Mukherjee and Gerhard Weikum. 2015. Leveraging joint interactions for credibility analysis in news communities. In _Proceedings of the 24th ACM International on Conference on Information and Knowledge Management_, CIKM '15, pages 353-362, Melbourne, Australia.
* Nguyen et al. (2018) An T. Nguyen, Aditya Kharosekar, Matthew Lease, and Byron C. Wallace. 2018. An interpretable joint graphical model for fact-checking from crowds. In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence_, AAAI '18, New Orleans, LA, USA.
* Pan et al. (2018) Jeff Z. Pan, Siyana Pavlova, Chenxi Li, Ningxi Li, Yangmei Li, and Jinshou Liu. 2018. Content based fake news detection using knowledge graphs. In _Proceedings of the International Semantic Web Conference_, ISWC '18, Monterey, CA, USA.
* Perez-Rosas et al. (2018) Veronica Perez-Rosas, Bennett Kleinberg, Alexandra Lefevre, and Rada Mihalcea. 2018. Automatic detection of fake news. In _Proceedings of the 27th International Conference on Computational Linguistics_, COLING '18, pages 3391-3401, Santa Fe, NM, USA.
* Pitoura et al. (2018) Evaggelia Pitoura, Panayiotis Tsaparas, Giorgos Flouris, Irini Fundulaki, Panagiotis Papadakos, Serge Abiteboul, and Gerhard Weikum. 2018. On measuring bias in online information. _SIGMOD Rec._, 46(4):16-21.
* Popat et al. (2016) Kashyap Popat, Subhabrata Mukherjee, Jannik Strotgen, and Gerhard Weikum. 2016. Credibility assessment of textual claims on the web. In _Proceedings of the 25th ACM International on Conference on Information and Knowledge Management_, CIKM '16, pages 2173-2178, Indianapolis, IN, USA.
* Popat et al. (2017) Kashyap Poparat, Subhabrata Mukherjee, Jannik Strotgen, and Gerhard Weikum. 2017. Where the truth lies: Explaining the credibility of emerging claims on the Web and social media. In _Proceedings of the 26th International Conference on World Wide Web Companion_, WWW '17, pages 1003-1012, Perth, Australia.
* Popat et al. (2018) Kashyap Popat, Subhabrata Mukherjee, Jannik Strotgen, and Gerhard Weikum. 2018. CreditEye: A credibility lens for analyzing and explaining misinformation. In _Proceedings of The Web Conference 2018_, WWW '18, pages 155-158, Lyon, France.
* Potthast et al. (2018) Martin Potthast, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, and Benno Stein. 2018. A stylometric inquiry into hyperpartisan and fake news. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_, ACL '18, pages 231-240, Melbourne, Australia.
* Preotiuc-Pietro et al. (2017) Daniel Preotiuc-Pietro, Ye Liu, Daniel Hopkins, and Lyle Ungar. 2017. Beyond binary labels: Political ideology prediction of Twitter users. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, ACL '17, pages 729-740, Vancouver, Canada.
* Rashkin et al. (2017) Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, EMNLP '17, pages 2931-2937, Copenhagen, Denmark.
* Rosenthal et al. (2017a) Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017a. SemEval-2017 task 4: Sentiment analysis in Twitter. In _Proceedings of the 11th International Workshop on Semantic Evaluation_, SemEval '17, pages 502-518, Vancouver, Canada.
* Rosenthal et al. (2017b) Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017b. SemEval-2017 task 4: Sentiment analysis in Twitter. In _Proceedings of the 11th International Workshop on Semantic Evaluation_, SemEval '17, pages 502-518, Vancouver, Canada.
* Sim et al. (2013) Yanchuan Sim, Brice D. L. Acree, Justin H. Gross, and Noah A. Smith. 2013. Measuring ideological proportions in political speeches. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, EMNLP '13, pages 91-101, Seattle, WA, USA.
* Tavits and Letki (2009) Margit Tavits and Natalia Letki. 2009. When left is right: Party ideology and policy in Post-Communist Europe. _The American Political Science Review_, 103(4):555-569.
* Vsooughi et al. (2018) Soroush Vsooughi, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online. _Science_, 359(6380):1146-1151.

 

Robert Walecki, Ognjen Rudovic, Vladimir Pavlovic, and Maja Pantic. 2016. Copula ordinal regression for joint estimation of facial action unit intensity. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 4902-4910.
* Yu et al. (2006) Shipeng Yu, Kai Xu, Volker Tresp, and Hans-Peter Kriegel. 2006. Collaborative ordinal regression. In _Proceedings of the 23rd international conference on Machine learning_, pages 1089-1096. ACM.