Venue: NAACL 2024

# Reliability Estimation of News Media Sources: _Birds of a Feather Flock Together_

Sergio Burdisso\({}^{*,1}\), Dairazalia Sanchez-Cortes\({}^{1}\), Esau Villatoro-Tello\({}^{1}\) and Petr Motlicek\({}^{1,2}\)

\({}^{1}\)Idiap Research Institute, Martigny, Switzerland

\({}^{2}\)Brno University of Technology, Brno, Czech Republic

{sergio.burdisso,dairazalia.anchez-cortes,esau.villatoro,petr.motlicek}@idiap.ch

###### Abstract

Evaluating the reliability of news sources is a routine task for journalists and organizations committed to acquiring and disseminating accurate information. Recent research has shown that predicting sources' reliability represents an important first-prior step in addressing additional challenges such as fake news detection and fact-checking. In this paper, we introduce a novel approach for source reliability estimation that leverages reinforcement learning strategies for estimating the reliability degree of news sources. Contrary to previous research, our proposed approach models the problem as the estimation of a reliability degree, and not a reliability label, based on how all the news media sources interact with each other on the Web. We validated the effectiveness of our method on a news media reliability dataset that is an order of magnitude larger than comparable existing datasets. Results show that the estimated reliability degrees strongly correlates with journalists-provided scores (Spearman=0.80) and can effectively predict reliability labels (macro-avg. F\({}_{1}\) score=81.05). We release our implementation and dataset, aiming to provide a valuable resource for the NLP community working on information verification.

+
Footnote †: [https://reporterslab.org/fact-checking/](https://reporterslab.org/fact-checking/) (Oct. 2023)

+
Footnote †: [https://reporterslab.org/fact-checksing/](https://reporterslab.org/fact-checksing/) (Oct. 2023)

## 1 Introduction

As of 2023, the number of internet users is over 5.18 billion worldwide, meaning that around two-thirds of the global population is currently connected to the WWW [4]. The Web has democratized and radically changed how people consume and produce information by shifting the paradigm from a news-centred one to a user-centred one. Nowadays, any person on the Web can potentially be a "news medium" providing information either by creating websites, blogs and/or by making use of social media platforms.

Nevertheless, news media can no longer perform its role as "gatekeeper" deciding which stories to disseminate to the public or not [7] since most of the information on the Internet is unregulated by nature [6]. As a consequence, an enormous proliferation of misinformation has emerged leaving the public vulnerable to incorrect or misleading information about the state of the world which, among others, increased polarization and decreased trust in institutions and experts [9, 10]. The World Health Organization (WHO) recently declared a worldwide "infodemic" characterized by an overabundance of misinformation [11]. The best-known type of misinformation is _fake news_[1] defined as "false information intentionally created to mislead and/or manipulate a public through the appearance of a news format with an opportunistic structure to attract the reader's attention" [1].

In an attempt to limit the impact of fake news, a large number of initiatives have been undertaken by media, journalists, governments, and international organizations to identify true and false information across the globe [15]. For instance, the Duke University's center for journalism research, the Reporters' Lab, lists a total of 419 fact-checking active sites online1 from which FactCheck.org, Snopes, Full Fact and Politifact are the most well-known. These sites manually and systematically assess the validity of thousands of claims. However, human annotators will always be outnumbered by the claims that need to be verified, reducing the impact of such services in a large-scale scenario. Consequently, we have witnessed a growing interest in using different machine learning models, ranging from non-neural [16, 17] to deep learning-based ones [18, 19]

2017; Popat et al., 2018; Wang et al., 2018; Fajcik et al., 2023), to determine the validity of claims, news and online information. Nevertheless, these models' performance still has not reached confident accuracy values, limiting their applicability in real-world scenarios (Baly et al., 2018).

A more recent paradigm to fight fake news proposes to focus on the source rather than on the content (Baly et al., 2018, 2020), a task referred to as profiling news medium. The underlying hypothesis states that even though fake news spreads mainly through social media, they still need an initial website hosting the news. Hence, if information about websites is known in advance, identifying potentially fake news can be done by verifying the _reliability_ of the source. In fact, this activity is also performed by journalists, who often consult rating services for news websites like NewsGuard2 or MBFC3. Nonetheless, these services are not exhaustive and difficult to keep up-to-date as they rely on human evaluators, highlighting the need for scalable automatic solutions that can be applied in real-world scenarios.

Footnote 2: [https://www.newsguardtech.com/](https://www.newsguardtech.com/)

Footnote 3: [https://mediabiasfactcheck.com](https://mediabiasfactcheck.com)

Previous research has shown that predicting reliability is an important first-prior step for facchecking systems (Nguyen et al., 2018; Popat et al., 2017; Mukherjee and Weikum, 2015) and also the most important aspect that journalists consider when manually verifying the trustworthiness of the information (Baly et al., 2018). Thus, in this paper, we focus on the task of _source reliability estimation_, i.e., automatically analyzing the source that produces a given piece of information and determining its reliability degree. Concretely, we address the posed task by investigating the following research question: _to what extent can we predict the reliability of a news media source solely based on its interactions with other sources?_ Contrary to previous research, our proposed method represents a scalable and language-independent approach that can be further enriched via content-based features. Our performed experiments shed light on the immediate (positive) effects of profiling news mediums through its interactions with other sources and also in combination with traditional content-based attributes. Our research holds the potential to uncover deeper insights by incorporating more recent content-based technologies to further explore the nuanced dynamics of news websites, opening the door to a broader NLP research community.

Overall, the main contributions of this paper can be summarized as follows: _(i)_ we propose a methodology capable of modeling the _source reliability estimation_ problem in a real-world scale scenario that contrary to previous research, estimates the reliability degree (_i.e._ a continuous value) rather than a categorical value and does not depend on any third-party resources; _(ii)_ we pioneer the introduction and evaluation of different algorithms to estimate the reliability score, exploring a spectrum from vanilla reinforcement learning strategies to task-specific variations; _(iii)_ we build the largest news media reliability dataset available, orders of magnitude larger than existing datasets; _(iv)_ we present empirical evidence that establishes the feasibility of predicting the reliability of a news media source solely through its interactions with other sources (which further improves when content-based features are incorporated); and _(v)_ we release both the dataset and source code to the wider NLP research community.4

Footnote 4: [https://github.com/idiap/News-Media-Reliability](https://github.com/idiap/News-Media-Reliability)

## 2 Related Work

The task of determining information veracity has been approached from different angles and perspectives, from micro to macro, depending on the object of study. For instance, fact-checking focuses on validating a single statement, _i.e._ the claim; fake news detection analyses a whole document, _i.e._ the content of the news article. In this work, we focus on the source that produces a given piece of information, also known as _source reliability estimation_. Within social media, the sources are individual users creating the content, and previous work has focused on identifying different types of users such as _spammers_(Liubchenko et al., 2022; Stringhini et al., 2010), _bots_(Lei et al., 2023; Knauth, 2019), fake profiles (Roy and Chahar, 2020; Ramalingam and Chinaaiah, 2018), paid users (Mihaylov et al., 2015), and _trolls_(Tomaiuolo et al., 2020; Miao et al., 2020; Mihaylov et al., 2015), among others (Sansonetti et al., 2020; Burdisso et al., 2022). However, in the broader case of the WWW, sources are individual websites (Dong et al., 2015), and in our case, news media websites. Previous studies have tangentially addressed news media source reliability as part of the study of automatic fact-checking systems, either as a _prior 

in probabilistic graphical models (Nguyen et al., 2018; Mukherjee and Weikum, 2015) or as features for stance classification models (Popat et al., 2018, 2017, 2016). In these studies, reliability estimation relied on indirect measures since no gold labels were used. For instance, some works use the _AlexaRank5_ and _PageRank_(Brin and Page, 1998) scores of the websites as proxies for their reliability (Bay et al., 2018; Popat et al., 2016) while others the proportion of articles that refute false claims and support true claims (Popat et al., 2018, 2017). However, in the latter, authors rely on a fact-checking model and the selected true and false claims while, in the former, on scores that only capture the authority and popularity of the sources, not necessarily their trustworthiness -- for instance, think of popular unreliable gossip websites6 or satirical news websites, like _The Onion_,7 highly popular, attracting huge web traffic.

Footnote 5: [https://www.alexa.com/](https://www.alexa.com/)

Footnote 6: [http://www.ebizmba.com/articles/gossip-websites](http://www.ebizmba.com/articles/gossip-websites)

Footnote 7: [https://www.theonion.com/](https://www.theonion.com/)

Recently, Baly et al. (2020, 2019, 2018) addressed the source reliability estimation task on its own, modeling it as a classification task using source-level gold annotations. In particular, authors focused on predicting websites factual reporting and political bias using the values published by a news media rating service as ground truth. However, their proposed method relies on collecting and extracting information from multiple external and restricted sources (e.g. Twitter, Facebook, YouTube, etc.) for generating content-based, audience-based, and metadata-based features for the classification model, limiting its practical use on a large-scale scenario. In this paper, we also address the task using gold annotations, however, we adopt an easier-to-scale approach. Specifically, we model the problem as estimating a continuous value (i.e. the reliability _degree_) based simply on how all news media sources interact with each other on the World Wide Web.

## 3 Methodology

### Problem Formulation

Let \(S\) be the set of all news media sources on the Web, and \(G=\langle S,E,w\rangle\) be the weighted directed graph where there is an edge \((s,s^{\prime})\in E\) if source \(s\) contains articles (hyper) linked to \(s^{\prime}\) and where the weight \(w(s,s^{\prime})\in[0,1]\) is the proportion of total hyperlinks in \(s\) linked to \(s^{\prime}\). Given two disjoint subsets \(\hat{S}^{+},\hat{S}^{-}\subset S\) containing, respectively, some known reliable and unreliable news sources, the goal is to estimate the _reliability degree_\(\rho(s)\)_for all_\(s\in S\). More precisely, a total function \(\rho:S\mapsto\mathbb{R}\) such that:

1. \(\rho(s)>0\) if \(s\) is reliable
2. \(\rho(s)\leq 0\) if \(s\) is unreliable
3. \(\rho(s)<\rho(s^{\prime})\) if \(s^{\prime}\) is more reliable than \(s\).

The underlying intuition behind using hyperlinks to build the graph is that the more frequently one source links to another (i.e., the higher \(w(s,s^{\prime})\)), the higher the chances of a random reader to (click and) reach the reliable/unreliable source \(s^{\prime}\) from \(s\). Notably, hyperlinks also serve as a proxy for content-based interactions, as they are typically used to cite content from the referred article. Thus, a higher \(w(s,s^{\prime})\) also implies a stronger content-based relationship. Therefore, this simple weighted, hyperlink-based, and source-centered approach potentially captures both interaction types among news sources simultaneously, while being relatively easy to scale.

### Reinforcement Learning Strategy

Our reinforcement learning reliability framework models reliability in terms of a Markov Decision Process (MDP). An MDP is defined by a 4-tuple \(\langle\mathbb{S},A,P_{a},r_{a}\rangle\) where \(\mathbb{S}\) is a set of states, \(A\) a set of actions, \(P_{a}(s,s^{\prime})\) is the probability that action \(a\) in state \(s\) will lead to state \(s^{\prime}\), and \(r_{a}(s,s^{\prime})\) is the immediate reward perceived after taking action \(a\) in state \(s\) leading to state \(s^{\prime}\)(Sutton and Barto, 2018; Puterman, 2014; Kaelbling et al., 1996).

Given an MDP, a decision process is then specified by defining a policy \(\pi\) that provides the probability of taking action \(a\) in state \(s\). In turn, given a policy \(\pi\) we can estimate the value of each state, \(V^{\pi}(s)\), in terms of how good it is to be in that state following the policy. In particular, the value \(V^{\pi}(s)\) is given by the _Bellman equation_ which is defined, for any state \(s\in\mathbb{S}\), recursively as:

\[V^{\pi}(s)=\sum_{s^{\prime}\in\mathbb{S}}P^{\pi}(s,s^{\prime})[r(s^{\prime})+ \gamma V^{\pi}(s^{\prime})] \tag{1}\]

where \(\gamma\in[0,1)\) is known as the _discount factor_ and \(r(s)\) is the immediate reward received when reaching \(s\). Thus, we address the reliability estimation as an MDP \(\langle\mathbb{S},A,P,r\rangle\) such that: (a) The set 

of states \({\mathbb{S}}\) are all the news media websites on the Web --_i.e._ we have \({\mathbb{S}}=S\); (b) The set of actions \(A\) contains only one element, the _"move to a different news media website"_ action; (c) The probability \(P\) of moving from \(s\) to \(s^{\prime}\) will be given by the proportion of hyperlinks in \(s\) connecting to \(s^{\prime}\) --_i.e._ we have \(P(s,s^{\prime})=w(s,s^{\prime})\); and (d) The reward \(r\) of moving to a source is determined only by the source itself, and it will be positive or negative for known reliable or unreliable sources respectively --_i.e._ we have \(r(s,s^{\prime})=r(s^{\prime})\) where \(r:S\mapsto{\mathbb{R}}\) such that \(r(s)=1\) if \(s\in S^{+}\); \(r(s)=-1\) if \(s\in S^{-}\); \(r(s)=0\) otherwise. In simple words, we can think of modeling the problem as if there was a "virtual user" browsing from one news media source to another with probability proportional to how strongly connected they are, and who will perceive a positive or negative signal (the reward) when arriving to known reliable or unreliable sources, respectively. Given this framework, now the challenge is how to estimate the _reliability scores_\(\rho(s)\).

#### 3.2.1 Perceived Future Reliability

Under this simple framework, our initial approach involves estimating reliability by "looking to the future". To be more precise, we will assume the reliability degree \(\rho(s)\) is proportional to the _expected_ perceived reliability (reward) by the virtual user. Consequently, a source is considered more reliable (or unreliable) if it is expected to guide the virtual user to well-known reliable (or unreliable) sources.

To achieve this, we can simply set \(\rho(s)=V(s)\), as Equation 1 defines \(V(s)\) as the discounted long-term future rewards received following a policy \(\pi\). Note that, given that we only have one possible action in \(\mathbb{A}\), the policy \(\pi\) is trivial and thus \(P^{\pi}(s,s^{\prime})=P(s,s^{\prime})\). Therefore, a source \(s\) will have a higher (lower) \(\rho(s)\) the more positive (negative) its total expected future reward \(V(s)\). In other words, it reflects how much \(s\) is expected to guide us to known reliable (unreliable) sources, as intended. We will refer to this strategy as _"F-Reliability"_. In practice, the computation of \(V(s)\) can be done using the _Value Iteration_ algorithm [4]. Thus, we compute \(\rho(s)\) for our specific MDP as shown in Alg. 1.

```
Set \(\forall s\in S,\rho(s)=0\) repeat \(\Delta=0\) forall\(s\in S\)do \(\rho^{\prime}(s)=r(s)+\gamma\sum_{s^{\prime}\in S}P(s^{\prime},s)\rho(s^{ \prime})\) \(\Delta=max(\Delta,|\rho^{\prime}(s)-\rho(s)|)\) \(\rho=\rho^{\prime}\) until\(\Delta\)is small enough
```

**Algorithm 2**_P-Reliability_ strategy for \(\rho(s)\).

#### 3.2.2 Accumulated Past Reliability

An alternative approach is to estimate reliability by "looking to the past" rather than the future. Specifically, we assume that the reliability degree \(\rho(s)\) is proportional to the accumulated reliability (reward) perceived by the virtual user in reaching the current source \(s\). Consequently, a source becomes more reliable (unreliable) as more known reliable (unreliable) sources lead to it.

To formalize the above intuition, we leverage the _reverse Bellman equation_ introduced by Yao and Schuurmans (2013). This equation is recursively defined for any state \(s\in{\mathbb{S}}\) as:

\[R^{\pi}(s)=r(s)+\gamma\sum_{s^{\prime}\in{\mathbb{S}}}P^{\pi}(s^{\prime},s)R^{ \pi}(s^{\prime}) \tag{2}\]

In contrast to Equation 1 that looks forward from a state to define its value, this equation looks backward to define it --note \(P^{\pi}(s,s^{\prime})\) is swapped to \(P^{\pi}(s^{\prime},s)\). More precisely, while \(V(s)\) defines the value of a state based on the forward accumulated reward, \(R(s)\) does so in terms of the historical accumulated reward. Therefore, by setting \(\rho(s)=R(s)\), a source \(s\) will have a higher (lower) \(\rho(s)\) the more positive (negative) is the accumulated reward \(R(s)\) --_i.e._ the more known reliable (unreliable) sources lead to \(s\), as intended. We will refer to this strategy as _"P-Reliability"_. In practice, we can again employ _Value Iteration_ to compute \(\rho(s)\) using \(R(s)\) as shown in Algorithm 2.

#### 3.2.3 Past and Future Perceived Reliability

Lastly, we can explore an approach that combines both "the future and the past". Intuitively, we can argue that the transfer of reliability between news 

media sources and their neighboring sources is asymmetric. Specifically, the impact on the reliability, \(\rho(s)\), of a source \(s\) when referencing a reliable source is not equivalent to the effect of a reliable source referencing \(s\).8 Moreover, this asymmetry extends to both reliable and unreliable sources. That is, a reliable source referencing \(s\) carries a different weight than an unreliable one referencing \(s\), and vice versa.9 In a broader sense, we can think of a source \(s\) increasing its reliability \(\rho(s)\) as more reliable sources link to it, while losing reliability as it links to more unreliable sources.

Footnote 8: _e.g._ it is not the same for your reputation as a news media if The New York Times references you as you referencing it.

To formalize this asymmetric behavior, we can incorporate both \(R(s)\) and \(V(s)\) into our reliability model. More precisely, let \(V^{-}(s)\) be \(V(s)\) where only negative rewards \(r(s)\) are allowed, and analogously, \(R^{+}(s)\) with only positive rewards, then we can define \(\rho(s)\) as:

\[\rho(s)=V^{-}(s)+R^{+}(s) \tag{3}\]

As a result, a source \(s\) will have a higher reliability \(\rho(s)\) the more reliable sources link to it (_i.e._ the higher \(R^{+}(s)\)), and lower reliability the more it links to unreliable sources (_i.e._ the lower \(V^{-}(s)\)). We will refer to this strategy as "_FP-Reliability_".

### Reliability Investment Strategy

A well-established algorithm used in the field of _truth discovery_Li et al. (2016) is the _Investment_ algorithm Pasternack and Roth (2010). This algorithm is an iterative method in which two interdependent steps are repeated: (1) sources uniformly "invest" their trustworthiness among their claimed values; (2) sources collect credits back from the claimed values which update, in turn, their trustworthiness. Inspired by this "invest and collect" intuition, we now formulate an algorithm based on the same principle. Initially, each source will distribute its reliability \(\rho(s)\) among neighboring sources in proportion to the strength of their links, _i.e._\(\propto w(s,s^{\prime})\). In essence, during the investment step, the total credits invested in each source \(s\) is defined as follows:

\[totalcrefits(s)=\sum_{s^{\prime}\in S}w(s^{\prime},s)\cdot\rho(s^{\prime}) \tag{4}\]

In the subsequent credit collection step, the total credits are distributed among investors, \(s^{\prime}\), in proportion to their contribution to the source \(s\):

\[credits_{s^{\prime}}(s)=w_{s^{\prime}}(s)\cdot totalcedits(s) \tag{5}\]

Here, \(w_{s^{\prime}}(s)\in[0,1]\) represents the proportion of total _inbound_ hyperlinks in \(s\) originating from \(s^{\prime}\). The reliability degree is then updated by collecting the credits back in proportion to the invested percentage:

\[\rho(s)=\rho(s)+\sum_{s^{\prime}\in S}w(s,s^{\prime})\cdot credits_{s}(s^{ \prime}) \tag{6}\]

Finally, we repeat this process \(n\) times to update \(\rho(s)\) considering values from up to \(n\)-hop-away sources in the graph, as illustrated in Algorithm 3.

```
Set \(\forall s\in S,\rho(s)=r(s)\) repeatfor all\(s\in S\)do\(\triangleright\) Investment step \(totalcrefits(s)=\sum_{s^{\prime}\in S}w(s^{\prime},s)\rho(s^{\prime})\) forall\(s\in S\)do\(\triangleright\) Credit collection step \(profit=\sum_{s^{\prime}\in S}w(s,s^{\prime})credits_{s}(s^{\prime})\) \(\rho(s)=\rho(s)+profit until\(n\)times
```

**Algorithm 3**_I-Reliability_ strategy for \(\rho(s)\).

## 4 Data

### A real-world scale news media graph

The _Common Crawl Foundation_10 maintains the _Common Crawl News Dataset_ (_CC-News_), the world's largest collection of news articles crawled from global news web sites since 2016. The data is updated daily and published as a series of snapshots organized by year and month.

Footnote 10: [https://commoncarwl.org](https://commoncarwl.org)

We developed a Python CC-News processing pipeline that takes care of building the news media graph, \(G\), from CC-News snapshots (details in Appendix C). Similar to the _CCNet_ pipeline Wenzek et al. (2020), our pipeline utilizes the language classifier from fastText Joulin et al. (2017); Grave et al. (2018) to categorize news articles into 176 languages. Consequently, for a given CC-NEWS

\begin{table}
\begin{tabular}{c l|c r} \hline \multicolumn{2}{c}{**CC-News**} & \multicolumn{2}{c}{**Graph**} \\
**snapshot** & _\#articles_ & _\#nodes_ & _\#edges_ \\ \hline
2019/08 & 17M & 6,799 & 171,810 \\
2020/08 & 23M & 11,427 & 276,666 \\
2021/08 & 28M & 10,938 & 315,447 \\
2022/08 & 35M & 10,607 & 354,386 \\ \hline _all above_ & 103M & 17,057 & 909,354 \\ \hline \end{tabular}
\end{table}
Table 1: CC-News snapshots and the obtained graphs. The last row corresponds to our final graph.

 

snapshot URL, the pipeline generates one graph for each supported language showing how news sources relate to each other in that language. However, in this paper, we focus exclusively on the English graph due to the predominance of available ground truth data for experimentation in this language. Specifically, for experimentation, we will use the English graph obtained from joining four different _CC-News_ snapshots corresponding to August over the past four years (2019 to 2022).11 As indicated in Table 1, this process resulted in a unified graph containing around 17k English-speaking news media sources and nearly 1M connections --graph shown in Figure 2 (Appendix B).

Footnote 11: By selecting the same month, we ensure a consistent 4-year time span while limiting the processed news articles to approximately 100M.

### Ground truth datasets

To facilitate a comparative analysis with previous studies on the source reliability classification task, we employ the largest dataset published earlier by Bay et al. (2018). This dataset encompasses 1066 annotated news media URL domains extracted from _Media Bias/Fact Check_ (MBFC) --refer to the first row of Table 2 for details.12 Furthermore, for a more comprehensive evaluation, we employ an extended dataset meticulously created by merging ground truth labels from various sources, as outlined below:

\(\bullet\)**MBFC:** we followed a similar process as in Baby et al. (2018) but crawling the entire MBFC website to extract 4138 ground truth labels. Following Gruppi et al. (2020), we aggregated these labels into three classes: "reliable" for sources with high or very high factual reporting, "unreliable" for sources flagged as conspiracy, pseudoscience, or with low/very low factual reporting, and "mixed" for sources with mixed factual reporting.

\(\bullet\)**Wikipedia's perennial sources:** the platform hosts a list of sources discussed by the community regarding their reliability and use on the platform.13 We extracted 553 ground truth labels from this list applying the following policy: sources marked as _generally reliable_ were labeled as "reliable"; sources marked as _generally unreliable_, _deprecated_, or _blacklisted_ were labeled as "unreliable"; and sources marked as _no consensus_, _stale discussions_ or _discussion in progress_ as "mixed";

Footnote 12: Original faculty labels were transformed into reliability labels following Gruppi et al. (2020) strategy.

Footnote 13: [https://en.wikipedia.org/wiki/Wikipedia](https://en.wikipedia.org/wiki/Wikipedia): Reliable_sources/Perennial_sources

\begin{table}
\begin{tabular}{c c c c} \hline \hline  & \multicolumn{3}{c}{**Label distribution**} \\

- \\ _NewsGuard_ & _25_ &
- & _60_ \\ \hline \hline \end{tabular}
\end{table}
Table 2: Datasets details. Bottom part shows individual contributions to our final dataset.
 

from Baly et al. (2018).

\(\bullet\)**ExpsetB:** 1630 _reliable_ and 487 _unreliable_ sources from our dataset.

\(\bullet\)**ExpsetB-:** 1630 _reliable_ and 166 _unreliable_ sources. A simpler version of _ExpsetB_ removing "mixed" from _unreliable_ sources.

## 5 Experiments and Evaluation Results

For experimentation, we define the reward values based on ground truth labels as \(r(s)=1\) if the label is "reliable", \(r(s)=-1\) if "unreliable", and \(r(s)=0\) otherwise. In addition, selecting appropriate hyperparameter values is crucial. For _I-Reliability_, \(n\) controls how far to look in the neighborhood for investments (how many nodes away). Similarly, in the reinforcement learning strategies, the discount factor \(\gamma\) controls the distance of looking back/forward; \(\gamma\approx 0\) focuses mostly on present reward \(r(s)\), while \(\gamma\approx 1\) considers all history/future to compute \(\rho(s)\). We performed a grid search over \(n\in[1,10]\) and \(\gamma\in[0.05,0.95]\) to determine the best hyperparameter values on each of the three experimental sets. The grid search was performed using 5-fold cross validation selecting the \(n\) and \(\gamma\) that obtained the best _macro avg._\(F_{1}\) on the reliability classification task, as described in Section 5.1. We observed that, independently of the dataset, _better reliability estimation is achieved when looking mostly at nearby sources_, as better performance was obtained with small \(n\) (\(n\leq 2\)) and \(\gamma\) (\(\gamma<0.5\)) values --details in Appendix A.

### Reliability Classification Results

In this section, we focus on evaluating the first two conditions for \(\rho(s)\) given in Section 3.1. These conditions allow us to measure the ability of \(\rho(s)\) to distinguish reliable from unreliable sources. For comparison, we follow the evaluation procedure from Baly et al. (2018) and report results for 5-fold cross-validation. More precisely, in each k-fold iteration, we only use ground truth rewards \(r(s)\) from four folds to compute \(\rho(s)\) for all 17k sources in the graph, and using conditions 1 and 2, all \(s\) in the hold-out fold are classified as _reliable_ (\(\rho(s)>0\)) or _unreliable_ (\(\rho(s)\leq 0\)).

Table 3 shows the evaluation results obtained on the three experimentation sets along with two naive baselines for reference, random and majority class classifiers.15 In addition, for _ExpsetA_, we also report the results obtained using the classification models introduced in previous works (Baly et al., 2018, 2020). These classifiers combine multiple content-based, audience-based, and metadata-based features about the sources. Authors released the pre-computed features values for the Bay et al. (2018) dataset and thus, using their source code,16 we were able to train and evaluate their classifiers on the _ExpsetA_ set. However, since building these features relies on multiple external sources (_e.g._ Twitter, Facebook, YouTube, etc.), we were not able to evaluate their method in our new dataset given current API restrictions to access them.

Footnote 15: For a comprehensive view of additional metrics, such as precision, recall, and confidence intervals, refer to Table 7 in the Appendix.

Observing the performance across the different datasets, all four strategies outperform the random and majority class baselines by a statistically significant difference (paired \(t\)-test with \(p\)-_value_\(<0.02\)). In addition, both _P-Reliability_ and _I-Reliability_ consistently outperform other strategies, including those presented in previously published works (\(p\)-_value_\(<0.03\)), however, the difference between these two strategies is not statisti

\begin{table}
\begin{tabular}{c c r r r r} \hline \hline  & \multicolumn{1}{l}{**Strategy**} & \multicolumn{3}{c}{\(\mathbf{F}_{1}\) **score**} \\ \cline{3-6}
**Data** & & _macro avg._ & _reliable_ & _unreliable_ & **Acc.** \\ \hline \multirow{6}{*}{**Model**} & M-BL & 42.33 & 84.66 & 0.00 & 73.44 \\  & R-BL & 48.85 & 61.76 & 35.94 & 52.33 \\  & Baly18 & 67.87 & 84.81 & 50.92 & 76.95 \\  & Baly20 & 65.24 & 82.99 & 47.50 & 74.37 \\ \cline{2-6}  & _F-R_ & 61.52 & 87.62 & 35.42 & 79.26 \\  & _P-R_ & **72.67** & **90.05** & **55.29** & **83.79** \\  & _FP-R_ & 69.28 & 89.23 & 49.34 & 82.29 \\  & _I-R_ & **72.81** & **90.03** & **55.60** & **83.77** \\ \hline \multirow{6}{*}{**Model**} & M-BL & 43.50 & 87.00 & 0.00 & 77.00 \\  & R-BL & 47.48 & 62.17 & 32.80 & 51.63 \\ \cline{2-6}  & _F-R_ & 61.85 & 79.72 & 43.98 & 70.34 \\ \cline{2-6}  & _P-R_ & **74.69** & **88.29** & **61.10** & **82.00** \\ \cline{2-6}  & _FP-R_ & 55.95 & 68.08 & 43.82 & 59.38 \\ \cline{2-6}  & _I-R_ & **75.51** & **89.30** & **61.72** & **83.28** \\ \hline \multirow{6}{*}{**Model**} & M-BL & 47.58 & 95.15 & 0.00 & 90.76 \\  & R-BL & 39.17 & 63.04 & 15.31 & 48.55 \\ \cline{1-1} \cline{2-6}  & _F-R_ & 62.18 & 90.23 & 34.12 & 83.02 \\ \cline{1-1}  & _P-R_ & **78.90** & **95.83** & **61.97** & **92.48** \\ \cline{1-1}  & _FP-R_ & 59.20 & 84.66 & 33.74 & 75.11 \\ \cline{1-1}  & _I-R_ & **81.05** & **96.71** & **65.39** & **93.99** \\ \hline \hline \end{tabular}
\end{table}
Table 3: 5-fold cross-validation average results for reliability classification. The best-performing values are **underlined**, while the 2nd-best results appear in **bold** font. R-BL and M-BL refer to random and majority class baselines; Baly18 and Baly20 refer to Baly et al. (2018) and Baly et al. (2020); and *-R stands for *-Reliability.

 

(\(p\)-_value_\(>0.5\)). From the results we can also see that, regardless of the chosen strategy and dataset, the \(F_{1}\) score for the _unreliable_ class consistently remains lower when compared to the _reliable_ class. This suggests that identifying unreliable sources is more challenging, likely due to the dataset imbalance favoring reliable sources --note that in both _ExpsetA_ and _ExpsetB_, only approximately \(25\%\) of the sources are unreliable. This imbalance results in models having fewer negative signals (_i.e._ rewards \(r(s)=-1\)) to learn to identify unreliable sources effectively. Another contributing factor is the inclusion of "mixed" labels in the _unreliable_ group, making the task more challenging by incorporating unreliable sources whose reliability is not clearly defined. This hypothesis is supported by the results from _ExpsetB\({}^{-}\)_, where the removal of "mixed" labels results in improvements across all metrics ---note that the highest _unreliable_\(F_{1}\) score is achieved while the dataset is significantly more unbalanced (ten times fewer unreliable sources than reliable ones). Concerning the reinforcement learning strategies, _P-Reliability_ significantly (\(p\)-_value_\(\leq 0.02\)) outperformed _F-Reliability_ suggesting that the reliability of a source _is more significantly influenced by its origins than by the destinations it reaches_. On the other hand, _FP-Reliability_ shows poor performance, mainly due to the different nature of \(V^{-}(s)\) and \(R^{+}(s)\) in Equation 3 --note that \(V(s)\) is defined as an expectation, whereas \(R(s)\) is not.17

Footnote 17: Transition probabilities are not normalized in both directions, only in the forward direction. Consequently, there is no inherent mathematical symmetry that ensures \(V^{-}(s)\) and \(R^{+}(s)\) will equilibrate to zero (\(\rho(s)=0\)) when they correspond to an equivalent number of unreliable and reliable sources

Finally, to assess the complementarity of our graph-based strategies with content-based ones, we performed an additional experiment: a simple voting ensemble between our strategies and Baly's models. Specifically, sources were classified as reliable only when both models agreed on the classification. Table 4 presents the obtained results for our two best-performing models, _P-Reliability_ and _I-Reliability_, on _ExpsetA_.18 The ensemble approach enabled the models to further improve their performance, particularly in detecting unreliable sources, achieving the highest _macro avg._\(F_{1}\) score on this dataset (77.47).

Footnote 18: Full results included in Table 7 (Appendix).

### Correlation with human judgment

In this section, we focus on evaluating the final condition in the definition of \(\rho(s)\) in Section 3.1. This condition enables \(\rho(s)\) to assess the reliability of _s relative_ to other sources, allowing the ranking of sources based on their reliability degrees. For evaluation, we use the _NewsuGard_ dataset introduced in Section 4.2 containing the 85 ground truth reliability scores provided by trained journalists. The score ranges from 0 to 100 and is obtained by answering 9 questions that address different journalistic criteria --details in Appendix D. Table 5 shows examples of NewsGuard scores and their estimated reliability degree.19

Footnote 19: For ease of comparison, in this table, \(\rho(s)\) is normalized in the range \([-1,1]\) by dividing it by the maximum (when \(\rho(s)\geq 0\)) and minimum value (when \(\rho(s)<0\)).

We measure the correlation between these human-provided scores and their estimated reliability degree by computing the _Pearson correlation coefficient_ (PCC) and the _Spearman's rank correlation coefficient_ (SRCC). PCC measures the linear relationship between two variables, whereas SRCC assesses the monotonic rela 

it _more suitable for evaluating the relative reliability_ of sources, as it captures ranked associations regardless of the exact numerical values (condition 3 for \(\rho\)). In particular, we perform the evaluation under two scenarios, when sources are known to be (un)reliable and, more challenging, when their reliability is not known in advance.20 In other words, we perform the evaluation following, respectively, two experimental settings: (\(\clubsuit\)) we use all the ground truth rewards from the largest experimental set, _ExpsetB_, to learn the reliability degree \(\rho(s)\) of all 17k sources in the graph; and (\(\diamondsuit\)) we repeat the same process but removing the rewards for all the 85 domains used for evaluation.

Footnote 20: For instance, is \(\rho(cnbc.com)<\rho(bbc.co.uk)?\) that is, is “bbc.co.uk” more reliable than the “cnbc.com”? knowing in advance that both are reliable (\(r(cnbc.com)=r(bbc.co.uk)=1\)) vs. not knowing it (\(r(cnbc.com)=r(bbc.co.uk)=0\)).

Table 6 shows the obtained results along with two baselines for reference, random and _PageRank_ algorithm [4], scatter plots in Appendix E. We can observe that, as expected, correlations are weaker under the hardest scenario without rewards, specially in terms of PCC which is more sensitive to the bias introduced by the rewards.22 Nevertheless, in both scenarios, obtained correlation coefficients are statistically significant (\(p\text{-}value\leq 5\text{e-}07\) for PCC, \(p\text{-}value\leq 1\text{e-}06\) for SRCC) and higher than the baselines, except for _F-Reliability_. In general, the strategies that correlate more strongly with the journalist-provided scores are _P-Reliability_ and _FP-Reliability_, showing both a strong linear (PCC) and ranking-based (SRCC) relationship independently of whether rewards were used or not.23_FP-Reliability_ results suggest that combining _F-Reliability_ and _P-Reliability_ strategies could be advantageous for estimating _relative_ reliability.24 Overall, results are inspiring considering that the learning process for all \(\rho(s)\) values in the graph leverages only a subset of binary ground truth rewards (\(r(s)=-1\) or \(1\)), _without any explicit notion of ground truth score or degree_. In contrast to reliability scores derived from various qualitative journalistic criteria, \(\rho(s)\) approximates the reliability degree solely based on the propagation of these initial rewards throughout the network's structure.25

Footnote 23: Note that “pageRank” is unstable for classification experiments as its non-negative scores always predicted the positive class, reducing it to a majority-class classifier. Hence, its exclusion from Table 3.

Footnote 22: Note that sources with \(r(s)=1\) will naturally tend to have a final \(\rho(s)\) close to \(1\) while sources with \(r(s)=-1\) close to \(-1\), this bias is heavily reduced when instead of using the actual \(\rho(s)\) value we use its ranking (as in SRCC).

## 6 Conclusion and Future Work

In this study, we introduced an approach for assessing the reliability of news media through their network interactions. This approach diverges from previous models that depend on content, audience feedback, and/or metadata. Moreover, unlike in previous works, our method estimates a _reliability degree_ rather than a reliability label. We assessed the quality of the estimated values in terms of reliability classification and correlation with journalists-provided scores. We found that a source's origins is more indicative of its reliability than its reach and show evidence that it is feasible to predict the reliability of news media using only their network interactions, providing an easier-to-scale approach than prior methods. As future work, we plan to expand the study by building a larger graph and designing more sophisticated strategies that leverage content-based features. Additionally, we aim to explore the estimation of other news source properties, such as political bias, using the same approach. Finally, we intend to investigate the use of the estimated reliability values in downstream tasks like fact-checking and fake news detection.

 

Ethical Considerations

The work presented in this paper has been done in the scope of the CRiTERIA project26 that follows the H2020 ethical standards and guidelines. The Consortium Agreement includes the partners' commitment to FAIR (findable, accessible, interoperable and re-usable) data management practices and responsible research practices. The framework of the research questions and preliminary results were reviewed by the Project Ethics Check and Audit committee in the form of an on-going work deliverable.

Footnote 26: [https://www.project-criteria.eu/](https://www.project-criteria.eu/)

In the present research paper there is no gender bias to be investigated or addressed. The data comes from hyperlinks and URL domain names that can not be associated to a gender. As described in Section 4, there is no intentional collection of personal data in any form, and as a consequence, there is no need for data anonymisation or pseudonymisation. Similarly, there is no need of informed and singed consents since there is no direct human participation in the construction of the graphs to calculate the reliability values.

Regarding data and processing security, a snapshot of CC-News data is transferred, held in the local servers and processed for the reliability estimation. The data can be deleted at any time and easily downloaded from the original public CC-News sources (as described in Section 4). Any further data processing carried out, is going to be publicly available, along with the reference to this paper.

Regarding other sources, only data collected by other sources (under Apache or open source licenses) with well described data collection methodology, validated with published results and that is publicly available was used. The overview of the datasets is described in Section 4.2, the description includes the annotations distribution in Table 2, and the original sources are properly acknowledge along the paper.

The estimated reliability values were obtained based on initial ground truth labels. This labels capture mainly the factuality of reporting and do not consider other aspects like, for instance, political bias or press freedom rating. Therefore, computed reliability values should not be considered as _de facto_ values.

From a societal perspective, this paper brings a positive impact, improving the situational awareness of decision makers, including fact-checkers. The mathematically defined algorithms are robust to content-related biases since they are both language and content independent (political, religious, racial, etc.).

## 8 Limitations

### Methodology

The main constraint of the proposed methodology lies in the requirement for news media sources to be included in the graph for their reliability to be calculated. This limitation may arise due to temporal or size constraints in the data used to construct the graph. For instance, a recently emerged news source might not be referenced by others until some time has passed. To address this limitation to a certain degree, assigning \(\rho(s)=0\) to such sources can be considered. This implies that their reliability is indeterminate, indicating an unknown or undetermined status, meaning they are neither reliable nor unreliable.

### Experimentation

The main three limitations of the present work regarding the experimentation and evaluation of the proposed approach are:

1. **Only English-speaking news sources:** the proposed methodology is content- and language-independent. However, we focused exclusively on the English-speaking news sources due to the predominance of available ground truth data for experimentation in this language. Further studies need to be done with ground truth for non-English-speaking sources to assess the robustness of the methodology across languages.
2. **Restricted graph size:** the graph we used was built processing around 100M news articles from 4 months spanning a 4-year time window. This imposes not only a temporal limitation but also restricts the number of sources in the graph. However, along with this paper, we release and open source, under Apache 2.0 license, the Python CC-News processing pipeline and the dataset for the community to reproduce the proposed methodology in larger scale.
3. **Corpus used to build the graph:** although CC-News is continuously growing on a daily 

basis, the crawling of news articles started in 2016. Therefore, CC-News does not contain articles prior to 2016 and news sources that existed before 2016 but not after, will not be reachable. Consequently, for these sources to be included, their articles need to be crawled from a different corpus or manually from the Web.

## 9 Acknowledgments

This work was supported by CRiTERIA, EU project funded under the Horizon 2020 program, grant agreement number 101021866. We would also like to express our sincere gratitude to the anonymous reviewers for their valuable feedback, which has helped us enhance the quality of this work.

## References

* Baly et al. (2018) Ramy Baly, Georgi Karadzhov, Dimitar Alexandrov, James Glass, and Preslav Nakov. 2018. Predicting factuality of reporting and bias of news media sources. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 3528-3539, Brussels, Belgium. Association for Computational Linguistics.
* Baly et al. (2020) Ramy Baly, Georgi Karadzhov, Jisun An, Haewoon Kwak, Yoan Dinkov, Ahmed Ali, James Glass, and Preslav Nakov. 2020. What was written vs. who read it: News media profiling using text analysis and social media context. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 3364-3374, Online. Association for Computational Linguistics.
* Baly et al. (2019) Ramy Baly, Georgi Karadzhov, Abdelrhman Saleh, James Glass, and Preslav Nakov. 2019. Multi-task ordinal regression for jointly predicting the trustworthiness and the leading political ideology of news media. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 2109-2116, Minneapolis, Minnesota. Association for Computational Linguistics.
* Baptista and Gradim (2022) Joao Pedro Baptista and Anabela Gradim. 2022. A working definition of fake news. _Encyclopedia_, 2(1).
* Brin and Page (1998) Sergey Brin and Lawrence Page. 1998. The anatomy of a large-scale hypertextual web search engine. _Computer Networks and ISDN Systems_, 30(1):107-117. Proceedings of the Seventh International World Wide Web Conference.
* Burdisso et al. (2022) Sergio Burdisso, Juan Pablo Zuluaga-Gomez, Esau Villatoro-Tello, Martin Fajcik, Muskean Singh, Pavel Smrz, and Petr Motlicek. 2022. IDIAPers @ causal news corpus 2022: Efficient causal relation identification through a prompt-based few-shot approach. In _Proceedings of the 5th Workshop on Challenges and Applications of Automated Extraction of Sociological Events from Text (CASE)_, pages 61-69, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
* Cuan-Baltazar et al. (2020) Jose Yunam Cuan-Baltazar, Maria Jose Munoz-Perez, Carolina Robledo-Vega, Maria Fernanda Perez-Zepeda, and Elena Soto-Vega. 2020. Misinformation of covid-19 on the internet: infodemiology study. _JMIR public health and surveillance_, 6(2):e18444.
* Dong et al. (2015) Xin Luua Dong, Evgeniy Gabrilovich, Kevin Murphy, Van Dang, Wilko Horn, Camillo Lugaresi, Shaohua Sun, and Wei Zhang. 2015. Knowledge-based trust: Estimating the trustworthiness of web sources. _Proc. VLDB Endow._, 8(9):938-949.
* Fajick et al. (2023) Martin Fajicki, Peter Motlicek, and Pavel Smrz. 2023. Claim-dissector: An interpretable fact-checking system with joint re-ranking and veracity prediction. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 10184-10205, Toronto, Canada. Association for Computational Linguistics.
* Grave et al. (2018) Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_, Miyazaki, Japan. European Language Resources Association (ELRA).
* Gruppi et al. (2020) Mauricio Gruppi, Benjamin D. Horne, and Sibel Adali. 2020. NELA-GT-2019: A large multi-labelled news dataset for the study of misinformation in news articles. _CoRR_, abs/2003.08444.
* ISD (2020) ISD. 2020. _Anatomy of a Disinformation Empire: Investigating NaturalNews_. Institute for Strategic Dialogue.
* Joulin et al. (2017) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_, pages 427-431, Valencia, Spain. Association for Computational Linguistics.
* Kaelbling et al. (1996) Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. 1996. Reinforcement learning: A survey. _Journal of artificial intelligence research_, 4:237-285.
* Knauth (2019) Jurgen Knauth. 2019. Language-agnostic Twitter-bot detection. In _Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019)_, pages 550-558, Varna, Bulgaria. INCOMA Ltd.
* Kwon et al. (2013) Sejeong Kwon, Meeyoung Cha, Kyomin Jung, Wei Chen, and Yajun Wang. 2013. Aspects of rumor 

spreading on a microblog network. In _Social Informatics: 5th International Conference, SocInfo 2013, Kyoto, Japan, November 25-27, 2013, Proceedings_ 5, pages 299-308. Springer.
* Lazer et al. (2018) David MJ Lazer, Matthew A Baum, Yochai Benkler, Adam J Bernsky, Kelly M Greenhill, Filippo Menczer, Miriam J Metzger, Brendan Nyhan, Gordon Pennycook, David Rothschild, et al. 2018. The science of fake news. _Science_, 359(6380):1094-1096.
* Lei et al. (2023) Zhenyu Lei, Herun Wan, Wenqian Zhang, Shangbin Feng, Zilong Chen, Jundong Li, Qinghua Zheng, and Minnan Luo. 2023. BIC: Twitter bot detection with text-graph interaction and semantic consistency. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 10326-10340, Toronto, Canada. Association for Computational Linguistics.
* Lewandowsky et al. (2017) Stephan Lewandowsky, Ullrich KH Ecker, and John Cook. 2017. Beyond misinformation: Understanding and coping with the "post-truth" era. _Journal of applied research in memory and cognition_, 6(4):353-369.
* Li et al. (2016) Yaliang Li, Jing Gao, Chuishi Meng, Qi Li, Lu Su, Bo Zhao, Wei Fan, and Jiawei Han. 2016. A survey on truth discovery. _SIGKDD Explor. Newsl._, 17(2):1-16.
* Liubchenko et al. (2022) Nataliia Liubchenko, Andrii Podrozhniak, and Vasyl Oliinyk. 2022. Research application of the spam filtering and spammer detection algorithms on social media. In _CEUR Workshop Proceedings_, volume 3171, pages 116-126.
* Ma et al. (2016) Jing Ma, Wei Gao, Prasenjit Mitra, Sejeong Kwon, Bernard J. Jansen, Kam-Fai Wong, and Meeyoung Cha. 2016. Detecting rumors from microblogs with recurrent neural networks. In _Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence_, IJCAI'16, page 3818-3824. AAAI Press.
* Miao et al. (2020) Lin Miao, Mark Last, and Marina Litvak. 2020. Detecting troll tweets in a bilingual corpus. In _Proceedings of the Twelfth Language Resources and Evaluation Conference_, pages 6247-6254, Marseille, France. European Language Resources Association.
* Mihaylov et al. (2015a) Todor Mihaylov, Georgi Georgiev, and Preslav Nakov. 2015a. Finding opinion manipulation trolls in news community forums. In _Proceedings of the Nineteenth Conference on Computational Natural Language Learning_, pages 310-314, Beijing, China. Association for Computational Linguistics.
* Mihaylov et al. (2015b) Todor Mihaylov, Ivan Koychev, Georgi Georgiev, and Preslav Nakov. 2015b. Exposing paid opinion manipulation trills. In _Proceedings of the International Conference Recent Advances in Natural Language Processing_, pages 443-450, Hissar, Bulgaria. INCOMA Ltd. Shoumen, BULGARIA.
* Mukherjee and Weikum (2015) Subhabrata Mukherjee and Gerhard Weikum. 2015. Leveraging joint interactions for credibility analysis in news communities. In _Proceedings of the 24th ACM International on Conference on Information and Knowledge Management_, CIKM '15, page 353-362, New York, NY, USA. Association for Computing Machinery.
* Munger (2020) Kevin Munger. 2020. All the news that's fit to click: The economics of clickbait media. _Political Communication_, 37(3):376-397.
* Nguyen et al. (2018) An Nguyen, Aditya Kharosekar, Matthew Lease, and Byron Wallace. 2018. An interpretable joint graphical model for fact-checking from crowds. _Proceedings of the AAAI Conference on Artificial Intelligence_, 32(1).
* Norregaard et al. (2019) Jeppe Norregaard, Benjamin D Horne, and Sibel Adahl. 2019. Nela-gt-2018: A large multi-labelled news dataset for the study of misinformation in news articles. In _Proceedings of the international AAAI conference on web and social media_, volume 13, pages 630-638.
* Pasternack and Roth (2010) Jeff Pasternack and Dan Roth. 2010. Knowing what to believe (when you already know something). In _Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010)_, pages 877-885, Beijing, China. Coling 2010 Organizing Committee.
* statistics & facts. _Statista_.
* Popat et al. (2016) Kashyap Popat, Subhabrata Mukherjee, Jannik Strotgen, and Gerhard Weikum. 2016. Credibility assessment of textual claims on the web. In _Proceedings of the 25th ACM International on Conference on Information and Knowledge Management_, CIKM '16, page 2173-2178, New York, NY, USA. Association for Computing Machinery.
* Popat et al. (2017) Kashyap Popat, Subhabrata Mukherjee, Jannik Strotgen, and Gerhard Weikum. 2017. Where the truth lies: Explaining the credibility of emerging claims on the web and social media. In _Proceedings of the 26th International Conference on World Wide Web Companion_, WWW '17 Companion, page 1003-1012, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
* Popat et al. (2018a) Kashyap Popat, Subhabrata Mukherjee, Jannik Strotgen, and Gerhard Weikum. 2018a. Credeye: A credibility lens for analyzing and explaining misinformation. In _Companion Proceedings of the The Web Conference 2018_, WWW '18, page 155-158, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
* Popat et al. (2018b) Kashyap Popat, Subhabrata Mukherjee, Andrew Yates, and Gerhard Weikum. 2018b. DeClare: Debunking fake news and false claims using evidence-aware deep learning. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 22-32, Brussels, Belgium. Association for Computational Linguistics 

Martin L Puterman. 2014. _Markov decision processes: discrete stochastic dynamic programming_. John Wiley & Sons.
* Ramalingam and Chinnaiah (2018) Devakunchari Ramalingam and Valliyammai Chinnaiah. 2018. Fake profile detection techniques in large-scale online social networks: A comprehensive review. _Computers & Electrical Engineering_, 65:165-177.
* Roy and Chahar (2020) Pradeep Kumar Roy and Shivam Chahar. 2020. Fake profile detection on social networking websites: A comprehensive review. _IEEE Transactions on Artificial Intelligence_, 1(3):271-285.
* Sansonetti et al. (2020) Giuseppe Sansonetti, Fabio Gasparetti, Giuseppe D'aniello, and Alessandro Micarelli. 2020. Unreliable users detection in social media: Deep learning techniques for automatic detection. _IEEE Access_, 8:213154-213167.
* Shaar et al. (2020) Shaden Shaar, Nikolay Babulkov, Giovanni Da San Martino, and Preslav Nakov. 2020. That is a known lie: Detecting previously fact-checked claims. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 3607-3618, Online. Association for Computational Linguistics.
* Stringhini et al. (2010) Gianluca Stringhini, Christopher Kruegel, and Giovanni Vispa. 2010. Detecting spammers on social networks. In _Proceedings of the 26th annual computer security applications conference_, pages 1-9.
* Stromback et al. (2020) Jesper Stromback, Yariv Tsfati, Hajo Boomgaarden, Alyt Damstra, Elina Lindgren, Rens Vliegenthart, and Torun Lindholm. 2020. News media trust and its impact on media use: Toward a framework for future research. _Annals of the International Communication Association_, 44(2):139-156.
* Sutton and Barto (2018) Richard S Sutton and Andrew G Barto. 2018. _Reinforcement learning: An introduction_. MIT press.
* Tomaiuolo et al. (2020) Michele Tomaiuolo, Gianfranco Lombardo, Monica Mordonini, Stefano Cagnoni, and Agostino Poggi. 2020. A survey on troll detection. _Future Internet_, 12(2).
* Van Der Linden (2022)Sander Van Der Linden. 2022. Misinformation: susceptibility, spread, and interventions to immunize the public. _Nature Medicine_, 28(3):460-467.
* Wang (2017) William Yang Wang. 2017. "liar, liar pants on fire": A new benchmark dataset for fake news detection. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 422-426, Vancouver, Canada. Association for Computational Linguistics.
* Wang et al. (2018) Yaqing Wang, Fenglong Ma, Zhiwei Jin, Ye Yuan, Guangxu Xun, Kishlay Jha, Lu Su, and Jing Gao. 2018. Eann: Event adversarial neural networks for multi-modal fake news detection. In _Proceedings of the 24th acm sigkdd international conference on knowledge discovery & data mining_, pages 849-857.
* Wenzek et al. (2020) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzman, Armand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In _Proceedings of the Twelfth Language Resources and Evaluation Conference_, pages 4003-4012, Marseille, France. European Language Resources Association.
* Yao and Schuurmans (2013) Hengshuai Yao and Dale Schuurmans. 2013. Reinforcement ranking. _arXiv preprint arXiv:1303.5988_.

 

Hyperparameter optimization

We performed a grid search to determine the best hyperparameter values on each of the three experimental sets. More precisely, as in Section 5.1, the evaluation was performed using 5-fold cross validation on the reliability classification task. For reinforcement learning strategies, we evaluated \(\gamma\) from \(0.05\) to \(0.95\) in increments of \(0.05\) (_i.e.,_\(\gamma\in\{0.05,0.1,0.15,\ldots,0.95\}\)). For _I-Reliability_, \(n\) from \(1\) to \(10\) (_i.e.,_\(n\in\{1,2,3,\dots,10\}\)). Finally, the hyperparameter values obtaining the best _macro avg. F\({}_{1}\) score_ were the one selected on each experimental set. Namely, the selected values for each strategy were:

* **F-Reliability:**\(\gamma=0.05\) for _ExpsetA_ and _ExpsetB\({}^{-}\)_, \(\gamma=0.5\) for _ExpsetB_.
* **P-Reliability:**\(\gamma=0.15\) for _ExpsetaA_, \(\gamma=0.3\) for _ExpsetB_, and \(\gamma=0.2\) for _ExpsetB\({}^{-}\)_.
* **FP-Reliability:**\(\gamma=0.1\) for _ExpsetA_, \(\gamma=0.05\) for _ExpsetB_ and _ExpsetB\({}^{-}\)_.
* **I-Reliability:**\(n=1\) for _ExpsetA_ and _ExpsetB_, and \(n=2\) for _ExpsetB\({}^{-}\)_.

As shown in Figure 1, we can observe that _better reliability estimation is achieved when looking mostly at nearby sources_, as better performance is obtained with small \(n\) and \(\gamma\) values, namely \(n\leq 2\) and \(\gamma<0.5\). Furthermore, _P-Reliability_ (orange line) outperforms the other reinforcement learning strategies, consistently, while being more robust to the choice of \(\gamma\), except when \(\gamma>0.7\) from which performance starts to decrease.

Finally, for the Baly et al. (2018) and Bay et al. (2020) classifiers in Table 3, we follow the same process described by the authors to tune the SVM hyperparameters, i.e., the cost \(C\), the kernel type, and the kernel width \(\gamma\) using the 5-fold cross validation maximizing the _F\({}_{1}\) score_ as with our methods.27

Footnote 27: We used the author’s source code containing the hyperparameter search in it (github.com/rambbaly/News-Media-Reliability).

## Appendix B Temporal Ablation Analysis

To evaluate the robustness of our proposed approach concerning both the graph size and the temporal span used in its construction, a temporal ablation study was conducted. In addition to the graph used for experimentation, illustrated in Figure 2, we generated four different graphs, each corresponding to one of the four CC-News snapshots (refer to Table 1 in Section 4.1 for detailed information on each graph). Subsequently, employing each of these four graphs, we replicated the evaluation procedure described in Section 5.1. These evaluations allowed us to measure how the performance of the proposed strategies changed, when changing the graph, compared to the reported values in Table 3.

Figure 3 shows, without loss of generality, the results obtained with the two best-performing strategies reported in Table 3, _P-Reliability_ and _I-Reliability_, on the largest experimental set _ExpsetB_. We observed that, independently of the strategy and the dataset, the best results were always obtained with the largest (in size and time) graph joining all the snapshots. We also observed that not all strategies exhibit equal robustness to changes in the graph. For instance, we can see in Figure 3 how _P-Reliability_ is more sensitive than _I-Reliability_ to the choice of snapshot for graph construction. Despite this variation, both strategies demonstrate improvement and achieve their best results with reduced uncertainty when considering all snapshots.

## Appendix C Graph Construction Steps

The _Common Crawl News Dataset_ (_CC-News_)28 is published as _WARC_ files29 grouped by year and month, called snapshots.30 To construct the news media graph, \(G=\langle S,E,w\rangle\), from CC-News snapshots, we follow to the subsequent steps:

Footnote 29: [https://commoncrawl.org/blog/news-dataset-available](https://commoncrawl.org/blog/news-dataset-available)

Footnote 29: A file format resembles the raw HTTP request and response of each crawled web page.

Footnote 30: [https://data.commoncrawl.org/crawl-data/CC-NEWS/index.html](https://data.commoncrawl.org/crawl-data/CC-NEWS/index.html)

1. Download each WARC file and parse each news article in it to extract its URL and all the hyperlinks in its body. At the end of this step, we have a set of news article URLs, \(U\), and a set of hyperlinks \(L_{u}\) for each article \(u\in U\).
2. Generate the graph nodes \(S\) from \(U\) simply as \(S=\{domain(u):u\in U\}\) which contains the URL domain names (_e.g._ "nytimes.com", "cnn.com", etc.) of all processed news articles.
3. For each domain \(s\in S\) create the list of all its hyperlinks, \(L_{s}\), by aggregating the hyperlinks of all its articles, _i.e._\(L_{s}=\bigcup_{u\in U:domain(u)=s}L_{u}\).

 

4. Finally, generate the graph edges \((s,s^{\prime})\in E\) for each \(s\in S\) by creating an edge to each unique domain \(s^{\prime}\) in its hyperlinks \(L_{s}\) weighted by the proportion of links whose domain is \(s^{\prime}\), _i.e._\(w(s,s^{\prime})=|\{l\in L_{s}:domain(l)=s^{\prime}\}|/|L_{s}|\).

## Appendix D NewsGuard Score Details

In Section 5.2, we used the scores from the _NewswGuard_ dataset introduced in Section 4.2 to measure the correlation with estimated \(\rho(s)\) values. NewsGuard employs a team of journalists and experienced editors to produce these reliability scores for news and information websites. The score ranges from 0 to 100 and NewsGuard is transparent about the methodology used to compute it. Namely, they compute this reliability score based on the following nine apolitical criteria, each is worth the indicated number of points, based on importance:

1. **Does not repeatedly publish false content?** (22 points)
2. **Gathers and presents information responsible?** (18 points)
3. **Regularly corrects or clarifies errors?** (12.5 points)

Figure 1: Performance variation across searched values of \(n\) (left side) and \(\gamma\) (right side) on the _ExpsetB_ (solid line) and _ExpsetB\({}^{-}\)_ (dashed line) datasets. The lines represent the mean values across the 5 folds, and \(95\%\) confidence intervals are depicted. Markers highlight selected hyperparameter values.

Figure 3: 5-fold cross-validation results obtained on the _ExpsetB_ dataset with the two best strategies, _P-Reliability_ and _I-Reliability_, using different graphs. The x-axis represents the CC-News snapshot used to build the graph, and the y-axis the _macro averaged \(F_{1}\) score_.

Figure 2: News media graph built from all four CC-News snapshot (only English-speaking sources) and used for experimentation.

 

4. _Handles the difference between news and opinion responsibly? (12.5 points)_
5. _Avoids deceptive headlines? (10 points)_
6. _Website discloses ownership and financing? (7.5 points)_
7. _Clearly labels advertising? (7.5 points)_
8. _R Reveals who's in charge, including any possible conflicts of interest? (5 points)_
9. _Provides information about content creators? (5 points)_

More details can be found in the "Rating Process and Criteria" section of their website.31

Footnote 31: [https://www.newsguardtech.com/ratings/](https://www.newsguardtech.com/ratings/) rating-process-criteria/

## Appendix E Detailed Results

Table 7 shows the detailed evaluation results obtained on the three experimentation sets along with two naive baselines for reference, random and majority class classifiers. Furthermore, for _ExpsetA_, we also report the results obtained using the classification models introduced in previous works [1] along with the ensemble results. We can see that on _ExpsetA_, unlike on the other experimental sets, our strategies have 100% precision for the _unreliable_ sources but its recall on the same sources is quite low (from 22.20% to 38.72%) showing the models are detecting only a small portion of unreliable sources but with high precision (probably only the easiest cases). By performing the ensemble with Bay models, the recall for the _unreliable_ group increases allowing the graph-based strategies to identify more unreliable sources, in turn improving the macro average F\({}_{1}\) scores.

Finally, in Figure 4 is shown the scatter plot showing the correlation between the ranking obtained by PageRank values and the one obtained by the News Guard scores. Likewise, Figures 5, 6, 7, and 8 show the scatter plots for _F-Reliability_, _P-Reliability_, _FP-Reliability_, _I-Reliability_, respectively.

Figure 4: Scatter plot showing the correlation between the rankings obtained by PageRank values (y-axis) and News Guard scores (x-axis).

 

* [14] M. C. R. Jones, J. S. Laird, and D. A. Smith, _The New York Times of the Association for Computational Statistics_, vol. 2003, p. 579-684, 2000.

[MISSING_PAGE_POST]

 

Figure 5: Scatter plot showing the correlation between the rankings obtained by _F-Reliability_ values (y-axis) and News Guard scores (x-axis). Left side without rewards and right side with rewards.

Figure 6: Scatter plot showing the correlation between the rankings obtained by _P-Reliability_ values (y-axis) and News Guard scores (x-axis). Left side without rewards and right side with rewards.

 

Figure 8: Scatter plot showing the correlation between the rankings obtained by _I-Reliability_ values (y-axis) and News Guard scores (x-axis). Left side without rewards and right side with rewards.

Figure 7: Scatter plot showing the correlation between the rankings obtained by _FP-Reliability_ values (y-axis) and News Guard scores (x-axis). Left side without rewards and right side with rewards.