Venue: NAACL 2019

# Fake News Detection using Deep Markov Random Fields

Duc Minh Nguyen\({}^{*\dagger}\), Tien Huu Do\({}^{*\dagger}\), Robert Calderbank\({}^{\ddagger}\), Nikos Deligiannis\({}^{*\dagger}\),

\({}^{*}\)Vrije Universiteit Brussel, Pleinlaan 2, B-1050 Brussels, Belgium

\({}^{\dagger}\)imec, Kapeldreef 75, B-3001 Leuven, Belgium

\({}^{\ddagger}\)Duke University, Durham, North Carolina

\({}^{*\dagger}\){mdnguyen, thdo, ndelligia}@etrovub.be

\({}^{\ddagger}\)robert.calderbank@duke.edu

###### Abstract

Deep-learning-based models have been successfully applied to the problem of detecting fake news on social media. While the correlations among news articles have been shown to be effective cues for online news analysis, existing deep-learning-based methods often ignore this information and only consider each news article individually. To overcome this limitation, we develop a graph-theoretic method that inherits the power of deep learning while at the same time utilizing the correlations among the articles. We formulate fake news detection as an inference problem in a Markov random field (MRF) which can be solved by the iterative mean-field algorithm. We then unfold the mean-field algorithm into hidden layers that are composed of common neural network operations. By integrating these hidden layers on top of a deep network, which produces the MRF potentials, we obtain our deep MRF model for fake news detection. Experimental results on well-known datasets show that the proposed model improves upon various state-of-the-art models.

## 1 Introduction

The term "fake news" refers to news articles that is intentionally and verifiably false Shu et al. (2017). The problem of fake news has existed since the appearance of the printing press, but only gained a lot of momentum and visibility during the age of social media. This is due to the large audience, easy access and fast dissemination mechanism of social media, where more and more users are consuming news in a daily basis Shu et al. (2017). Traditional methods for verifying the veracity of news that rely on human experts, despite being reliable, do not scale well to the massive volume of news nowadays. This renders the automatic detection of fake news on social media an important problem, drawing a lot of attention from both the academic and industrial communities.

The recent literature has witnessed the success of deep learning models in detecting fake news on social media Ma et al. (2016); Yu et al. (2017); Ruchansky et al. (2017); Rashkin et al. (2017); Ma et al. (2018a); Kochkina et al. (2018). By leveraging the capability of deep networks in learning high-level representations, these models have achieved state-of-the-art performance on various benchmark datasets. Nevertheless, one limitation of existing deep-learning-based methods is that they often ignore the correlations among news articles, which have been proved to be effective for analysing online events Freire et al. (2016); Fairbanks et al. (2018).

To overcome this limitation, we aim at a model that leverages the capability of deep neural networks while effectively incorporating the correlations among articles when determining their credibility. To this end, we first model the relationship between two articles by the number of the com

Figure 1: Modeling the relationship among news articles (or events): the dash lines represent engagements of users to articles, the solid lines represent the relationships between articles with weights determined by the number of common engaged users.



mon users that engage to them, e.g., by means of tweeting, re-tweeting, commenting. An example is illustrated in Fig. 1: the articles \(a_{2}\) and \(a_{3}\) have a strong relationship as they are engaged to by \(5\) common users, whereas, there is no relationship between the articles \(a_{1}\) and \(a_{2}\) because there is no user engage to both of them. With this modeling, we can construct a _news graph_, where each node corresponds to an article and an edge encodes the relationship between two articles.

Our underlying assumption is that if there exists a strong relationship between two articles, they are likely to share the same labels. To verify this assumption, we calculate the percentages of the edges that connect two articles with the same labels, among those whose weights are equal to certain values, and plot the results for the news graph constructed from the Weibo dataset Ma et al. (2016) in Fig. 2. It is clear from Fig. 2 that the higher the edge weight, the more likely it is that the corresponding articles share the same labels (_fake_ or _true_). Similar patterns are observed on other datasets such as the Twitter Ma et al. (2016) and the PHEME Zubiaga et al. (2017) datasets. Evidently, these results support our assumption.

In order to incorporate the correlations among news articles, we formulate fake news detection as an inference problem in a Markov random field (MRF). Our motivation behind this formulation is to leverage the capability of MRF in capturing dependencies among random variables. We solve the resulting inference problem using the mean-field algorithm Koller and Friedman (2009). We then propose a method to unfold this algorithm into hidden layers that can be integrated on top of a deep network that computes the potentials of the MRF. By doing this, we obtain our deep MRF model for detecting fake news, referred to as the DMFN model. To the best of our knowledge, this is the first integration of deep networks and MRF for detecting fake news. Our main contributions are as follows:

* We formulate fake news detection as an inference problem in an MRF model. This allows us to incorporate the correlations among news articles when deciding their credibility.
* We propose a method to unfold the mean-field algorithm into specially-designed neural network layers, and build a deep MRF model for detecting fake news.
* We carry out comprehensive experiments on widely-used datasets collected from popular social networks. Experimental results show the effectiveness of the proposed model compared to various state-of-the-art models.

The remainder of the paper is organized as follows: we review the related work in Section 2, and describe our formulation of fake news detection as an inference problem MRF in Section 3. In Section 4, we describe our model in detail. We present our experimental studies in Section 5 and finally draw the conclusions in Section 6.

## 2 Related Work

Early work in fake news detection focused on finding a good set of features that are useful for separating fake news from genuine news. Linguistic patterns, such as special characters, specific keywords and expression types, have been explored to spot fake news Castillo et al. (2011); Liu et al. (2015); Zhao et al. (2015). Different feature types have also been considered, such as the characteristics of users involved in spreading the news, e.g. the number of followers, the users' ages and genders Castillo et al. (2011); Yang et al. (2012), and the news' propagation patterns Castillo et al. (2011); Kwon et al. (2013). Instead of relying on a single feature type, existing works normally made use of multiple types of feature at the same time.

Recent years have witnessed the use of deep learning for fake news detection. The idea is to leverage deep neural networks to overcome the limitation of the _shallow_ hand-craft features employed in the earlier methods Ma et al. (2016).

Figure 2: Percentage of edges having certain weights, which connects two articles with the same labels on the news graph constructed from the Weibo dataset Ma et al. (2016).

 

Many works proposed to represent news articles as multivariate time series using the timestamp information and to formulate fake news detection as a sequence classification problem [16, 20, 34, 59]. As it is common in the literature to utilize multiple types of features in detecting fake news, deep networks with multiple branches to incorporate various feature types were also proposed [7, 8, 21]. In general, deep learning based methods yield higher accuracy compared to shallow-feature-based approaches, leading to state-of-the-art performance.

The existing deep-learning models, however, often ignore the correlations among news articles which haven been shown to be effective cues for analysing online news and events [14, 15]. Freire et al. proposed a method to detect breaking news on Wikipedia by exploring the graph of related events [14]. This graph is built by connecting any pair of pages on Wikipedia which were edited by the same users in a small time window. In [15], a graph of news was constructed by connecting the corresponding web pages using the links between them in form of html tags. The credibility of the news was assessed by applying the loopy belief propagation algorithm to perform semi-supervised learning on this graph. In their experiments, the authors showed that the correlations among the news, encoded in the constructed graph, were more effective than the textual content of the news for predicting their credibility.

In this work, we propose a deep MRF model for fake news detection. Apart from leveraging the power of deep networks, our model incorporates the correlations among news articles when determining their credibility. In this regard, our method shares a similar motivation with the graph-based methods for social event analysis as mentioned above. Nevertheless, while these graph-based methods do not utilize any information of the news articles beyond the labels, our model is more generic and allows incorporating an arbitrary number of features.

## 3 Fake News Detection on MRF

We employ a Markov random field (MRF) model [14] to incorporate the correlations among the events due to its capability in capturing dependencies among random variables. To this end, we first construct an event graph \({\mathcal{G}}=({\mathcal{V}},{\mathcal{E}})\), with \({\mathcal{V}}\) the set of vertices and \({\mathcal{E}}\) the set of edges as described in Section 1. We then define an MRF model over \({\mathcal{G}}\). In this model a node \(k\) is associated with a random variable \(X_{k}\), which represents the label of the \(k\)-th event. The random variables \(X_{k},\forall k\in\left\{1,\ldots,n\right\}\) have domain \({\mathcal{L}}=\left\{{\mathcal{L}}_{1},{\mathcal{L}}_{2}\ldots{\mathcal{L}}_{ s}\right\}\), which represent the \(s\) possible labels. For notation brevity, we refer to the nodes in \({\mathcal{V}}\) by their indices, namely, \(k\in\left\{1,\ldots,n\right\}\). We are interested in inferring the distribution \(P(X)\) of the MRF, from which the labels for the events can be obtained. In the MRF, the probability \(P(X={\rm x})\), with \({\rm x}\) a set of values of the random variables, is given by [13]

\[P(X={\rm x})=\frac{1}{Z}\exp(-E({\rm x})), \tag{1}\]

with \(Z\) the partition function ensuring a valid distribution and \(E({\rm x})\) the energy of the MRF, which has the following form

\[E({\rm x})=\sum_{k\in{\mathcal{V}}}\Phi\left({\rm x}^{u}_{k}\right)+\lambda \sum_{k,l\in N}\Psi\left({\rm x}^{u}_{k},{\rm x}^{v}_{l}\right). \tag{2}\]

In (2), \(N\) is the set of nodes that are connected in the MRF . The _unary potential_, \(\Phi\left({\rm x}^{u}_{k}\right)\), measures the cost of assigning the label \({\mathcal{L}}_{u}\) to the node \(k\), while the _pairwise potential_, \(\Psi({\rm x}^{u}_{k},{\rm x}^{v}_{l})\), measures the cost of assigning the nodes \(k\) and \(l\) respectively the labels \({\mathcal{L}}_{u}\) and \({\mathcal{L}}_{v}\). As such, the pairwise potentials capture the dependencies among the nodes of the MRF. \(\lambda\) is a hyperparameter.

As exactly computing \(P(X)\) is intractable, we employ the mean-field algorithm [13] to approximate \(P(X)\) by a fully-factorized proposal distribution \(Q(X)=\prod_{\forall k\in{\mathcal{V}}}Q_{k}(X_{k})\). \(Q_{k}(X_{k}={\mathcal{L}}_{u})\) is the probability that the node \(k\) have the label \({\mathcal{L}}_{u}\) according to the distribution \(Q_{k}\). Denote \(Q_{k}(X_{k}={\mathcal{L}}_{u})\) as \(q^{u}_{k}\), the mean-field algorithm iteratively calculates \(q^{u}_{k},\forall k\in\left\{1,\ldots,n\right\}\), \(u\in\left\{1,\ldots,s\right\}\) according to [13]

\[\begin{split} q^{u}_{k}&=\frac{1}{Z_{k}}\exp\left\{- \left(\Phi\left({\rm x}^{u}_{k}\right)\right.\right.\\ &\quad\quad+\left.\lambda\sum_{l\in N_{k}}\sum_{v\in{\mathcal{ L}}}q^{v}_{l}\Psi\left({\rm x}^{u}_{k},{\rm x}^{v}_{l}\right)\right)\right\}, \end{split}\] (3) 

with \(N_{k}\) the set of nodes connected to \(k\) and \(Z_{k}=\sum_{u\in\mathcal{L}}q^{u}_{k}\) the normalization factor ensuring \(q^{u}_{k}\), \(\forall u\in\{1,\ldots,s\}\), add up to \(1\). The generic mean-field update in (3) requires the unary potentials and pairwise potentials. In Section 4, we show how we realize these potentials.

## 4 Deep MRF Model for Fake News Detection

In this section, we present our realizations of the unary and pairwise potentials, with which we obtain the final mean-field update equation. After that, we present a method to unfold the mean-field update into specially-designed neural network layers, and describe our deep MRF model for fake news detection.

### Computing the Unary and Pairwise Potentials

We compute the unary potential \(\Phi(\mathrm{x}^{u}_{k})\) of the MRF as the negative log likelihood:

\[\Phi(\mathrm{x}^{u}_{k})=-\ln p(X_{k}=\mathcal{L}_{u}), \tag{4}\]

with \(p(X_{k}=\mathcal{L}_{u})\) the likelihood that \(X_{k}\) has label \(\mathcal{L}_{u}\). As such, \(\Phi(\mathrm{x}^{u}_{k})\) will be high if this node is not likely to have the label \(\mathcal{L}_{u}\), and vice versa. The likelihood \(p(X_{k}=\mathcal{L}_{u})\) is computed as \(p(X_{k}=\mathcal{L}_{u})=\mathcal{F}_{\theta}(\mathcal{O}_{k})\), where \(\mathcal{F}_{\theta}\) is a non-linear function represented by a deep neural network parameterized by \(\theta\) and \(O_{k}\) is the observations, i.e, features associated to the \(k\)-th event. The design of this network is described later in Section 4.

The pairwise potential is computed by

\[\Psi(\mathrm{x}^{u}_{k},\mathrm{x}^{v}_{l})=a(k,l)\times\mu(u,v), \tag{5}\]

where \(a(k,l)\) is the weight of the edge between the nodes \(k\) and \(l\), namely, \(a(k,l)\) represents how strong the relationship between the nodes \(k\) and \(l\) is; and \(\mu(u,v)\) is the _label compatibility_, which is calculated using the Pott model Boykov et al. (1998):

\[\mu(u,v)=\begin{cases}1,&\text{if $u\neq v$},\\ 0,&\text{otherwise}.\end{cases} \tag{6}\]

Substituting the pairwise potential in (5) into (3), we obtain the final mean-field update equation:

\[\begin{split} q^{u}_{k}&=\frac{1}{Z_{k}}\exp\left\{-\Phi \left(x^{u}_{k}\right)\right.\\ &\quad-\lambda\sum_{l\in N_{k}}a(k,l)\sum_{v\in\mathcal{L}}q^{v}_{ l}\times\mu\left(u,v\right)\right\}.\end{split} \tag{7}\]

We refer to the term \(\sum_{v\in\mathcal{L}}q^{v}_{l}\times\mu(u,v)\) in (7) as the _compatibility transform_ step, and to the term \(\sum_{l\in N_{k}}a(k,l)\sum_{v\in\mathcal{L}}q^{lv}_{l}\times\mu\left(u,v\right)\) summing up information from the nodes connected to the node \(k\) as the _message passing_ step.

### Mean-Field Updates as Neural Network Layers

We now describe a method to implement the update equation in (7) using operations that are common in the deep learning literature. Abusing the notation, we denote also by \(Q\in\mathbb{R}^{n\times s}\) a matrix, with an entry \(Q_{k,u}\) corresponding to the value \(q^{u}_{k}\), i.e., the probability that the node \(k\) has the label \(\mathcal{L}_{u}\). Denote by \(A\in\mathbb{R}^{n\times n}\) the adjacency matrix, containing all the edge weights in \(\mathcal{G}\). We set \(A_{k,k}=0,\forall k\in\{1,\dots,n\}\). We further denote by \(M\in\mathbb{R}^{s\times s}\) a square matrix whose the entry \(M_{u,v}\) corresponds the label compatibility \(\mu(u,v)\) calculated using (6), and by \(\Phi\in\mathbb{R}^{nx\times s}\) the matrix containing the unary potentials \(\Phi(\mathrm{x}^{u}_{k})\), for all \(k\in\{1,\dots,n\}\) and \(u\in\{1,\dots,s\}\). \(\Phi\) is calculated by taking the logarithm of \(Q\) element-wise.

#### 4.2.1 The Compatibility Transform Step

The compatibility transform in (7) can be performed via a 1-D convolutional layer applied on the matrix \(Q\). This convolutional layer has \(s\) filters of kernel size \(1\times s\). The weights of the \(u\)-th filter are set equal to the values along the \(u\)-th row of \(M\). We do not employ any padding and set the stride to \(1\). When applying this operation, the \(u\)-th filter slides vertically across \(Q\), and calculates the inner product between its weights and the rows of \(Q\). The output is denoted as \(\mathcal{Q}^{{}^{\prime}}\in\mathbb{R}^{n\times s}\), with an entry \(Q^{{}^{\prime}}_{l,u}\) given by

\[Q^{{}^{\prime}}_{l,u} =\sum_{v\in\mathcal{L}}Q_{l,v}\times M_{u,v}\] \[=\sum_{v\in\mathcal{L}}q^{v}_{l}\times\mu(u,v), \tag{8}\]

for all \(l\in\{1,\dots,n\}\), \(u\in\{1,\dots,s\}\), which is equal to the result of the compatibility transform step in (7) 



#### 4.2.2 The Message Passing Step

Given the adjacency matrix and the output from the compatibility transform step, the message passing step can be performed simply by multiplying with . This multiplication results in , in which:

(9)

As , the operation in is equivalent to the message passing step in (7).

#### 4.2.3 The Mean-field Layer

To finish the update in (7), one needs to element-wise multiply with , add and negate the result. The exponential and normalization can be performed jointly using the _softmax_ function, resulting in the new matrix whose entries correspond to the values of after one mean-field iteration, for all , . We can consider these operations together with those implementing the compatibility transform and the message passing steps, as the operations of specially-designed a neural network layer, which we call the mean-field (MF) layer. As all the component operations are differentiable, the operation implemented by the MF layer is also differentiable. Each MF layer, hence, implements one iteration of the mean-field algorithm. Clearly, by stacking MF layers, we can implement the mean-field algorithm with iterations.

### The DMFN Model

#### 4.3.1 Model Architecture

The architecture of the DMFN model is illustrated in Figure 3, with two blocks: the first block corresponds to the deep network that produces the unary potentials, and the second block is composed of MF layers, which implements the -iteration mean-field algorithm. The deep network takes an inputs several feature types extracted to represent the given set of events. Each feature type is processed by a feature branch with a number of fully-connected (FC) layers. The outputs of the last layers in all feature branches are concatenated to produce high-level representations for all the events. These representations are then fed to another set of FC layers and a softmax function to produce the label probabilities. These label probabilities form the matrix , as described in Section 4.2. All the FC layers in the model are followed by a batch normalization layer Ioffe and Szegedy (2015), a ReLU activation function Glorot et al. (2011), and the dropout regularization Srivastava et al. (2014).

In the second block, all the MF layers share the same parameters, namely, the matrix , the adjacency matrix that encodes the relationships

Figure 3: The architecture of the DMFN model: the first block consists of three feature branches, each with configurable numbers of fully-connected (FC) layers to process a type of feature, a concatenation layer and a number of FC layers with softmax activation function at the end to produce labels’ probability matrix ; Each (MF) layer consists of four main operations, namely convolution (conv), matrix multiplication (), point-wise addition () and the softmax function . MF layers are stacked one after another to implement the mean-field algorithm with iterations. All MF layers share the same set of parameters, namely, the matrix of the unary potentials , which is computed from , the adjacency matrix encoding the relationships among the input events, and the compatibility matrix .



among the given events, and the label compatibility matrix \(M\). The first MF layer takes as input the matrix \(Q\), whereas, the later MF layers operate on the outputs from their previous MF layers. The output after the last MF layer is the final label probabilities predicted for the given events. It should be noted that the parameters of the MF layers are pre-computed and shared. As such, adding MF layers after the layers in the first block does not increase the the risk of overfitting of the model.

#### 4.3.2 Feature Extraction

We extract multiple types of features to capture the characteristics of the events, namely, the textual contents and social engagements. For the textual features, we first group all the tweets related to an event into a document. We preprocess the documents by removing stop words, converting the words into lower case and tokenizing them. From the pre-processed documents, we extract the _term frequency - inverse document frequency_ (tf-idf) Wu et al. (2008), and _word2vec_ Mikolov et al. (2013) feature vectors. We use the word2vec model pre-trained on the Google News dataset to map each word in a document to a \(300-\)dimensional embedding vector, and then obtain the embedding for the document by taking the average of the word embeddings.

We use a graph embedding technique to capture the social engagements associated to the events. Concretely, we first build a graph of users from the given dataset. Two users are connected by an edge if they engage to at least one event in common, with the edge's weight determined by the total number of common engaged events. We then employ the _node2vec_ algorithm Grover Leskovec (2016) to learn an embedding for each user. The social engagement feature of an event is then calculated by taking the average of the embeddings of all users engaged to it. It is worth noting that the graph of users used in this step is different from the graph of events, constructed as described in Section 1.

#### 4.3.3 Training and Testing the DMFN Model

We employ the weighted cross entropy loss function to train the DMFN model. When calculating the loss, the weight given to the training samples of a particular label is inversely proportional to the number of samples in the current batch which have this label. This technique is highly beneficial when dealing with imbalanced dataset, e.g., the PHEME dataset Zubiaga et al. (2016). The model's parameters are learned by using the SGD algorithm with the Adam parameter update Kingma and Ba (2015).

At the testing stage, we select for the testing event \(k\) the label \({\mathcal{L}}_{\hat{u}}\in{\mathcal{L}}\) with \(\hat{u}\) determined by \(\hat{u}=\operatorname*{arg\,max}_{u\in\{1,...,s\}}q^{u}_{k}\). As we want to utilize the correlations among events in the prediction, we provide an adjacency matrix encoding the relationships among a set of events and run a forward pass with all their features as input. Without the adjacency matrix or when setting the adjacency matrix to the identity matrix, the model predicts the labels for each events without considering their correlations.

## 5 Experiments

### Experimental Settings

We employ three well-known benchmark datasets, namely the Twitter, Weibo Ma et al. (2016) and PHEME datasets Zubiaga et al. (2017) for our experiments. The Twitter dataset consists of \(992\) events, involving \(233.7\) thousand users and \(592.4\) thousand tweets. The Weibo dataset is larger, with \(4,664\) events, \(2.8\) million users and \(3.8\) million posts. Events in these datasets are labeled as either _fake_ or _true_, and the two labels are relatively balanced. The PHEME dataset consists of \(5,802\) comment threads collected from Twitter, with approximately \(103\) thousand tweets in total. This dataset is imbalanced, with \(1,972\) threads labeled as _rumour_ and \(3,830\) threads labeled as _non-rumour_.

### Hyperparameter Selection

For the DMFN model, we employ one hidden layer in each feature branch, and one hidden layer after the concatenation layer, all with \(100\) hidden units. We train the model for maximum \(100\) epochs with learning rate \(0.001\) and stop training early if the validation loss does not improve over the average of those of the previous \(25\) epochs. To control overfitting, we employ dropout with high dropping rate of \(0.9\).

We determine the values of \(\lambda\) which balance the weight between the unary and pairwise potentials in the MRF, and of the number of MF layers, \(T\) by cross validation on a separate split on the Weibo dataset. First, we fix \(T\) to \(30\), with which the mean-field algorithm is highly likely to converge Krahenbuhl Koltun (2011), and experi 

ment with different values of \(\lambda\). The result of this is presented in Table 1.

Fixing \(\lambda\) to \(0.05\), we experiment with different number of MF layers by varying \(T\). The results are summarized in Table 2. As can be seen from the table, employing multiple MF layers improves the results over just one MF layer. Even though we still observe improvements in the performance with more than \(5\) MF layers, the difference is small. As a result, we select \(T=5\) as it produces the best trade-off between accuracy and computational complexity.

### Experimental Results

We compare the results of different models in two experimental settings, namely _late detection_ and _early detection_. The former setting allows the models to use all the available users posts in the entire time span of the given datasets, whereas in the latter setting, the models are only allowed to use posts that have appeared within a specific deadline (in hours) since the appearances of the events.

#### 5.3.1 Late Detection Setting

We compare the results of the proposed models in the late detection setting with those of reference models, including the decision tree classifier (DTC) Castillo et al. (2011), the SVM classifier (SVM-RBF) Yang et al. (2012), the random forest classifier (RFC) Kwon et al. (2013), the SVM classifier with timeseries features (SVM-TS) Ma et al. (2015), the 2-layer GRU model (GRU-2) Ma et al. (2016) and the convolutional neural network (CAMI) Yu et al. (2017), the Tree-structured Recursive Neural Networks (TD-RvNN) Ma et al. (2018b), and the CRF and Naive Bayes classifiers Liuzubiaga et al. (2017). The performance of the models is assessed using four metrics, namely, accuracy, average precision, average recall and macro F1 score. Similar to Yu et al. (2017), on the Twitter and Weibo datasets, we randomly reserve \(10\%\) of the samples for parameter tuning and perform four-fold cross validation on the remaining. On the PHEME dataset, we follow the leave-one-event-out approach as in Zubiaga et al. (2017). We report the average results for the models.

The results for different models on the Twitter and the Weibo datasets Ma et al. (2016) are shown in Table 3. On these two datasets, we do not include the results of the TD-RvNN model Ma et al. (2018b) as this model requires a tree-like connections among the tweets to represent an event. As can be seen from the table, the DMFN model consistently outperforms the reference models on both datasets. The results of all the models are better on the Weibo dataset than on the Twitter dataset. This is possibly because number of posts available on the Weibo dataset is much larger than that available on the Twitter dataset.

The results on the PHEME dataset is illustrated in Table. 4. As this dataset is imbalanced, the prediction accuracy is not a good metric for comparison. Similar to Zubiaga et al. (2017), we focus on the other metrics, namely, precision, recall and macro F1 score in the comparison. The CRF model with content features Zubiaga et al. (2017) yields the highest precision, whereas the Naive Bayes classifiers yield the highest recalls.. While the reference models are either biased toward high precision or high recall scores, the DMFN model produces more balanced precision and recall, and the best Macro F1 score. The DF-RvNN model Ma et al. (2018a) also achieves balanced precision and recall, nevertheless its performance is lower than that of the DMFN model in all metrics (with p-values of \(0.004\), \(0.04\), \(0.03\) respectively for the precision, recall and macro f1 scores in a pairwise t-test).

On the PHEME dataset, the average number of tweets per event is \(17.8\), which is much smaller than those on the Twitter and Weibo datasets (\(805\) and \(815\) posts per event respectively). The lower number of tweets, and the class imbalance render this dataset highly challenging. As such, the performance of all the considered models on this dataset is lower than that on the Twitter and Weibo datasets. Overall on all the considered datasets, we observe consistent performance of the DMFN model: The variance of the Macro F1 score over 10 repetitions are relatively low, which are equal to

\begin{table}
\begin{tabular}{|c|c|c|c|c|} \hline \(T\) & \(1\) & \(5\) & \(10\) & \(15\) \\ \hline Weibo & \(0.949\) & \(0.960\) & \(0.960\) & \(0.960\) \\ \hline \end{tabular}
\end{table}
Table 2: Accuracy of the DMFN model when varying \(T\) on the Weibo datasets Ma et al. (2016).

 

\(1{e}-4\), \(2{e}-4\) and \(6{e}-5\) respectively on the Twitter, Weibo and PHEME datasets.

#### 5.3.2 Early Detection Setting

We perform the early detection experiments on the Twitter and Weibo datasets with different deadlines, in \(\{1,5,12,24,36,48,72,96\}\) (hours). Fig. 3(a) and Fig. 3(b) illustrate the results on the Twitter and the Weibo datasets, respectively. As can be seen, on both datasets, the DMFN model performs the best among the selected models, followed by the CAMI model. On the Weibo dataset, the average number of posts per event is approximately \(168\), \(353\), \(497\) within the 1-hour, 5-hour and 12-hour deadlines respectively.These figures suggest that Weibo users are responsive and quickly react to a newly broadcasted event. The large number of posts per event, even at 1-hour deadline, gives enough information for the DMFN, as well as the CAMI models to produce good results, even within short deadlines.

#### 5.3.3 The Effects of Jointly Training the Deep Network with Mean-field Inference

An advantage of the proposed model is that it allows training the deep network that produces the unary potentials with feedback from the mean-field inference. To verify this argument, we compare the performance when using different variants of the proposed model: (_i_) training and testing without the MF inference, (_ii_) training without the MF inference and testing with the MF inference, and (_iii_) training and testing with the MF inference (the full DMFN model). We denote the three variants as _DMFN-base_, _DMFN-separate_ and _DMFN_. As can be seen from the table, applying the MF inference improves the results of the base model even if it has been trained without the MF inference. This proves the benefits of enforcing the dependencies between the events in the MRF when making predictions. Nevertheless, training and testing with MF inference consistently yields the best results among the three variants. This proves

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline Model & Prec. & Rec. & Macro F1 \\ \hline \hline Naive Bayes Content & \(0.309\) & \(\mathbf{0.723}\) & \(0.433\) \\ \hline CRF Content & \(\mathbf{0.683}\) & \(0.545\) & \(0.606\) \\ \hline Naive Bayes Content + Social & \(0.310\) & \(\mathbf{0.723}\) & \(0.434\) \\ \hline CRF Content + Social & \(0.667\) & \(0.556\) & \(0.607\) \\ \hline TD-RvNN & \(0.616\) & \(0.612\) & \(0.609\) \\ \hline DMFN & \(0.667\) & \(0.670\) & \(\mathbf{0.657}\) \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 4: Results for different models on the PHEME datasets [2].

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|c|c|c|} \hline \multirow{2}{*}{Model} & \multicolumn{4}{c|}{Twitter} & \multicolumn{4}{c|}{Webio} \\ \cline{2-9}  & Acc. & Prec. & Rec. & Macro F1 & Acc. & Prec. & Rec. & Macro F1 \\ \hline \hline SVM-RBF & \(0.715\) & \(0.720\) & \(0.710\) & \(0.709\) & \(0.818\) & \(0.819\) & \(0.818\) & \(0.818\) \\ \hline DTC & \(0.718\) & \(0.718\) & \(0.718\) & \(0.718\) & \(0.831\) & \(0.831\) & \(0.831\) \\ \hline RFC & \(0.728\) & \(0.728\) & \(0.728\) & \(0.728\) & \(0.849\) & \(0.866\) & \(0.849\) & \(0.847\) \\ \hline SVM-TS & \(0.745\) & \(0.741\) & \(0.741\) & \(0.740\) & \(0.857\) & \(0.859\) & \(0.858\) & \(0.859\) \\ \hline GRU-2 & \(0.757\) & \(0.760\) & \(0.757\) & \(0.771\) & \(0.910\) & \(0.914\) & \(0.910\) & \(0.910\) \\ \hline CAMI & \(0.777\) & \(0.782\) & \(0.777\) & \(0.776\) & \(0.933\) & \(0.933\) & \(0.933\) & \(0.933\) \\ \hline DMFN & \(\mathbf{0.800}\) & \(\mathbf{0.803}\) & \(\mathbf{0.803}\) & \(\mathbf{0.799}\) & \(\mathbf{0.962}\) & \(\mathbf{0.963}\) & \(\mathbf{0.962}\) & \(\mathbf{0.970}\) \\ \hline \end{tabular}
\end{table}
Table 3: Results for different models on the Twitter and Weibo datasets [11].

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|c|} \hline \multirow{2}{*}{Model} & \multicolumn{2}{c|}{Twitter} & \multicolumn{2}{c|}{Weibo} & \multicolumn{2}{c|}{PHEME} \\ \cline{2-7}  & Acc. & Macro F1 & Acc. & Macro F1 & Acc. & Macro F1 \\ \hline DMFN-base & \(0.763\) & \(0.762\) & \(0.944\) & \(0.944\) & \(0.682\) & \(0.607\) \\ \hline DMFN-separate & \(0.789\) & \(0.788\) & \(0.958\) & \(0.959\) & \(0.690\) & \(0.641\) \\ \hline DMFN & \(\mathbf{0.800}\) & \(\mathbf{0.799}\) & \(\mathbf{0.962}\) & \(\mathbf{0.970}\) & \(\mathbf{0.703}\) & \(\mathbf{0.657}\) \\ \hline \end{tabular}
\end{table}
Table 5: Results of different variants of the DMFN model on the three datasets.

 

the benefits of unfolding the MF inference and incorporate it on top of the base network.

## 6 Conclusion

We formulated the fake news detection on social media as an inference problem in an MRF model that can be solved using the mean-field algorithm. By translating each update step in this algorithm into common operations in the deep learning literature, we can unfold it into hidden layers that can be integrated on top of another deep neural network. This results in our deep MRF model (DMFN) for detecting fake news. As such, the DMFN carries the advantages of both deep neural networks in learning high-level representations, and of MRF in incorporating correlations among the news articles. Experiments on well-known benchmark datasets show that the proposed model consistently improves over the state of the art in fake news detection in both the late and early detection settings.

## Acknowledgments

The authors acknowledge the financial support from the Vrije Universiteit Brussel (PhD bursary - Duc Minh Nguyen), the Fonds Wetenschappelijk Onderzoek Vlaanderen (FWO) and the Francqui Foundation (2016-2017 International Francqui Chair - Robert Calderbank).

## References

* Boykov et al. (1998) Yuri Boykov, Olga Veksler, and Ramin Zabih. 1998. Markov random fields with efficient approximations. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 648-655.
* Castillo et al. (2011) Carlos Castillo, Marcelo Mendoza, and Barbara Poblete. 2011. Information credibility on twitter. In _ACM International Conference on World Wide Web (WWW)_, pages 675-684.
* Fairbanks et al. (2018) James Fairbanks, Natalie Fitch, Nathan Knauf, and Erica Briscoe. 2018. Credibility assessment in the news : Do we need to read ? In _Misinformation and Misbehavior Mining on the Web Workshop (MIS2)_, pages 1-8.
* Freeman et al. (2000) William T. Freeman, Egon C. Pasztor, and Owen T. Carmichael. 2000. Learning low-level vision. _International Journal of Computer Vision_, 40(1):25-47.
* Freire et al. (2016) Ana Freire, Matteo Manca, Diego Saez-Trumper, David Laniado, Ilaria Bordino, Francesco Gullo, and Andreas Kaltenbrunner. 2016. Graph-based breaking news detection on wikipedia. In _AAAI Conference on Web and Social Media (ICWSM) Workshops_, pages 1-2.
* Glorot et al. (2011) Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Deep sparse rectifier neural networks. In _International Conference on Artificial Intelligence and Statistics (AISTATS)_, pages 315-323.
* Grover and Leskovec (2016) Aditya Grover and Jure Leskovec. 2016. Node2vec: Scalable feature learning for networks. In _ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_, pages 855-864.
* Ioffe and Szegedy (2015) Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In _International Conference on Machine Learning (ICML)_, pages 448-456.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: A method for stochastic optimization. In _International Conference on Learning Representations (ICLR)_.

Figure 4: Accuracy of different models when considering only posts within specific deadlines on the Twitter and Weibo datasets.

 

* Kochkina et al. (2018) Elena Kochkina, Maria Liakata, and Arkaitz Zubiaga. 2018. All-in-one: Multi-task learning for rumour verification. In _International Conference on Computational Linguistics (COLING)_.
* Koller and Friedman (2009) Daphne Koller and Nir Friedman. 2009. In _Probabilistic Graphical Models: Principles and Techniques_. MIT Press.
* Krahenbuhl and Koltun (2011) Philipp Krahenbuhl and Vladlen Koltun. 2011. Efficient inference in fully connected crfs with gaussian edge potentials. In _Advances in Neural Information Processing Systems (NIPS)_, pages 109-117.
* Kwon et al. (2013) Sejeong Kwon, Meeyoung Cha, Kyomin Jung, Wei Chen, and Yajun Wang. 2013. Prominent features of rumor propagation in online social media. In _IEEE International Conference on Data Mining (ICDM)_, pages 1103-1108.
* Liu et al. (2015) Xiaomo Liu, Armineh Nourbakhsh, Quanzhi Li, Rui Fang, and Sameena Shah. 2015. Real-time rumor debunking on twitter. In _ACM International on Conference on Information and Knowledge Management (CIKM)_, pages 1867-1870.
* Liu and Wu (2018) Yang Liu and Yi-Fang Brook Wu. 2018. Early detection of fake news on social media through propagation path classification with recurrent and convolutional networks. In _AAAI Conference on Artificial Intelligence_, pages 1-8.
* Ma et al. (2016) Jing Ma, Wei Gao, Prasenjit Mitra, Sejeong Kwon, Bernard J. Jansen, Kam-Fai Wong, and Meeyoung Cha. 2016. Detecting rumors from microblogs with recurrent neural networks. In _Joint Conference on Artificial Intelligence (IJCAI)_, pages 3818-3824.
* Ma et al. (2015) Jing Ma, Wei Gao, Zhongyu Wei, Yueming Lu, and Kam-Fai Wong. 2015. Detect rumors using time series of social context information on microblogging websites. In _ACM International on Conference on Information and Knowledge Management (CIKM)_, pages 1751-1754.
* Ma et al. (2018a) Jing Ma, Wei Gao, and Kam-Fai Wong. 2018a. Detect rumor and stance jointly by neural multi-task learning. In _The Web Conference (WWW)_, pages 585-593.
* Ma et al. (2018b) Jing Ma, Wei Gao, and Kam-Fai Wong. 2018b. Rumor detection on twitter with tree-structured recursive neural networks. In _Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 1980-1989.
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In _International Conference on Neural Information Processing Systems (NIPS)_, pages 3111-3119.
* Rashkin et al. (2017) Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2931-2937.
* Ruchansky et al. (2017) Natali Ruchansky, Sungyong Seo, and Yan Lin. 2017. CSI : A hybrid deep model for fake news detection. In _ACM International on Conference on Information and Knowledge Management (CIKM)_, pages 797-806.
* Shu et al. (2017) Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017. Fake news detection on social media: A data mining perspective. _SIGKDD Explorations Newsletter_, 19(1):22-36.
* Srivastava et al. (2014) N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. _Journal of machine learning research_, 15:1929-1958.
* Volkova et al. (2017) Svitlana Volkova, Kyle Shaffer, Jin Yea Jang, and Nathan Hodas. 2017. Separating facts from fiction: Linguistic models to classify suspicious and trusted news posts on twitter. In _Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 647-653.
* Wu et al. (2008) Ho Chung Wu, Robert Wing Pong Luk, Kam Fai Wong, and Kui Lam Kwok. 2008. Interpreting TF-IDF term weights as making relevance decisions. _ACM Transactions on Information Systems_, 26(3):13:1-13:37.
* Yang et al. (2012) Fan Yang, Yang Liu, Xiaohui Yu, and Min Yang. 2012. Automatic detection of rumor on sina weibo. In _ACM SIGKDD Workshop on Mining Data Semantics_, pages 13:1-13:7.
* Yang et al. (2018) Yang Yang, Lei Zheng, Jiawei Zhang, Qingcai Cui, Xiaoming Zhang, Zhoujun Li, and Philip S. Yu. 2018. TI-CNN : Convolutional neural networks for fake news detection. _ArXiv e-prints 1806.00749_.
* Yu et al. (2017) Feng Yu, Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. 2017. A convolutional approach for misinformation identification. In _International Joint Conference on Artificial Intelligence (IJCAI)_, pages 3901-3907.
* Zhao et al. (2015) Zhe Zhao, Paul Resnick, and Qiaozhu Mei. 2015. Enquiring minds: Early detection of rumors in social media from enquiry posts. In _ACM International Conference on World Wide Web (WWW)_, pages 1395-1405.
* Zubiaga et al. (2017) Arkaitz Zubiaga, Maria Liakata, and Rob Procter. 2017. Exploiting context for rumour detection in social media. In _Social Informatics_, pages 109-123.
* Zubiaga et al. (2016) Arkaitz Zubiaga, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Peter Tolmie. 2016. Analysing how people orient to and spread rumours in social media by looking at conversational threads. _PLOS ONE_, 11(3):1-29