Detecting Fake News with Capsule Neural Networks

Abstract: Fake news is dramatically increased in social media in recent years. This has prompted the need for effective fake news detection algorithms. Capsule neural networks have been successful in computer vision and are receiving attention for use in Natural Language Processing (NLP). This paper aims to use capsule neural networks in the fake news detection task. We use different embedding models for news items of different lengths. Static word embedding is used for short news items, whereas non-static word embeddings that allow incremental up-training and updating in the training phase are used for medium length or large news statements. Moreover, we apply different levels of n-grams for feature extraction. Our proposed architectures are evaluated on two recent well-known datasets in the field, namely ISOT and LIAR. The results show encouraging performance, outperforming the state-of-the-art methods by 7.8% on ISOT and 3.1% on the validation set, and 1% on the test set of the LIAR dataset.

Flexibility and ease of access to social media have resulted in the use of online channels for news access by a great number of people. For example, nearly two-thirds of American adults have access to news by online channels BIBREF0, BIBREF1. BIBREF2 also reported that social media and news consumption is significantly increased in Great Britain.
In comparison to traditional media, social networks have proved to be more beneficial, especially during a crisis, because of the ability to spread breaking news much faster BIBREF3. All of the news, however, is not real and there is a possibility of changing and manipulating real information by people due to political, economic, or social motivations. This manipulated data leads to the creation of news that may not be completely true or may not be completely false BIBREF4. Therefore, there is misleading information on social media that has the potential to cause many problems in society. Such misinformation, called fake news, has a wide variety of types and formats. Fake advertisements, false political statements, satires, and rumors are examples of fake news BIBREF0. This widespread of fake news that is even more than mainstream media BIBREF5 motivated many researchers and practitioners to focus on presenting effective automatic frameworks for detecting fake news BIBREF6. Google has announced an online service called “Google News Initiative” to fight fake news BIBREF7. This project will try to help readers for realizing fake news and reports BIBREF8.
Detecting fake news is a challenging task. A fake news detection model tries to predict intentionally misleading news based on analyzing the real and fake news that previously reviewed. Therefore, the availability of high-quality and large-size training data is an important issue.
The task of fake news detection can be a simple binary classification or, in a challenging setting, can be a fine-grained classification BIBREF9. After 2017, when fake news datasets were introduced, researchers tried to increase the performance of their models using this data. Kaggle dataset, ISOT dataset, and LIAR dataset are some of the most well-known publicly available datasets BIBREF10.
In this paper, we propose a new model based on capsule neural networks for detecting fake news. We propose architectures for detecting fake news in different lengths of news statements by using different varieties of word embedding and applying different levels of n-gram as feature extractors. We show these proposed models achieve better results in comparison to the state-of-the-art methods.
The rest of the paper is organized as follows: Section SECREF2 reviews related work about fake news detection. Section SECREF3 presents the model proposed in this paper. The datasets used for fake news detection and evaluation metrics are introduced in Section SECREF4. Section SECREF5 reports the experimental results, comparison with the baseline classification and discussion. Section SECREF6 summarizes the paper and concludes this work.
Fake news detection has been studied in several investigations. BIBREF11 presented an overview of deception assessment approaches, including the major classes and the final goals of these approaches. They also investigated the problem using two approaches: (1) linguistic methods, in which the related language patterns were extracted and precisely analyzed from the news content for making decision about it, and (2) network approaches, in which the network parameters such as network queries and message metadata were deployed for decision making about new incoming news.
BIBREF12 proposed an automated fake news detector, called CSI that consists of three modules: Capture, Score, and Integrate, which predicts by taking advantage of three features related to the incoming news: text, response, and source of it. The model includes three modules; the first one extracts the temporal representation of news articles, the second one represents and scores the behavior of the users, and the last module uses the outputs of the first two modules (i.e., the extracted representations of both users and articles) and use them for the classification. Their experiments demonstrated that CSI provides an improvement in terms of accuracy.
BIBREF13 introduced a new approach which tries to decide if a news is fake or not based on the users that interacted with and/or liked it. They proposed two classification methods. The first method deploys a logistic regression model and takes the user interaction into account as the features. The second one is a novel adaptation of the Boolean label crowdsourcing techniques. The experiments showed that both approaches achieved high accuracy and proved that considering the users who interact with the news is an important feature for making a decision about that news.
BIBREF14 introduced two new datasets that are related to seven different domains, and instead of short statements containing fake news information, their datasets contain actual news excerpts. They deployed a linear support vector machine classifier and showed that linguistic features such as lexical, syntactic, and semantic level features are beneficial to distinguish between fake and genuine news. The results showed that the performance of the developed system is comparable to that of humans in this area.
BIBREF15 provided a novel dataset, called LIAR, consisting of 12,836 labeled short statements. The instances in this dataset are chosen from more natural contexts such as Facebook posts, tweets, political debates, etc. They proposed neural network architecture for taking advantage of text and meta-data together. The model consists of a Convolutional Neural Network (CNN) for feature extraction from the text and a Bi-directional Long Short Term Memory (BiLSTM) network for feature extraction from the meta-data and feeds the concatenation of these two features into a fully connected softmax layer for making the final decision about the related news. They showed that the combination of metadata with text leads to significant improvements in terms of accuracy.
BIBREF16 proved that incorporating speaker profiles into an attention-based LSTM model can improve the performance of a fake news detector. They claim speaker profiles can contribute to the model in two different ways. First, including them in the attention model. Second, considering them as additional input data. They used party affiliation, speaker location, title, and credit history as speaker profiles, and they show this metadata can increase the accuracy of the classifier on the LIAR dataset.
BIBREF17 presented a new dataset for fake news detection, called ISOT. This dataset was entirely collected from real-world sources. They used n-gram models and six machine learning techniques for fake news detection on the ISOT dataset. They achieved the best performance by using TF-IDF as the feature extractor and linear support vector machine as the classifier.
BIBREF18 proposed an end-to-end framework called event adversarial neural network, which is able to extract event-invariant multi-modal features. This model has three main components: the multi-modal feature extractor, the fake news detector, and the event discriminator. The first component uses CNN as its core module. For the second component, a fully connected layer with softmax activation is deployed to predict if the news is fake or not. As the last component, two fully connected layers are used, which aims at classifying the news into one of K events based on the first component representations.
BIBREF19 developed a tractable Bayesian algorithm called Detective, which provides a balance between selecting news that directly maximizes the objective value and selecting news that aids toward learning user's flagging accuracy. They claim the primary goal of their works is to minimize the spread of false information and to reduce the number of users who have seen the fake news before it becomes blocked. Their experiments show that Detective is very competitive against the fictitious algorithm OPT, an algorithm that knows the true users’ parameters, and is robust in applying flags even in a setting where the majority of users are adversarial.
In this section, we first introduce different variations of word embedding models. Then, we proposed two capsule neural network models according to the length of the news statements that incorporate different word embedding models for fake news detection.
Dense word representation can capture syntactic or semantic information from words. When word representations are demonstrated in low dimensional space, they are called word embedding. In these representations, words with similar meanings are in close position in the vector space.
In 2013, BIBREF20 proposed word2vec, which is a group of highly efficient computational models for learning word embeddings from raw text. These models are created by training neural networks with two-layers trained by a large volume of text. These models can produce vector representations for every word with several hundred dimensions in a vector space. In this space, words with similar meanings are mapped to close coordinates.
There are some pre-trained word2vec vectors like 'Google News' that was trained on 100 billion words from Google news. One of the popular methods to improve text processing performance is using these pre-trained vectors for initializing word vectors, especially in the absence of a large supervised training set. These distributed vectors can be fed into deep neural networks and used for any text classification task BIBREF21. These pre-trained embeddings, however, can further be enhanced.
BIBREF21 applied different learning settings for vector representation of words via word2vec for the first time and showed their superiority compared to the regular pre-trained embeddings when they are used within a CNN model. These settings are as follow:
Static word2vec model: in this model, pre-trained vectors are used as input to the neural network architecture, these vectors are kept static during training, and only the other parameters are learned.
Non-static word2vec model: this model uses the pre-trained vectors at the initialization of learning, but during the training phase, these vectors are fine-tuned for each task using the training data of the target task.
Multichannel word2vec model: the model uses two sets of static and non-static word2vec vectors, and a part of vectors fine-tune during training.
Although different models based on deep neural networks have been proposed for fake news detection, there is still a great need for further improvements in this task. In the current research, we aim at using capsule neural networks to enhance the accuracy of fake news identification systems.
The capsule neural network was introduced by BIBREF22 for the first time in the paper called “Dynamic Routing Between Capsules”. In this paper, they showed that capsule network performance for MNIST dataset on highly overlapping digits could work better than CNNs. In computer vision, a capsule network is a neural network that tries to work inverse graphics. In a sense, the approach tries to reverse-engineer the physical process that produces an image of the world BIBREF23.
The capsule network is composed of many capsules that act like a function, and try to predict the instantiation parameters and presence of a particular object at a given location.
One key feature of capsule networks is equivariance, which aims at keeping detailed information about the location of the object and its pose throughout the network. For example, if someone rotates the image slightly, the activation vectors also change slightly BIBREF24. One of the limitations of a regular CNN is losing the precise location and pose of the objects in an image. Although this is not a challenging issue when classifying the whole image, it can be a bottleneck for image segmentation or object detection that needs precise location and pose. A capsule, however, can overcome this shortcoming in such applications BIBREF24.
Capsule networks have recently received significant attention. This model aims at improving CNNs and RNNs by adding the following capabilities to each source, and target node: (1) the source node has the capability of deciding about the number of messages to transfer to target nodes, and (2) the target node has the capability of deciding about the number of messages that may be received from different source nodes BIBREF25.
After the success of capsule networks in computer vision tasks BIBREF26, BIBREF27, BIBREF28, capsule networks have been used in different NLP tasks, including text classification BIBREF29, BIBREF30, multi-label text classification BIBREF31, sentiment analysis BIBREF18, BIBREF32, identifying aggression and toxicity in comments BIBREF33, and zero-shot user intent detection BIBREF34.
In capsule networks, the features that are extracted from the text are encapsulated into capsules (groups of neurons). The first work that applied capsule networks for text classification was done by BIBREF35. In their research, the performance of the capsule network as a text classification network was evaluated for the first time. Their capsule network architecture includes a standard convolutional layer called n-gram convolutional layer that works as a feature extractor. The second layer is a layer that maps scalar-valued features into a capsule representation and is called the primary capsule layer. The outputs of these capsules are fed to a convolutional capsule layer. In this layer, each capsule is only connected to a local region in the layer below. In the last step, the output of the previous layer is flattened and fed through a feed-forward capsule layer. For this layer, every capsule of the output is considered as a particular class. In this architecture, a max-margin loss is used for training the model. Figure FIGREF6 shows the architecture proposed by BIBREF35.
Some characteristics of capsules make them suitable for presenting a sentence or document as a vector for text classification. These characteristics include representing attributes of partial entities and expressing semantic meaning in a wide space BIBREF29.
For fake news identification with different length of statements, our model benefits from several parallel capsule networks and uses average pooling in the last stage. With this architecture, the models can learn more meaningful and extensive text representations on different n-gram levels according to the length of texts.
Depending on the length of the news statements, we use two different architectures. Figure FIGREF7 depicts the structure of the proposed model for medium or long news statements. In the model, a non-static word embedding is used as an embedding layer. In this layer, we use 'glove.6B.300d' as a pre-trained word embedding, and use four parallel networks by considering four different filter sizes 2,3,4,5 as n-gram convolutional layers for feature extraction. In the next layers, for each parallel network, there is a primary capsule layer and a convolutional capsule layer, respectively, as presented in Figure FIGREF6. A fully connected capsule layer is used in the last layer for each parallel network. At the end, the average polling is added for producing the final result.
For short news statements, due to the limitation of word sequences, a different structure has been proposed. The layers are like the first model, but only two parallel networks are considered with 3 and 5 filter sizes. In this model, a static word embedding is used. Figure FIGREF8 shows the structure of the proposed model for short news statements.
Several datasets have been introduced for fake news detection. One of the main requirements for using neural architectures is having a large dataset to train the model. In this paper, we use two datasets, namely ISOT fake news BIBREF17 and LIAR BIBREF15, which have a large number of documents for training deep models. The length of news statements for ISOT is medium or long, and LIAR is short.
In 2017, BIBREF17 introduced a new dataset that was collected from real-world sources. This dataset consists of news articles from Reuters.com and Kaggle.com for real news and fake news, respectively. Every instance in the dataset is longer than 200 characters. For each article, the following metadata is available: article type, article text, article title, article date, and article label (fake or real). Table TABREF12 shows the type and size of the articles for the real and fake categories.
As mentioned in Section SECREF2, one of the recent well-known datasets, is provided by BIBREF15. BIBREF15 introduced a new large dataset called LIAR, which includes 12.8K human-labeled short statements from POLITIFACT.COM API. Each statement is evaluated by POLITIFACT.COM editor for its validity. Six fine-grained labels are considered for the degree of truthfulness, including pants-fire, false, barely-true, half-true, mostly-true, and true. The distribution of labels in this dataset are as follows: 1,050 pants-fire labels and a range of 2,063 to 2,638 for other labels.
In addition to news statements, this dataset consists of several metadata as speaker profiles for each news item. These metadata include valuable information about the subject, speaker, job, state, party, and total credit history count of the speaker of the news. The total credit history count, including the barely-true counts, false counts, half-true counts, mostly-true counts, and pants-fire counts. The statistics of LIAR dataset are shown in Table TABREF14. Some excerpt samples from the LIAR dataset are presented in Table TABREF15.
The experiments of this paper were conducted on a PC with Intel Core i7 6700k, 3.40GHz CPU; 16GB RAM; Nvidia GeForce GTX 1080Ti GPU in a Linux workstation. For implementing the proposed model, the Keras library BIBREF36 was used, which is a high-level neural network API.
The evaluation metric in our experiments is the classification accuracy. Accuracy is the ratio of correct predictions to the total number of samples and is computed as:
Where TP is represents the number of True Positive results, FP represents the number of False Positive results, TN represents the number of True Negative results, and FN represents the number of False Negative results.
For evaluating the effectiveness of the proposed model, a series of experiments on two datasets were performed. These experiments are explained in this section and the results are compared to other baseline methods. We also discuss the results for every dataset separately.
As mentioned in Section SECREF4, BIBREF17 presented the ISOT dataset. According to the baseline paper, we consider 1000 articles for every set of real and fake articles, a total of 2000 articles for the test set, and the model is trained with the rest of the data.
First, the proposed model is evaluated with different word embeddings that described in Section SECREF1. Table TABREF20 shows the result of applying different word embeddings for the proposed model on ISOT, which consists of medium and long length news statements. The best result is achieved by applying the non-static embedding.
BIBREF17 evaluated different machine learning methods for fake news detection on the ISOT dataset, including the Support Vector Machine (SVM), the Linear Support Vector Machine (LSVM), the K-Nearest Neighbor (KNN), the Decision Tree (DT), the Stochastic Gradient Descent (SGD), and the Logistic regression (LR) methods.
Table TABREF21 shows the performance of non-static capsule network for fake news detection in comparison to other methods. The accuracy of our model is 7.8% higher than the best result achieved by LSVM.
The proposed model can predict true labels with high accuracy reaching in a very small number of wrong predictions. Table TABREF23 shows the titles of two wrongly predicted samples for detecting fake news. To have an analysis on our results, we investigate the effects of sample words that are represented in training statements that tagged as real and fake separately.
For this work, all of the words and their frequencies are extracted from the two wrong samples and both real and fake labels of the training data. Table TABREF24 shows the information of this data. Then for every wrongly predicted sample, stop-words are omitted, and words with a frequency of more than two are listed. After that, all of these words and their frequency in real and fake training datasets are extracted. In this part, the frequencies of these words are normalized. Table TABREF25 and Table TABREF28 show the normalized frequencies of words for each sample respectably. In these tables, for ease of comparison, the normalized frequencies of real and fake labels of training data and the normalized frequency for each word in every wrong sample are multiplied by 10.
The label of Sample 1 is predicted as fake, but it is real. In Table TABREF25, six most frequent words of Sample 1 are listed, the word "tax" is presented 2 times more than each of the other words in Sample 1, and this word in the training data with real labels is obviously more frequent. In addition to this word, for other words like "state", the same observation exists.
The text of Sample 2 is predicted as real news, but it is fake. Table TABREF28 lists six frequent words of Sample 2. The two most frequent words of this text are "trump" and "sanders". These words are more frequent in training data with fake labels than the training data with real labels. "All" and "even" are two other frequent words, We use "even" to refer to something surprising, unexpected, unusual or extreme and "all" means every one, the complete number or amount or the whole. therefore, a text that includes these words has more potential to classify as a fake news. These experiments show the strong effect of the sample words frequency on the prediction of the labels.
As mentioned in Section SECREF13, the LIAR dataset is a multi-label dataset with short news statements. In comparison to the ISOT dataset, the classification task for this dataset is more challenging. We evaluate the proposed model while using different metadata, which is considered as speaker profiles. Table TABREF30 shows the performance of the capsule network for fake news detection by adding every metadata. The best result of the model is achieved by using history as metadata. The results show that this model can perform better than state-of-the-art baselines including hybrid CNN BIBREF15 and LSTM with attention BIBREF16 by 3.1% on the validation set and 1% on the test set.
Figure FIGREF32 shows the confusion matrix of the best classification using the proposed model for the test set. The model classifies false, half-true, and mostly-true news with more accuracy. Nevertheless, it is difficult to distinguish between true and mostly-true and also between barely-true and false. The worst accuracy is for classifying pants-fire. For these labels, detecting the correct label is more challenging, and many pants-fire texts are predicted as false.
In this paper, we apply capsule networks for fake news detection. We propose two architectures for different lengths of news statements. We apply two strategies to improve the performance of the capsule networks for the task. First, for detecting the medium or long length of news text, we use four parallel capsule networks that each one extracts different n-gram features (2,3,4,5) from the input texts. Second, we use non-static embedding such that the word embedding model is incrementally up-trained and updated in the training phase.
Moreover, as a fake news detector for short news statements, we use only two parallel networks with 3 and 5 filter sizes as a feature extractor and static model for word embedding. For evaluation, two datasets are used. The ISOT dataset as a medium length or long news text and LIAR as a short statement text. The experimental results on these two well-known datasets showed improvement in terms of accuracy by 7.8% on the ISOT dataset and 3.1% on the validation set and 1% on the test set of the LIAR dataset.

Figure and table captions:
Figure 1: The architecture of capsule network proposed by Yang et al. [35] for text classification
Figure 2: The architecture of the proposed non-static capsule network for detecting fake news in medium or long news statements.
Figure 3: The architecture of the proposed static capsule network for detecting fake news in short news statements.
Table 1: Type and size of every articles per category for ISOT Dataset provided by Ahmed et al. [2]
Table 2: The LIAR dataset statistics provided by Wang [32]
Table 3: Three random excerpts from the LIAR dataset.
Table 4: Result of proposed model with different word embedding models
Table 5: Comparison of non-static capsule network result with Result of Ahmed et al. [2]
Table 6: Two samples with wrong prediction
Table 7: The number of word tokens and word types of training data and samples
Table 8: Normalized frequency for words in sample 1 and training data with fake and real label
Table 9: Normalized frequency for words in sample 2 and training data with fake and real label
Table 10: Comparison of capsule network result with other baseline
Figure 4: Confusion matrix of classification using proposed model for LIAR dataset

Questions about this paper:
What are state of the art methods authors compare their work with? 

Answers:
ISOT dataset: LLVM
Liar dataset: Hybrid CNN and LSTM with attention