Fake Statements Identification
By: Anqi Lou
Tags: Machine Learning, Naturallanguageprocessing, Fake News, Lime, Deep Learning

An Approach to Fake News Classification Using Natural Language Processing

By: Matthew Leong, Anqi Lou, Chirag Ramesh, Yue Tian, Qinwen Zhou

Abstract

We used Natural Language processing to build a variety of predictive machine learning and deep learning models to predict whether or not a statement is true or false. We collected over 17,000 real or fake labeled statements from IEEEDataPort who sourced them from the PolitiFact API. Then we preprocessed the data by removing punctuation, removing stopwords, and using lemmatization. We used scikit’s train-test split package to split the data into training and testing sets. In our preliminary exploratory analysis, we found a 50% baseline accuracy; identified speakers with the most real and fake statements; and created a word cloud showcasing the differences between the real and fake statements. We also visualized our 2 dimensional projections for bag-of-words, tf-idf, and Word2Vec. We found that the train and test statements were fairly similar.

For our traditional models, we built the following: TF-IDF Logistic regression; bag of words Naive Bayes; TF-IDF Random Forest; and bag of words Random Forest. For our deep learning models, we built both the LSTM and RNN with tokenization and padding. We found that our models performed very similarly but our overall best model was the bag of words Random Forest under the criteria of test accuracy of 63.8%. We additionally tried out more ensemble methods to improve our accuracy and found that the logistic regression meta learner ensemble achieved a test accuracy of 63.6%. We also tried to interpret our models using LIME and tried out TF-IDF Random Forest as a proof of concept. Lastly, we finished the project by identifying LIME issues, identifying the reasons for low accuracy, and thought of potential improvements to be made in the future.

Introduction and Background

Fake news has been an issue that relates to every aspect of people’s lives in modern years, especially for socio-political topics. Fake information arises from blog posts, social media posts, and even the news. These messages are published deliberately with the intent to distort the truth and mislead people. As a result, people’s confidence in the authenticity of online information has been undermined shown in a study conducted by Statista showcasing over a quarter of Americans stating that they rarely ever trusted the news they read online.

We wanted to combat this lack of trust by exploring machine learning and deep learning models based on natural language processing. We aimed to build models to help people identify trustworthy and true information. For this project, we focused on predicting the veracity of statements reasoning that a paragraph can contain a mixture of true and false information while a statement is typically entirely true or false.

Data Collection/Description

Data Source

We obtained our dataset from IEEEDataPort. It consists of:

15,212 training samples

1,058 validation samples

1,054 test samples

This dataset is crawled from a fact-checking website PolitiFact.com and contains statements from politicians and other sources. Each statement is analyzed by PolitiFact and given a Truth-O-Meter rating that reflects its relative accuracy. The original Truth-O-Meter has six ratings which have been preprocessed into binary class labels (real or fake) in our dataset. The dataset contains the following columns:

id : matches the id in the PolitiFact website API

: matches the id in the PolitiFact website API date : the time each article that analyzed the statements was published on the website

: the time each article that analyzed the statements was published on the website speaker : the person or organization who made the statement

: the person or organization who made the statement statement : a claim published in the media that has been investigated by the PolitiFact article

: a claim published in the media that has been investigated by the PolitiFact article sources : the sources used to analyzed the statement

: the sources used to analyzed the statement paragraph_based_content : the PolitiFact article stored as paragraphs in a list

: the PolitiFact article stored as paragraphs in a list fullText_based_content : full text of the PolitiFact article that analyzed the statements

: full text of the PolitiFact article that analyzed the statements label_fnn: the class label for each statement

In this project, we will focus on the statement as the model input and label column as the target variable.

Data Pre-Processing & Exploration

Data Pre-Processing

Before we trained the models, we preprocessed the data into a machine-readable format. We first concatenated the training, validation and test samples into one large dataframe and confirmed there are no null statements in the dataset.

1.Convert labels into dummy targets

As mentioned earlier, our class labels are made up of strings [“real”,”fake”]. We converted the labels into 1 and 0 with 1 indicating a real statement and vice versa.

2. Text Pre-processing

We went through several steps to clean the text:

Remove all punctuations, special characters and white spaces

Convert text into lowercase

Tokenization

Remove stopwords

Lemmatization with NLTK WordNet Lemmatizer

This guide serves as a great reference for text pre-processing.

3. Re-splitting Training and Test set

We re-splitted our data into an 80/20 split consisting of 13,859 training samples and 3,465 testing samples. This split resulted in a 50/50 balance of real and fake statements in both training and test sets.

Word Representation

We tried one-hot encoding, TF-IDF scores, Word2vec embeddings to convert the statement sentences into numerical values which can be used as inputs into our Machine Learning models.

One-hot encoding (aka Bag of Words)

We have overall 14,420 unique words in the statement column of the training set. With the bag of words model, each word in the vocabulary is associated with an index and each statement is represented as a vector of length 14,420. The number of times each word appears in the given statement will be stored in the corresponding index.

The statements mostly have a length of 15 words with the maximum length being 73 words. That means we will have sparse vectors containing a lot of zero scores. We further projected the vectors down to two dimensions using truncated SVD to see how much information the bag-of-words model is able to capture and if two classes can be separated.

TF-IDF

TF-IDF reflects the importance of a word to the document in our statement collections. The score increases as the word appears more often in the document and is offset by the number of documents that contain the word. Thus if a word appears in almost every document like stopwords, it will not obtain a high TF-IDF importance.

Word2vec Embeddings

Word embeddings are one of the most popular word representation methods as it is able to capture the semantic meanings of each word. Trained on a large corpus, word embeddings will know what words are more likely to occur together and which words are similar to each other. With word embeddings, we don’t need to worry that new words in the test set which haven’t appeared in the training set before will lead to the model classifying the statement inaccurately.

We used pre-trained word vectors trained on Google News dataset which contain 300-dimensional vectors for 3 million words and phrases. And then we represented each statement by a vector with averaged Word2vec scores for each word in the statement.

Reference code

For the three visualizations above, we found little to no separation in the class labels.

Exploratory Analysis

This graph is basically telling us that the label of the statement has nothing to do with its length, so we cannot use the length as an input to train the model.

These are graphs showing us the top 10 speakers with the most fake statements and top 10 speakers with real statements. We can see that Donald trump, Bloggers and Facebook posts have the most fake statements. And Obama, trump, and Hillary have a lot of real statements. Since trump is appearing in both the categories, we can see that a lot of the statements in our dataset are from him. However, since the statements that PolitiFact are trying to study are not randomly pulled from the the internet, the graph below might not be able to accurately reflect the nature of statements that a certain speaker makes.

We also created word cloud for real and fake statements seperately. However, just as the graph below shows, there is no clear differences in the word usages for the two classes of statements.

This is our unigrams, bigrams, and trigrams. Here we see that health and health care appear a lot, it makes sense cause its one of the most frequently talked about topics in politics.

Modeling

Logistic Regression

We started with the simplest model which can be easily interpreted. GridSearchCV was used to tune the regularization parameter.

The best performing logistic regression has a training accuracy of 69.7% and a testing accuracy of 63.2% with the TF-IDF representation.

Naive Bayes

Another simple model that we tried out is Multinomial naive bayes with bag-of-words representation. It takes way less time to train than Neural Networks and can sometimes perform better than Logistic Regression if the independence assumption holds.

The training accuracy is 78.7% and testing accuracy is 62.3%.

Random Forest

Among the various classical scikit classifications models we tried, we found that we obtained the best results with Random Forest, an ensemble learning method. an ensemble learning method. As a quick reminder, ensemble methods generally work better when every model is accurate but uncorrelated. As we have plenty of possible parameters and there are many possible ways to build a decision tree, we thought that this model would be able to get high accuracy scores from our data. We used the GridSearchCV package to facilitate the testing of optimal parameters.

The code above shows an example of how this was accomplished. We did not choose to limit the depth of the tree as we were desiring better accuracy scores at the cost of increasing model complexity. Under similar logic, we estimated that we would need a high number of trees due to the amount of features inherent in our text data. We found that 350 estimators were optimal under all of our possible inputs to the model and obtained the best test accuracy score of 63.8% as well as test AUROC score of 0.688 from the bag of words random forest model.

Recurrent Neural Network (RNN)

A major advantage of the RNN model is that it has a memory; it contains a temporal loop that allows the hidden layer to not only give outputs, but also feed itself as well, which makes it very suitable for analyzing speeches.

Therefore, we also tried to run an RNN model to classify the labels of the political statements. However, instead of trying to train the model with all the other inputs that we used on other models, i.e., TF-IDF, bag of words, etc., we trained the RNN model with the tokenized and padded data, which saved us a significant amount of running time. More precisely, we tokenized each statement to transform it into a sequence of vectors, and then we zero-padded all the statements in the end to make sure they were all of the same length.

In order to find the optimal parameters, we first tried to set up the model with some random numbers, and then we tweaked the parameters for the best-performing model repeating the process until the performance of the model did not significantly improve. During this repetition, we found evidence of overfitting as the training accuracy increased rapidly and the testing accuracy fell within a short number of epochs. Therefore, to deal with this problem, the number of epochs that we set when fitting the training data was limited to three, and we also added a dropout layer, which combats overfitting by probabilistically removing inputs to a layer. After multiple attempts, we found our best performing RNN model with the following parameters:

This model gave us a training accuracy of 79.66% and a test accuracy of 59.58%, which was actually not as good as what we had expected.

We were puzzled as to why the model had performed poorly and conjectured that our short statements may have lacked context and sequence information. Therefore, the advantage of RNN could not be effectively taken. Also, due time constraints, we believe that our parameters could still be further improved. It might be possible that with the right set of parameters, the RNN model could have the potential to outperform the other classical ones.

Long Short-term Memory (LSTM)

Since RNN is vulnerable to the vanishing gradient problem, we tried the LSTM network in hope of achieving a better test accuracy rate.. Vanishing gradient problem refers to the issue that the gradients of the error function decreases exponentially in the backpropagation process, which will make the network harder to be trained. LSTM overcomes this problem by remembering information for a prolonged period of time so that the memory becomes more stable. Similar to the RNN model, we trained LSTM with the simple tokenized and padded data.

The process of finding the optimal parameter was built upon the result of RNN. We took the best model that we found with RNN and applied the same set of parameters on LSTM. We then attempted to tweak the parameters to see whether the accuracy improved.

The classification accuracy was slightly better than that of the RNN model, just as what we had expected. We achieved a training accuracy of 75.05% and a test accuracy of 63.26%. However, for similar reasons as RNN, LSTM did not have a good enough performance as what we imagined either, and it failed to outperform the classical machine learning models.

Ensemble — Stacked Model

Our final attempt to improve the model performance was to use stacked models, an ensemble technique where the prediction results of several models were combined together to form a new set of input to train a meta-learner. By doing so, it was hoped that the meta-learner would be able to learn from the outputs of each sub-model and thus find a way to best combine the predictions to arrive at a more accurate classification.

To build the stacked model, we utilized the results of four models that we previously built and trained: Logistic Regression, Random Forest, Naïve Bayes, and LSTM. We did not use RNN because it was similar to LSTM in nature, but its test accuracy was lower than that of LSTM. The parameters for all the above models were also left the same as the optimal parameters that we had found for each of them. However, since we were unsure of the standard for choosing the meta-learner, we went through the process of building the stack model four times, each with a different model being the meta-learner and the rest three being the sub-models. In the end, it turned out the test accuracy was highest when we selected logistic regression as the meta learner.

However, the model overfitted the data evident by an almost 100% training accuracy. We thought this might be caused by the possibility that the mistakes that all the sub-models made were very random. So even though the meta-learner was able to learn from all the mistakes in the classification for the training data, when dealing with the test data, it would encounter new types of mistakes and it will not know how to handle that. In addition, in comparison to the individual models, this number did not improve by much. To find out the reason for this, we then constructed a correlation matrix of all the outputs of the four models that were involved in this process.

As illustrated by the above graph, all the outputs are very highly correlated, which means that when one model classifies a statement as fake, the other ones are very likely to do so as well. Thus, there is little we can learn by putting all the outputs together.

Results

Key Findings and Evaluation

Below is the table summarizing the performance of all the five models we tried.

Our best performing model is random forest under both the test accuracy and AUROC criteria. Therefore, we created a confusion matrix for the classification result of the random forest model to see whether there was anything we could learn from to further improve the model performance.

According to the confusion matrix, we had slightly more false positive cases than false negative ones. However, when we tried to extract some misclassified cases and dig further into those errors, we were not able to find any patterns that could be extracted and help the prediction. This, on the other hand, also testified our reasonings for why the stacked model suffered greatly from overfitting. All the errors were randomly made which made the ensemble technique lose its advantage.

Another interesting thing that we found was that even though we chose random forest as our best model, all the models we tried actually achieved roughly the same test accuracy, which was about 63%. Although this number was not very high, when we tried to manually label a sample of 150 statements ourselves, we only achieved an accuracy rate of 60%, and this is also taking into account that we had prior knowledge of the topics being classified. Therefore, judging from this perspective, we believed that our model achieved a very satisfying result.

However, we still looked for some reasons to potentially explain why it was hard for us to improve the accuracy rate, and we believe that the primary reason might be that all our statements are very short and lacking context to learn from. In addition, both the real and fake statements are very similar, further increasing the difficulty of classification. As shown in our exploratory analysis, we found that our visualizations showed no clear distinctions between the two classes. Therefore, given the high similarity in the wording and structure of the statements, we suspect that when the models were predicting the labels for the statements, the main thing that they looked at might just be the use of extreme words (most, exactly, number one, etc.), since there is a greater possibility that sentences containing those words are fake. If all the models were utilizing this information to make the judgement, then it is not hard to explain why they all arrived at a similar test accuracy in the end.

Interpreting our Models

Bringing things back to the business context of the problem, we wanted to predict real or fake statements made in politics. In that regard, getting the model is only half the battle. The other half is being able to convince people to trust the model and understand what it’s doing. While our first half left much to be desired, we also wanted to get started on the second half. The typical approach towards interpreting a random forest model is to get the feature importance and then plot it as seen below.

However, we considered this method insufficient for the purposes of our work. While feature importance reveals which words are important for the model, it does not do much else. Feature importance does not showcase how a word influences a model; for example, does percent predict fake or does it help predict real? Additionally, as politics have become more and more divisive, we needed a way to showcase the reasoning of why our model predicts certain outcomes in a certain way.

To that end, we looked into LIME which stands for local interpretable model-agnostic explanations. How these explanations work is that they treat the model they are interpreting as a black box and instead focus on the inputs and the outputs. From there, a sample or instance is given and can be approximated using a linear model. The linear model gives weights obtained from neighboring samples which can be interpreted to be the approximate probabilities of why our model predicts a certain class. For example in the statement, “A national organization says Georgia has one of America’s toughest ethics laws.” the following plot is obtained:

We can see that the statement is predicted to be fake and the particular reasoning as to why it’s predicted to be fake. Namely that law, america, organization, toughest, national give very high negative weights. We also see the potential reasoning as to why the model is classifying it as positive namely that the word Georgia gives it an approximate 7% positive weighting.

However, our professor pointed out to us that LIME had issues. We thus did some additional research on the method and found that LIME is not particularly robust meaning that explanations of similar examples might be completely different. Additionally, the current implementation suffers from low explanation fidelity meaning that it does not perform well when train and test data differ significantly. In our case, our train and test statements are very similar due to how politifact tends to pick sensationalized statements to prove or disprove. As a result, we didn’t actually see that issue come up. We found that while LIME was looking promising for us at first, we believe that in practice, it may actually undermine the confidence in our model as it gives contradictory explanations.

Conclusion

For this project, we tried to identify the veracity of the statements with six models: Logistic Regression; Naive Bayes; Random Forest; RNN; LSTM; and stacked model. We found that Random Forest was the best performing one with both the highest test accuracy and highest AUROC. However, all the models that we built failed to achieve a very high accuracy score, which we believe might be caused by the fact that the statements we were trying to classify were all very short, making it hard to extract useful information. Therefore, there is still much room for us to improve the classification accuracy, and we will talk about the potential approaches in the section below.

Future Improvement

One way to improve our models is to include more meta-data, such as the platform on which the statements are posted and whether it is second-hand information or not. Also, trying more advanced classification techniques like BERT, might be helpful as well. If possible, more training data can be sourced from platforms other than Politifact such as news article titles, etc. Since Politifact has a specific process of choosing the statement to fact-check, there could be the issue of dataset bias.

Data issues aside, we’ve also learned that just because a model interpretation package seems promising, doesn’t mean it will work in practice. Thus, in the future to improve this project we aim to comprehensively look into the advantages and disadvantages of any future model interpretability tool. In particular, we’re looking at SHAP values and the interpretML package as potential ways to help interpret our model.

If we are able to achieve a higher accuracy rate and good interpretability in the future, a potential business use of our model is to partner with social media platforms to be used as a filtering tool to decrease the possibility of a fake statement being posted and spread online.