Machine Learning detects Fake News. | Towards Data Science

IntroductionExploratory Data AnalysisFeature EngineeringModelAccuracyFeature ImportanceSummaryReferences
Introduction
Exploratory Data Analysis
Feature Engineering
Model
Accuracy
Feature Importance
Summary
References
Deciphering if a news article is valid or not can be difficult. As a new author on Towards Data Science, Analytics Vidhya, and DataSeries, I have gained some intuition into developing an eye for what is fake or real. Instead of purely relying on a hunch, I wanted to test my theory using data science and machine learning. I have compiled Python code that constructs a Random Forest model to predict whether or not an article is fake. As a Data Scientist, I have always been curious about creativity and automation — I believe my article and work on this model combines both facets of science and art to predict news’ validity. Below, I outline the code that I used for this model, as well as commentary on further work, and a summary for non-technical users who are interested in the characteristics of a true article.
The dataset used in this model was created from my dummy data. I am using several features or characteristics about common articles and trying to predict if they are fake or not. In my case, false is determined by a predetermined label, as well as true. For the sake of this tutorial, this data is randomly generated from my own text. The great thing about this data and model is that it can be scaled and redefined, say you compile several articles from more sources, your labels can be real-world articles from Facebook, Twitter, or even the New York Times. Now that we know about my target variable, fake (or true), let us look into which features I used to build the model:
title — text/words used in the article title ( Main 20 Reasons to..., etc.)
subtitle — text/words used in the article subtitle (A construction zone..., etc.)
title_characters — ( “:”, “-”, special characters to break up the text)
image_count — number of images used
Below are the first five rows of the dataframe:
Because both text and numeric features are used in this model, natural language processing (NLP) is beneficial for prediction. I used the common term-frequency inverse document frequency vectorizer to transform the count of text into a weighed matrix of term importance. I then compiled both the TextTransformer and TfidfVectorizer for the text features like title and subtitle. Next, I performed a similar method, but for numeric features using the NumberTransformerr class. Lastly, I combined all of the features, text and numeric together. The code [3] used below is for feature engineering, NLP, and ultimately to prepare/preprocess the data so that it can be inputted into the model correctly.
# use the term-frequency inverse document frequency vectorizer to transform count of text# into a weighed matrix of term importancevec_tdidf = TfidfVectorizer(ngram_range=(1,2), analyzer='word', norm='l2')# compile both the TextTransformer and TfidfVectorizer # to the text featurestitle = Pipeline([                ('transformer', TextTransformer(key='title')),                ('vectorizer', vec_tdidf)                ])subtitle = Pipeline([                ('transformer', TextTransformer(key='subtitle')),                ('vectorizer', vec_tdidf)                ])# compile the NumberTransformer# to the numeric featurestitle_character = Pipeline([                ('transformer', NumberTransformer(key='title_character')),                ])image_count = Pipeline([                ('transformer', NumberTransformer(key='image_count')),                ])# combine all of the features, text and numeric togetherfeatures = FeatureUnion([('title', title),                      ('subtitle', subtitle),                      ('title_character', title_character),                      ('image_count', image_count)                      ])
The next part of the data science process, and perhaps the most fun, is to create a model that predicts your target variable. In this case, the model is a Random Forest classifier. I created a general classifier, then united all the features and classifier together in a pipeline. I isolated the features from the variable, y, and stored those accordingly for later reference. The next step was performing a train-test split on the data, while establishing a train and test size 70:30, and then incorporating a random state for reproducibility. To see how the model performs on new data, develop this split, and ultimately test on data that is completely unseen (which would be the real-world application of a model).
The next step was to create a parameter grid for GridSearchCV to search through. This grid is used to test several listed parameters and to see which is the best one. For my model, I looked at the n_estimators. I created an object that stores the estimator which is the pipeline already created, with the param_grid, a cv (cross-validation) fold of 2, n_jobs of 1 (for faster computing), verbose of 1 (to see textual output), and return_train_score for model diagnostics. Not only did I look at the mean_train_score, but I also wanted to see how useful the model was at looking at new data with the mean_test_score as well. I found out which parameters were best as well with grid_search.best_params that helped to distinguish that 30 number of estimators were best.
Below is the code used for the modeling part of this process:
# create the classfier from RFclf = RandomForestClassifier()# unite the features and classfier togetherpipe = Pipeline([('features', features),                 ('clf', clf)                 ])# split the data into train and test# isolate the features from the target fieldtext_numeric_features = ['title', 'subtitle', 'title_character', 'image_count']y = 'fake_true'X_train, X_test, y_train, y_test = train_test_split(df[text_numeric_features], df[y],                                                     test_size=0.30,                                                    random_state=42                                                   )# parameter grid to scan throughparam_grid = {     'clf__n_estimators': [10, 20, 30, 40, 50, 100]}grid_search = GridSearchCV(estimator = pipe, param_grid = param_grid,                           cv = 2, n_jobs = 1, verbose = 1, return_train_score=True)grid_search.fit(X_train, y_train)grid_search.cv_results_['mean_train_score']grid_search.cv_results_['mean_test_score']grid_search.best_params_clf_test = grid_search.best_estimator_
Perhaps the most gratifying part of the data science model process is seeing your results. I predicted on my X_test and assigned that to preds. I used metrics.accuracy_score for determining my accuracy.
My model accuracy was 91%
To use the code referenced, see below:
# test statspreds = clf_test.predict(X_test)# print out accuracyprint("Accuracy:", metrics.accuracy_score(y_test, preds))
In non-data science vocabulary, feature importance refers to which parts of a dataset are great for predicting the response variable, or in this case, an article’s features are important and with how much weight are they important. Because this model is new and the data is small, I looked at plotting the highest features (most were text), since the features used are all important.
I called grid_search.best_estimator_ to return the feature importances of a fake article. In the dataframe, there are several words that do not lead to a fake article. However, there are plenty that are helpful in determining a fake article. Important words include:
Article Text — 1000,000 (misformattted numbers)
Image Count — 20 image for one article (too high)
The code below is used for the feature importance analysis:
pd.DataFrame(grid_search.best_estimator_.named_steps["clf"].feature_importances_)importances = list(zip(grid_search.best_estimator_.named_steps["clf"].feature_importances_, df.columns))importances.sort(reverse=True)print(grid_search.best_params_)print(grid_search.best_score_)feature_importances = grid_search.best_estimator_.named_steps["clf"].feature_importances_feat_importances = pd.Series(grid_search.best_estimator_.named_steps["clf"].feature_importances_,                              #index=df.columns                            )feat_importances.nlargest(10).plot(kind='barh')plt.title("Top 10 Important Features")plt.show()
With any data science model and dataset, it is important to note that not one feature is the main reason for a prediction, or in this case, if an article is fake. It is the combination of them that works to create an article that is false or true. While my analysis and model are great at predicting, it is mostly great at classifying on my own data. To change this model, not much work would be needed, other than collecting a variety of fake and true articles to train. The key attributes of an article to look at are:
article title, article subtitle, article special characters, and image count
Other characters to test in the future include:
body text, image quality, image classification (what is the photo?), amount of readers, day of the week, time of day, placement of photo or code, capitalization of letters, and several more
I hope that this article was both interesting and useful, and gave you some ideas on how to classify fake news, as well as ideas and code for building your own data science model.
For all the code used in my model, see the gist [4] below:
[1] freestocks, Unsplash, (2016)
[2] Markus Winkler on Unsplash, (2020)
[3] M.Przybyla, screenshots (2020)
[4] M.Przybyla, gist (2020)
sklearn, sklearn (2020)
pandas, pandas (2020)