Building a Fake News Classification app using Keras & Streamlit
By: Anderson Ho Yong Kian
Tags: Naturallanguageprocessing, Web App Development, Deep Learning, Machine Learning, Fake News

Building a Fake News Classification app using Keras & Streamlit

Introduction

The objective of this article is to outline end-to-end steps on building and training a Long Short Term Memory (LSTM) model to classify fake and true news using a Keras framework and deploying this model using Streamlit. The dataset that will be used will be Kaggle’s Fake News dataset from the InClass Prediction Competition. All notebooks and scripts used can be found on my GitHub repo. This article will illustrate 5 steps: Exploratory data analysis (EDA), Cleaning and pre-processing the dataset, Creating the classifiers pipelines and training & evaluating the classifiers, creating the LSTM model and training & evaluating the LSTM and creating the Streamlit code for model deployment.

Step 1: Exploratory Data Analysis (EDA)

Just like any data science or machine learning project, EDA serves as an essential foundation to not only begin creating the necessary models for classification but to also get a better understanding of the data we are working with.

Import necessary modules

The necessary modules can be found and installed using pip in the requirements.yml file in the GitHub repo.

import pandas as pd

import numpy as np

import seaborn as sns

import matplotlib.pyplot as plt from sklearn.feature_extraction.text import CountVectorizer

from sklearn import feature_extraction, linear_model, model_selection, preprocessing

from sklearn.metrics import accuracy_score

from sklearn.model_selection import train_test_split

from sklearn.feature_extraction.text import TfidfTransformer

from sklearn.pipeline import Pipeline

from sklearn.svm import LinearSVC

from sklearn.linear_model import LogisticRegression

from sklearn.naive_bayes import GaussianNB

from sklearn.naive_bayes import BernoulliNB

from sklearn.naive_bayes import MultinomialNB

from sklearn.metrics import plot_confusion_matrix from sklearn.ensemble import GradientBoostingClassifier

from sklearn.tree import DecisionTreeClassifier

from sklearn.neighbors import KNeighborsClassifier

from sklearn.linear_model import SGDClassifier

from sklearn.ensemble import RandomForestClassifier

from sklearn.preprocessing import LabelEncoder

from sklearn.gaussian_process import GaussianProcessClassifier

from sklearn.metrics import accuracy_score, confusion_matrix,classification_report,roc_curve,auc

from sklearn.utils import shuffle import nltk

import nltk as nlp

import string

import re

import pickle

from textblob import TextBlob from nltk.tokenize import word_tokenize

import re

import string

from nltk.corpus import stopwords

from nltk.stem.wordnet import WordNetLemmatizer from tqdm import tqdm, tqdm_notebook

tqdm.pandas(tqdm_notebook) MAX_NB_WORDS = 100000 # max no. of words for tokenizer

MAX_SEQUENCE_LENGTH = 200 # max length of each entry (sentence), including padding

VALIDATION_SPLIT = 0.2 # data for validation (not used in training)

EMBEDDING_DIM = 100

GLOVE_DIR = "glove/glove.6B."+str(EMBEDDING_DIM)+"d.txt"

Preparing the dataset

The dataset from Kaggle is provided in 2 CSV files which are already classified between true and fake news. In order to make data cleaning, pre-processing, EDA and model-building operations easier moving forward, it would be prudent to combine the 2 CSV files into one Pandas dataframe where functions can be implemented on the whole dataset.

true = pd.read_csv(r”data/True.csv”)

fake = pd.read_csv(r”data/Fake.csv”) true[“target”] = 0

fake[“target”] = 1 df = pd.concat([true,fake]) df = shuffle(df)

df.head()

Cleaning the data

The news body text has to be cleaned for EDA and pre-processing. This will be done using the basic_text_cleaning() function below.

wordnet = WordNetLemmatizer()

regex = re.compile('[%s]' % re.escape(string.punctuation)) def basic_text_cleaning(line_from_column):

tokenized_doc = word_tokenize(line_from_column) new_review = []

for token in tokenized_doc:

new_token = regex.sub(u’’, token)

if not new_token == u’’:

new_review.append(new_token) new_term_vector = []

for word in new_review:

if not word in stopwords.words(‘english’):

new_term_vector.append(word) final_doc = []

for word in new_term_vector:

final_doc.append(wordnet.lemmatize(word)) return ‘ ‘.join(final_doc)

The basic_text_cleaning() function which has 4 features. First, it tokenises the news text, second, it removes punctuations using regex, third, it removes stopwords and finally lemmatises the text. This function can be applied to the dataset’s “text” column to clean the text using .progress_map() function and the progress can be monitored using tqdm. As the cleaning process took an hour, it would be prudent to save the cleaned dataframe so that it can be used later without repeating the process again.

tqdm.pandas(tqdm_notebook)

df[“clean_text”] = df[“text”].progress_map(basic_text_cleaning)

df["clean_title"] = df["title"].progress_map(basic_text_cleaning)

df.to_csv("models/clean_news.csv")

Feature Engineering

As text data is hard to visualise in itself, we should create additional features which we can use to generate more insights about the text data. For example, Textblob can be used to generate sentiment values on the news text to ascertain if the news has a positive or negative sentiment. The word and character count for both the titles and news body text may also be useful later on in the EDA.

df[‘polarity’] = df[‘clean_text’].progress_map(lambda text: TextBlob(str(text)).sentiment.polarity)

df[‘text_len’] = df[‘clean_text’].astype(str).progress_map(len)

df[‘text_word_count’] = df[‘clean_text’].progress_map(lambda x: len(str(x).split()))

df[‘title_len’] = df[‘clean_title’].astype(str).progress_map(len)

df[‘title_word_count’] = df[‘clean_title’].progress_map(lambda x: len(str(x).split()))

EDA

For this EDA, it would be more useful to compare the descriptive statistics between news that are labelled as fake and true. We will use certain functions for this EDA. We would be using ECDF (empirical cumulative distribution function) in addition to exploring the distribution of the features created in the feature engineering above to reduce the effect of bining bias. We can also compare the top 10 most frequent words from both the news text and titles.

def ecdf(data):

length= len(data)

x = np.sort(data)

y = np.arange(1, length+1)/length



return x,y def generate_hist_ECDF(data,x_label,title):

x_1, y_1 = ecdf(data=data[df[“target”] == 1])

x_0, y_0 = ecdf(data=data[df[“target”] == 0])



fig, (ax1, ax2) = plt.subplots(1,2,figsize=(15,5))

fig.suptitle(“ECDF and histogram plots for col — {} comparision between true and false”.format(title))



_ = ax1.plot(x_1, y_1, marker=”.”, linestyle=”none”)

_ = ax1.plot(x_0, y_0, marker=”.”, linestyle=”none”)

ax1.set(xlabel=x_label, ylabel=”CDF”)



_ = ax2.hist(data[df[“target”] == 1],normed=True,bins=50)

_ = ax2.hist(data[df[“target”] == 0],normed=True,bins=50)

ax2.set(xlabel=x_label, ylabel=”Probability”)



plt.legend()

plt.show()



def get_top_n_words(corpus, n=None):



corpus_fake = corpus[df[“target”] == 1].astype(str)

corpus_true = corpus[df[“target”] == 0].astype(str)



vec = CountVectorizer(stop_words = ‘english’).fit(corpus_fake)

bag_of_words = vec.transform(corpus_fake)

sum_words = bag_of_words.sum(axis=0)

words_freq_fake = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]

words_freq_fake =sorted(words_freq_fake, key = lambda x: x[1], reverse=True)



vec = CountVectorizer(stop_words = ‘english’).fit(corpus_true)

bag_of_words = vec.transform(corpus_true)

sum_words = bag_of_words.sum(axis=0)

words_freq_true = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]

words_freq_true =sorted(words_freq_true, key = lambda x: x[1], reverse=True)



df_fake = pd.DataFrame(words_freq_fake[:n],columns = [“text”, “count”])

df_true = pd.DataFrame(words_freq_true[:n],columns = [“text”, “count”])



fig, (ax1, ax2) = plt.subplots(1,2,figsize=(20,8))

ax1.bar(df_fake[“text”], df_fake[“count”])

ax1.set_xticklabels(df_fake[“text”])

ax1.set(xlabel=”top 10 most frequent terms for fake news”, ylabel=”count”)

ax2.bar(df_true[“text”], df_true[“count”])

ax2.set_xticklabels(df_true[“text”])

ax2.set(xlabel=”top 10 most frequent terms for true news”, ylabel=”count”)

plt.suptitle(“Comparision between the top 10 most frequent terms (fake/true)”)



plt.show()



def get_top_n_bigram(corpus, n=None):



corpus_fake = corpus[df[“target”] == 1].astype(str)

corpus_true = corpus[df[“target”] == 0].astype(str)



vec = CountVectorizer(ngram_range=(2, 2)).fit(corpus_fake)

bag_of_words = vec.transform(corpus_fake)

sum_words = bag_of_words.sum(axis=0)

words_freq_fake = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]

words_freq_fake =sorted(words_freq_fake, key = lambda x: x[1], reverse=True)



vec = CountVectorizer(ngram_range=(2, 2)).fit(corpus_true)

bag_of_words = vec.transform(corpus_true)

sum_words = bag_of_words.sum(axis=0)

words_freq_true = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]

words_freq_true =sorted(words_freq_true, key = lambda x: x[1], reverse=True)



df_fake = pd.DataFrame(words_freq_fake[:n],columns = [“text”, “count”])

df_true = pd.DataFrame(words_freq_true[:n],columns = [“text”, “count”])



fig, (ax1, ax2) = plt.subplots(1,2,figsize=(20,8))

ax1.bar(df_fake[“text”], df_fake[“count”])

ax1.set_xticklabels(df_fake[“text”])

ax1.set(xlabel=”top 10 most frequent terms for fake news”, ylabel=”count”)

ax2.bar(df_true[“text”], df_true[“count”])

ax2.set_xticklabels(df_true[“text”])

ax2.set(xlabel=”top 10 most frequent terms for true news”, ylabel=”count”)

plt.suptitle(“Comparision between the top 10 most frequent terms (fake/true)”)



plt.show()

Comparing descriptive statistics of created features between true & fake news

f, axes = plt.subplots(1, 5, figsize=(24,8)) sns.boxplot(y=”polarity”, x= “target”, data=df, orient=’v’ , ax=axes[0], showfliers = False)

axes[0].set_title(“Polarity”, fontsize=17)

sns.boxplot(y=”text_len”, x= “target”, data=df, orient=’v’ , ax=axes[1], showfliers = False)

axes[1].set_title(“Length of News characters”, fontsize=17)

sns.boxplot(y=”text_word_count”, x= “target”, data=df, orient=’v’ , ax=axes[2], showfliers = False)

axes[2].set_title(“Length of News words”, fontsize=17)

sns.boxplot(y=”title_len”, x= “target”, data=df, orient=’v’ , ax=axes[3], showfliers = False)

axes[3].set_title(“Length of News title characters”, fontsize=17)

sns.boxplot(y=”title_word_count”, x= “target”, data=df, orient=’v’ , ax=axes[4], showfliers = False)

axes[4].set_title(“Length of News title words”, fontsize=17) plt.show()

Exploring differences in the length of news text between true & fake news

generate_hist_ECDF(data=df[“text_len”],x_label = “text_len”, title=”text_len”)

Exploring differences in the length of news titles between true & fake news

generate_hist_ECDF(data=df[“title_len”],x_label = “title_len”, title=”title_len”)

Exploring differences in the sentiment polarity of news text between true & fake news

generate_hist_ECDF(data=df[“polarity”],x_label = “polarity”, title=”polarity”)

Based on the respective boxplots, ECDF and histogram plots, there is not much difference in the polarity and news text length between true and fake news except for title length. Based on the ECDF and histogram plot for title length, news that is labelled as fake generally tend to be shorter compared to their true counterparts.

Comparison of the top 10 most frequent words in news text between true & fake news

get_top_n_words(corpus=df[“clean_text”], n=10)

Comparison of the top 10 most frequent words in news titles between true & fake news

get_top_n_words(corpus=df[“clean_title”], n=10)

Based on the comparison between the top 10 frequent words in titles and news text, we can infer that both fake and true news is dominated by news relating to politics and more specifically, the subject being heavily related to American politics is shared between true and fake news. This may be an issue when creating the models later on as the models may not be able to generalise well during training.

Step 2: Creating the training & validation data

The news text and target labels have to be split into training and validation datasets and preprocessed before training the LSTM model. This will be done in 4 stages, passing the dataset through 4 different functions for preprocessing and segmentation: get_dummies(), prep_features(), prep_tokenizer() and prepare_training_test_data(). The functions are as stated below.

def get_dummies(df, col_target):

df_y = pd.get_dummies(df[col_target])

df_new = df.join(df_y)

df_new = df_new.drop(col_target,axis=1)

return df_new def prep_features(df, labels, text):

y = df[labels].values

comments_train = df[text]

comments_train = list(comments_train)



return comments_train, y def prep_tokenizer(texts,MAX_NB_WORDS):

tokenizer = Tokenizer(num_words=MAX_NB_WORDS)

tokenizer.fit_on_texts(texts)



print(“Tokeniser created — Saving Tokeniser”)



with open(‘tokenizer.pickle’, ‘wb’) as handle:

pickle.dump(tokenizer, handle, protocol=pickle.HIGHEST_PROTOCOL)



print(“Tokeniser saved”)



def prepare_training_test_data(texts,tokenizer,y):

sequences = tokenizer.texts_to_sequences(texts)

word_index = tokenizer.word_index

print(‘Vocabulary size:’, len(word_index))



print(“Padding sequences”)



data = pad_sequences(sequences, padding = ‘post’, maxlen = MAX_SEQUENCE_LENGTH)

print(‘Shape of data tensor:’, data.shape)



indices = np.arange(data.shape[0])

np.random.shuffle(indices)

data = data[indices]

labels = y[indices]



num_validation_samples = int(VALIDATION_SPLIT*data.shape[0])

x_train = data[: -num_validation_samples]

y_train = labels[: -num_validation_samples]

x_val = data[-num_validation_samples: ]

y_val = labels[-num_validation_samples: ]



print(‘Number of entries in each category:’)

print(‘training: ‘, y_train.sum(axis=0))

print(‘validation: ‘, y_val.sum(axis=0))



print(‘Tokenized sentences:

’, data[0])

print(‘One hot label:

’, labels[0])



return x_train, y_train, x_val, y_val

One-hot encoding the target labels

First, the get_dummies() function will take in 2 arguments; our dataframe and the binarized Fake/True label column. The purpose is to one-hot encode the target values and replace them with the existing labels.

df_new = get_dummies(df=df, col_target=”target”)

df_new.head()

Generating the X and Y values needed for training

Second, the prep_features() function will take in 3 arguments; our dataframe, labels created from the get_dummies() function earlier and the news body text column. The purpose is to create our X and y values for training.

labels = [0,1]

x_train, y_train = prep_features(df=df_new, labels=[0,1], text=”clean_text”)

print(x_train[0])

print(y_train[0])

Creating the tokenizer object

Third, the prep_tokenizer() function will take in 2 arguments; the X data that was generated from the prep_features() function and the maximum number of words for the tokenizer. The purpose is to create the tokenizer by fitting the tokenizer on the text and finally saving the tokenizer as a pickle file.

prep_tokenizer(texts=x_train,MAX_NB_WORDS=MAX_NB_WORDS)

Segmenting the X & Y values into training & validation datasets

Finally, the prepare_training_test_data() function will take in 3 arguments; the X and y data generated using the prep_features() function and the tokenizer that was created using the prep_tokenizer() function. The purpose is to convert the X data into sequences using the tokenize.texts_to_sequences() method and pad the sequences using the pad_sequences method. Afterwhich, the X and y data is split into training and testing data and will return the pre-processed X_train, X_test, y_val, y_val values.

with open(‘tokenizer.pickle’, ‘rb’) as handle:

tokenizer = pickle.load(handle) x_train,y_train,x_test,y_test = prepare_training_test_data(texts=x_train,tokenizer=tokenizer,y=y_train)

Step 3: Creating, training and evaluating the LSTM model

Creating the embedding layer

For the LSTM model, we will be using GloVe (global vectors for word representation) to generate the embedding layer for the LSTM model. The script below will open the glove.6B.100d file, load the embedding matrix and create the embedding layer.

embeddings_index = {}

word_index = tokenizer.word_index

f = open(GLOVE_DIR, encoding=”utf8")

print(‘Loading GloVe from:’, GLOVE_DIR,’…’, end=’’)

for line in f:

values = line.split()

word = values[0]

embeddings_index[word] = np.asarray(values[1:], dtype=’float32')

f.close()

print(“Done.

Proceeding with Embedding Matrix…”, end=””) embedding_matrix = np.random.random((len(word_index) + 1, EMBEDDING_DIM))

for word, i in word_index.items():

embedding_vector = embeddings_index.get(word)

if embedding_vector is not None:

embedding_matrix[i] = embedding_vector

print(“ Completed!”) sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')

embedding_layer = Embedding(len(word_index) + 1,

EMBEDDING_DIM,

weights = [embedding_matrix],

input_length = MAX_SEQUENCE_LENGTH,

trainable=False,

name = 'embeddings')

embedded_sequences = embedding_layer(sequence_input)

Creating the LSTM model

Below shows the code for model creation and summary, for the activation function in the final dense layer as well as the loss function when compiling the model, the sigmoid and binary_crossentropy functions are selected as this is a binary classification problem.

x = LSTM(60, return_sequences=True,name=’lstm_layer’)(embedded_sequences)

x = GlobalMaxPool1D()(x)

x = Dropout(0.1)(x)

x = Dense(50, activation=”relu”)(x)

x = Dropout(0.1)(x)

preds = Dense(2, activation=”sigmoid”)(x) model = Model(sequence_input, preds)

model.compile(loss = 'binary_crossentropy',

optimizer='adam',

metrics = ['accuracy'])

model.summary()

LSTM model summary

Training the model

As this is a fairly large dataset, to speed up the training process, A GPU will be used for training. For training, we will set the epochs to 20 and batch size to 32 as the GPU has a limited VRAM of 3GB.

history = model.fit(x_train, y_train, epochs = 10, batch_size=32, validation_data=(x_test, y_test))

Evaluating the model

loss = history.history['loss']

val_loss = history.history['val_loss'] acc = history.history['acc']

val_acc = history.history['val_acc'] epochs = range(1, len(loss)+1) fig, (ax1,ax2) = plt.subplots(1,2,figsize=(15,5)) ax1.plot(epochs, loss, label='Training loss')

ax1.plot(epochs, val_loss, label='Validation loss')

ax1.set_title('Training and validation loss')

ax1.set_xlabel('Epochs')

ax1.set_ylabel('Loss')

ax1.legend() ax2.plot(epochs, acc, label='Training accuracy')

ax2.plot(epochs, val_acc, label='Validation accuracy')

ax2.set_title('Training and validation accuracy')

ax2.set_xlabel('Epochs')

ax2.set_ylabel('Accuracy')

ax2.legend() plt.show()

After training the model, it became clear that there is over-fitting as evident from the training and validation loss & accuracy plots where the training loss starts going below the validation loss and training accuracy starts going above validation accuracy at around 2–3 epochs respectively.

Step 4: Creating, training & evaluating 6 classifiers pipelines

For the app, apart from using the LSTM model, we would also use 6 traditional machine learning classifiers from the Scikit-learn libraries to classify between real and fake news; Logistic Regression Classifier, Bernoulli Naive Bayes Classifier, Multinomial Naive Bayes Classifier, Gradient Boosting Classifier, Decision Tree Classifier and Random Forest Classifier.

We will train and evaluate these 6 models using 3 functions; load_up_classifier(), confusion_matrixes() and generate_all_curves().

def generate_all_curves(model_list, probs_list):

fpr_list = []

tpr_list = []

threshold_list = []

roc_auc_list = []



for model,probs in zip(model_list,probs_list):

preds = probs[:,1]

fpr, tpr, threshold = roc_curve(y_test, preds)

roc_auc = auc(fpr, tpr)



fpr_list.append(fpr)

tpr_list.append(tpr)

threshold_list.append(threshold)

roc_auc_list.append(roc_auc)



fig, axs = plt.subplots(2, 3, figsize=(15,10)) #subplot_kw=dict(polar=True))

#for roc_values in zip(fpr_list,tpr_list,threshold_list,roc_auc_list):

axs[0, 0].plot(fpr_list[0], tpr_list[0], 'b', label = 'AUC = %0.2f' % roc_auc_list[0])

axs[0, 0].legend(loc = 'lower right')

axs[0, 0].plot([0, 1], [0, 1],'r--')

axs[0, 0].set_title('Logistic regression classifier')

axs[0, 0].set_ylabel('True Positive Rate')

axs[0, 0].set_xlabel('False Positive Rate')



axs[0, 1].plot(fpr_list[1], tpr_list[1], 'b', label = 'AUC = %0.2f' % roc_auc_list[1])

axs[0, 1].legend(loc = 'lower right')

axs[0, 1].plot([0, 1], [0, 1],'r--')

axs[0, 1].set_title('Multinomial Naive Bayes classifier')

axs[0, 1].set_ylabel('True Positive Rate')

axs[0, 1].set_xlabel('False Positive Rate')



axs[0, 2].plot(fpr_list[2], tpr_list[2], 'b', label = 'AUC = %0.2f' % roc_auc_list[2])

axs[0, 2].legend(loc = 'lower right')

axs[0, 2].plot([0, 1], [0, 1],'r--')

axs[0, 2].set_title('Bernoulli Naive Bayes classifier')

axs[0, 2].set_ylabel('True Positive Rate')

axs[0, 2].set_xlabel('False Positive Rate')



axs[1, 0].plot(fpr_list[3], tpr_list[3], 'b', label = 'AUC = %0.2f' % roc_auc_list[3])

axs[1, 0].legend(loc = 'lower right')

axs[1, 0].plot([0, 1], [0, 1],'r--')

axs[1, 0].set_title('Gradient Boosting classifier')

axs[1, 0].set_ylabel('True Positive Rate')

axs[1, 0].set_xlabel('False Positive Rate')



axs[1, 1].plot(fpr_list[4], tpr_list[4], 'b', label = 'AUC = %0.2f' % roc_auc_list[4])

axs[1, 1].legend(loc = 'lower right')

axs[1, 1].plot([0, 1], [0, 1],'r--')

axs[1, 1].set_title('Decision Tree classifier')

axs[1, 1].set_ylabel('True Positive Rate')

axs[1, 1].set_xlabel('False Positive Rate')



axs[1, 2].plot(fpr_list[5], tpr_list[5], 'b', label = 'AUC = %0.2f' % roc_auc_list[5])

axs[1, 2].legend(loc = 'lower right')

axs[1, 2].plot([0, 1], [0, 1],'r--')

axs[1, 2].set_title('Random Forest classifier')

axs[1, 2].set_ylabel('True Positive Rate')

axs[1, 2].set_xlabel('False Positive Rate')



#plt.title("ROC AUC plots for all 6 Classifiers")



plt.show() def confusion_matrixes(model_list, model_names):

fig, axes = plt.subplots(nrows=2, ncols=3, figsize=(15,10))

for cls, ax, model_name in zip(model_list, axes.flatten(),model_names):

plot_confusion_matrix(cls,

x_test,

y_test,

ax=ax,

cmap='Blues',

display_labels=["True","Fake"])

ax.title.set_text(model_name)



plt.tight_layout()

plt.show() def load_up_classifier(classifier, x_train, x_test, y_train, y_test):



pipe = Pipeline([('vect', CountVectorizer()),

('tfidf', TfidfTransformer()),

('model', classifier)])



model = pipe.fit(x_train, y_train)

prediction = model.predict(x_test)

probs = model.predict_proba(x_test)



#plot_con_matrix(classifier=model,X_test=x_test,y_test=y_test)



return model, probs

The plot_con_matrix() function will plot out the confusion matrix while the generate_roc_curve() function will plot out the AUC-ROC curve for model evaluation for all 6 models. He load_up_classifier() function will first split the news text data into train and test sets, after which, it will initiate a pipeline that will pass the data through the CountVectorizer() and TfidfTransformer() for preprocessing before fitting the pre-processed data onto the classifier.

Creating & training classifier pipelines

x_train,x_test,y_train,y_test = train_test_split(df['text'], df["target"], test_size=0.25, random_state=2020) classifier_list = [LogisticRegression(),MultinomialNB(),BernoulliNB(),GradientBoostingClassifier(),DecisionTreeClassifier(),RandomForestClassifier()] model_list = []

probs_list = [] for classifier in classifier_list:

train_model, probs = load_up_classifier(classifier,x_train, x_test, y_train, y_test)

model_list.append(train_model)

probs_list.append(probs)

Evaluation — Confusion Matrix

Confusion Matrixes are useful model evaluation tools that display the precision and recall values of the model. They are more informative compared to metrics such as accuracy as they illustrate how well a model is able to accurately predict each of the target classes.

model_names = [“Logistic Regression”,’Multinomial Naive Bayes Classifier’,’Bernoulli Naive Bayes Classifier’,’Gradient Boost Classifier’,’Decision Tree’,’RFC Classifier’] confusion_matrixes(model_list=model_list,model_names=model_names)

Confusion matrix for the 6 classifiers

Based on the comparison between the confusion matrix of the respective models, it would seem that Gradient Boost Classifier, Decision Tree Classifier, RFC Classifier and Logistic Regression Classifier performed better compared to Multinomial Naive Bayes Classifier and Bernoulli Naive Bayes Classifier. Despite this, all 6 models perform fairly well.

Evaluation — AUC - ROC Curves

AUC (Area Under The Curve)- ROC (Receiver Operating Characteristics) curve is an effective evaluation measurement to highlight how well a model can classify between different classes. The higher the AUC, the better the model at predicting true positives and negatives.

generate_all_curves(model_list = model_list, probs_list = probs_list)

AUC ROC plots for the 6 Classifiers

Saving the classifiers

The 6 Classifiers can be saved as a pickle file using pickle which needs to be loaded when the Streamlit app is being built

model_file_list = [r”models/LR_model.pkl”,

r”models/MNVBC_model.pkl”,

r”models/BNBC_model.pkl”,

r”models/GBC_model.pkl”,

r”models/DT_model.pkl”,

r”models/RFC_model.pkl”] for model, filename in zip(model_list, model_file_list):

pickle.dump(model, open(filename, ‘wb’))

Step 5: Creating the Streamlit code for deployment

Streamlit is an amazing framework for rapid deployment of machine learning applications. In order to begin, we will load up the modules needed and create the variables required in the Streamlit script below.

from keras import backend as K

from tensorflow.keras.models import Model, load_model

import streamlit as st

from nltk.tokenize import word_tokenize

from tensorflow.keras.preprocessing.sequence import pad_sequences

import pickle

import re

import string

from nltk.corpus import stopwords

from nltk.stem.wordnet import WordNetLemmatizer

import pandas as pd

import numpy as np

import seaborn as sns

import matplotlib.pyplot as plt import plotly.express as px

import plotly.graph_objects as go MODEL_PATH = r”models/LSTM_model_1.h5"

MAX_NB_WORDS = 100000 # max no. of words for tokenizer

MAX_SEQUENCE_LENGTH = 200 # max length of each entry (sentence), including padding

VALIDATION_SPLIT = 0.2 # data for validation (not used in training)

EMBEDDING_DIM = 100

tokenizer_file = “tokenizer.pickle” wordnet = WordNetLemmatizer()

regex = re.compile(‘[%s]’ % re.escape(string.punctuation)) model_list = [“Logistic Regression”,’Multinomial Naive Bayes Classifier’,’Bernoulli Naive Bayes Classifier’,’Gradient Boost Classifier’,’Decision Tree’,’RFC Classifier’]

model_file_list = [r”models/LR_model.pkl”,r”models/MNVBC_model.pkl”,r”models/BNBC_model.pkl”,r”models/GBC_model.pkl”,r”models/DT_model.pkl”,r”models/RFC_model.pkl”]

Next, we will need to load up the basic_text_cleaning() function as well as the tokenizer that was created during the pre-processing step earlier. These will be needed to pre-process the news that is inputted by the user. A Load_model() function will also be needed to load up the LSTM model and session

with open(tokenizer_file, ‘rb’) as handle:

tokenizer = pickle.load(handle)

def Load_model():

model = load_model(MODEL_PATH)

model._make_predict_function()

model.summary() # included to make it visible when model is reloaded

session = K.get_session()

return model, session @st .cache(allow_output_mutation=True)def Load_model():model = load_model(MODEL_PATH)model._make_predict_function()model.summary() # included to make it visible when model is reloadedsession = K.get_session()return model, session

Finally, we will have to create the structure of the Streamlit app. For this component, there will be 3 parts. First, the U.I that the user will use to input his news content and interact with the model. The second would be presenting the prediction results for the LSTM model and third would be presenting the prediction results for the 6 classifiers.

U.I

The U.I should be informative for the user and yet simple to use. Therefore, the app does not have too many features. It would include a title, some information of the app, a news text input container and a button for prediction.

if __name__ == ‘__main__’:

st.title(‘Fake News Classification app ‘)

st.write(“A simple fake news classification app utilising 6 traditional ML classifiers and a LSTM model”)

st.info(“LSTM model, tokeniser and the 6 traditional machine learning models loaded “)

st.subheader(“Input the News content below”)

sentence = st.text_area(“Enter your news content here”, “Some news”,height=200)

predict_btt = st.button(“predict”)

model, session = Load_model()

A Simple U.I

Prediction results for LSTM model

The news input has to be cleaned using the basic_text_cleaning() function, converted to sequences and apply padding with the same maximum sequence length as used in training. Finally, input the processed data into the loaded LSTM model provide its prediction.

if predict_btt:

clean_text = []

K.set_session(session)

i = basic_text_cleaning(sentence)

clean_text.append(i)

sequences = tokenizer.texts_to_sequences(clean_text)

data = pad_sequences(sequences, padding = ‘post’, maxlen = MAX_SEQUENCE_LENGTH)

prediction = model.predict(data)

Since the LSTM model’s prediction will return an array with a 1x2 list, with the [0] value of the array being the probability that the prediction is real while the [1] value of the array being the probability that the prediction is fake, we will have to extract these values as well as the final prediction using the .argmax() function.

prediction_prob_real = prediction[0][0]

prediction_prob_fake = prediction[0][1]

prediction_class = prediction.argmax(axis=-1)[0]

We can present the respective probabilities using a bar chart from Plotly. First, a Dataframe containing the respective probabilities have to be created. After which, a bar chart can be created using Streamlit’s inbuilt plotly_chart() function. The probabilities can be presented dynamically using the chart title and an infobox in Streamlit.

st.header(“Prediction using LSTM model”) if prediction_class == 0:

st.success(‘This is not a fake news’)

if prediction_class == 1:

st.warning(‘This is a fake news’) class_label = [“fake”,”true”]

prob_list = [prediction[0][1]*100,prediction[0][0]*100]

prob_dict = {“true/fake”:class_label,”Probability”:prob_list}

df_prob = pd.DataFrame(prob_dict)

fig = px.bar(df_prob, x=’true/fake’, y=’Probability’)

model_option = “LSTM”

if prediction[0][1] > 0.7:

fig.update_layout(title_text=”{} model — prediction probability comparison between true and fake”.format(model_option))

st.info(“The {} model predicts that there is a higher {} probability that the news content is fake compared to a {} probability of being true”.format(model_option,prediction[0] [1]*100,prediction[0][0]*100))

elif prediction[0][0] > 0.7:

fig.update_layout(title_text=”{} model — prediction probability comparison between true and fake”.format(model_option))

st.info(“The {} model predicts that there is a higher {} probability that the news content is true compared to a {} probability of being fake”.format(model_option,prediction[0][0]*100,prediction[0][1]*100))

else:

fig.update_layout(title_text=”{} model — prediction probability comparison between true and fake”.format(model_option))

st.info(“Your news content is rather abstract, The {} model predicts that there a almost equal {} probability that the news content is true compared to a {} probability of being fake”.format(model_option,prediction[0][0]*100,prediction[0][1]*100))

st.plotly_chart(fig, use_container_width=True)

Streamlit output example — LSTM model

Prediction results for the 6 classifiers

The prediction probabilities for all 6 classifiers will have to be acquired. Hence, a For loop is created to load the classifier pickle file, predict the news text input by the user and append that prediction into a list. Similar to the LSTM model above, a dataframe will have to be created to contain the respective prediction of the 6 classifiers. Afterwhich, the prediction can be presented using a pie chart to compare the prediction results across all 6 classifiers.

st.header(“Prediction using 6 traditional machine learning model”)

predictions = []

for model in model_file_list:

filename = model

model = pickle.load(open(filename, “rb”))

prediction = model.predict([sentence])[0]

predictions.append(prediction) dict_prediction = {“Models”:model_list,”predictions”:predictions}

df = pd.DataFrame(dict_prediction) num_values = df[“predictions”].value_counts().tolist()

num_labels = df[“predictions”].value_counts().keys().tolist() dict_values = {“true/fake”:num_labels,”values”:num_values}

df_prediction = pd.DataFrame(dict_values)

fig = px.pie(df_prediction, values=’values’, names=’true/fake’)

fig.update_layout(title_text=”Comparision between all 7 models: Prediction proportion between True/Fake”)

st.plotly_chart(fig, use_container_width=True)

st.table(df)

Streamlit output example — 6 Classifiers

Conclusion

Even with the optimistic precision scores from both the classifiers and the accuracy and loss scores from the LSTM model, there are further improvements that can be made to the 6 classifiers and the LSTM model. First, the hyperparameters from the 6 classifiers have not been tweaked, tweaking the parameters and conducting a GridSearchCV can help identify an optimal classifier out of the 6 used. There was also evidence of overfitting for the LSTM model. Moreover, trying out other RNNs could also be explored.