Another ‘boring’ thing I’ve learnt about 2016 elections
By: Laura Vlad
Tags: 2016 Election, Word Cloud, Fake News, Data Science

CODE

I started with downloading an extensive dataset from University of Victoria containing two type of articles: fake and real News. The dataset was collected in 2015 and 2016. It never crossed my mind that 2016 was the election year until the night before I had to present. it. The truthful articles were collected from Reuters and the fake ones were flagged by Politifact as fake.

I understand that considering Reuters and Politifact a reliable source of information is a personal choice. Some might chose to label all Reuters articles as fake.

First we had to import useful packages:

!pip install wordcloud import re

import numpy as np

import pandas as pd

import matplotlib

import matplotlib.pyplot as plt

import seaborn as sns

%matplotlib inline import nltk

from nltk import FreqDist

from nltk import word_tokenize

import string from nltk.collocations import *

nltk.download('wordnet')

nltk.download('stopwords')

from nltk.corpus import stopwords

from wordcloud import WordCloud, STOPWORDS

Then we need to download the zips , unzip and extract the files:

#unzip True.csv file

from zipfile import ZipFile

ZipFile("True.csv.zip").extractall("") #unzip Fake.csv file.

from zipfile import ZipFile

ZipFile("Fake.csv.zip").extractall("")

We use padas to read the csv file.

#load True.csv and turn it into a dataframe.

df_true = pd.read_csv('True.csv')

print(len(df_true))

df_true.head()

Now let’s examine and see if we observe any patterns in our datasets and and then transform them in files that classifiers like to model.

I’m only going to show the procedure on the dataset with real news. You can do the same for the fake news dataset.

First we drop all columns we don't need , and keep only the ‘text’ one, then we add a ‘label’ column with the value ‘true’ for real articles.

df_true['label'] = 'true' df_true = df_true.drop(['date'], axis = 1)

Let’s find duplicate rows.

duplicateRowsData = df_true[df_true.duplicated()]

duplicateRowsData

Get indexes for all duplicate rows and delete them from our dataset..

duplicateIndex = []

for row in duplicateRowsData.index:

duplicateIndex.append(row)

print(duplicateIndex)

len(duplicateIndex) df_true = df_true.drop(duplicateIndex) STOPWORDS = stopwords.words('english')

STOPWORDS += list(string.punctuation)

STOPWORDS += ['washington', 'reuters']

Stopwords are generally the most common words used in English like: “the”, “is”, “in”, “for”, “where”, “when”, “to”, “not” etc.” In most of the circumstances they don’t add a lot of information to the text.

We’ll also add punctuation and the words “washington” and “reuters” to our the stopword bag. We observed that they appear in all truthful articles so we will take them to make the model more accurate

We’re going to set up our Stopword bag:

STOPWORDS = stopwords.words('english')

STOPWORDS += list(string.punctuation)

STOPWORDS += ['washington', 'reuters']

Let’s clean it even further by removing special characters using ‘re.’

df_true['Clean_text'] = df_true['Clean_text'].str.replace("[^a-zA-Z' ]","")

df_true['Clean_text'] = df_true['Clean_text'].str.replace(re.compile(r"(^| ).( |$)"), " ")

In the next step we will tokenize each article text.

In this case tokenization means breaking the text apart into individual linguistic units.

df_true['Clean_text'] = df_true['Clean_text'].str.split()

Now we can remove the stopwords.

df_true['Clean_text'] = df_true['Clean_text'].apply(lambda text: [word for word in text if word not in STOPWORDS])

Normally the next step would lemanizing or stemming or both but we want to see the whole word on the graph so we will just combine the article words together back in texts.

df_true['Clean_text'] = df_true['Clean_text'].apply(lambda text: ' '.join(text))

Now that we will create a list with all the texts and then create a bag with all the words in each text.