Exploring the Trump Twitter Archive with SpaCy
By: Ednalyn C. De Dios
Tags: Data Science, NLP, Towards Data Science, Spacy, Exploring Trump

For this project, we’ll be using pandas for data manipulation, spaCy for natural language processing, and joblib to speed things up.

Let’s get started by firing up a Jupyter notebook!

Housekeeping

Let’s import pandas and also set the display options so Jupyter won’t truncate our columns and rows. Let’s also set a random seed for reproducibility.

# for manipulating data

import pandas as pd # setting the random seed for reproducibility

import random

random.seed(493) # to print out all the outputs

from IPython.core.interactiveshell import InteractiveShell

InteractiveShell.ast_node_interactivity = "all" # set display options

pd.set_option('display.max_columns', None)

pd.set_option('display.max_rows', None)

pd.set_option('display.max_colwidth', -1)

Getting the Data

Let’s read the data into a dataframe. If you want to follow along, you can download the cleaned dataset here along with the file for stop words¹. This dataset contains Trump’s tweets from the moment he took office on January 20, 2017 to May 30, 2020.

df = pd.read_csv('trump_20200530_clean.csv', parse_dates=True, index_col='datetime')

Let’s take a quick look at the data.

df.head()

df.info()

Using spaCy

Now let’s import spaCy and begin natural language processing.

# for natural language processing: named entity recognition

import spacy

import en_core_web_sm

We’re only going to use spaCy’s ner functionality or named-entity recognition so we’ll disable the rest of the functionalities. This will save us a lot of loading time later.

nlp = spacy.load('en_core_web_sm', disable=['tagger', 'parser', 'textcat'])

Now let’s load the contents stopwords file into the variable stopswords . Note that we converted the list into a set to also save some processing time later.

with open('twitter-stopwords — TA — Less.txt') as f:

contents = f.read().split(',')

stopwords = set(contents)

Next, we’ll import joblib and define a few functions to help with parallel processing.

In the code above², the function preprocess_parallel executes the other function process_chunks in parallel to help with speed. The function process_chunks iterates through a series of texts — in our case, the column 'tweet' of our the df dataframe — and inspects the entity if it belongs to either NORP, PERSON, FAC, ORG, GPE, LOC, PRODUCT, or EVENT. If it is, the entity is then appended to 'preproc_pipe' and subsequently returned to its caller. Prashanth Rao has a very good article on making spaCy super fast.

Let’s call the main driver for the functions now.

df['entities'] = preprocess_parallel(df['tweet'], chunksize=1000)

Doing a quick df.head() will reveal the new column 'entities' that we added earlier to hold the entities found in the 'tweet' column.

Prettifying the Results

In the code below, we’re making a list of lists called 'entities' and then flattening it for easier processing. We’re also converting it into a set called 'entities_set' .

entities = [entity for entity in df.entities if entity != []]

entities = [item for sublist in entities for item in sublist] entities_set = set(entities)

Next, let’s count the frequency of the entities and append it to the list of tuples entities_counts . Then let’s convert the results into a dataframe df_counts .

df_counts = pd.Series(entities).value_counts()[:20].to_frame().reset_index()

df_counts.columns=['entity', 'count']

df_counts

For this step, we’re going to reinitialize an empty list entity_counts and manually construct a list of tuples with a combined set of entities and the sum of their frequencies or count.

Let’s take a quick look before continuing.

Finally, let’s convert the list of tuples into a dataframe.

df_ner = pd.DataFrame(entity_counts, columns=["entity", "count"]).sort_values('count', ascending=False).reset_index(drop=True)

And that’s it!

We’ve successfully created a ranking of the named entities that President Trump most frequently talked about in his tweets since taking office.