Context-specific Pre-processing for NLP with spaCy: Tweets
By: Wei-Ting Yap
Tags: Getting Started, Spacy, Editors Pick, NLP, Data Science

NLP with the spaCy library

spaCy is an open-source python library for natural language processing. It integrates well with the rest of the python machine learning libraries (scikit-learn, TensorFlow, PyTorch) and more, and uses a object-oriented approach to keep its interface readable and easy to use. Notably, it its model returns Document type data, which consists of tokens with various useful annotations (e.g. its lemma, whether it's a stopword) as attributes.

Let’s import spaCy, download the model for American English, and load the relevant spaCy model.

# download spaCy model for American English

!python3 -m spacy download en_core_web_sm import spacy

import en_core_web_sm

nlp = en_core_web_sm.load()

How well does out-of-the-box spaCy do with tweets?

Before we customize spaCy, we can take a look at how the out-of-the-box model tokenizes tweets with its default rules. I created a tweet that included a number, a contraction, a hashtag, a mention and a link.

As shown below, out-of-the-box spaCy already breaks up contractions and gives us the relevant lemmas. It also recognizes numbers, mentions and URLs as their own tokens according to the default rules. That leaves us with hashtags, which are split into a ‘#’ punctuation and hashtag content, instead of it staying as a whole token.

# Let's see what spaCy does with numbers, contractions, #hashtags, @mentions and URLs

s = "2020 can't get any worse #ihate2020 @bestfriend <https://t.co>"

doc = nlp(s) # Let's look at the lemmas and is stopword of each token

print(f"Token\\t\\tLemma\\t\\tStopword")

print("="*40)

for token in doc:

print(f"{token}\\t\\t{token.lemma_}\\t\\t{token.is_stop}"

This prints:

Token Lemma Stopword

========================================

2020 2020 False

ca can True

n't not True

get get True

any any True

worse bad False

# # False

ihate2020 ihate2020 False

@bestfriend @bestfriend False

<https://t.co> <https://t.co> False

Modifying spaCy

We can modify spaCy’s model to recognize hashtags as entire tokens.

spaCy’s tokenizer can be modified (you can also build a custom tokenizer if you want!) by redefining its default rules. spaCy’s tokenizer prioritizes rules in the following order: token match pattern, prefix, suffix, infixes, URL, special cases (see How spaCy’s Tokenizer Works).

In our case, we’ll modify the tokenizer’s pattern matching regex pattern (read more about regex here: A simple intro to Regex with Python) by appending '#\\w+' , which is a pattern for the hash symbol and a word.

# We want to also keep #hashtags as a token, so we will modify the spaCy model's token_match import re # Retrieve the default token-matching regex pattern

re_token_match = spacy.tokenizer._get_regex_pattern(nlp.Defaults.token_match) # Add #hashtag pattern

re_token_match = f"({re_token_match}|#\\w+)"

nlp.tokenizer.token_match = re.compile(re_token_match).match # Now let's try again

s = "2020 can't get any worse #ihate2020 @bestfriend <https://t.co>"

doc = nlp(s) # Let's look at the lemmas and is stopword of each token

print(f"Token\\t\\tLemma\\t\\tStopword")

print("="*40)

for token in doc:

print(f"{token}\\t\\t{token.lemma_}\\t\\t{token.is_stop}")

Our code prints:

Token Lemma Stopword

========================================

2020 2020 False

ca can True

n't not True

get get True

any any True

worse bad False

#ihate2020 #ihate2020 False

@bestfriend @bestfriend False

<https://t.co> <https://t.co> False

Pre-processing algorithm

We can then proceed to create a preprocessing algorithm, and put it in a function so it can be called on every tweet in the training set. In the following preprocess function, each tweet:

Is changed to lower case

Is tokenized with our modified spaCy model

Has its lemmas of tokens set unioned with our features set

set Has its bag-of-words representation constructed in a dictionary freq

Has its hashtags, mentions and URLs counted

# Create a pre-process function for each tweet

def preprocess(s, nlp, features):

"""

Given string s, spaCy model nlp, and set features (lemmas encountered),

pre-process s and return updated features and bag-of-words representation dict freq

- changes s to lower-case

- tokenize s using nlp to create a doc

- update features with lemmas encountered in s

- create bag-of-words representation in dict type freq, including counts for hashtags, mentions and URLs

""" # To lowercase

s = s.lower() # Creating a doc with spaCy

doc = nlp(s)

lemmas = []

for token in doc:

lemmas.append(token.lemma_) # Union between lemmas and our features set

features |= set(lemmas) # Constructing a bag of words for the tweet

freq = {'#':0,'@':0,'URL':0}

for word in lemmas:

freq[str(word)] = 0

for token in doc:

if '#' in str(token): freq['#'] += 1 # Count number of hashtags, regardless of hashtag

if '@' in str(token): freq['@'] += 1 # Count number of mentions, regardless of mention

if 'http://' in str(token): freq['URL'] += 1 # Count number of URLs, regardless of URL

freq[str(token.lemma_)] += 1



return features, freq

Setting it up

We’ll create a copy our de-duplicated data as a best practice, so that any pre-processing changes does not affect the original state of our training data. Then, we will initialize a python set features , which will contain all features of each tweet. In addition to all lemmas encountered via tokenization of each tweet, features will include number of hashtags ( # ), number of mentions ( @ ), and number of URLs ( URL ).

preprocess_df = df # Duplicate for preprocessing

features = set({'#','@','URL'}) # Using set feature to contain all words (lemmas) seen

Using our preprocess function, we'll preprocess every tweet, updating features each time with new lemmas seen. With each tweet, the tweet's bag of words representation freq is also appended to an array of bag of words for each tweet ( bow_array ).

# Array bow_array of bow representations for each tweet;

# bow_array[i] is the bow representation for tweet id (i+1)

bow_array = []

for i in range(len(preprocess_df)):

features, freq = preprocess(preprocess_df.iloc[i]['text'],nlp,features)

bow_array.append(freq)

len(bow_array) # 7561

With all lemmas encountered across all tweets collected in features , we can create a dataframe bow to represent the features of all the tweets.

# Create dataframe for bag of words representation for each tweet

bow = pd.DataFrame('0', columns=features,index=range(len(preprocess_df)))

bow['id']=preprocess_df.index

bow.set_index('id',drop=True,inplace=True)

Now, let’s update our dataframe with the feature values of each tweet.

# Update bow[i] with bag-of-words freq of the tweet id (i+1)

for i in range(len(preprocess_df)):

freq = bow_array[i]

for f in freq:

bow.loc[i+1,f]=freq[f]

Finally, we will join our training data dataframe with our bag-of-words. pandas Dataframe’s join method allows us to add columns from one dataframe to another for rows where the index matches. Note that we append '_data' as a suffix to all columns from the 'left' dataframe, which is preprocess_df . This is to avoid conflicts between the keyword given as part of the training data, and 'keyword' as a lemma-token-feature. Remember to save the preprocessed .csv file for easier next steps!