Supercharging word vectors
By: Josh Taylor
Tags: Fasttext, Modern Slavery, Naturallanguageprocessing, Python, Data Science

Supercharging word vectors

A simple technique to boost fastText and other word vectors in your NLP projects

Over the last few years, word vectors have been transformative in their ability to create semantic linkages between words. It is now the norm for these to be fed into deep learning models for tasks such as classification or sentiment analysis.

Despite this, I have always been impressed with the simple power of term-frequency based analysis. Term Frequency Inverse Document Frequency (TF-IDF) can be easily explained to non-technical audiences and the results are straightforward to interpret.

This article explores how combining TF-IDF with word vectors can give you outputs which are both easily interpretable and can also capture the subtle semantic relationships which exist in language.

Whilst the internet is littered with examples of both ‘bag-of-words’ analysis using TF-IDF (see here)and word vector approaches (see here), these are almost always seen as mutually exclusive techniques. This article explores how combining TF-IDF with word vectors can give you outputs which are both easily interpretable and can also capture the subtle semantic relationships which exist in language. Using the fastText method for creating word vectors, we will also be able to create a model which can handle out-of-vocabulary words as well as being robust to spelling mistakes and typos.

fastText word vectors

This article assumes prior knowledge of word vectors but it is worth touching on fastText and how this differs from the more widely known word2vec approach to creating vector representations of words. FastText was developed by Facebook with a stable release being open-sourced in 2017.

The most noticeable difference between fastText and word2vec is that fastText splits out words using n-gram characters. For example, ‘Lincolnshire’, (a county in the UK) would be split into:

Lin, inc, nco, col, oln, lns, nsh, shi, hir, ire

Where n=3. This approach is a significant improvement over word2vec for two reasons:

The ability to infer out-of-vocabulary words. For example, the above model would understand that ‘Lancashire’ (also a county in the UK) is related to Lincolnshire due to the overlap of ‘shire’ (or ‘shi’, ‘hir’, and ‘ire’) between the two words. The robustness to spelling mistakes and typos. It is easy to see that the same character level modelling also means that fastText is robust enough to handle spelling variations. This is particularly useful in the analysis of Social Media content.

A detailed review of how fastText works can be viewed here.

Show me the code!

The rest of this article walks through a simple example which will train a fastText model on a series of documents, apply TF-IDF to these vectors and use this to perform further analysis.

The documents in question are Modern Slavery statements submitted by companies to explain the steps they are taking to eradicate Modern Slavery both internally and within their supply chains. The below article shows how this data has been cleaned prior to analysis:

If you would like to follow along, a colab notebook can be found here containing all of the code.

Step 1. tokenize text and create phrases

We will use spaCy to split each of the documents into a list of words (tokenization). We will also clean the data by removing stop words, punctuation and converting to lowercase using the Gensim library:

#The dataframe is called 'combined' it has a column called 'text' containing the text data for each company #creates a list of documents with a list of words inside:

text = []

for i in combined.text.values:

doc = nlp(remove_stopwords(strip_punctuation(strip_non_alphanum(str(i).lower()))))

tokens = [token.text for token in doc]

text.append(tokens)

We will then stick together common terms. For example, as each of the documents writes about modern slavery, it is useful to combine these two words into a single phrase ‘modern_slavery’. The Gensim library makes this easy and the code is below:

common_terms = ["of", "with", "without", "and", "or", "the", "a"]

# Create the relevant phrases from the list of sentences:

phrases = Phrases(text, common_terms=common_terms, threshold = 10, min_count=5)

# The Phraser object is used from now on to transform sentences

bigram = Phraser(phrases)

# Applying the Phraser to transform our sentences is simply

tokens = list(bigram[text])

We now have a list of documents, each containing a list of words. Common phrases have been combined into words.

Step 2. Train the fastText model

This requires only a single line of code using the Gensim library’s fastText model. You can set the n-gram size (‘window’) as well as the dimension size of the vectors (‘size’). Here we have created vectors with 100 dimensions and used an n-gram size of 3:

model = FastText(tokens, size=100, window=3, min_count=1, iter=10, sorted_vocab=1)

We can check the model using the following:

similarities = model.wv.most_similar(restrict_vocab=10000, positive=['contract'])

This returns:

subcontract 0.9493274688720703

sub_contract 0.9349175095558167

contractual 0.9346154928207397

contracts 0.9312876462936401

contractor 0.9068889617919922

Just to show how fastText can handle out-of-vocabulary words we can try misspelling contract as ‘contracct’. From the below, we can see that the model has no problem interpreting this word even though it has never seen it before!

contract 0.9389102458953857

contracts 0.9058693051338196

contrary 0.9027011394500732

contractor 0.8995087742805481

contractual 0.885408341884613

Step 3. Apply TF-IDF to the vectors

This step first creates TF-IDF information for each document using Scikit. It then applies the weighting to each word vector in a document and then averages these across the document:

#TF-IDF

text = []

for i in tqdm(tokens):

string = ' '.join(i)

text.append(string)

tf_idf_vect = TfidfVectorizer(stop_words=None)

final_tf_idf = tf_idf_vect.fit_transform(text)

tfidf_feat = tf_idf_vect.get_feature_names() #Applying TF-IDF scores to the model vectors tfidf_sent_vectors = []; # the tfidf-w2v for each sentence/review is stored in this list

row=0;

errors=0

for sent in tqdm(tokens): # for each review/sentence

sent_vec = np.zeros(100) # as word vectors are of zero length

weight_sum =0; # num of words with a valid vector in the sentence/review

for word in sent: # for each word in a review/sentence

try:

vec = model.wv[word]

# obtain the tf_idfidf of a word in a sentence/review

tfidf = final_tf_idf [row, tfidf_feat.index(word)]

sent_vec += (vec * tfidf)

weight_sum += tfidf

except:

errors =+1

pass

sent_vec /= weight_sum

#print(np.isnan(np.sum(sent_vec))) tfidf_sent_vectors.append(sent_vec)

row += 1

print('errors noted: '+str(errors))

We now have our supercharged TF-IDF word vectors for every document in our corpus! Time to test it out…

Results

Now that we have our fastText vectors combined with out TF-IDF weightings, it is easy to sense check the results using cosine similarity. The below code finds a matching company based on a query string and then produces a list of the most similar companies based on their Modern Slavery returns.

compName = 'Deloitte' query = [combined.loc[combined.Company.str.contains(compName)].iloc[0]['FT_tfidf']]

query = np.array(list(query))

query = np.nan_to_num(query) vectors = np.array(list(combined.FT_tfidf.values))

vectors = np.nan_to_num(vectors) cosine_similarities = pd.Series(cosine_similarity(query, vectors).flatten()) for i,j in cosine_similarities.nlargest(10).iteritems():

print(str(i) + '-' + combined.Company.iloc[i] + " " + str(j))

The example here was Deloitte, a professional services firm. From the results below we can see that the most similar returns are from related companies (namely Deloitte’s own return for another financial year as well as DLA, a law partnership and Ernst & Young, another of the ‘Big 4’ accounting firms):

Deloitte LLP 1.0000000000000002

Deloitte LLP 0.949480726712043

DLA Piper International LLP 0.8635928917765654

Travers Smith LLP 0.8495683187822698

EBSCO International Inc. 0.8405915834557236

Vink UK 0.8356471225683573

Ernst & Young LLP 0.8345225966043321

IFS UK Ltd 0.8288755547154663

First Central Insurance 0.8279939308519769

TalkTalk Telecom Group PLC 0.8260778120395709

Hopefully this post has demonstrated that combining word vectors with TF-IDF techniques can produce powerful results which are also simple to explain and interpret. The document vectors produced in this walk-through can now be used in text classification, clustering as well as further analysis (for example, discovering trends in how company returns have changed over time or across industries).