Word Distance between Word Embeddings | by Edward Ma | Towards Data Science

Word Mover’s Distance (WMD) is proposed fro distance measurement between 2 documents (or sentences). It leverages Word Embeddings power to overcome those basic distance measurement limitations.
WMD[1] was introduced by Kusner et al. in 2015. Instead of using Euclidean Distance and other bag-of-words based distance measurement, they proposed to use word embeddings to calculate the similarities. To be precise, it uses normalized Bag-of-Words and Word Embeddings to calculate the distance between documents.
After reading this article, you will understand:
Earth Mover Distance (EMD)
Word Mover’s Distance (WMD)
Relaxed Word Moving Distance (RWMD)
WMD Implementation
Take Away
Before introducing WMD, I have to share the idea of Earth Mover Distance (EMD) first because the core part of WMD is EMD.
EMD [2] solves transportation problem. For instance, we have m and n while m and n denote a set of suppliers and warehouses. The target is going to minimize transportation cost such that shipping all goods from m to n. Given that there are constraints:
Only allowing transport from m to n. Not allowing transport from n to m
Total number of sending cargoes cannot exceed total capacity of m
Total number of receiving cargoes cannot exceed total capacity of n
Maximum number of transportation is the minimum between total cargoes in m and total cargoes in n
The denotations are:
p: Set of origin
q: Set of destination
f(i,j): flow from i to j
m: Number of origin
n: Number of destination
w(i, j): Number of cargo transport from i to j
To optimal flow F, the linear formula is
In the previous blog, I shared how we can use simple way to find the “similarity” between two documents (or sentences). At that time, Euclidean Distance, Cosine Distance and Jaccard Similarity are introduced but it has some limitations. WMD is designed to overcome synonym problem.
The typical example is
Sentence 1: Obama speaks to the media in Illinois
Sentence 2: The president greets the press in Chicago
Except the stop words, there is no common words among two sentences but both of them are taking about same topic (at that time).
WMD use word embeddings to calculate the distance so that it can calculate even though there is no common word. The assumption is that similar words should have similar vectors.
First of all, lower case and removing stopwords is an essential step to reduce complexity and preventing misleading.
Sentence 1: obama speaks media illinois
Sentence 2: president greets press chicago
Retrieve vectors from any pre-trained word embeddings models. It can be GloVe, word2vec, fasttext or custom vectors. After that it using normalized bag-of-words (nBOW) to represent the weight or importance. It assumes that higher frequency implies that it is more important.
It allows transfer every word from sentence 1 to sentence 2 because algorithm does not know “obama” should transfer to “president”. At the end it will choose the minimum transportation cost to transport every word from sentence 1 to sentence 2.
The best average time of solving WMD is about O(p3 log p) while p is number of unique word. It is a little bit slow so there are two approaches to improve the reduce computation time. First one is Word Centroid Distance (WCD) which is summarizing the lower bound distance between. Second approach is Relaxed Word Moving Distance (RWMD) which is using the closet distance without considering there are multiple words transforming to single words.
Taking the previous sentence as an example. Assuming that shortest word in sentence of all word in sentence 1 is “president”, it will use summarize these score instead of pairing one by one. So that the time complexity reduce to O(p2).
By using gensim, we only need to provide two list of tokens then it will take the rest of calculation
subject_headline = news_headlines[0]subject_token = headline_tokens[0]print('Headline: ', subject_headline)print('=' * 50)print()for token, headline in zip(headline_tokens, news_headlines):    print('-' * 50)    print('Comparing to:', headline)    distance = glove_model.wmdistance(subject_token, token)    print('distance = %.4f' % distance)
Output
Headline:  Elon Musk's Boring Co to build high-speed airport link in Chicago==================================================--------------------------------------------------Comparing to: Elon Musk's Boring Co to build high-speed airport link in Chicagodistance = 0.0000--------------------------------------------------Comparing to: Elon Musk's Boring Company to build high-speed Chicago airport linkdistance = 0.3589--------------------------------------------------Comparing to: Elon Musk’s Boring Company approved to build high-speed transit between downtown Chicago and O’Hare Airportdistance = 1.9456--------------------------------------------------Comparing to: Both apple and orange are fruitdistance = 5.4350
In gensim implementation, OOV will be removed so that it will not throw an exception or using random vector.
For source code, you may check out from my github repo.
The advantage of WMD are hyper-parameter free and overcoming synonym problem.
Same as those simple approaches, WMD does not consider ordering.
The time complexity is an issue. The original version is O(p3 log p) while the enhanced version is still O(p2).
Pre-train vectors may not apply to all scenario.
I am Data Scientist in Bay Area. Focusing on state-of-the-art in Data Science, Artificial Intelligence , especially in NLP and platform related. You can reach me from Medium Blog, LinkedIn or Github.
[1] Kusner Matt J., Sun Yu, Kolkin Nicholas I., Weinberger Kilian Q. From Word Embeedings To Document Distance. 2015. http://proceedings.mlr.press/v37/kusnerb15.pdf
[2] EMD Theory: https://en.wikipedia.org/wiki/Earth_mover%27s_distance