How to Rank Text Content by Semantic Similarity | by Rupert Thomas | Towards Data Science

Search is a standard tool for finding content — whether it's online or on-device — but what if you wanted to go deeper and find content based on the meaning of words? Tools developed for Natural Langage Processing can help.
In this article we will cover two methods of calculating the similarity of text:
Term Frequency-inverse document frequency (TF-idf): this looks at words that appear in both pieces of text, and scores them based on how often they appear. It is a useful tool if you expect the same words to appear in both pieces of text, but some words are more important that others.Semantic similarity: this scores words based on how similar they are, even if they are not exact matches. It borrows techniques from Natural Language Processing (NLP), such as word embeddings. This is useful if the word overlap between texts is limited, such as if you need ‘fruit and vegetables’ to relate to ‘tomatoes’.
Term Frequency-inverse document frequency (TF-idf): this looks at words that appear in both pieces of text, and scores them based on how often they appear. It is a useful tool if you expect the same words to appear in both pieces of text, but some words are more important that others.
Semantic similarity: this scores words based on how similar they are, even if they are not exact matches. It borrows techniques from Natural Language Processing (NLP), such as word embeddings. This is useful if the word overlap between texts is limited, such as if you need ‘fruit and vegetables’ to relate to ‘tomatoes’.
For the first part, we’ll use the TF-idf implementation in scikit-learn in isolation, as its really simple and only requires a couple of lines of code. For semantic similarity, we’ll use a number of functions from gensim (including its TF-idf implementation) and pre-trained word vectors from the GloVe algorithm. Also, we’ll need a few tools from nltk. These packages can be installed using pip:
pip install scikit-learn~=0.22pip install gensim~=3.8pip install nltk~=3.4
Alternatively, you can grab the example code from the GitHub repository and install from the requirements file:
git clone https://github.com/4OH4/doc-similaritycd doc-similaritypip install -r requirements.txt
github.com
Run the demo code in examples.ipynb,Use the tfidf.rank_documents(search_terms: str, documents: list) function to score documents based on overlapping content,Use the docsim.DocSim() class to score documents on similarity using doc2vec and the GloVe word embedding model.
Run the demo code in examples.ipynb,
Use the tfidf.rank_documents(search_terms: str, documents: list) function to score documents based on overlapping content,
Use the docsim.DocSim() class to score documents on similarity using doc2vec and the GloVe word embedding model.
Document: a piece of text, in the form of a string. This could be just a few words, or a whole novel.
Corpus: a collection of documents.
Term: a word in a document.
Term Frequency-inverse document frequency (or TF-idf) is an established technique for scoring document similarity based on the importance of the words that they share. Very high level summary:
if a term (word) appears frequently in a document, that term is probably important in that document.
but, if a term appears frequently in many documents, that term is probably less important generally.
A TF-idf score estimates the trade-off between those two heuristics, based on how frequently a word appears. There is a more detailed summary here.
We can create and fit a TF-idf vectorizer model from scikit-learn with only a few lines of code:
Here, we create the model and ‘fit’ using the text corpus. TfidfVectorizer handles the pre-processing using its default tokenizer — this converts strings into lists of single word ‘tokens’. It produces a sparse matrix of document vectors containing the term frequencies.
We then take the dot product (linear kernel) of the first vector (that contains the search terms) with the documents to determine the similarity. We have to ignore the first similarity result ([1:]) as that is comparing the search terms to themselves.
This works well for a simple example, but may fall down in real-world cases for a number of reasons.
Firstly, it doesn’t make much sense to involve words such as [‘and’, ‘on’, ‘the’, ‘are’] in the matching process, as these stopwords don’t contain contextual information. These words should be stripped out before determining similarity.
Secondly, we would want ‘fruit’ and ‘fruits’ to be recognised as related, although the model above only find exact matches. One approach to this problem is to reduce each word down to its simplest lemma — for that we need a lemmatizer.
In this case, ‘tomatoes’ in the second document is reduced to ‘tomato’ by the lemmatizer (tokenizer), which is then matched to the same word in the search terms.
Conceptually, each document is a point in a high-dimensional space, where the number of dimensions is equal to the number of unique words in the text corpus. This space will usually be quite empty, with large distances between points (documents) as documents contain a wide variety of words. In this scenario, the angle between points makes more sense as a metric of similarity than, for example, a distance-related metric such as Euclidean distance.
The similarity here is referred to as the cosine similarity. The output from TfidfVectorizer is (by default) L2-normalized, so then the dot product of two vectors is the cosine of the angle between the points denoted by the vectors.
It’s fast and works well when documents are large and/or have lots of overlap.
It looks for exact matches, so at the very least you should use a lemmatizer to take care of the plurals.
When comparing short documents with limited-term variety — such as search queries — there is a risk that you will miss semantic relationships where there isn’t an exact word match.
A move advanced approach is to compare documents based on how similar their words are. For example, ‘apples’ and ‘oranges’ might be regarded as more similar than ‘apples’ and ‘Jupiter’. Judging word similarity at scale is difficult — one widely used approach is to analyse a large corpus of text and rank words that appear together often as being more similar.
This is the basis of the word embedding model GloVe: it maps words into numerical vectors — points in a multi-dimensional space so that words that occur together often are near each other in space. It is an unsupervised learning algorithm, developed at Stanford University in 2014.
In the previous example, we used WordNetLemmatizer from the nltk package to stem and tokenize (convert to a list of single word strings) our data. Here we do similar preprocessing in gensim, and also remove any HTML tags that may be present, such as if we have scraped data from the web:
Then we create a similarity matrix, that contains the similarity between each pair of words, weighted using the term frequency:
Finally, we calculate the soft cosine similarity between the query and each of the documents. Unlike the regular cosine similarity (which would return zero for vectors with no overlapping terms), the soft cosine similarity considers word similarity as well.
There is a full example in the notebook in the code repository. There is also a self-contained DocSim class, that can be imported as a module and used to run semantic similarity queries without additional code:
from docsim import DocSimdocsim = DocSim(verbose=True)similarities = docsim.similarity_query(query_string, documents)
The GloVe word embedding models can be quite large — on my machine it took about 40 seconds to load from disk. Once it's loaded, however, subsequent operations are faster. The multi-threaded version of the class loads the model in the background, to avoid locking the main thread for a significant period of time. It is used in a similar way, although will raise an exception if the model is still loading so the status of the model_ready property should be checked first.
from docsim import DocSim_threadeddocsim = DocSim_threaded(verbose=True)similarities = docsim.similarity_query(query_string, documents)
It is more flexible as it doesn’t rely on finding exact matches.
There is a lot more computation involved so it can be slower, and the word embedding models can be quite large and take a while to prepare for first use. This scales well, but running a single query is slow.
Most words have some degree of similarity to other words, so almost all documents will have some non-zero similarity to other documents. Semantic similarity is good for ranking content in order, rather than making specific judgements about whether a document is or is not about a specific topic.
We’ve looked at two methods for comparing text content for similarity, such as might be used for search queries or content recommender systems. The first (TF-idf) scores document relationships based on the frequency of occurrence of shared words. It is fast and works well when documents are large and/or have lots of overlapping terms. The second technique looks for shared words that address similar concepts, but it does not require an exact match: for example, it links ‘fruit and vegetables’ with the word ‘tomato’. This is slower, and can sometimes give less clear-cut results, but is good with shorter search queries or documents with low word overlap.
I examined both of these methods for a real-world application: I needed to know if any of a <collection of documents> related to a small number of <search terms>. Initially, I thought I would need to use semantic similarity matching — the search terms came from uncontrolled user input and so there might be some pretty tenuous relationships. Document ranking was a useful feature, but the overall aim was to apply a decision threshold so that a binary yes/no result could be generated.
For that particular application, I found that using TF-idf alone was sufficient. The collections of documents were usually large enough that — in the cases when a match should indeed be found—there was sufficient overlap with the search terms. TF-idf gave a clearer separation between the positive and negative cases, although it did miss a few documents that would also be relevant.
Its worth taking a data-driven approach, and seeing what works best in your application.
What are your thoughts? Have you used these tools in your work, or do you favor a different approach? Leave your comments below!
Rupert Thomas is a technology consultant specialising in machine learning, machine vision, and data-driven products. @rupertthomas
Sci-kit Learn: TfidfVectorizer
GloVe: Global Vectors for Word Representation
Gensim: Soft Cosine Tutorial