Semantic Similarity Using Transformers | by Raymond Cheng | Towards Data Science

Semantic Similarity, or Semantic Textual Similarity, is a task in the area of Natural Language Processing (NLP) that scores the relationship between texts or documents using a defined metric. Semantic Similarity has various applications, such as information retrieval, text summarization, sentiment analysis, etc.
There have been a lot of approaches for Semantic Similarity. The most straightforward and effective method now is to use a powerful model (e.g. transformer) to encode sentences to get their embeddings and then use a similarity metric (e.g. cosine similarity) to compute their similarity score. The similarity score indicates whether two texts have similar or more different meanings. This post will show how to implement Semantic Similarity using Transformers, which is a powerful NLP architecture that has resulted in state-of-the-art performance for various NLP tasks.
So without ado, let’s get started!
Install Dependencies and LibrariesImport LibraryModel Selection and InitializationCalculate semantic similarity between two sentencesCalculate semantic similarity between two lists of sentencesRetrieve Top K most similar sentences from a corpus given a sentence
Install Dependencies and Libraries
Import Library
Model Selection and Initialization
Calculate semantic similarity between two sentences
Calculate semantic similarity between two lists of sentences
Retrieve Top K most similar sentences from a corpus given a sentence
The main library that we are going to use to compute semantic similarity is SentenceTransformers (Github source link), a simple library that provides an easy method to calculate dense vector representations (e.g. embeddings) for texts. It contains many state-of-the-art pretrained models that are fine-tuned for various applications. One of the primary tasks that it supports is Semantic Textual Similarity, which is the one we will focus on in this post.
To install SentenceTransformers, you will have to install the dependencies Pytorch and Transformers first.
Go to the Pytorch official website and follow the instructions to install Pytorch.
To install transformers, run:
pip install transformers
Now, have you have installed Pytorch and transformers, you can install SentenceTransformers by:
pip install sentence-transformers
Note: SentenceTransformers recommends Python 3.6 or higher, PyTorch 1.6.0 or higher, and transformers v3.1.0 or higher.
After successfully installing the SentenceTransformers library and its dependencies, we can start using the library. We can import it by:
from sentence_transformers import SentenceTransformer, utilimport numpy as np
SentenceTransformers supports a variety of pretrained models fine-tuned for different tasks out of the box. To find a list of models optimized for semantic textual similarity, you can see it here.
As of the time this post is written, stsb-roberta-large, which uses ROBERTA-large as the base model and mean-pooling, is the best model for the task of semantic similarity. Thus, we use this model to demonstrate.
After selecting our model, we can initialize it by:
model = SentenceTransformer('stsb-roberta-large')
After defining our model, we can now compute the similarity score of two sentences. As discussed in the introduction, the approach is to use the model to encode the two sentences, and then calculating the cosine similarity of the resulting two embeddings. The final result will be the semantic similarity score.
In general, we can use different formulas to calculate the final similarity score (e.g. dot product, Jaccard, etc.), but in this case, we are using cosine similarity due to its properties. The more important factor is the embeddings, which is produced by the model, so it is important to use a decent encoding model.
To calculate the semantic similarity score using the approach discussed, we can run the following:
sentence1 = "I like Python because I can build AI applications"sentence2 = "I like Python because I can do data analytics"# encode sentences to get their embeddingsembedding1 = model.encode(sentence1, convert_to_tensor=True)embedding2 = model.encode(sentence2, convert_to_tensor=True)# compute similarity scores of two embeddingscosine_scores = util.pytorch_cos_sim(embedding1, embedding2)print("Sentence 1:", sentence1)print("Sentence 2:", sentence2)print("Similarity score:", cosine_scores.item())
We first define two sentences, sentence1 and sentence2, then encode them using our previously defined model. We convert our final embeddings to tensor so that they can be processed faster by the GPU. This is not required for small data like in our case but still a good practice to have.
Then, we can compute the cosine similarity scores between the two embeddings conveniently using the pytorch_cos_sim function provided by the util, thanks to Sentence-Transformers.
Finally, we can see the similarity score. We can see that the score for sentence1 and sentence2, in this case, is close to 1, which means they are pretty similar.
When you want to compare more sentences with each other, you can put the sentences into two lists and use the same code as above to calculate the similarity scores between them. The final result will be a similarity score matrix where the i, j element contains the similarity score between sentence i in list 1 and sentence j in list 2. To do this, run the following:
sentences1 = ["I like Python because I can build AI applications", "The cat sits on the ground"]   sentences2 = ["I like Python because I can do data analytics", "The cat walks on the sidewalk"]# encode list of sentences to get their embeddingsembedding1 = model.encode(sentences1, convert_to_tensor=True)embedding2 = model.encode(sentences2, convert_to_tensor=True)# compute similarity scores of two embeddingscosine_scores = util.pytorch_cos_sim(embedding1, embedding2)for i in range(len(sentences1)):    for j in range(len(sentences2)):        print("Sentence 1:", sentences1[i])        print("Sentence 2:", sentences2[j])        print("Similarity Score:", cosine_scores[i][j].item())        print()
You can see that these two sentence pairs (“I like Python because I can build AI applications”, “I like Python because I can do data analytics”) and (“The cat sits on the ground”, “The cat walks on the sidewalk”) are relatively similar. Therefore, the outputted similarity score is also relatively higher. On the other hand, Python are vastly dissimilar with a cat, and vice versa, so the other two sentence pairs have a lower similarity score.
A popular use case of semantic similarity is to find the top most relevant sentences in a corpus given a query sentence. This can also be called as semantic search. To conduct semantic search, we will need a corpus of sentences and a sentence that acts as the query. Then, we can encode both the corpus and our sentence using our model and then calculate the cosine similarity scores between our sentence and each sentence in the corpus, using the same method as previously. Finally, we can get the top_k most similar sentences by getting the top_k highest similarity scores.
corpus = ["I like Python because I can build AI applications",          "I like Python because I can do data analytics",          "The cat sits on the ground",         "The cat walks on the sidewalk"]# encode corpus to get corpus embeddingscorpus_embeddings = model.encode(corpus, convert_to_tensor=True)sentence = "I like Javascript because I can build web applications"# encode sentence to get sentence embeddingssentence_embedding = model.encode(sentence, convert_to_tensor=True)# top_k results to returntop_k=2# compute similarity scores of the sentence with the corpuscos_scores = util.pytorch_cos_sim(sentence_embedding, corpus_embeddings)[0]# Sort the results in decreasing order and get the first top_ktop_results = np.argpartition(-cos_scores, range(top_k))[0:top_k]print("Sentence:", sentence, "\n")print("Top", top_k, "most similar sentences in corpus:")for idx in top_results[0:top_k]:    print(corpus[idx], "(Score: %.4f)" % (cos_scores[idx]))
In our example, our corpus has 4 sentences, and we set top_k to 2 to retrieve the top 2 most similar sentences with our query sentence. Our query sentence is “I like Javascript because I can build web applications” and the two sentences about Python are returned because Javascript is more similar to Python than cats.
And here you have it! Now, you should be able to understand how to compute semantic similarity between sentences, list of sentences, and also retrieve the top most similar data from a corpus. For your convenience, here is the entire Jupyter code of this post:
Please let me know if you have any questions in the comments below!
If you like my work, feel free to browse my other articles :)
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
[1] Pytorch Official Website, Facebook
[2] Transformers Github, Huggingface
[3] SentenceTransformers Website, Ubiquitous Knowledge Processing Lab
[4] SentenceTransformers Github, Ubiquitous Knowledge Processing Lab
[5] Liu, Yinhan, et al. “Roberta: A robustly optimized bert pretraining approach.” arXiv preprint arXiv:1907.11692 (2019).