Inverted Index for Document Similarity Computation | Towards Data Science

My current Google search showed over 3 million questions on Quora, but still, they can instantly show you similar questions even while you are typing a new question. And I instantly accessed this information by just typing to the search bar of Google, and the algorithm instantly showed me the most related content based on my search query.
Although there are many state-of-the-art concepts responsible for this magic, I will show you an old and easy method for handling pairwise similarity computation for documents more efficiently in some scenarios. I will compare it with the brute-force approach, and I will present some Python code examples.
You can find the full code here: GitHub link.
If we can explain any information with vectors, we can find hidden connections in this domain by making some similarity comparisons.
Similarity comparison methods are heavily used for clustering vectors. It is used for topic detection, document clustering, language translation, or checking plagiarism. It is also used for recommending youtube videos or friend suggestions on social media websites.
There are many more application fields. In short, if we can explain any information with vectors, we can exploit it for information retrieval.
Imagine that you have tons of documents with many words in them. In this structure, the words are collected based on the document they belong to. In other words, documents are parents, and the words are children. The inverted index method reverses this relation by making words the parents and the documents the children. I will demonstrate this with an example.
Imagine that you have documents like this:
doc1: "Today is a beautiful, and a sunny day to start my workout."doc2: "I will not be able to come today to meet with him."doc3: "Our class meeting starts soon!"doc4: "My class starts at 6."
After tokenization, removing stopwords, and some cleaning, the documents will look like this:
doc1: ["today", "beautiful", "sunny" "day", "start", "workout"]doc2: ["come", "today", "meet"]doc3: ["class", "meet", "start", "soon"]doc4: ["class", "start"]
Now we can invert the documents and the words.
today     : [doc1, doc2]beautiful : [doc1]sunny     : [doc1]day       : [doc1]start     : [doc1, doc3]workout   : [doc1]come      : [doc2]meet      : [doc2, doc3]class     : [doc3, doc4]soon      : [doc3]
Now we can see which words appear in which documents. Alternatively, we can even demonstrate the words' positions in the document, but we will use this demonstration for simplicity.
How do we benefit from an inverted index structure?
Imagine that you are trying to cluster these documents by their feature embeddings (a numerical representation of the document that can be created with word2vec, doc2vec, BERT, or TF-IDF, etc.), and you will do this by the cosine similarities of these vectors. In a brute-force approach, you have to do 6 different similarity checks since there are four documents.
Inverted index structure helps us detect the documents that have a common word. We will compare a document with others only if they have at least one common word.
We will iterate over the documents in numerical order for more efficiency. For document 1, it is present in the sets of the words “today” and “start” with another document. A combination of these sets gives us [doc1, doc2, doc3]. Thus, we will calculate the similarity score between (doc1, doc2), and (doc1, doc3). We can move on to the second document.
Since we already covered the first document, we will not consider it anymore. Since document 2 is present in the “meet” set with document 4, we will only find the similarity (doc2, doc4).
For document 3, since it is present in the “class” set with document 4, we will only find the similarity (doc3, doc4).
Since we are only left with one document, we do not need to do any computation. We have already covered all possible pairs in previous steps.
As a result, we have calculated.
(doc1, doc2)    (doc1, doc3)    (doc2, doc4)    (doc3, doc4)
The number of comparisons is two less than the brute-force approach! This might seem a little effect on the performance. So, let’s try this approach with a real dataset.
We will load a text dataset from here1, which contains Turkish text data from different topics.
Data looks like this:
  category  text0 siyaset 3 milyon ile ön seçim vaadi mhp nin 10 olağan...1 siyaset mesut_yılmaz yüce_divan da ceza alabilirdi pr...2 siyaset disko lar kaldırılıyor başbakan_yardımcısı ar...3 siyaset sarıgül anayasa_mahkemesi ne gidiyor mustafa_...4 siyaset erdoğan idamın bir haklılık sebebi var demek ...
We will clear the data before using it.
Now it is time to generate some feature vectors. Currently, the best method I know to generate the feature vectors is by using Sentence-Bert. You can find more about it by clicking here. But, it is out of the scope of this blog post. Thus, we will use simple TF-IDF for demonstration2.
Now let’s try the brute-force and inverted index approaches.
Brute-Force Approach:
Inverted Index Approach:
Results are as follows:
Brute-Force Approach:
Inverted Index Approach:
There is no significant difference between the two approaches, not as promised. Now it is time to talk about the bottlenecks of the inverted index approach3.
The inverted index approach is useful only if the documents are distinct enough and the number of words in each document is small. Otherwise, it might be slower than the brute-force approach because of the extra computations for generating an inverted index structure.
The most common solution to this problem is to reduce the data in each document. There are many different feature selection methods, and these methods are out of the scope of this blog post. To solve this issue, we will extract 50 keywords from each document that describe it and use them to represent the document.
Extract Keywords
The results are as follows:
Where “B.F.” stands for “Brute-Force” and “I.I.” stands for “Inverted Index”. Raw data shows the results before the keyword extraction, and simplified data shows the results after the keyword extraction.
After keyword extraction, the brute-force approach's number of comparisons did not change, and the run time slightly differed. However, the inverted index has improved its performance significantly! It is almost 50 seconds faster than the brute-force approach with 1000 lines of data. If you are working with thousands of lines of data, there will be a dramatic difference in runtime.
You can find the full code here: GitHub link.
This blog post demonstrated a common and easy way to speed up the pairwise similarity comparison operations and showed it with python examples. The results show that the inverted index approach only useful if the documents are short or very distinct. Still, using data reduction techniques with long data makes a significant difference.
Today, there may be better and more suitable approaches depending on the data and the challenge. However, the inverted index is an easy method to implement and understand.
That’s it. I hope you enjoy it and thank you very much for reading my blog post.
If you want, you can contact me from these links:
Twitter: https://twitter.com/etolga_ayan
Linkedin: https://www.linkedin.com/in/emre-tolga-ayan-7b9a0a149/
GitHub: https://github.com/tolgayan/
Cheers,
Emre Tolga Ayan
[1] S. Yıldırım, A Benchmark Data for Turkish Text Categorization, https://www.kaggle.com/savasy/ttc4900
[2] K. Ganesan, Tutorial: Extracting Keywords with TF-IDF and Python’s Scikit-Learn, https://kavita-ganesan.com/extracting-keywords-from-text-tfidf/#.YDNHzegzZnJ
[3] Efficient document similarity with inverted index, sketching and sampling, https://cosinesimilarity.org/content/inverted-index-sketching-sampling.html