Developing a text retrieval system
By: Leon Šamec
Tags: Text Retreival, Query, Document, Metrics, Computer Science

An example of text retrieval system is google search. It functions like this. You write an inquiry string and based on that you get documents sorted by relevance. From the most relevant to the least relevant. Of course, as there are many many documents on the internet you won’t get all of them back when you make a request, just the most relevant.

You can see that for this sorting you need some kind of metrics that connects your inquiry string and a document. So, let’s develop that metrics.

There is a certain presumtion, or maybe it is better to say, simplification, that makes this fairly simple. Both documents and the inquiry can be looked as “bag of words”. What does that mean? It means that order of words in these texts is neglected. Texts are represented only by the words they are made up of. If there are two identical words we just put a mark that this word has appeared twice. So, in pythonic terms, text is reperesented by a dictionary where key is a word and value is number of times this word has appeared in the text.

So, having this fairly simple representation, let’s develop certain principles that correspond to the real life, which will be later turned into mathematical formulas. Lets call the metrics we are developing “matching metrics”. Here are the principles:

The more words are identical in the document and the inquiry the higher is the metrics.

The higer the count of the identical word is in document the bigger the merics.

The bonus for higher count shouldn’t be linear, but punish the bonus for every new same word. So, it should be sublinear. Maybe logarithmic would be good. It has decreasing derivation, so it fits the request.

Punish the word for appearing in many different documents. So rarer words give bigger increase to the metrics. However, also don’t make the punishment linear, it shouldn’t go into infinity. Alright, maybe not very fast. Logarithmic should fit this.

Let’s now conjure up a mathematical formula for “matching metrics” from these principles.

As we have a principle that needs information on in how many documents a word has appeared, we have to make this data structure.

So, the metric goes like this. Go over all words that appear both in the document and inquiry and make a caluculation for this specific word. Give it logarithmical increase for number of appearances in the document, and logarithmical decrease for number of appearances in different documents. So, formula for a single word goes like this:

( log( number of appearances in the document ) + 1 ) / ( log( number of appearances in different documents ) + 1 )

I add a plus 1, so there won’t be a division with 0, and because it seems that when number of appearances is 1, the increase part should return 1.

Now, sum this calculation for every single word that appears both in the document and the inquiry. And voila! We have the “matching metrics”.

Calculate this for every single document you have and sort documents by decreasing order considering this calculation. The first document in the list should be the most relevant and the last the least relevant.

Check the implementation of this on my github: https://github.com/lsamec/myProjects/blob/master/textRetrieval.py

One more thing, there is something called stemming and lemmatization. These operations on the bag of words make all the words with the same root the same. For example, “climbing”, “climbs” and “climbed” would be all treated as the same word. Quite a neat features that probably improve your system because what actually matters in the retrieval is the essence of a word and these operations extract it for you.