Alternatives to full text queries (part I)
By: Blog Moove-It
Tags: Xapian, Full Text, Solr, Lucene, Sphinx

When it comes to data storage and data handling, we developers, are quite used to working with database engines (MySQL, Oracle, SQLite, etc) and database query languages.

Depending on the application being developed, one of these solutions can be more than enough to meet our needs and that is what we usually end up using.

But there are times, when the amount of information to be handled is so big (we’re talking about millions of rows of information) and the needed response times are so low (we’re talking about a few milliseconds time) that we need solutions designed specifically for searching large amount of information instead of generic data handling ones.

What’s so good about them?

While most database engines work great with very dynamic loads of information (i.e.: where new information needs to be shown in real-time), and managing highly complex relational data, sometimes what we need is actually a solution that can search through a great amount of data in a very short time, giving the user a quick response.

The solutions we’ll be talking about here, do just that, they give the developers a way of accessing information at a striking speed, but at a price: the information used is not always the most updated one1. Another lose to consider here, is the fact that these searching solutions do not manage relational information like relational databases do. So you need to be careful when choosing one over the other, and consider exactly what your needs are.

How do they work?

The way they work, is, roughly, by indexing all the information, using some of several indexing mechanics (like the use of stop words2, stemming3, etc). Once the information has been indexed, it is there (on the index) where the search takes place, instead of searching directly on the unprocessed data.

Some of the most interesting features presented by these APIs are:

Ranked search: The results can be returned ordered by the relevance of the match.

Proximity search: Because sometimes, the searched terms are not written correctly, these APIs can return results that are close to the original ones.

The use of wildcards: When searching, not always do we know the exact words, that’s when wildcards come into place, allowing us to specify a basic structure for our search and letting the search engine find the matching results.

Comparing some of them

Considering this scenario, today we’ll be discussing some alternative options for data handling and querying. I’ve chosen these projects based on general opinion on internet, according to most places, these guys are the most known/used alternatives out there (there are many others of course).

Our options are:

Lucene: An Apache developed project, written entirely in Java.

An Apache developed project, written entirely in Java. Xapian: An Open Source Search Engine Library, released under theGPL and written in C++.

An Open Source Search Engine Library, released under theGPL and written in C++. Sphinx: An open source full text search server with a high level of integration with database engines.

An open source full text search server with a high level of integration with database engines. Solr: Solr is a standalone enterprise search server with a REST-like API that uses Lucene as its base full-text search engine.

What am I looking for?

Before even comparing products, it is important to understand that there is rarely a single solution to our problems. That almost every time, there is more than one product that will meet our needs.

Knowing this, I’ll be comparing products based on an “out-of-the-box” basis. This means, that no addons or plugins or anything will be taken into account for comparison purposes (we have to draw the line somewhere).

My chosen criteria

I won’t be benchmarking product’s speed against each others, if that’s what you’re looking for then stop reading here. Even though it might be a good measure of how good a product is (in some cases) I feel that in this particular case, where we don’t really have a specific problem to solve, those numbers would not bring anything of value to the table.

What I will be taking into account is the amount of features (search related features actually) that come with a product out of the box. I fell that this approach could give a basic understanding of which product stands over others for any generic given problem. In the end, based solely on this information, my resolution can be wrong (or misleading) for some specific cases, but then again, that’s not what I’m trying to solve, since in order to tackle specific scenarios this article would have to be extended into a small book.

Another point of notice, is the fact that all of the presented solutions (as we’ll see in a minute), are long term projects, with active support and under constant development (and improvement).

The projects

Now that we know what we’ll be comparing and what to look for on our projects, lets take a look at them:

Lucene