Introduction to full text search engines

Today, we are going to continue working on our database, but in a
different area. All applications that store content must provide a
search capability.

For many types of web sites it is possible to just let Google, Bing,
etc. index all the content and provide the search results. This works
well for sites that have mostly static pages, like a forum. In our
little microblog application the basic unit of content is just a short
user post, not a whole page. The type of search results that we want are
dynamic. For example, if we search for the word "dog" we want to see
blog posts from any users that include that word. It is obvious that
until someone searches for that word there is no page that the big
search engines could have indexed with these results, so clearly we have
no choice other than rolling our own search.

Introduction to full text search engines

Unfortunately support for full text search in relational databases is
not well standardized. Each database implements full text search in its
own way, and SQLAlchemy at this time does not have a full text search
abstration.

We are currently using SQLite for our database, so we could just create
a full text index using the facilities provided by SQLite, bypassing
SQLAlchemy. But that isn't a good idea, because if one day we decide to
switch to another database we would need to rewrite our full text search
capability for another database.

So instead, we are going to let our database deal with the regular data,
and we are going to create a specialized database that will be dedicated
to text searches.

There are a few open source full text search engines. The only one that
to my knowledge has a Flask extension is Whoosh, an engine also written
in Python. The advantage of using a pure Python engine is that it will
install and run anywhere a Python interpreter is available. The
disadvantage is that search performance will not be up to par with other
engines that are written in C or C++. In my opinion the ideal solution
would be to have a Flask extension that can connect to several engines
and abstract us from dealing with a particular one in the same way Flask-
SQLAlchemy gives us the freedom to use several database engines, but
nothing of that kind seems to be available for full text searching at
this time. Django developers do have a very nice extension that supports
several full text search engines called django-haystack. Maybe one day
someone will create a similar extension for Flask.

But for now, we'll implement our text searching with Whoosh. The
extension that we are going to use is Flask-WhooshAlchemy, which
integrates a Whoosh database with Flask-SQLAlchemy models.

If you don't have Flask-WhooshAlchemy installed in your virtual
environment yet go ahead and install it now. Windows users should run
this:

flask\Scripts\pip install Flask-WhooshAlchemy
All others must run this:

flask/bin/pip install Flask-WhooshAlchemy
Configuration

Configuration for Flask-WhooshAlchemy is pretty simple. We just need to
tell the extension what is the name of the full text search database
(file config.py):

WHOOSH_BASE = os.path.join(basedir, 'search.db')
Model changes

Model changes

Since Flask-WhooshAlchemy integrates with Flask-SQLAlchemy, we indicate
what data is to be indexed for searching in the proper model class (file
app/models.py):

from app import app
import flask.ext.whooshalchemy as whooshalchemy

class Post(db.Model):
    __searchable__ = ['body']

    id = db.Column(db.Integer, primary_key = True)
    body = db.Column(db.String(140))
    timestamp = db.Column(db.DateTime)
    user_id = db.Column(db.Integer, db.ForeignKey('user.id'))

    def __repr__(self):
        return '<Post %r>' % (self.text)

whooshalchemy.whoosh_index(app, Post)
The model has a new __searchable__ field, which is an array with all the
database fields that will be in the searchable index. In our case we
only want to index the body field of our posts.

We also have to initialize the full text index for this model by calling
the whoosh_index function.

Since this isn't a change that affects the format of our relational
database we do not need to record a new migration.

Unfortunately any posts that were in the database before the full text
engine was added will not be indexed. To make sure the database and the
full text engine are synchronized we are going to delete all posts from
the database and start over. First we start the Python interpreter. For
Windows users:

flask\Scripts\python
And for everyone else:

flask/bin/python
Then in the Python prompt we delete all the posts:

>>> from app.models import Post
>>> from app import db
>>> for post in Post.query.all():
...    db.session.delete(post)
>>> db.session.commit()
Searching

Searching

And now we are ready to start searching. First let's add a few new posts
to the database. We have two options to do this. We can just start the
application and enter posts via the web browser, as regular users would
do, or we can also do it in the Python prompt.

From the Python prompt we can do it as follows:

>>> from app.models import User, Post
>>> from app import db
>>> import datetime
>>> u = User.query.get(1)
>>> p = Post(body='my first post', timestamp=datetime.datetime.utcnow(),
author=u)
>>> db.session.add(p)
>>> p = Post(body='my second post',
timestamp=datetime.datetime.utcnow(), author=u)
>>> db.session.add(p)
>>> p = Post(body='my third and last post',
timestamp=datetime.datetime.utcnow(), author=u)
>>> db.session.add(p)
>>> db.session.commit()
The Flask-WhooshAlchemy extension is nice, because it hooks up into Flask-
SQLAlchemy commits automatically. We do not need to maintain the full
text index, it is all done for us transparently.

Now that we have a few posts in our full text index we can issue
searches:

>>> Post.query.whoosh_search('post').all()
[<Post u'my second post'>, <Post u'my first post'>, <Post u'my third and
last post'>]
>>> Post.query.whoosh_search('second').all()
[<Post u'my second post'>]
>>> Post.query.whoosh_search('second OR last').all()
[<Post u'my second post'>, <Post u'my third and last post'>]
As you can see in the examples above, the queries do not need to be
limited to single words. In fact, Whoosh supports a pretty powerful
search query language.

Integrating full text searches into the application

To make the searching capability available to our application's users we
have to add just a few small changes.

Configuration

As far as configuration, we'll just indicate how many search results
should be returned as a maximum (file config.py):

MAX_SEARCH_RESULTS = 50
Search form

Search form

We are going to add a search form to the navigation bar at the top of
the page. Putting the search box at the top is nice, because then the
search will be accessible from all pages.

First we add a search form class (file app/forms.py):

class SearchForm(Form):
    search = TextField('search', validators = [Required()])
Then we need to create a search form object and make it available to all
templates, since we will be putting the search form in the navigation
bar that is common to all pages. The easiest way to achieve this is to
create the form in the before_request handler, and then stick it in
Flask's global g (file app/views.py):

@app.before_request
def before_request():
    g.user = current_user
    if g.user.is_authenticated():
        g.user.last_seen = datetime.utcnow()
        db.session.add(g.user)
        db.session.commit()
        g.search_form = SearchForm()
Then we add the form to our template (file app/templates/base.html):

<div>Microblog:
    <a href="{{ url_for('index') }}">Home</a>
    {% if g.user.is_authenticated() %}
    | <a href="{{ url_for('user', nickname = g.user.nickname) }}">Your
Profile</a>
    | <form style="display: inline;" action="{{url_for('search')}}"
method="post"
name="search">{{g.search_form.hidden_tag()}}{{g.search_form.search(size=20)}}<input
type="submit" value="Search"></form>
    | <a href="{{ url_for('logout') }}">Logout</a>
    {% endif %}
</div>
Note that we only display the form when we have a logged in user.
Likewise, the before_request handler will only create a form when a user
is logged in, since our application does not show any content to guests
that are not authenticated.

Search view function

The action field of our form was set above to send all search requests
the the search view function. This is where we will be issuing our full
text queries (file app/views.py):

@app.route('/search', methods = ['POST'])
@login_required
def search():
    if not g.search_form.validate_on_submit():
        return redirect(url_for('index'))
    return redirect(url_for('search_results', query =
g.search_form.search.data))
This function doesn't really do much, it just collects the search query
from the form and then redirects to another page passing this query as
an argument. The reason the search work isn't done directly here is that
if a user then hits the refresh button the browser will put up a warning
indicating that form data will be resubmitted. This is avoided when the
response to a POST request is a redirect, because after the redirect the
browser's refresh button will reload the redirected page.

Search results page

Once a query string has been received the form POST handler sends it via
page redirection to the search_results handler (file app/views.py):

@app.route('/search_results/<query>')
@login_required
def search_results(query):
    results = Post.query.whoosh_search(query, MAX_SEARCH_RESULTS).all()
    return render_template('search_results.html',
        query = query,
        results = results)
The search results view function sends the query into Whoosh, passing a
maximum number of search results, since we don't want to be presenting a
potentially large number of hits, we are happy showing just the first
fifty.

The final piece is the search results template (file
app/templates/search_results.html):

<!-- extend base layout -->
{% extends "base.html" %}

{% block content %}
<h1>Search results for "{{query}}":</h1>
{% for post in results %}
    {% include 'post.html' %}
{% endfor %}
{% endblock %}
And here, once again, we can reuse our post.html sub-template, so we
don't need to worry about rendering avatars or other formatting
elements, since all of that is done in a generic way in the sub-
template.

Final words

We now have completed yet another important, though often overlooked
piece that any decent web application must have.