Full-text search and indexing with Bleve
By: Nassor Paulino Da Silva
Tags: Bleve, Elasticsearch, Golang

Let’s code!

What? Don’t you have Go on your machine? Shame on you! https://golang.org/doc/install

First, setup our project workspace.

# My Go studies project/folder

mkdir -p ~/Workspace/Go

cd ~/Workspace/Go

# Exporting the env var GOPATH to the actual directory

export GOPATH=`pwd`

# Creating and accessing the project folder

mkdir -p ~/Workspace/Go/src/github.com/nassor/studies-blevesearch

cd ~/Workspace/Go/src/github.com/nassor/studies-blevesearch

Now let’s download the package to our $GOPATH environment. The purpose isn’t database implementation, so to not loose too much time working with the SQL database, we will use the ORM package called gorm.

go get github.com/blevesearch/bleve/... # bleve package

go get github.com/mattn/go-sqlite3 # sqlite3 package

go get github.com/jinzhu/gorm # orm package

Everything is ready, so let’s learn how to create an index connection. We’ll create a BleveConn method to create or connect to the indexing persistence. The Bleve Search uses the BoltDB as the default persistence, but you can choose others.

package conn



import "github.com/blevesearch/bleve"



var bleveIdx bleve.Index



// Bleve connect or create the index persistence

func Bleve(indexPath string) (bleve.Index, error) {



// with bleveIdx isn't set...

if bleveIdx == nil {

var err error

// try to open de persistence file...

bleveIdx, err = bleve.Open(indexPath)

// if doesn't exists or something goes wrong...

if err != nil {

// create a new mapping file and create a new index

mapping := bleve.NewIndexMapping()

bleveIdx, err = bleve.New(indexPath, mapping)

if err != nil {

return nil, err

}

}

}



// return de index

return bleveIdx, nil

}

Ref: github.com/nassor/studies-blevesearch/conn/bleve.go Ref: github.com/nassor/studies-blevesearch/conn/bleve.go

Let’s start with our model. It is a classic Event data:

// Event is an event! wow! ;D

type Event struct {

ID int

Name string

Description string

Local string

Website string

Start time.Time

End time.Time

}



// Index is used to add the event in the bleve index.

func (e *Event) Index(index bleve.Index) error {

err := index.Index(string(e.ID), e)

return err

}

Ref: github.com/nassor/studies-blevesearch/models/event.go

I add a method called Index, it receives as a parameter a bleve.Index struct and return an error. The method is in charge to add the event in the index.

The bleve.Index.Index() method accept only string to identify. Because we are using the Default IndexMapping, all the fields in the type Event will be indexed.

Below, the code I use to test the functionality:

func TestIndexing(t *testing.T) {

_, eventList := dbCreate()

idx := idxCreate()



err := eventList[0].Index(idx)

if err != nil {

t.Error("Wasn't possible create the index", err, ballotX)

} else {

t.Log("Should create an event index", checkMark)

}



idxDestroy()

dbDestroy()

}

Ref: github.com/nassor/studies-blevesearch/models/event_test.go

Simple, isn’t? To retrieve the data we need a little bit more steps, but not too much. :)

func TestFindByAnything(t *testing.T) {

db, eventList := dbCreate()

idx := idxCreate()

indexEvents(idx, eventList)



// We are looking to an Event with some string which match with dotGo

query := bleve.NewMatchQuery("dotGo")

searchRequest := bleve.NewSearchRequest(query)

searchResult, err := idx.Search(searchRequest)

if err != nil {

t.Error("Something wrong happen with the search", err, ballotX)

} else {

t.Log("Should search the query", checkMark)

}



if searchResult.Total != 1 {

t.Error("Only 1 result are expected, got ", searchResult.Total, ballotX)

} else {

t.Log("Should return only one result", checkMark)

}



event := &Event{}

db.First(&event, &searchResult.Hits[0].ID)

if event.Name != "dotGo 2015" {

t.Error("Expected \"dotGo 2015\", Receive: ", event.Name)

} else {

t.Log("Should return an event with the name equal a", event.Name, checkMark)

}



idxDestroy()

dbDestroy()

}

Ref: github.com/nassor/studies-blevesearch/models/event_test.go

You have access to the code here. Pull the code, change it and have fun. \o/

And that’s it! I’m here studying how to make more complex searches. When I have something new, I’ll write the second part.

P.S.: Writing this article I had the idea to use the BlotDB as the main persistence, a simple key/value instead a SQL/NoSQL/NewSQL database. The point came with this thought: “why I need a second query structure if I’ll use Bleve to do queries?”. I’ll write an article about my studies about that too.