Full Text Search Part 1: How to create MongoDB full and partial text search
By: Coding In Depth
Tags: Fulltext Search, Mongodb, Nodejs

Preface:

I am working on the application where search is required. This search is for the attachments. Many users go to the website create folder structure with sub folders and upload the documents. These document generally Microsoft document, docx, xlsx, xls, .csv and PDF. I architect the document to save on AWS S3 bucket using Node application. At the same time when reader is writing on S3 bucket we are reading the content and saving it in MongoDB. The documents are secured in S3 bucket using envelope encryption. I have already written an article to upload encrypted file in s3 bucket in this link.

This article is about MongoDB full text search based on indexing. It is not about Elastic search, Redis based search or Amazon cloud search.

This search is light weight search for simple applications. This search is heavily rely on indexing and SQL like search. This search do not use search engine optimization or web crawling etc features. See the Full text search Part 2 for Redis search.

Features used:

As MongoDB latest version provide text search featured by searches on text content. Support case insensitive search on text content. Provide stop-word searches i.e. and, or. Also provide language based stemming rule. i.e. argue, argued, argues, arguing, and argus to the stem argu. Search similar sounding words i.e. LEONARD, LEON etc. Rank searches based on the score. Score is based on certain algorithm based on frequency of word and many other factors. For in memory autocomplete in Mongoose use this link.

Create Indexing:

Create indexing in mongoose schema. Here I am creating a simple fileContentSchema which is storing file content, file metadata and then creating indexing on metadata and content.

If you want to include all string in the searches then create wildcard search which ‘$**’

fileContentSchema .index({'$**': 'text'});

Search API:

Once schema and indexing is ready, create model and import model in API. We created fileContents model and exporting to the API. Now search model with find. $or will take array of search parameters. At the end sort the query based on the textScore.

Wildcard search API can be done like this:

fileContents .find({$text: {$search: request.searchtext }})

.skip(20)

.limit(10)

.exec(function(err, data) { //send response here });

One of the important method is explain, if chained with sort() it explain how the search is done.

.explain(true)

Cost, Drawback and Performance:

Full text searches are new in MongoDB world and slowly evolving to much better.

Full text searches are based on indexes which itself come with cost. Indexes rearrange when we new document is added.

Full text searches do not cache most recent searches, i.e. in Redis it cache the search and provide better performance and availability of data.

Wrapping up:

In this article we have seen the scenario and the need of MongoDB search. We have seen how to create mongoose schema and indexed the content. We have also created API with search using $text. We have also seen the wildcard searches and the drawbacks of index search. In part we will cover Full text search with Redis. Redis provide very fast module to do Full text search.

I have attached references, I have tried to cover those point which are not very clear in these documentation.

References: