Elastic Search Simplified: Part 1
By: Nitin Agarwal
Tags: Elasticsearch

Recently we used elastic search to implement wildcard search functionality. After spending a week with this, we realized while it's easy to get started, but having a good understanding of the basics of elastic search, can help implement the solution in a much more optimized manner.

Elastic Search has multiple use cases like log analysis, visualizing and aggregating data, but this series is relevant for the “Search” use case.

We will discuss below items

What is Text Search Inverted Index TF/IDF Text Analysis Mapping Data Types Query Types Substring/Wildcard search

What is Text Search

Imagine you want to search for mobile on amazon with specific requirements like 128gb storage, 6gb ram, and around 15k INR price.

Consider we had these details stored in SQL table like below

Product details SQL Table

And we had to achieve this in a SQL table we might have to create a query something like below

select * from Products where details like “%mobile” and details like “%4gb%” and details like “%128gb%” and details like “%6gb” and details like “%prices 10000%

As you can see this is an extremely inefficient query doing a full table scan.

This query may still not work, as the query had a typo “prices” instead of “price”.

In the above case, we can also see that a “limited time deal” has been given higher preference. Adding this functionality in a conventional database will be difficult.

Considering all these things, it becomes obvious we require a custom solution for text search.

Inverted Index

An inverted index is a simple mapping of individual terms and their overall frequency in all the indexed documents. The term “winter” appears once in document 1, term “the” appears twice, so they have been mapped to the corresponding document ids.

TF/IDF

TF — Term Frequency

term frequency is simply the number of times a term appeared in the individual given document. This is further normalized for longer and shorter documents

IDF — Inverse Document Frequency

It simply indicates how rare a given word is all the indexed documents

When we run a query we require to get the most relevant results returned first. For calculating the score we use the concept of TF/IDF. Let's understand it with the help of an example using layman maths.

E lastic Search does provide a lot of flexibility to help calculate the score like using boosts etc, but TF/IDF will help get the idea across.

"The sky is blue.

"The sun is bright today."

"The sun in the sky is bright."

"We can see the shining sun, the bright sun." Inverted Index

+--------+----------------------+-------------------+

| Term | total term frequency | document_id |

+--------+----------------------+-------------------+

| sky | 2 | (1,1) (3,1) |

| blue | 1 | (1,1) |

| sun | 4 | (2,1) (3,1) (4,2) |

| bright | 3 | (2,1) (3,1) (4,1) |

| shine | 1 | (4,1) |

| today | 1 | (2,1) |

+--------+----------------------+-------------------+ The documents have been normalised by removing common terms like is, a, the etc and also words like shining have been stemmed to shine. More on this in the article below Term Frequency Table +-------------+------+--------+-------+-----+------+-------+

| document_id | blue | bright | shine | sky | sun | today |

+-------------+------+--------+-------+-----+------+-------+

| 1 | 1 | 0 | 0 | 1 | 0 | 0 |

| 2 | 0 | 1 | 0 | 0 | 1 | 1 |

| 3 | 0 | 1 | 0 | 1 | 1 | 0 |

| 4 | 0 | 1 | 1 | 0 | 2 | 0 |

| n_t | 1 | 3 | 1 | 2 | 3 | 1 |

+-------------+------+--------+-------+-----+------+-------+

*n_t -> total occurances in all documents Inverse Document Frequency Table +------+--------+-------+-----+------+-------+

| blue | bright | shine | sky | sun | today |

+------+--------+-------+-----+------+-------+

| 4/1 | 4/3 | 4/1 | 4/2 | 4/3 | 4/1 |

+------+--------+-------+-----+------+-------+

TF * IDF score for query "sky sun" +-------------+-------------+---------------+-------------+------+

| document_id | TF * IDF | matching word | final score | rank |

+-------------+-------------+---------------+-------------+------+

| 1 | 1 * 2 | sky | 2 | 3 |

| 2 | 1* 4/3 | sun | 1.33 | 4 |

| 3 | 1*2 + 1*4/3 | sky,sun | 3.33 | 1 |

| 4 | 2*4/3 | sun | 2.66 | 2 |

+-------------+-------------+---------------+-------------+------+ As we can see term "sky" is less common compared to term "sun", so sky has been given higher weightage when calculating score

This has been simplified for ease of understanding, the real formula is a different, but the idea is the same, you can check link to understand in depth

Text Analysis

Text analysis enables Elasticsearch to perform a full-text search, where the search returns all relevant results rather than just exact matches

ElasticSearch provides analyzers for some of the common use cases and also gives the ability to define custom analyzers. We will walk through a standard analyzer below and discuss how to create a custom analyzer later in this article.

Character filters

assume the case where we have some HTML elements in the documents, Character filters can help remove those filters. These are optional.

Tokenizers

tokenizers are used to break a document into individual tokens(terms). For example, we will use “whitespace” for a document in the English language

Token filters

token filters can be used to convert tokens to lowercase. A stop token filter can be used to remove common stop words like “is”, “the” “a” etc in the English language. Another common requirement can be to consider “shining” and “shine” as the same term, which is can be done y using the stemmer filter.

Many more filters and tokenizers exists, which you can find in the elastic search documentation.

query localhost:9200/_analyze {

"char_filter": [

"html_strip"

],

"tokenizer": "standard",

"filter": [

"stop",

"lowercase",

"stemmer"

],

"text": "<b>the sky is Blue bright and shining<b>"

} { "details" : "<b>the sky is Blue bright and shining<b>" } Output

{

"tokens": [{

"token": "sky",

"start_offset": 7,

"end_offset": 10,

"type": "<ALPHANUM>",

"position": 1

},

{

"token": "blue",

"start_offset": 14,

"end_offset": 18,

"type": "<ALPHANUM>",

"position": 3

},

{

"token": "bright",

"start_offset": 19,

"end_offset": 25,

"type": "<ALPHANUM>",

"position": 4

},

{

"token": "shine",

"start_offset": 30,

"end_offset": 40,

"type": "<ALPHANUM>",

"position": 6

}

]

}

ElasticSearch provides common stop words out of the box, at the same time giving the flexibilty to define your own list.

Text analysis occurs at two times

Index Analyzers

Search Analysers

We will walk through an example of this later in the article.

Mapping

Mapping is the process of defining how a document, and the fields it contains, are stored and indexed. Lucene only understands Inverted index, elastic search converts, and creates the appropriate inverted indices based on the mappings provided.

ElasticSearch Indices are actually Lucene Indices, ElasticSearch is a distributed system on top of Lucene.

PUT /my-index-000001

{

"mappings": {

"properties": {

"age": { "type": "integer" },

"email": { "type": "keyword" },

"name": { "type": "text" }

}

}

}

Fields

A mapping contains a list of fields or properties pertinent to the document, with each field having its own data type.

Dynamic Mapping

It is not necessary to define mapping when creating an index, elastic search itself will create mapping dynamically

PUT data/_doc/1

{ "count": 5 } count field with data type long gets created

Data Types

A complete list of data types supported can be found at this link. We will discuss two important data types

Keywords

Keyword field type usually contains structured content such as IDs, email addresses, zip codes, tags, etc, and can be used for filtering, sorting.

Text

The text field type is used for text analysis and we can define the index and search analyzers for each individual text field.

Query types

ElasticSearch query can consist of two types of clauses. For detail use this link

Leaf query clauses

lead queries look for a particular value in a particular field, such as match, term or range

GET /_search

{

"query": {

"term": {

"name": {

"value": "foobar",

"boost": 1.0

}

}

}

}

Compound query clauses

Compound query clauses wrap other leaf or compound queries and are used to combine multiple queries in a logical fashion

Example of boolean query POST _search

{

"query": {

"bool" : {

"must" : {

"term" : { "user.id" : "kimchy" }

},

"filter": {

"term" : { "tags" : "production" }

},

"must_not" : {

"range" : {

"age" : { "gte" : 10, "lte" : 20 }

}

},

"should" : [

{ "term" : { "tags" : "env1" } },

{ "term" : { "tags" : "deployed" } }

]

}

}

}

SubString Search/N-gram Filters

We looked at using elastic search for our use case of being able to do substring/wildcard search.

Consider an index of documents with names

{

"name": "nitin agarwal"

}, {

"name": "nitish"

}

In case we want to search “nit*”, in a conventional database, the normal index on the field would return the results since we are doing a “starts with query” (name ≥ nit and name < niu).

Edge N-gram

But as we use inverted indexes in elastic search, and we would need to use something called “edge N-gram”.

Edge N-gram for word "nitin" doc_id -> 1 unigram - ['n'] -> 1

bigram - ['ni'] -> 1

trigram - ['nit']

quadgram - ['niti']

hexagram - ['nitin'] An inverted index is created with the above configuration for a quick lookup.

Wildcard Queries

Let's take the example of a “*iti*” query. Elastic Search does provide the functionality of wildcard queries

GET /_search

{

"query": {

"wildcard": {

"name": {

"value": "*iti*",

}

}

}

}

But this query is actually doing a full index scan and will be extremely slow.

N-gram

We can define the max and min length of chunks to be created from a given term and token will be created like below

min_len = 2, max_len = 5, term = ["nitin"]

["ni", "it", "ti", "in"]

["nit", "iti", "tin"]

["niti", "itin"]

["nitin"]

With Version 7.9 wildcard field type has been added, this is much more storage efficient compared to creating N-grams. Wildcard filed type did not completely solve our use case, so we went ahead with our customized N-gram solution

Index and Search Analysers

We had the below requirements for our search implementation

wildcard search

name can contain Unicode characters, for example, names like “ϟTHOST⁷⁷” should be searchable if the user searches for “ϟth”

whitespace splitting NOT required, for example, names like “op jap”, should be searchable if the user searches for “op ja”

the max search chars length will be 10 characters, otherwise, the user will search for the exact match.

We had to create a custom Index analyzer for our specific requirements.

Custom Index Analysers

{ "analysis": {

"analyzer": {

"ngram_custom_analyser": {

"tokenizer": "ngram_tokenizer",

"filter": [

"lowercase" ----------------> added lowercase filter

]

}

},

"tokenizer": {

"ngram_tokenizer": {

"type": "ngram", -----------> defines type as ngram

"min_gram": 2, -----------> defines the min len

"max_gram": 10, -----------> defines the max len

"token_chars": [----> defines signs which should be included

"letter",

"punctuation",

"whitespace", ------------> allow whitespace

"digit",

"symbol" --------------> allow symbols

]

}

}

}

} {

"mappings": {

"properties": {

"name": {

"type": "text",

"analyzer": "ngram_custom_analyser",

"search_analyzer": "custom_search_analyzer"

}

}

}

}

Custom Search Analysers

We will also need to define a custom search analyzer, on the field, as elastic search by default would split the search query into N-gram tokens defined on the field, for example, “nitin ag” would be split into its N-grams like “ni”, “it” etc and these would be searched which would give wrong results.

{

"analyzer": {

"custom_search_analyzer": {

"tokenizer": "ngram_search_tokenizer",

"filter": [

"lowercase"

]

}

},

"tokenizer": {

"ngram_search_tokenizer": {

"type": "char_group",

"token_chars": [ -----> defines signs which should be inclued

"letter",

"punctuation",

"whitespace", ------> allow whitespace in search query

"digit",

"symbol" --------> allow symbols in search query

]

}

}

}

Advantages of this Approach

The search would be extremely fast, as we will doing a direct match.

Disadvantages of this Approach

this will take exponentially more storage, compared to a normal standard analyzer, but in our use case, we can delete documents that are more than 1 month old.

With Elastic Search, you will need to play around to find the solution that best fits your requirement.

Other interesting topics related to search which were not covered include

Part2 of this article we talk about ElasticSearch Cluster and about the Journey of an index and search request.