ElasticSearch
By: Jay Prakash
Tags: Elasticsearch Cluster, Elasticsearch, Elasticsearch Queries

ElasticSearch

Elasticsearch is a distributed, free and open search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured.

Why ElasticSearch?

Elasticsearch allows you to store, search, and analyze huge volumes of data quickly and in near real-time and give back answers in milliseconds. It’s able to achieve fast search responses because instead of searching the text directly, it searches an index.

Indexing

Elasticsearch is able to achieve fast search responses because, instead of searching the text directly, it searches an index instead.

This is like retrieving pages in a book related to a keyword by scanning the index at the back of a book, as opposed to searching every word of every page of the book.

This type of index is called an inverted index, because it inverts a page-centric data structure (page->words) to a keyword-centric data structure (word->pages).

Elasticsearch uses Apache Lucene to create and manage this inverted index.

For indexing documents, ElasticSearch uses an analyzer.

Indexing Process

Analyzer is a combination of tokenizer and filters that can be applied to any field for analyzing in ElasticSearch. There are already built-in analyzers available in Elasticsearch. You can create your custom analyzers as well. You can read built-in analyzers here.

Document 1

{

"name" : "my name in luv"

}

Document 2

{

"name": "my father name is ram"

}

After the standard analyzer, the terms in the documents would be as follows

Document 1

Field Terms

"name": "my","name","is","luv"

document 2

Field Terms

"name" : "my","father","name","is","ram"

Now let us combine the above two table of terms for the field “name” and make it a single one like below:

Visual Representation of an Inverted Index

Now the above table is called the inverted index for the field “name”. The advantage of using an inverted index is that the search term can be looked up in the “terms” column and then, if there is a match, finding the documents where the search term is present is a very simple job of a look-up at the corresponding column.

You can check which analyzer is applied at any field in your index like this

GET <index>/_analyze

{

"text": "software engineer blr",

"field": "<ur_field>"

}

ElasticSearch Basic Storage Concepts

Documents are JSON objects that are stored within an Elasticsearch index and are considered the base unit of storage.

An example of a document:

{

"_id": 3,

“_index”: [“your index name”],

"_source":{

"age": 28,

"name": ["daniel”],

"year":1989,

}

Mapping It defines the fields for documents of a specific type — the data type (such as keyword, text, and long) and how the fields should be indexed and stored in Elasticsearch.

{

"mappings": {

"properties": {

"name": {

"type": "keyword"

},

"age": {

"type": "long"

},

"year": {

"type": "long"

}

}

}

}

Index Indices, the largest unit of data in Elasticsearch, are logical partitions of documents and can be compared to a database in the world of relational databases.

Shards Put simply, shards are a single Lucene index. They are the building blocks of Elasticsearch and what facilitates its scalability.

curl -XPUT localhost:9200/example -d '{

"settings" : {

"index" : {

"number_of_shards" : 3,

"number_of_replicas" : 1

}

}

}'

Replicas are the copy of the primary shard, to prevent data loss in case of hardware failure. By default, each index in Elastisearch is allocated 5 primary Shards and 1 replica which means that if you have at least two nodes in your cluster, your index will have 5 primary shards and another 5 replica shards (1 complete replica) for a total of 10 shards per index.

Node Any time that you start an instance of Elasticsearch, you are starting a node. For further reading refer to this.

Cluster A collection of connected nodes is called a cluster.

Typical Elastic CLuser

Query Example For each example we will create a new index that will have data customized to its need. I am using Kibana because they are easy to configure and comes with a pretty intuitive UI.

We will be using the same index for all three queries “Term”, “Text”, “Nested”.

PUT /test-index PUT /test-index/_mapping

{

"properties": {

"firstname": {

"type": "keyword"

},

"lastname": {

"type": "keyword"

},

"fullname": {

"type": "text"

},

"age": {

"type": "long"

},

"friends": {

"type": "nested"

}

}

} POST /test-index/_doc

{

"firstname": "Jay",

"lastname": "Prakash",

"fullname": "Jay Prakash",

"age": 30,

"friends":[{"friendname": "Adarsh"}, {"friendname": "Jaismin"}

]

} POST /test-index/_doc

{

"firstname": "Viral",

"lastname": "Prasad",

"fullname": "Viral Prasad",

"age": 30,

"friends":[{"friendname": "Lovenish"}, {"friendname": "Sandeep"}, {"friendname": "Shashank"}

]

} POST /test-index/_doc

{

"firstname": "Sandeep",

"lastname": "Dash",

"fullname": "Viral Prasad",

"age": 30,

"friends":[{"friendname": "Jaismin"}, {"friendname": "Adarsh"}, {"friendname": "Shashank"}

]

}

Term Query Returns documents that contain an exact term in a provided field. This is mostly used for the “keyword” type field.

Get User whose first name Jay

GET /test-index/_search

{

"query": {

"bool": {

"must": [

{"term": {

"firstname": {

"value": "Jay"

}

}}

]

}

}

}

Result

{

"took" : 4,

"timed_out" : false,

"_shards" : {

"total" : 5,

"successful" : 5,

"skipped" : 0,

"failed" : 0

},

"hits" : {

"total" : {

"value" : 1,

"relation" : "eq"

},

"max_score" : 0.2876821,

"hits" : [

{

"_index" : "test-index",

"_type" : "_doc",

"_id" : "lSUVtHgBw7rVVucQG7xH",

"_score" : 0.2876821,

"_source" : {

"firstname" : "Jay",

"lastname" : "Prakash",

"fullname" : "Jay Prakash",

"age" : 30,

"friends" : [

{

"friendname" : "Adarsh"

},

{

"friendname" : "Jaismin"

}

]

}

}

]

}

}

Query on first and last name would be case sensitive as well. Because keywords are stored in ElasticSearch index as it is.

GET /test-index/_search

{

"query": {

"bool": {

"must": [

{"term": {

"firstname": {

"value": "Jay"

}

}}

]

}

}

}

The above query will give you zero hits.

2. Querying over multiple values on the same field.

GET /test-index/_search

{

"query": {

"bool": {"must": [

{"terms": {

"firstname": [

"Jay",

"Viral"

]

}}

]}

}

}

Result

{

"took" : 171,

"timed_out" : false,

"_shards" : {

"total" : 5,

"successful" : 5,

"skipped" : 0,

"failed" : 0

},

"hits" : {

"total" : {

"value" : 2,

"relation" : "eq"

},

"max_score" : 1.0,

"hits" : [

{

"_index" : "test-index",

"_type" : "_doc",

"_id" : "lSUVtHgBw7rVVucQG7xH",

"_score" : 1.0,

"_source" : {

"firstname" : "Jay",

"lastname" : "Prakash",

"fullname" : "Jay Prakash",

"age" : 30,

"friends" : [

{

"friendname" : "Adarsh"

},

{

"friendname" : "Jaismin"

}

]

}

},

{

"_index" : "test-index",

"_type" : "_doc",

"_id" : "y04VtHgBBK0Ei3S1KHhq",

"_score" : 1.0,

"_source" : {

"firstname" : "Viral",

"lastname" : "Prasad",

"fullname" : "Viral Prasad",

"age" : 30,

"friends" : [

{

"friendname" : "Lovenish"

},

{

"friendname" : "Sandeep"

},

{

"friendname" : "Shashank"

}

]

}

}

]

}

}

Full-Text Query The full-text queries enable you to search analyzed text fields such as the body of an email. The query string is processed using the same analyzer that was applied to the field during indexing.

All queries below will give input because “fullname” is stored in the index after analyzing with a standard analyzer. ElasticSearch by default uses fuzzy matching for text fields if nothing specified.

GET /test-index/_search

{

"query": {"bool": {

"must": [

{

"match": {

"fullname": "jay"

}

}

]

}}

} GET /test-index/_search

{

"query": {"bool": {

"must": [

{

"match": {

"fullname": "JAy"

}

}

]

}}

} GET /test-index/_search

{

"query": {"bool": {

"must": [

{

"match": {

"fullname": "prakash"

}

}

]

}}

}

Result

{

"took" : 5,

"timed_out" : false,

"_shards" : {

"total" : 5,

"successful" : 5,

"skipped" : 0,

"failed" : 0

},

"hits" : {

"total" : {

"value" : 1,

"relation" : "eq"

},

"max_score" : 0.2876821,

"hits" : [

{

"_index" : "test-index",

"_type" : "_doc",

"_id" : "lSUVtHgBw7rVVucQG7xH",

"_score" : 0.2876821,

"_source" : {

"firstname" : "Jay",

"lastname" : "Prakash",

"fullname" : "Jay Prakash",

"age" : 30,

"friends" : [

{

"friendname" : "Adarsh"

},

{

"friendname" : "Jaismin"

}

]

}

}

]

}

}

Nested Query Wraps another query to search nested fields.

Get all the users who are friend with Jasimin

GET /test-index/_search

{

"query": {"bool": {

"must": [

{

"nested": {

"path": "friends",

"query": {

"term": {

"friends.friendname.keyword": {

"value": "Jaismin"

}

}

}

}

}

]

}}

}

Result

{

"took" : 1023,

"timed_out" : false,

"_shards" : {

"total" : 5,

"successful" : 5,

"skipped" : 0,

"failed" : 0

},

"hits" : {

"total" : {

"value" : 2,

"relation" : "eq"

},

"max_score" : 0.9808291,

"hits" : [

{

"_index" : "test-index",

"_type" : "_doc",

"_id" : "604rtHgBBK0Ei3S1aXjk",

"_score" : 0.9808291,

"_source" : {

"firstname" : "Sandeep",

"lastname" : "Dash",

"fullname" : "Viral Prasad",

"age" : 30,

"friends" : [

{

"friendname" : "Jaismin"

},

{

"friendname" : "Adarsh"

},

{

"friendname" : "Shashank"

}

]

}

},

{

"_index" : "test-index",

"_type" : "_doc",

"_id" : "lSUVtHgBw7rVVucQG7xH",

"_score" : 0.6931471,

"_source" : {

"firstname" : "Jay",

"lastname" : "Prakash",

"fullname" : "Jay Prakash",

"age" : 30,

"friends" : [

{

"friendname" : "Adarsh"

},

{

"friendname" : "Jaismin"

}

]

}

}

]

}

}

Suggester Suggests similar-looking terms based on a provided text by using a suggester. If you see the ElasticSearch document you will see this text “Parts of the suggested feature are still under development.” But I have gone through multiple changelogs of ElasticSearch from 6.4 onwards but I don’t see anything except bug fixes so I believe it is pretty stable to use.

Three types of the suggester

Completion Term Phrase

Completion Suggestor

Provides autocomplete Only works based on prefix Stores data in a special data structure called Finite-state transducer for speed. This is stored in memory

We will create an index “ test-completion ” with two fields title and suggest as bellow.

PUT /test-completion

PUT /test-completion/_mapping

{

"properties" :{

"title":{

"type": "keyword"

},

"suggest": {

"type": "completion"

}

}

}

Let’s insert four documents in our index

POST /test-completion/_doc

{

"title": "Software Developer",

"suggest": ["Software", "Developer"]

} POST /test-completion/_doc

{

"title": "Software Engineer",

"suggest": ["Software", "Engineer"]

} POST /test-completion/_doc

{

"title": "Software",

"suggest": ["Software"]

} POST /test-completion/_doc

{

"title": "Abs",

"suggest": ["Abs is aabsolute system"]

}

We can see the result by searching everything

GET /test-completion/_search{}

You will be seeing output something like this.

{

"took" : 3,

"timed_out" : false,

"_shards" : {

"total" : 5,

"successful" : 5,

"skipped" : 0,

"failed" : 0

},

"hits" : {

"total" : {

"value" : 4,

"relation" : "eq"

},

"max_score" : 1.0,

"hits" : [

{

"_index" : "test-completion",

"_type" : "_doc",

"_id" : "c6aFs3gBlUt7Kuo2Rfuc",

"_score" : 1.0,

"_source" : {

"title" : "Software Engineer",

"suggest" : [

"Software",

"Engineer"

]

}

},

{

"_index" : "test-completion",

"_type" : "_doc",

"_id" : "fE6Ks3gBBK0Ei3S1NXjA",

"_score" : 1.0,

"_source" : {

"title" : "Abs",

"suggest" : [

"Abs is aabsolute system"

]

}

},

{

"_index" : "test-completion",

"_type" : "_doc",

"_id" : "baaEs3gBlUt7Kuo2Bfuk",

"_score" : 1.0,

"_source" : {

"title" : "Software Developer",

"suggest" : [

"Software",

"Developer"

]

}

},

{

"_index" : "test-completion",

"_type" : "_doc",

"_id" : "a06Es3gBBK0Ei3S1MXiI",

"_score" : 1.0,

"_source" : {

"title" : "Software",

"suggest" : [

"Software"

]

}

}

]

}

}

autocomplete with prefix soft

GET /test-completion/_search

{

"suggest": {

"autocoplete": {

"prefix":"soft",

"completion":{

"field": "suggest",

"skip_duplicates": true

}

}

}

}

Result

{

"took" : 5,

"timed_out" : false,

"_shards" : {

"total" : 5,

"successful" : 5,

"skipped" : 0,

"failed" : 0

},

"hits" : {

"total" : {

"value" : 0,

"relation" : "eq"

},

"max_score" : null,

"hits" : [ ]

},

"suggest" : {

"autocoplete" : [

{

"text" : "soft",

"offset" : 0,

"length" : 4,

"options" : [

{

"text" : "Software",

"_index" : "test-completion",

"_type" : "_doc",

"_id" : "c6aFs3gBlUt7Kuo2Rfuc",

"_score" : 1.0,

"_source" : {

"title" : "Software Engineer",

"suggest" : [

"Software",

"Engineer"

]

}

}

]

}

]

}

}

2. autocomplete using prefix Abs

GET /test-completion/_search

{

"_source": false,

"suggest": {

"autocoplete": {

"prefix":"abs",

"completion":{

"field": "suggest",

"skip_duplicates": true

}

}

}

}

Result

{

"took" : 3,

"timed_out" : false,

"_shards" : {

"total" : 5,

"successful" : 5,

"skipped" : 0,

"failed" : 0

},

"hits" : {

"total" : {

"value" : 0,

"relation" : "eq"

},

"max_score" : null,

"hits" : [ ]

},

"suggest" : {

"autocoplete" : [

{

"text" : "abs",

"offset" : 0,

"length" : 3,

"options" : [

{

"text" : "Abs is aabsolute system",

"_index" : "test-completion",

"_type" : "_doc",

"_id" : "fE6Ks3gBBK0Ei3S1NXjA",

"_score" : 1.0

}

]

}

]

}

}

Autocomplete support fuzziness as well. Which really helps in case of typo.

3. autocomplete with typo

GET /test-completion/_search

{

"_source": false,

"suggest": {

"autocoplete": {

"prefix":"dve",

"completion":{

"field": "suggest",

"skip_duplicates": true,

"fuzzy":{

"fuzziness": "auto"

}

}

}

}

}

Result

{

"took" : 7,

"timed_out" : false,

"_shards" : {

"total" : 5,

"successful" : 5,

"skipped" : 0,

"failed" : 0

},

"hits" : {

"total" : {

"value" : 0,

"relation" : "eq"

},

"max_score" : null,

"hits" : [ ]

},

"suggest" : {

"autocoplete" : [

{

"text" : "dve",

"offset" : 0,

"length" : 3,

"options" : [

{

"text" : "Developer",

"_index" : "test-completion",

"_type" : "_doc",

"_id" : "baaEs3gBlUt7Kuo2Bfuk",

"_score" : 1.0

}

]

}

]

}

}

If you are closely following you will notice the last two results don’t have _source field. This is a good technique when you are doing autocomplete for large datasets.

Term Suggestor

Used for correcting user’s spelling Based on edit distance. (Edit distance is basically how many characters need to change to match the term. For example, “softwwere” is one distance from software.)

Create an index “ test-term ” and create the mapping for it.

PUT /test-term PUT /test-term/_mapping

{

"properties":{

"title": {

"type": "text"

}

}

}

Insert some document in it.

POST /test-term/_doc

{

"title": "Senior Software Engineer"

} POST /test-term/_doc

{

"title": "Senior Software Engineer Test"

} POST /test-term/_doc

{

"title": "Senior Software Anlyst Test"

}

See all documents in it.

GET /test-term/_search {} {

"took" : 353,

"timed_out" : false,

"_shards" : {

"total" : 5,

"successful" : 5,

"skipped" : 0,

"failed" : 0

},

"hits" : {

"total" : {

"value" : 3,

"relation" : "eq"

},

"max_score" : 1.0,

"hits" : [

{

"_index" : "test-term",

"_type" : "_doc",

"_id" : "kaams3gBlUt7Kuo2YPtl",

"_score" : 1.0,

"_source" : {

"title" : "Senior Software Anlyst Test"

}

},

{

"_index" : "test-term",

"_type" : "_doc",

"_id" : "iU6ms3gBBK0Ei3S1U3gQ",

"_score" : 1.0,

"_source" : {

"title" : "Senior Software Engineer Test"

}

},

{

"_index" : "test-term",

"_type" : "_doc",

"_id" : "kKams3gBlUt7Kuo2Rvv4",

"_score" : 1.0,

"_source" : {

"title" : "Senior Software Engineer"

}

}

]

}

}

Here we will pass incorrect spelling of term and see what ElasticSearch suggests.

GET /test-term/_search

{

"suggest": {

"spellcheck": {

"text": "softewre",

"term": {

"field": "title"

}

}

}

}

Result

{

"took" : 326,

"timed_out" : false,

"_shards" : {

"total" : 5,

"successful" : 5,

"skipped" : 0,

"failed" : 0

},

"hits" : {

"total" : {

"value" : 0,

"relation" : "eq"

},

"max_score" : null,

"hits" : [ ]

},

"suggest" : {

"spellcheck" : [

{

"text" : "softewre",

"offset" : 0,

"length" : 8,

"options" : [

{

"text" : "software",

"score" : 0.75,

"freq" : 3

}

]

}

]

}

}

If you look closely result you will find “freq” which is basically how many times this term has been found in our dataset. With help of this, we suggest the closest correct term.

Phrase Suggester

Similar to term suggester. Based on n-gram models

N-gram model is a way of the tokenizing string. Some people call shingles instead of gram.

Example

Indexing string “domain” with gram size 2 then every two consecutive characters will be one term like “do”, “om”, “ma”, “ai”, “in”. “Quick Brown Fox” with minim gram size 2 and max gram size 3 will be “Quick Brown” “Brown Fox”, “Quick Brown Fox”.

For phrase suggester, we are required to create a custom analyzer as per our requirement. We will create an index and pass the setting with our custom “analyzer”.

PUT /test-phrase

{

"settings": {

"analysis": {

"analyzer": {

"trigram": {

"type": "custom",

"tokenizer": "standard",

"filter": [

"shingle", "lowercase"

]

}

},

"filter": {

"shingle": {

"type": "shingle",

"min_shingle_size": 2,

"max_shingle_size": 3

}

}

}

},

"mappings": {

"properties":{

"title":{

"type": "text",

"fields":{

"trigram": {

"type": "text",

"analyzer": "trigram"

}

}

}

}

}

}

Let’s insert some documents in our index.

POST /test-phrase/_doc

{

"title": "Domain Driven Design"

} POST /test-phrase/_doc

{

"title": "Domain Driven Design by Jay"

}

Just want to suggest the correct phrase

GET /test-phrase/_search

{

"suggest": {

"phrase-fix": {

"text": "domian driven design",

"phrase": {

"field": "title.trigram"

}

}

}

}

Result

{

"took" : 15,

"timed_out" : false,

"_shards" : {

"total" : 5,

"successful" : 5,

"skipped" : 0,

"failed" : 0

},

"hits" : {

"total" : {

"value" : 0,

"relation" : "eq"

},

"max_score" : null,

"hits" : [ ]

},

"suggest" : {

"phrase-fix" : [

{

"text" : "domian driven design",

"offset" : 0,

"length" : 20,

"options" : [

{

"text" : "domain driven design",

"score" : 0.40580386

}

]

}

]

}

}

Phrase suggester has multiple interesting features.

Want to highlight the correction phrase — “highlight” Want to match return suggestion against the document in the index — “collate with a query against the returned suggestion” Differentiate matched and none matched suggestion in the index — using “prune”

GET /test-phrase/_search

{

"suggest": {

"phrase-fix": {

"text": "domian driven design",

"phrase": {

"field": "title.trigram",

"confidence": 0,

"collate":{

"query": {

"source":{

"match_phrase":{

"{{field_name}}" : "{{suggestion}}"

}

}

},

"params": {"field_name" : "title"},

"prune": true

},

"highlight":{

"pre_tag": "<em>",

"post_tag": "</em>"

}

}

}

}

}

Result

{

"took" : 16,

"timed_out" : false,

"_shards" : {

"total" : 5,

"successful" : 5,

"skipped" : 0,

"failed" : 0

},

"hits" : {

"total" : {

"value" : 0,

"relation" : "eq"

},

"max_score" : null,

"hits" : [ ]

},

"suggest" : {

"phrase-fix" : [

{

"text" : "domian driven design",

"offset" : 0,

"length" : 20,

"options" : [

{

"text" : "domain driven design",

"highlighted" : "<em>domain</em> driven design",

"score" : 0.40580386,

"collate_match" : true

},

{

"text" : "domian driven design",

"highlighted" : "domian driven design",

"score" : 0.065872006,

"collate_match" : false

}

]

}

]

}

}

For further reading on suggester please refer this page