ElasticSearch: Things you should Know about ES to be Useful
By: Hansraj Choudhary
Tags: Search, Elasticsearch, Lucene, Docker, Inverted Index

The purpose of this writing is to give basics about ElasticSearch. After completing this 6 mins read, you should feel comfortable about ES architecture, the way it stores the documents, indexing and near realtime search response.

Let’s understand about ES step by step:

1. High level architecture

2. Document storage and indexing

3. Near real-time search

4. Try it on local

Relational databases likes mysql etc. are comparatively slow when it comes to huge data and fetching search results through queries. ES is a document-oriented database, designed to store, retrieve, and manage JSON data. It is schema-less, uses Lucene as a primary data store engine.

ES: High level architecture

ES provides REST APIs to insert / delete / get / search data. The documents are getting stored in ES indices and sharded based on the hash value of document_id. Every shard is actually a Lucene index and store the document in segments. To understand Elasticsearch in detail; we need to understand its core concepts and terminologies.

Index

Index is a logical partition of documents and can be compared to a database in the world of relational database.

The index is a collection of documents that have similar characteristics. For example, we can have an index type for employee data and another one for department information. An index is identified by a unique name that refers to the index when performing indexing, search, update, and delete operations. In a single cluster, we can define as many indexes as we want.

Shards

A shard is a subset of documents of an index. An index can be divided into many shards. They are the building block of Elasticsearch and facilitate scalability part of it.

Sharding allows you to horizontally scale by volume and space; it enhances the performance of a cluster by running parallel operations and distributing loads across different shards. By default; Elasticsearch adds 5 primary shards for an index. This can be manually configured to suit your requirements.

Nodes

A node is a server; it is a single unit of the cluster. If it is a single node cluster; all data will be stored in that single node; else the data will be distributed amongst n nodes which are the part of that cluster. Nodes participate in a cluster’s search and indexing capabilities. Depending upon the type of query fired; they will collaborate and will return the matching response.

Storing documents

An Elasticsearch cluster can contain multiple indices, which in turn contain multiple types. These types hold multiple documents, and each document has multiple fields.

ES usage Lucene as the backend data store and retrieval engine. Lucene’s functionalities can be grouped primarily into two categories: indexing and searching. When a document is getting inserted, diagram below shows the summary of the entire indexing process.

Tokeniser and Analyser used to tokenise and parse the raw text data. And then we have Lucene index package contains classes that records and provides the basic functionalities of inverted indexing, i.e., counting, recording and sorting mechanism of terms of document. It divide document, especially long documents, into several segments, creates sorted inverted index and term metadata, and then merge the sorted segments at the end. Finally, we have the Lucene store package that takes care of the final writing of δ-encoded postings onto the disk.

Lucene performs copy on write when updating and deleting a document. It means the document is never deleted from the index. Instead, it marks the document as deleted and creates another one when an update is triggered. During a merge, Lucene takes 2 segments, and moves the content into a third, new one. Then the old segments are deleted from the disk. It means Lucene needs enough free space on the disk to create a segment the size of both segments it needs to merge.

Searching a document

Search functionality, on the other hand, involves parsing of query, reading the relevant data from disk and returning the results.

The Lucene index provides a mapping from terms to documents. This is called an inverted index because it reverses the usual mapping of a document. The same string in two different fields is considered a different term. Thus terms are represented as a pair of strings, the first naming the field, and the second naming text within the field. Each segment index maintains Term Vectors : For each field in each document, the term vector is stored. A term vector consists of term text and term frequency.

A user expresses an ES query with a simple language, Query DSL. A query examines one or many target values, and scores each of the elements in the results according to how close they match the focus of the query.

Index Versus Index Versus Index: [i] Index (noun): is like a database in a traditional relational database. [ii] Index (verb): To index a document is to store a document in an index (noun). [iii]Inverted index: Indexing a token in order to improve the speed of data retrieval.

Near real-time search

Instead of having to search through the entire document or row space for a given value, the system can find that value in its internal index and immediately know which documents or rows contain it. This, of course, makes querying significantly faster.

In Elasticsearch the data is distributed and stored in different clusters. So when a change is made on an index it may not be readily available; a latency of few milliseconds is expected! New segment is written to the filesystem cache first — which is cheap — and only later is it flushed to disk — which is expensive. But once a file is in the cache, it can be opened and read which makes it searchable.

When to use elasticsearch

1) When we need to manage huge amount of data and search from it quickly.

Enterprise search, site search etc.

2) Geospatial data store: If we have lot of geographical data and want to search nearby locations / places, ES is suited best for the use case.

3) JSON document storage: It’s the basic information unit that can be indexed in ES. Auto complete / suggest / data aggregation can be achieved out of the box.

4) Logging and Analysis along with Logstash

Some real-world examples of organisations that rely on Elasticsearch to search and query data are Wikipedia, StackOverflow, Github, The Guardian etc

Try ES on local machine (Using docker)

The easiest way to understand what Elasticsearch can do for you is to play with it, so let’s get started! Make sure you already have docker installed in your machine.

1. Pull ES image

Pull the specified version and check if the image is present in available images list.

docker pull elasticsearch:6.7.0

docker images

2. Start ES

Elasticsearch instance should be up and running once you execute the following command. Check the cluster info using browser: http://localhost:9200

docker run -p -d 9200:9200 --name=elasticsearch1 elasticsearch:6.7.0

3. Index document

This command will put a document with id=1 in type=detail and index = employee in ES data store.

4. Fetch / search document

The GET command will fetch the employee details having id=1. Search command will try to match the keyword ‘sachin’ in all the available tokens and return the ranked results. Play with indexing more documents and check the results.

curl -XGET localhost:9200/employee/detail/1

curl -XPOST localhost:9200/employee/_search?q=sachin

5. Delete document

Delete won’t immediately deletes a document from the data store. The document is marked for deletion and will be filtered out from subsequent search query results. When merge segments is triggered, ES ignores the deleted documents and create a new segment with merged data. The old segments then gets deleted from the data store. We can also delete a full index or document type using this command.

curl -XDELETE localhost:9200/employee/detail/1

Further Reads

=> https://www.elastic.co/blog/a-practical-introduction-to-elasticsearch

=> https://markheath.net/post/exploring-elasticsearch-with-docker

=> http://elasticsearch-cheatsheet.jolicode.com/

=> http://exploringelasticsearch.com/searching_data.html

=> https://www.youtube.com/watch?v=oPObRc8tHgQ

=> http://www.dcs.bbk.ac.uk/~dell/teaching/cc/book/ditp/ditp_ch4.pdf