An Overview on Elasticsearch: Part I
By: Chhaya Kumari
Tags: Search, Elasticsearch, Search Engine, Dbms, Inverted Index

Search engines — “an answering machine”, an online tool that discovers, understands and organizes the internet’s content in order to offer the most appropriate results to the questions searchers are asking.

Elasticsearch is a search engine that is built on top of Apache Lucene and java. Lucene library is an open-source, high-performance library suitable for nearly any application that requires full-text search, especially cross-platform.

Initially, elasticsearch started as a scalable version of the Lucene open-source search framework and then added the capability to horizontally scale Lucene indices.

At its core, we can think of elastic search as a server that processes JSON requests and returns JSON data.

The basic structure of elasticsearch is comprised of documents instead of tables and schemas along with extensive REST APIs for storing and searching the data.

Elasticsearch attains a fast search response because rather than searching the text directly, it searches the index. This makes it an incredibly unique search engine and one of the 10 most liked DBMS.

How does it work?

In this section, we will try to understand the building blocks of a simple engine created by elastic co-founder Shay Bannon for his wife’s cooking recipes that became today’s most famous enterprise search engine.

Logical Components

Documents:

Documents are the primary unit of information used for indexing in elasticsearch. In elasticsearch, a document can be any structured data encoded in JSON with a variety of different data types in it. Each document comes with a unique ID and a given data type, which describes the entity of the document.

Indices:

A collection of documents with similar types of characteristics is termed as an index. In elasticsearch, the index is the highest level entity used for querying. Documents present in an index are logically related. For example: in the case of an eCommerce platform, we can have an index for customers, products, orders, payments, etc.

An index is always defined by a name that refers to the index while performing various operations like indexing, search, update and delete against the document in it.

In a nutshell, we can say that the Index in elasticsearch is similar to a table in a relational database schema.

Inverted Index:

The underlying data structure is known as the Inverted Index. The mechanism of an inverted index aims to store text in a structure that permits very efficient and fast full-text searches.

If a user wants to find articles that talk about the automobile, he/she may search using the word “automobile”. If the user doesn’t have a special indexing technique, then all the records will be scanned to find a match, which is an inefficient technique. “inverted index” offers the solution for this.

The following example elaborates on the searching using Elasticsearch. Let’s consider these documents to be indexed with the contents in it.

Document1: “Elasticsearch is fast”

Document2: “Elasticsearch is efficient”

Document3: “Elasticsearch offers high speed”

Now to create the inverted index for the above documents, the contents of each document will split into separate words. After that, a list gets generated with unique words and document ids of the document and the frequency.

Now, the inverted index generated for the above documents would be below:

If the query is fired to search the word “speed”, then Elasticsearch will look into its inverted index table. It will find that the word occurs in document 3 and will show that document. So due to the inverted index creation, the search becomes faster.

Concept of stopwords in elasticsearch:

Some words like is, the, on, this, etc seldom add value to a search. Elasticsearch has its own list of predefined stopwords. Stopwords are usually filtered out before indexing. The inverted index is then created over the terms of the document to make search faster.

Backend Components:

Cluster:

Elasticsearch is a group of multiple nodes, called clusters. The power of an elasticsearch cluster lies in the distribution of tasks, searching, and indexing, across all the nodes in the cluster.

Nodes:

A node is a single server(part of a cluster) which stores data and aids to the cluster’s indexing and searching capabilities:

There are three types of nodes in elasticsearch:

Master Node — This node executes all cluster-wide operations like creating/deleting an index and adding/removing nodes.

Data Node — This node is responsible for data storage and performs data-related operations such as search and aggregation.

Client Node — This node communicates with the master node and data nodes for cluster requests and data-related requests respectively.

Shards:

Data in an Elasticsearch index can grow into a humongous proportion. In order to keep it manageable, it subdivides the index into multiple pieces called shards. Each shard is in itself an independent and fully-functional “index” that can be hosted on any node within a cluster.

This ensures redundancy and also protects against hardware failure.

Having the right number of shards is important for performance. It is thus wise to plan in advance. The absolute number of shards should be determined based on the amount of data in an index.

Replica:

Elasticsearch facilitates you to make one or more copies of the index’s shards which are called “replica shards” or just “replicas”. A replica is a copy of the primary shard and each document in an index belongs to one primary shard. Each replica of the shard is located on a different node, ensures smooth accessibility to your data in case of an event failure.

There are a couple of differences in the prioritization of primary and replica shards. While both are capable of processing queries, indexing requests must go through primary shards first, and then it can replicate to the replica shards.

As discussed above, if a primary shard becomes unavailable — for example, due to a node failure or hardware failure — a replica is promoted to take over its role.

If the analysis process is based on an index, the retrieval process would be more efficient. Elasticsearch is indeed an excellent technology of big data for searching operations.

In this article, we attempted to answer the nuts and bolts of a booming technology through the lens of understanding of its logical components and architecture and we’re still only scarcely scratching the surface of learning everything there is about it.

In the next post, we will discuss its physical representation along with its internal functioning.

But, in a quick glance of its introduction, we can briefly conclude that Elasticsearch’s underlying architecture makes it fast and scalable, sitting at the heart of an ecosystem of complementary tools that together can be used for various uses cases.