Elasticsearch quick fire — a fun romp through the history and use cases
By: Isaac Reichman
Tags: Kibana, Data Analytics, History, Lucene, Elasticsearch

Finally, the meat of the post — How do I get started and use Elasticsearch?

From my perspective, one of the most beautiful aspects of Elasticsearch is that it is language agnostic. Sure, there are wrappers and extensions in various languages and various platforms, but you need nothing more than the four HTTP CRUD protocols to make it work for you.

GET

PUT

POST

DELETE

Now, the APIs for the Elastic stack arguably include those for Kibana, which is a beast unto itself, so let’s just focus on Elasticsearch specifically just for now. We’ll start with some quick usage patterns and then a helpful guide on transitioning from SQL to Elasticsearch.

For a quick start on installing Elasticsearch/Kibana, I recommend spinning up a cluster with three nodes using the docker compose from the official documentation.

Let’s have a REST™

In order to see the true value in Elasticsearch, you’ve gotta start with a search in mind. The format for searching, as well as a lot of other APIs, in Elasticsearch, is on a per-index basis (replace <variable> with the relevant value):

curl -XGET -u <user>:<pw> \ 'https://<yourelasticurl>:9201/<index_name>/_search?pretty'

This is a pretty generic request that will just grab the top 10,000 documents in our index. In the following, I’m submitting requests to an index in my cluster called “scraped_2019_2020”, which consists of web-scraped data.

What if we want to specify how many documents we want? We bring in our JSON body

curl -XGET 'http://localhost:9200/scraped_2019_2020/_search' \ --header 'Content-Type: application/json' \ --data-raw '{ "size":1 }'

Specific documents? Sounds like a candidate for search

curl -XGET 'http://localhost:9200/scraped_2019_2020/_search' \ --header 'Content-Type: application/json' \ --data-raw '{ "size":1, "query":{ "match":{ "title":"Google Cloud — Data Engineer Exam Study Guide" } } }'

What we’re doing in the above two command bodies specifying our search parameters. Our parameters aren’t just the term or phrase we’re looking for, it also includes how we look for terms. For example, in the above, we’ve used the match query. This query type is capable of doing fuzzy matching on a text field, such as wildcards and transpositions. For a full list of the different DSL query types, check out the official documentation.

That seems familiar? Is this Deja vu?

“What about my transferable knowledge from SQL?” You’re going to throw out some basic operations you may want to do based on SQL usage, and I’m gonna holla back with how you would do it in Elasticsearch. Note: it really isn’t an either/or between Elasticsearch and SQL engines, it’s a use case question.

I wanna find a specific entry/row/document in my table/collection/index

GET is roughly equivalent to your SELECT/DESCRIBE in SQL, coupled with a specific endpoint for your API (_search/_cat)

I wanna create a new table/collection/index/object

PUT is your friend here, used for creating (putting) new objects. Much like CREATE.

I wanna import data/append a specific entry/row/document in my table/collection/index

POST that data into Elastic for an equivalent to INSERT

I wanna delete/remove a specific entry/row/document in my table/collection/index

Hazard a guess (it’s DELETE, and you may remember your fears regarding the DROP command in SQL)

The great thing about transitioning from familiarity with SQL to familiarity with Elasticsearch is that you don’t have to do it alone. The kind folks over at Elastic have created an endpoint for Elasticsearch clusters that actually accepts SQL and translates it to Elasticsearch syntax. Unfortunately, there isn’t a one-to-one mapping; but I think this is good because fundamentally, Elasticsearch is not a relational database.

What do I mean by that? Let’s take a look:

Assuming you’ve installed and started ES, let’s double-check it’s working. Open a terminal and make a simple get request:

Fantastic, our cluster is up and running. In my case, I’ve got a three-node cluster deployed through docker compose so that I can deploy and tear it down with ease. Now, let’s completely forget that we’re using a NoSQL document store and post some SQL queries.

Great stuff, it even tells us the phenomenological mapping from SQL to Elasticsearch. So now that we’ve found a crutch when NoSQL gets too much for us, let’s talk about some use cases.

Well that’s all well and good, but why the hell should I use this?

Search underpins a lot of the use cases for Elasticsearch. Shay himself started this whole venture with a driving motive being:

“Search is something that any application should have (and search here is not just full text search)”

But what does this mean for you, a developer?

It means that, even though I just vomited a bunch of SQL analogues at you to assist your understanding of elastic, you may not need to pick one over the other; they are different things.

Here are a few situations where you may want to implement Elasticsearch over SQL, vice versa, and when you want both:

Search engine

A natural use, Elasticsearch can be used for creating a search engine for countless applications. It could be used for a blog, for an e-commerce store, for an internal database of documents, ephemeral data, the list goes on… The question to ask yourself is “How would I implement generic fuzzy text searching in a database?” This is possible with Elastic out of the box, being based on Lucene; so it’s a natural choice.

What’s important here is finding some way to ingest your data into your index, depending on if it’s streaming in or if it’s static.

A method for ingesting data is using a client/connector. An official Python client for Elastic enables you to create and use Elastic related objects that you can manipulate from within the script.

Observability (SIEM) and Anomaly detection

Tangential to the search function, one of the main uses for Elasticsearch is its ability to store and catalogue logs in such a way that they can be digested for visualisations and informed observability.

You can automatically ingest log data into Elasticsearch using Logstash and the different beat agents (filebeat, winlogbeat, etc…) made by Elastic for sitting on endpoints. These endpoints could be servers, networks, or even applications (you could ingest MySQL or PostgreSQL logs, for delicious irony). If the use case is simple enough — you don’t need too much processing or fine-grained index control— you can ingest directly from the beats.

But the data in your cluster will be somewhat nebulous unless you write your own data visualisation platform for graphing and monitoring your index data.

If you don’t want to do that, and would rather rely on a platform created by Elastic, read on.

Kibana — i.e. how to make your life easier as a user and developer (really, I mean it)

Kibana is to Elasticsearch what MySQL workbench is to MySQL; that is if you gave workbench steroids and also turned it into a data visualisation platform. Now, don’t get me wrong, I love workbench as much as the next person (so don’t @ me).

Visualisation

The main usage of Kibana is data visualisation and stack management. If you’re looking for anomalies or patterns in your data, you can use the discover application:

You also have it in your power to create custom visualizations based on your data and then save them to a dashboard.

Stack Management

You can also use Kibana for checking on different properties of your indices, including (but not limited to) their health, size, ILM, template…

Dev tools

If you’re the sort of developer who solely uses Vim as your main editor, you may be thinking, “I’m perfectly happy interacting with my Elastic instance via the terminal, I don’t need any interactive console”. While I respect that intensity, you can have your cake and eat it too with Kibana. Something I found to be incredibly useful, to the point of no return, with Kibana, was dev tools.

See what I mean? Being an API, Elasticsearch might be a natural candidate for developing using cURL based tools, like Postman. But you don’t need to concern yourself all the time with proper endpoints and typos when you can skip all the foray and just focus on the request.

It’s all for your ease of use, so why not make your life easy and use it?

Stay tuned for some cool applications of the Elastic stack…