From scratch to search: setup Elasticsearch under 4 minutes, load a CSV with Python and read some more about it | by Stanislav Prihoda 🔥 | Towards Data Science

Primary aim of this article is to provide a short and simple guidance on how to initially setup your first Elasticsearch “dev” environment in order to get going quickly and start exploring/exploiting what the technology offers. The introduction will be based around the foremost APIs that Elasticsearch offers which are fundamental for getting the data in and for execution of data queries. Secondary aim is to provide links on documentation and other interesting resources to be aware of other potential operations aspects, additional cool features and various tools (generally stuff I find useful when working with Elasticsearch). I have been missing this kind of an “intro with pointers further”, so this is an attempt for anyone alike :). These links will be summed up in dedicated What next sections in each chapter.
Intended audience could be individual data analyst or web developer, eventually small teams on budget that are having relevant data use-case and have already heard of Elasticsearch. What is not intended is to provide full technical overview of the technology (clusters, nodes, shards, replicas, Lucene, inverted indexes etc.) or a deep dive on some specific topic as there are many good resources on that... some of which will be linked in the article or in the What next sections.
If you’re just wondering how Elasticsearch can be useful for your purposes, then there are generally three main areas (among other) which can be effectively supported by the Elastic Stack
“big data” analytics & specific data science tasks which can revolve around having large semi-structured data (text and json objects) requiring highly available storage and fast full-text search capabilities. Interesting cases can be found for example in the NLP area, or when dealing with loads of GEO-based data with the need to perform geo-location queries and visualisations. Also it can bring value if you need just a ready-made exploration/dashboarding tool for some more complex business analytics queries.
time-series processing, storage and browsing relevant for dealing with streams of time-stamped data coming in. These can include Elastic’s power area log processing from vast amount of running services, or their monitoring from the health and uptime perspective and also for example processing any sensor data that can be ingested into the Elasticsearch etc.
full-text search of any large textual data (websites, enterprise stuff, document archives etc.) to make their content available for complex textual queries. These can (with correct setup) get returned under tens of milliseconds thus highly supporting user experience and ability to drill relevant information. Inspiration can be also found eg. in combination with other techniques such as OCR.
From my perspective the easiest way to start experimenting with Elasticsearch (and many other software technologies) is via Docker and thus the examples will utilize this approach. Generally the containerisation is (already for some time) taking over also the actual production deployment/ops so this can be two-in-one benefit. Large scale example of this is presented by Uber showing their fully containerised approach with hundreds of Elastic clusters. So let's do it...
I presume you are already familiar with Docker (if not... install Docker for Desktop for Mac, Windows or Linux distributions, read some intro stuff and you are good to go). Note: examples were tested on macOS Mojave, remember there might be some Docker specifics on Linux distros or Win.
When you have your docker environment ready just open your terminal and start Elasticsearch cluster with these commands:
docker network create elastic-network
This will create for us a basic communication “space” for our future containers.
Now if you want to go with just a single node deployment — just run:
docker run --rm --name esn01 -p 9200:9200 -v esdata01:/usr/share/elasticsearch/data --network elastic-network -e "node.name=esn01" -e "cluster.name=stanislavs-docker-cluster" -e "cluster.initial_master_nodes=esn01" -e "bootstrap.memory_lock=true" --ulimit memlock=-1:-1 -e ES_JAVA_OPTS="-Xms2g -Xmx2g" docker.elastic.co/elasticsearch/elasticsearch:7.3.0
Wait couple of seconds and in your log you should find message confirming setup success from your Elasticsearch cluster:
{“type”: “server”, “timestamp”: “2019–08–18T13:40:18,575+0000”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “stanislavs-docker-cluster”, “node.name”: “esn01”, “cluster.uuid”: “ilHqkY4UQuSRYnE5hFbhBg”, “node.id”: “KKnVMTDwQR6tinaGW_8kKg”, “message”: “started” }
Hooray... we have started our first Elasticsearch node (esn01) running in Docker container (named equally), listening on a port 9201 (docker -p parameter). Thanks to cluster.initial_master_nodes it has formed a new cluster (with name given in cluster.name) in the process known as cluster bootstrapping and immediately elected itself as the master of the cluster (what else to do when you are alone :)). Last thing worth mentioning is the -v parameter, which created new Docker volume esdata01 and bound it to a data directory of the running Elasticsearch — so that after restart our working data will be preserved. The rest of the parameters is more “system” related (and belong to important system settings) — we disable swapping with bootstrap.memory_lock, increase file/process limits with ulimit memlock and assigned initial/max heap size with ES_JAVA_OPTS (ie dedicated memory so adjust to your configuration... but remember that Elasticsearch utilizes also off-heap so do not set above 50% of available memory). Licensing note: it is by default running with a basic license but if you want to go with pure open-source flavor just add *-oss to the end of the image name.
Now if you want to try the taste of a distributed setup add two (or more) other nodes to your cluster. It is not needed for the rest of the tutorial (running on a local machine :)) nor even actual basic operations. Nevertheless as the distribution of the processing is actually one of the core value-adding capabilities of Elasticsearch (as the indexing part is mostly taken care by Apache Lucene library), it is at least good to be aware of it. So if you’re up to... here are our two other nodes (run in separate terminals):
docker run --rm --name esn02 -p 9202:9200 -v esdata02:/usr/share/elasticsearch/data --network elastic-network -e "node.name=esn02" -e "cluster.name=stanislavs-docker-cluster" -e "discovery.seed_hosts=esn01" -e "bootstrap.memory_lock=true" --ulimit memlock=-1:-1 -e ES_JAVA_OPTS="-Xms1g -Xmx1g" docker.elastic.co/elasticsearch/elasticsearch:7.3.0docker run --rm --name esn03 -p 9203:9200 -v esdata03:/usr/share/elasticsearch/data --network elastic-network -e "node.name=esn03" -e "cluster.name=stanislavs-docker-cluster" -e "discovery.seed_hosts=esn01,esn02" -e "bootstrap.memory_lock=true" --ulimit memlock=-1:-1 -e ES_JAVA_OPTS="-Xms1g -Xmx1g" docker.elastic.co/elasticsearch/elasticsearch:7.3.0
Now it starts to be more interesting as the nodes are joining our cluster. You can observe this process in logs, where you have messages about the additions (below from the master of our cluster about newly added node esn02). Generally the nodes are joining the cluster contacting other “already-present” nodes listed in discovery.seed_hosts parameter (to get an up-to-date info on the cluster setup) — this param belongs (along with aforementioned cluster.initial_master_nodes) to important discovery settings. All communication between nodes in the cluster is performed directly over the Transport layer (ie without the HTTP overhead) that’s why there is the port 9300 (not the port 920x we have exposed for “outside” communication with the cluster).
{“type”: “server”, “timestamp”: “2019–08–18T13:40:28,169+0000”, “level”: “INFO”, “component”: “o.e.c.s.ClusterApplierService”, “cluster.name”: “stanislavs-docker-cluster”, “node.name”: “esn01”, “cluster.uuid”: “ilHqkY4UQuSRYnE5hFbhBg”, “node.id”: “KKnVMTDwQR6tinaGW_8kKg”, “message”: “added {{esn02}{4xOwc-_fQDO_lwTlpPks2A}{FW-6YGVSSsmgEt3Mo_GY_Q}{172.18.0.3}{172.18.0.3:9300}{dim}{ml.machine_memory=8360488960, ml.max_open_jobs=20, xpack.installed=true},}, term: 1, version: 16, reason: Publication{term=1, version=16}” }
Everything is up and running so we can check couple of things about our Elasticsearch cluster (notice that we can contact any node in our cluster as generally any node can become coordinating node to process the incoming request via scatter-gather approach):
Basic info about the contacted node:
curl localhost:9200
returns:
{    "name" : "esn01",    "cluster_name" : "stanislavs-docker-cluster",    "cluster_uuid" : "ilHqkY4UQuSRYnE5hFbhBg",    "version" : {        "number" : "7.3.0",        "build_flavor" : "default",        "build_type" : "docker",        "build_hash" : "de777fa",        "build_date" : "2019–07–24T18:30:11.767338Z",        "build_snapshot" : false,        "lucene_version" : "8.1.0",        "minimum_wire_compatibility_version" : "6.8.0",        "minimum_index_compatibility_version" : "6.0.0-beta1"    },    "tagline" : "You Know, for Search"}
Basic info about the cluster status:
curl localhost:9202/_cluster/health?pretty
returns:
{    "cluster_name" : "stanislavs-docker-cluster",    "status" : "green",    "timed_out" : false,    "number_of_nodes" : 3,    "number_of_data_nodes" : 3,    "active_primary_shards" : 0,    "active_shards" : 0,    "relocating_shards" : 0,    "initializing_shards" : 0,    "unassigned_shards" : 0,    "delayed_unassigned_shards" : 0,    "number_of_pending_tasks" : 0,    "number_of_in_flight_fetch" : 0,    "task_max_waiting_in_queue_millis" : 0,    "active_shards_percent_as_number" : 100.0}
Further info about the nodes in the cluster:
curl localhost:9203/_cat/nodes?v
returns:
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name172.19.0.2 26 56 3 0.01 0.16 0.17 dim * esn01172.19.0.4 13 56 1 0.01 0.16 0.17 dim — esn03172.19.0.3 20 56 2 0.01 0.16 0.17 dim — esn02
Last response is probably the most interesting (as the above basically says we have an empty cluster :) . We can see all of the three nodes we have in our cluster (esn01–03). From node.role we can see that all nodes are of the type mdi — which means they are master-eligible (can be elected master if the current master “breaks down”), data (holds actual data ie shards and can perform CRUD/search operations) and ingest (can setup special ingest pipeline to pre-process ie transform/enrich the incoming data before actual indexing). We can also see our current master (marked with *).
DEPLOYMENT: think with your devops about the best option to run in your integration/production environment — options are: standard local installation on your servers, managed-service solution Elastic Cloud or AWS Elasticsearch etc. or sort of a “middle option” mentioned earlier with deploying the docker/OCI images into some containerization platform (Kubernetes, OpenShift, Swarm) running again either on-prem or in the cloud
CONFIGURATION: fundamental configuration resource is elasticsearch.yml file (located in config directory) which is present in any Elasticsearch instance — read more info on that (and other key params) in the important settings
SECURITY: first...! you can go with native ES Security option or with more complex setup with some Security plugin (SearchGuard, Open Distro Security). At least have TLS and role-based auth!
CLUSTERING: so distributed much fault-tolerant ... no seriously if you want to dive deeper into the clustering concepts read something about different node type ie master-eligible/data/ingest/ML and the dig into discovery modules and process (ie to find out why you should have three or more master-eligible nodes in your cluster)
SIZING: read more on sizing and capacity planning
DOCKER: further info on Docker setup, volumes, networks or if you are dealing with specific Elasticsearch version you can get matching image
LOGGING OF ELASTICSEARCH: change logging configuration (based on log4j) in specific hierarchy etc. useful when setting the thing up
LICENSING: it’s good to check under which specific licensing schema some specific desired feature falls (in this article I try to work with either pure oss under Apache 2.0 or basic-licensed stuff for testing)
APIs: Elasticsearch provides a good documentation of all their REST APIs so check it out
APIs: if you liked the cat API with info on your cluster (such as I do) know that there is plenty more covering nodes, shards, indices, health, segments etc. etc.
When you have your initial environment set up it is time to get our data to our running node(s) to be ready for searching. The basic principle of Elasticsearch is that any “unit” of our data that is put to it is called DOCUMENT. Required format of these documents is JSON. These documents are not stored individually but are grouped into specific collections that are called INDEXes. Documents in one index share similar characteristics (ie data types mappings, settings etc.), thus the index serves also as a management and API layer to the contained documents.
So let’s create an index with Create — Indices API and put ie index one document to it with Index — Document API...
curl -X PUT \  http://localhost:9200/test-csv \  -H 'Content-Type: application/json' \  -d '{    "settings" : {        "index" : {            "number_of_shards" : 3,             "number_of_replicas" : 2         }    }}'
returns:
{“acknowledged”:true,”shards_acknowledged”:true,”index”:”test-csv”}
Index creation... checked! We have created our first index named test-csv and defined an important settings of the number of shards and replicas. Primary shards and shard replicas (in correct terms) are a way to split our index (ie document collection) into multiple pieces thus ensuring redundancy. Actual values depend (in a basic sense) on how many nodes we have. Number of shards (ie to how many units our index will be split into) is fixed with index creation. Number of replicas determines how many times the primary shard (ie each of our 3) will be “copied”. We have defined 2 as we want to achieve a state where on each node we have one primary shard (1⁄3 of our data) but also copies of other 2⁄3 of our data ie the full dataset. This setup ensures that if we loose one of our nodes we won’t loose any data. Hint: if you don’t want for starter to mess with these settings you don’t have to because if you attempt to index a document without an existing index — it gets created automatically (with default 1 shard and 1 replica).
curl -X PUT \  http://localhost:9203/test-csv/_doc/1?pretty \  -H 'Content-Type: application/json' \  -d '{    "value": "first document"}'
returns:
{    "_index" : "test-csv",    "_type" : "_doc",    "_id" : "1",    "_version" : 1,    "result" : "created",    "_shards" : {        "total" : 3,        "successful" : 3,        "failed" : 0    },    "_seq_no" : 0,    "_primary_term" : 1}
Document indexing... checked! We have indexed our first document to our test-csv index, all shards responded correctly. We have indexed a very simple json document with only one field, but you can index basically any document (from the size, depth etc perspectives) as long as it is valid JSON — but try to keep it under 100MB per doc :). Now you can easily retrieve a single document by its id with Get API. I’ll leave that up to you.
Now obviously you don’t want to index your documents one by one with curl. The more when you probably already have this data somewhere in your application, database or storage drive. For these purposes there are official Elasticsearch clients for various programming languages ie Java, JS, Go, Python etc. You can grab any of these based on your technology stack and use its APIs to perform indexing/deleting/updating/... actions programmatically.
But if you just have the data and want to quickly get them in you have two options... I took the Python client and prepared very simple CLI script that allows you to index data from given file — supporting CSV format and NDJSON format (newline delimited JSON which is often used for logging, streaming processing etc.). Take it as a possible way how to tackle your data “problem” and write something better based on your needs :). You can browse many other methods and helpers matching the Elasticsearch APIs it offers in the client docs. In any way don’t forget to pip install elasticsearch (to install the module) before using the script.
Code/script freely available in this repo:
github.com
I won’t describe the whole code contents (as it always could be written better :)), but there are two relevant parts in the code:
1.It utilizes concept of Python generator as it goes line by line through the given file and yields one preprocessed element/row for the indexing execution. This “element-wise” operation was picked on purpose to ensure memory-efficiency to be able to process even large files.
def _csv_generator(self, es_dataset, **kwargs):    with open(es_dataset.input_file) as csv_file:        csv_dict_reader = csv.DictReader(csv_file)        for cnt, row in enumerate(csv_dict_reader):            yield self._prepare_document_for_bulk(es_dataset, row, cnt)
2. On the other hand as the actual indexing operation is somewhat costly it utilizes the BULK API to index in multiple-document chunks (default 500 items) which is aligned with recommendations for tuning for speed). For this operation it uses one of the provided helpers (which are convenience abstractions on top of the raw api) by the python elasticsearch client — elasticsearch.helpers.streaming_bulk.
for cnt, response in enumerate(streaming_bulk(self.client, generator, chunk_size)):    ok, result = response
After execution it is necessary to use the refresh API to make the documents immediately searchable.
self.client.indices.refresh(index=es_dataset.es_index_name)
From the data perspective there are wide options nowadays how to get some interesting data set (Kaggle datasets, AWS datasets or other curated lists of open datasets), but as we will want to demonstrate also some text searching capabilities let’s test/showcase our script on something from NLP area — nice collection can be found here. We can use for example Jeopardy questions dataset as it is not a usual demo dataset. Note: if you use this set also consider removal of whitespaces from the header to have more concise column/field names.
Now we can run following command in terminal and watch the stdout logs (for testing purposes I left the DEBUG level on):
python3 load_csv_or_json_to_elasticsearch.py _data/JEOPARDY_CSV.csv test-csv
In the end we should receive a following message:
2019–08–18 17:03:19,013 — load_csv_or_json_to_elasticsearch.py — INFO — Script execution FINALIZED in 44.15356087684631 seconds. Dataset from file _data/JEOPARDY_CSV.csv loaded to Elasticsearch at localhost:9200. Number of documents processed: 216929.
The execution took 44 seconds which is not that fast, but as we are running everything on a single machine it is (must be :)) acceptable. Actually quite a bit of that is the cost of the 3+2 sharding so so for test if go just with 1+0 (and single node) we are under 25 seconds. It could probably be further improved by self-optimized management of the chunks sizing as recommended in tune for speed documentation (ie starting at 100 and then gradually increased until the speed plateau), so you can do it as an interesting exercise. On the other hand if we were to index the documents one by one we are up to beautiful 1411 seconds :), so it seems the bulk api makes some sense.
Now we can recheck the /_cat/indices?v API to make sure all of our data are in place.
INDEX MAPPING/SETTINGS: check the data structure of your index with Get Mapping API and index settings with Get Settings API — there are various settings options.
INDEX TEMPLATING: for usecases where similar indexes get created on an ongoing basis (ie on some time periods) it is convenient to use a predefined structure for that — this is case for Index Templates
INDEX CHANGES: you can move the data between indexes with Reindex API (useful when you need to make changes in sharding — which is not possible after creation), this can be used in conjunction with Index Aliases that allow to create a “nickname” to which we can point our queries (this layer of abstraction allows to then change the indexes behind the alias more freely)
ANALYSIS: when we index or full-text search the query goes through the analysis process, read more about the Analyzers and its core building blocks character filters, tokenizers, and token filters or create your custom analyzer.
CLIENTS: official Elasticsearch clients for various programming languages ie Java, JS, Go, Python etc.
DATA INGESTION: usually with time-based data (such as logs, monitoring etc.) you might consider using other tools from the Elastic Stack such a Beats (multi-purpose lightweights data shippers) or/and Logstash for more sophisticated data pre-processing before indexing to Elasticsearch
MEMORY/STORAGE: if you find yourself with growing amounts of indexed data running low on disk space you might consider Index Lifecycle Management that allows based on defined rules to change or delete indices. Or/and if you are low on memory you can consider index freezing that will move various overhead data structures off the heap. Or just buy more disk space and memory :)
PERFORMANCE: tune for indexing speed or efficient disk usage
Now that we have indexed our data we can finally search and browse through our data. Elasticsearch uses specific JSON-based language Query DSL (Domain Specific Language) to perform various complex queries across the indexed data. As there are many options in the Query DSL world of Elasticsearch we will just go through couple and let you try the rest.
The simplest way to search in Elasticsearch though is without the Query DSL just providing the searched string as a query parameter in URI Search which is useful for quick curl tests as Elastic itself says :). So let’s start with that and search for documents consisting the term impressionism.
curl localhost:9200/test-csv/_search?q=question:impressionism
returns:
{    "took": 10,    "timed_out": false,    "_shards": {        "total": 3,        "successful": 3,        "skipped": 0,        "failed": 0    },    "hits": {        "total": {            "value": 4,            "relation": "eq"        },        "max_score": 13.689855,        "hits": [(...)        {            "_index": "test-csv",            "_type": "_doc",            "_id": "68959",            "_score": 12.518969,            "_source": {                "show-number": "4943",                "air-date": "2006–02–22",                "round": "Double Jeopardy!",                "category": "ART & ARTISTS",                "value": "$1200",                "question": "Impressionism got its name thanks to his painting \"Impression: Sunrise\"",                "answer": "(Claude) Monet"            }        }(...)
We can see that the query on our 216k documents took 10ms, has gone through all 3 shards successfully and 4 documents with the given term were found. The actual results are under the hits object in hits array. For each document ie hit we have received along with the actual contents in the _source field also the _score field which is a calculation of relevance
Now let’s use the Query DSL to search across multiple fields. For this purpose we will use the Multi-match query that allows exactly that. As can be seen in the linked docs there are various options to play with conditions, relevance calculation and other params. Let’s keep the topic and search for Claude Monet. We will specify the phrase as a type as we want to match both, Elasticsearch would otherwise also return documents matching only one of the terms (Claude but no Monet).
curl -X GET \  http://localhost:9203/test-csv/_search \  -H 'Content-Type: application/json' \  -d '{  "query": {    "multi_match" : {      "query":    "Claude Monet",       "fields": [ "category", "question", "answer" ],      "type": "phrase"     }  }}'
returns:
{"took": 16,    "timed_out": false,    "_shards": {        "total": 1,        "successful": 1,        "skipped": 0,        "failed": 0    },    "hits": {        "total": {            "value": 36,            "relation": "eq"        },        "max_score": 22.763008,        "hits": [        {            "_index": "test-csv",            "_type": "_doc",            "_id": "171125",            "_score": 22.763008,            "_source": {                "show-number": "2918",                "air-date": "1997–04–16",                "round": "Jeopardy!",                "category": "CROSSWORD CLUES \"I\"",                "value": "$400",                "question": "Claude Monet or Rich Little (13)",                "answer": "Impressionist"            }        },(...)
We have now 36 hits and the top 10 were returned under the hits field. Btw. the first result... I wonder what would Claude say to his fellow’s impressions :).
For the third query let’s go even more complex and combine the above two searches in a way that we seek again for Claude Monet (that’s a MUST) but also we value more result matching also the term impressionism (that’s a SHOULD)... The must/should stuff might seem weird to you but as we are going to use the Boolean query for our search we will be using exactly these terms. Bool query is the default from so called Compound queries that allow combining multiple leaf queries (which are the actual queries matching some field) or other compound queries.
curl -X GET \  http://localhost:9202/test-csv/_search \  -H 'Content-Type: application/json' \  -d '{  "query": {    "bool" : {      "must" : {        "multi_match" : {            "query":    "Claude Monet",             "fields": [ "category", "question", "answer" ],            "type": "phrase"         }      },      "should" : {           "match" : { "question" : "impressionism" }       }    }  }}'
returns:
{    "took": 6,    "timed_out": false,    "_shards": {        "total": 3,        "successful": 3,        "skipped": 0,        "failed": 0    },    "hits": {        "total": {            "value": 36,            "relation": "eq"        },        "max_score": 28.663502,        "hits": [        {            "_index": "test-csv",            "_type": "_doc",            "_id": "68959",            "_score": 28.663502,            "_source": {                "show-number": "4943",                "air-date": "2006–02–22",                "round": "Double Jeopardy!",                "category": "ART & ARTISTS",                "value": "$1200",                "question": "Impressionism got its name thanks to his painting \"Impression: Sunrise\"",                "answer": "(Claude) Monet"            }        },        {            "_index": "test-csv",            "_type": "_doc",            "_id": "171125",            "_score": 22.763008,            "_source": {                "show-number": "2918",                "air-date": "1997–04–16",                "round": "Jeopardy!",                "category": "CROSSWORD CLUES \"I\"",                "value": "$400",                "question": "Claude Monet or Rich Little (13)",                "answer": "Impressionist"            }        },(...)
We can see that we have received as a top score the original result as it is the only one matching all of our terms in the exact form.
QUERY DSL: check all the various options for the full-text queries
QUERY CONTEXT & FILTER CONTEXT: does the document match our query vs how well it matches
LEAF QUERIES & COMPOUND QUERIES: with Leaf queries look for a particular value in a particular field, such as the match, term or range queries, on the other hand with Compound queries you can wrap other compound or leaf queries together
AGGREGATIONS: instead of working with the data on a “document-level” basis you can aggregate your results to different types of aggregate outcomes such as buckets, metrices, matrixes, pipelines with wide range of options
COMMON OPTIONS: see the various options that can be applied to all of the REST APIs or specifically to the search query body
PERFORMANCE: tune for the max search speed
SCORE TUNING: you can play with relevance score with Function score query or Script score query (allowing custom script for score calc) or check out other tunning options (such as rank_feature) to make the results more relevant for your case
HIGHLIGHTING & SUGGESTERS: there is plentiful of other cool features such as Completion suggester for auto-complete/search-as-you-type functionality or Highlighting for wrapping the results in html tags to emphasise where the matches are
Up until now we have been interacting directly with our Elasticsearch cluster. In the first chapter via python client but later mostly with simple terminal cURL commands. cURL is fine for quick tests and putting things together but when you want to perform some data exploration with plotting, filtering etc. or need some tool for administration checks and updates on your cluster and indexes you might want to choose some more convenient UI-based app. Which is exactly what Kibana is or as Elastic puts it window to Elasticsearch. Without further ado let’s add it to our setup and see what we are up to:
docker run --rm --link esn01:elasticsearch --name kibana --network elastic-network -p 5601:5601 docker.elastic.co/kibana/kibana:7.3.0
The Kibana docker params are quite simple. We basically need to do two things — add Kibana to our docker network (with — network) and link it with our Elasticsearch cluster (eg. with — link pointed towards our esn01 node), we also keep the default port 5601 and pick the same version 7.3. as we are running our Elasticsearch on — must do for compatibility purposes. Note: these can also be overridden with bind-mounted custom kibana.yml or via environment variables. After it loads up you have your Kibana instance running at: http://localhost:5601 so don’t waste your time and check it out :). It will take you to your Home section and you can start browsing the Kibana neighbourhood.
Our goal is to start working with the data quickly so we will head straight to Management -> Index Patterns and select create our first index pattern. From the Kibana usage perspective the index patterns are likely the most important concept. They represent a logical structure over the Elasticsearch indexes and which tells Kibana what indexes (and related fields) to work with. Index pattern can match the name of a single index, or include a wildcard (*) to match multiple indices. So let’s create test-* index pattern (that shall match our test-csv index).
In the next step we will select a timestamp field which will be later used for time-based filtering. This can be omitted by selecting I don’t want to use Time Filter but if you have any timestamp field in your data I recommend not to so that we can do time-series visualizations. Select Create and we are done. It will take you to the index pattern overview page where you have list of all your fields (which can be refreshed in top-right corner if you added some more fields). You can change the formats of the individual fields and you can also add new Scripted (but as Elastic recommends for performance purposes Please familiarize yourself with script fields and with scripts in aggregations before using scripted fields.).
We can do other sorts of Elasticsearch administration/configuration stuff in the Kibana Management application eg. we can check out Index management section with info/actions on our indices (eg. freeze, delete, merge etc.) or setup automatic jobs and rules for Index Lifecycle Management (that will do this for us). On a last note all Kibana settings gets stored to Elasticsearch in the .kibana system indices so thanks to our docker volumes these will be preserved after restarts along with our other data.
The go-to section when you want to browse, query and filter your data is the Discover application (first in the left menu). When you open it you will see our default index pattern that we have created and the individual data points.
Here you can:
filter for required time range (remember our data is from 1984–2012)
filter out for required field or value (tip: use magnifier symbol next to the field or value)
use Kibana Query Language to perform searches on the data ie category: STUPID ANSWERS
see basic field statistics
view your single document data and list other document in its context (ie from time perspective)
Kibana is also useful when you’re in need to visually present your data findings or to provide interactive performance/status/metrics/KIPs overviews of the area you are dealing with. For these purposes you can create visualizations in the Kibana Visualize app and group these visualizations on rich Dashboards that can be also provided with explanatory markdowns etc. All of this is built on top of your indexed data and allows in real-time for further filtering and customizable views eg. for specific time-range.
As our showcase dataset is not really numeric in its nature we can’t really utilize all the options but let’s crate at least couple of document-counts visualizations and put them on a dashboard to have the sense of what it is about. So let’s go to the Visualize app (second on the left) and select Create new visualization.
Let’ do a count of documents (ie Questions we have in our data) grouped by year. Pick a Vertical Bar and select our test-* index pattern through which the data will be pulled. Then let’s add to Buckets and X-axis bucket with Date Histogram Aggregation, pick our air-date field and select a Yearly interval. Optionally you can drop partial buckets and in the top-right corner you should see “play arrow” button which will apply defined settings to our visualization. And... you’re done.
To put this on a Dashboard go to the Dashboard app (bellow Visualize) and Create new dashboard. In the top menu you have an Add option so use it to add a panel with your visualization. And... you’re done. Now you can try couple of these and I’m sure you can do much better than I did with my simple dashboard bellow :).
CONFIGURATION: at some point you will find the need to change other Kibana configurations so linked is a overview of the options
SECURITY: see the first What next section and check the linked resources as these are obviously must apply also for Kibana (generally there are equivalent plugins or native-capabilities also for Kibana)
VISUALISATION: create a visualization in the Visualize app, put multiple visualizations together with interactive Dashboard or share even more visually customizable reports with Canvas, if you have a GeoJSON data put the on Maps
SPACES: uses spaces to separate out different data areas/domains etc.
MONITORING: Stack Monitoring app is very useful especially when dealing with higher number of instances of the stack tools (having loads of servers with Beats deployed, multiple nodes in the cluster etc.)
OTHER KIBANA APPS: there are plenty depending on your stack setup eg. Uptime used with Heartbeats for endpoint monitoring
DEV TOOLS: use Dev Tools when liking the Kibana UI but still want to perform some request-based stuff
CSV FILE LOAD: you can perform the same CSV/NDJSON ingestion (we have done with our Python script) via still quite new Kibana tool (still with Experimental status) File Data Visualizer that allow index data up to 100MB — very good for quick data checks.
SAMPLE DATA: if you just want to play with Kibana use provided sample data
It wouldn’t be a full (distributed) picture without seeing a master election. You can play with these easily in Docker, but remember to put all back up afterwards:). We check who is the current master (eg. /_cat/nodes remember) and to kill it we can run:
docker stop esn01
This will immediately cause an exception in our cluster and initiate a master elections. Except the apparent exception (about the failed connection to the master) you can see in the logs that both nodes became candidates for the master (both are master-eligible as we already know) and execution of the elections with two candidating nodes. Everything happened under 200ms (for real life scenario also add some network overhead etc.).
{“type”: “server”, “timestamp”: “2019–08–18T18:31:15,485+0000”, “level”: “INFO”, “component”: “o.e.c.s.ClusterApplierService”, “cluster.name”: “stanislavs-docker-cluster”, “node.name”: “esn02”, “cluster.uuid”: “w16btQPhSQ-iE96DSM6PnQ”, “node.id”: “yPAolS5gTxes0mcltF5WvA”, “message”: “master node changed {previous [{esn01}{EHEDowSVRMenU2BaVM442g}{thFwnNprQs-QwOx75KUbUg}{172.18.0.2}{172.18.0.2:9300}], current []}, term: 1, version: 5, reason: becoming candidate: onLeaderFailure” }(...){“type”: “server”, “timestamp”: “2019–08–18T18:31:15,626+0000”, “level”: “INFO”, “component”: “o.e.c.s.MasterService”, “cluster.name”: “stanislavs-docker-cluster”, “node.name”: “esn03”, “cluster.uuid”: “w16btQPhSQ-iE96DSM6PnQ”, “node.id”: “rLAI506ATr6lvqw-aiKgHg”, “message”: “elected-as-master ([2] nodes joined)[{esn03}{rLAI506ATr6lvqw-aiKgHg}{pf-3B_9GTGayL4KzmGkqIQ}{172.18.0.4}{172.18.0.4:9300} elect leader, {esn02}{yPAolS5gTxes0mcltF5WvA}{kadMA9eFTO2WR4v0t4aXzQ}{172.18.0.3}{172.18.0.3:9300} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 2, version: 6, reason: master node changed {previous [], current [{esn03}{rLAI506ATr6lvqw-aiKgHg}{pf-3B_9GTGayL4KzmGkqIQ}{172.18.0.4}{172.18.0.4:9300}]}” }
Afterwards you can request the /_cat/nodes?v again to see who is the new boss. When you’re done you can clean things up for the fresh start with docker system prune — volumes which will remove unused containers, volumes, networks etc.
That’s it... we went through a simple “end-to-end” process with Elasticsearch starting with Docker setup, getting the data in the cluster and performing basic search queries. We then also added Kibana to our stack to have some browsing and administration comfort and useful visualisations features. From the original goals outlined in the introduction maybe the shortness wasn’t really reached :) but hopefully you gained some understanding what the technology can offer to have a basis for decision on how to eventually deploy it to your architecture. Feel free to share any additional interesting resources that can be appended to the What next sections lists.