ELK stack using Docker Compose
By: Amol Kokje
Tags: Elasticsearch, Kibana, Docker, Elk, Docker Compose

The ELK stack is a very widely accepted platform for data analytics. To comply with latest DevOps processes and mindset, Elasticsearch.co has provided docker images to deploy the application code in matter of few seconds. You can use these images as is, or customize them to your needs. You can add startup code, init scripts, etc to customize them as you wish for your environment.

In my work experience, I have seen a lot of different ways in which people deploy these docker containers — most common being shell scripts or python scripts which wrap a bunch of docker CLI and provide a wrapper with single-command deploy. Well, that works, but I don’t think that’s the best way to do it, when you already have Docker Compose! Not only do I feel that its the best, fastest, latest and most maintainable way to do it, it already has all the features the scripts are trying to build!

Plus, using docker compose enables you to align with the paradigm of Infrastructure as Code!

In this post, I am going to go over how I am using docker to quickly and easily spin up my Kibana and Elasticsearch environment. Deployment of environment is one aspect, and the other is being able to separate the configuration code from application code, and enabling a way to back up data for recovery.

Architecture:

As you may notice in the diagram for docker on the left, my application code is deployed in docker containers, which are configured using internal network and startup configuration scripts. The data from my elasticsearch cluster is mounted to a local drive. I choose to use this style, so that I can back up this drive using a cron job to my favorite backup AWS S3, where I can enable incremental backups. The other benefit is that I can use this backup to spin up the exact same environment in any other on-perm or virtualization environment of my choice, without having to re-index all the data, and needing to create all my Kibana dashboards all over again.

Elasticsearch, Kibana, Docker Compose

Docker Compose:

The above architecture(on left side in the Docker section) may seem complex to deploy, but its actually not that hard. With the awesome docker documentation, and few trial-error, I was able to write the basic code below. Each section in this code defines a micro-service, that is ‘contained’ in a docker container. You can specify the container-host port mapping, health check, and also define dependencies which are evaluated by docker engine during run-time to determine the deployment sequence. As you can see, I have also defined an internal network for the containers to communicate with each other, which is a pretty cool Docker feature, since using this you can setup your own isolated networks for the containers to communicate, and have tight control on it! :-)

All the original code files for this blog are located here. For convenience, the main code file is pasted below.

# all constants can be defined in the .env file in the same location as docker-compose.yml file

version: "2.1"

services: # For simplicity, running only a single node here

elasticsearch:

container_name: elasticsearch

hostname: elasticsearch

image: "docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION}"

environment:

- cluster.name=es-cluster

- node.name=es-node-1

- path.data=/usr/share/elasticsearch/data

- http.port=9200

- http.host=0.0.0.0

- transport.host=127.0.0.1

- bootstrap.memory_lock=true

- "ES_JAVA_OPTS=-Xms${ES_JVM_HEAP} -Xmx${ES_JVM_HEAP}"

mem_limit: ${ES_MEM_LIMIT}

ulimits:

memlock:

soft: -1

hard: -1

volumes:

- ${ES_MOUNT_DRIVE}:/usr/share/elasticsearch/data

ports:

- '9200:9200'

networks:

- 'elk_stack'

healthcheck:

test: ["CMD", "curl","-s" ,"-f", "-u", "elastic:${ES_PASSWORD}", "http://localhost:9200/_cat/health"] # Kibana container. localhost 5601 (ensure this is open on host) --> container 5601. Deployed after Elasticsearch.

kibana:

container_name: kibana

hostname: kibana

image: "docker.elastic.co/kibana/kibana:${ELASTIC_VERSION}"

environment:

- "ELASTICSEARCH_PASSWORD=${ES_PASSWORD}"

- server.port=127.0.0.1:5601

- elasticsearch.url="http://elasticsearch:9200"

- server.name="kibana"

ports:

- '5601:5601'

healthcheck:

test: ["CMD", "curl", "-s", "-f", "http://localhost:5601/login"]

retries: 6

networks:

- 'elk_stack'

depends_on:

- 'elasticsearch' # Configure Stack container. This container configures the stack after Kibana and Elasticsearch have been deployed.

configure_stack:

container_name: configure_stack

image: docker.elastic.co/beats/metricbeat:${ELASTIC_VERSION}

environment:

- 'ELASTIC_VERSION=${ELASTIC_VERSION}'

- 'ES_PASSWORD=${ES_PASSWORD}'

- 'ES_DEFAULT_INDEX_PATTERN=${ES_DEFAULT_INDEX_PATTERN}'

volumes:

- './init/configure-stack.sh:/usr/local/bin/configure-stack.sh:ro'

command: ['/bin/bash', '-c', 'cat /usr/local/bin/configure-stack.sh | tr -d "\r" | bash']

networks:

- 'elk_stack'

depends_on:

- 'elasticsearch'

- 'kibana'

Configure Stack:

Ideally, the first two sections are enough, but in this example, I have defined another short-lived container which you can use to run a bunch of commands to configure your stack as you wish to! If you are interested to dive deep, look up the whole source code here.

Data Backups, Disaster Recovery:

In any deployments, the server application code should be managed separately from the data i.e. there should not be any tight coupling. The problem with tight coupling is that if the server crashes, how do you recover? To solve this, you can mount the data directory of elasticsearch to a local drive, and back it up regularly(I use AWS S3, but you can use a mounted drive, USB hard drive, etc. whichever you prefer). So, when something very bad happens and your server crashes, all you need to do is deploy a new stack (few seconds if the images are available locally), and mount drive again (which is already included in the code, so don’t need to do anything! :-) :-)) i.e. system is back live again in a few minutes at max.

Since I have started using docker compose for my local deployments, I have never went back. It just makes is so easy to deploy and maintain your services/servers in code. Once you start playing with it, I am sure you will love it too! Here ELK stack is just one example, but this can be extended to any deployment.

Follow us on Twitter 🐦 and Facebook 👥 and join our Facebook Group 💬.

To join our community Slack 🗣️ and read our weekly Faun topics 🗞️, click here⬇