ELK deployment in kubernetes
By: Akash Tangde
Tags: Elasticsearch, Eck, Beats, Kubernetes, Kibana

This article describes the steps to deploy ELK components such as elasticsearch , kibana, apm-server , beats agents on to the kubernetes cluster. Though this whole setup is tested successfully , necessary tuning must be done in order to make it production ready.

Elastic Cloud on Kubernetes (ECK) built on kubernetes operator pattern extends the basic Kubernetes orchestration capabilities to support the setup and management of Elasticsearch, Kibana, APM Server, Enterprise Search, and Beats on Kubernetes.

With Elastic Cloud on Kubernetes we can manage critical operations related to ELK, such as:

Managing and monitoring multiple clusters Scaling cluster capacity and storage Performing safe configuration changes Securing clusters and agents with TLS certificates Setting up hot-warm-cold architectures with availability zone/environment awareness

Deploying ECK on kubernetes cluster

Steps to install

· Install custom resource definitions and the operator with its RBAC rules:

· Monitor the operator logs:

kubectl -n elastic-system logs -f statefulset.apps/elastic-operator

Above steps will create all the resources needed for ECK operator. A dedicated namespace called elastic-system will be created with one ECK pod running in it. For further details refer documentation

Deploying Elasticsearch Cluster nodes as Stateful Sets

Elasticsearch needs persistence in storage volumes to store its workloads. This feature is provided by kubernetes’ StatefulSets. Other feature that justifies using elasticsearch nodes as Statefulset is , pods created as statefulsets are not interchangeable and each maintain persistent identifier across any rescheduling.

Pre-requisites for deploying the elasticsearch cluster:

1. Load Balancer should be configured so that elasticsearch http service can use it as it is going to be load balancer type of service

2. Storage class which allows for volume expansion must be available.

Manifest file:

Deployment specifications:

1. This is elasticsearch kind of deployment which will be managed by ECK operator

2. Name of the elasticsearch cluster is “quickstart”

3. Elasticsearch cluster version is 7.13.4

4. Elasticsearch service which will be receiving data from beats and apm server is of load balancer type. Docs

5. It will be 2 node elasticsearch cluster with each node having master, data, ingest, ml role

6. 4Gb of heap memory is allocated

7. Atleast 16 Gb of RAM is made available to elasticsearch cluster

8. 500Gb of disk space is allocated to elasticsearch cluster using storage class “essc” which allows volume expansion . Docs

Steps to get cluster health and secrets

· To get overview of elasticsearch cluster’s health, version and number of nodes

kubectl get elasticsearch

· To see the pods running

kubectl get pods --selector='elasticsearch.k8s.elastic.co/cluster-name=quickstart

· To get the elasticsearch http service

kubectl get service quickstart-es-http

· To get elasticsearch password

PASSWORD=$(kubectl get secret quickstart-es-elastic-user -o go-template='{{.data.elastic | base64decode}}')

· To see if cluster is running

curl -u "elastic:$PASSWORD" -k https://<load-balancer-externalIP>:9200

· To get the tls certificate from the secret defined by ECK for accessing elasticsearch http service (this certificate is useful for beats agents to access elasticsearch cluster from outside the kubernetes cluster) Docs

kubectl get secret quickstart-es-http-certs-public -o go-template=’{{index .data “tls.crt” | base64decode }}’

Deploying Kibana instance

Kibana can be deployed as custom resource managed by ECK. Its upgradation, connectivity with elasticsearch using basic authentication and ssl authentication will be taken care by ECK.

Manifest file:

Deployment Specifications:

1. Name of the kibana instance is “quickstartkb”

2. Kibana version is 7.13.4 which should be same as elasticsearch version

3. Elasticsearch cluster name, with which kibana will connect with the help ECK operator

Steps to access Kibana:

· Get the service load balancer external IP using

kubectl get service quickstartkb-kb-http

· Get the kibana password which is same as elasticsearch password using

PASSWORD=$(kubectl get secret quickstart-es-elastic-user -o go-template='{{.data.elastic | base64decode}}')

· Open https://<load-balancerIp>:5601 in the local browser and give “elastic” as username and password from above to access kibana

Beats Agents Configuration

Configuration described here is for beats agents deployed in other kubernetes cluster and so do not come under ECK. The official manifest file for kubernetes is used for each beats agent. Metricbeat configuration is described as follows , similarly heartbeat and filebeat configuration can be done. Autodiscover section needs to be uncommented for filebeat-kubernetes.yaml.

Metricbeat

Metricbeat will collect data related to the Kubernetes nodes, pods, containers, persistent volumes, services, stateful sets, daemonset, replicaset when kubernetes module is activated . System module is enabled to get data related to cpu/memory/disk usage of kubernetes nodes. It will also load dashboards into kibana if kibana access credentials are provided. Below are the steps to configure and deploy metricbeat:

· Download metricbeat-kubernetes.yaml from this

· Create “quickstart-es-http-certs-public” secret in the same namespace where metricbeat daemonset will be running with following command (tls.crt is extracted as described in elasticsearch deployment section)

kubectl create secret generic quickstart-es-http-certs-public --from-file=ca.crt=tls.crt --from-file=tls.crt=tls.crt

· Add following configuration in “Volumes” specification of daemonset to create “certificate” volume where tls certificate will be stored which will be extracted from “quickstart-es-http-certs-public” secret

- name: certificate

secret:

secretName: quickstart-es-http-certs-public

· Mount the the above created volume in “volumeMounts” specification using

- name: certificate

mountPath: /home/<user>/certificate/ca.crt

readOnly: true

subPath: ca.crt

· Put below configuration in “metricbeat-daemonset-config” configmap , under elasticsearch.output.password specification

ssl.certificate_authorities:

- /home/<user>/certificate/ca.crt

· Add the appropriate values for “metricbeat” daemonset

env:

- name: ELASTICSEARCH_HOST

value: https://<IP>

- name: ELASTICSEARCH_PORT

value: "9200"

- name: ELASTICSEARCH_USERNAME

value: elastic

- name: ELASTICSEARCH_PASSWORD

value: <password>

· Deploy metricbeat using following command

Kubectl apply –f metricbeat-kubernetes.yaml

APM Server Configuration

APM server collects the metrics gathered by apm agents attached to the applications to be monitored and ingest this data into elasticsearch which further can be visualized in kibana’s APM app. In this case ECK will operate the APM server and manage its connection with kibana and elasticsearch

Manifest file:

Specifications:

Name of the APM server

2. APM http service type is loadbalancer to make it accessible outside the cluster

3. TLS security for APM http service is disabled as token can be used for authenticating apm agent

4. Name of the elasticsearch cluster to connect to

5. Name of the kibana instance to connect to

Important Secrets and Service

· quickstartapm-server-token : this secret stores the token needed to connect to APM server by apm agent. This token can be extracted by following command

kubectl get secret quickstartapm-apm-token -o go-template='{{index .data "secret-token" | base64decode }}'

· Following is the command to see the APM server http service which runs on port 8200

kubectl get svc

APM agent configuration

APM agent can be deployed as part of the application deployment without affecting the application’s image. Based on service name , apm agent configuration can be done from the Kibana APM app. Following are the steps to configure application’s deployment file to attach APM agent with it

Sample manifest for application with apm agent attached can be found here

Specifications:

APM server URL to send the APM agent data

- name: ELASTIC_APM_SERVER_URL

value: "http://<apm-server-apm-http>:8200"

2. Name of the application where APM agent is attached

- name: ELASTIC_APM_SERVICE_NAME

value: "petclinic"

3. Application packages

- name: ELASTIC_APM_APPLICATION_PACKAGES

value: "org.springframework.samples.petclinic"

4. Environment name where application is deployed

- name: ELASTIC_APM_ENVIRONMENT

value: test

5. Secret in which APM server token is embedded.

- name: ELASTIC_APM_SECRET_TOKEN

valueFrom:

secretKeyRef:

name: quickstartapm -apm-token

key: secret-token

This token can be created using following command

kubectl create secret generic quickstartapm-apm-token --from-literal=secret-token=<token> -n <namespace>

6. APM agent version

- name: elastic-java-agent

image: docker.elastic.co/observability/apm-agent-java:1.25.0

Conclusion

Deploying ECK operator provided by elasticsearch and supported by Kubernetes, is one of the simplest way to deploy ELK components. It also ensures tls layered security between elk components and also between the elasticsearch nodes thereby enhancing data security. Storage requirement changes for elasticsearch cluster can be done easily by changing some values in deployment file and redeploying it. Similarly necessary module activation can be done just by changing the manifest files of the ELK components.