Fluentbit Stream Processing with Kubernetes
By: Sudhindra Tirupati Nagaraj
Tags: Fluentbit, Kubernetes, Elasticsearch, Docker, Stream Processing

Fluentbit (https://fluentbit.io/) is becoming increasingly popular as a light-weight alternative to Fluentd for log collection, processing and forwarding in Kubernetes environments. Fluentbit is typically deployed as a DaemonSet(https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) in production environments, where a Fluentbit instance runs on every node in the cluster. In a typical EFK (Elastic Search + Fluentbit + Kibana) stack, this instance tails docker (usually the container runtime) container logs (written to stdout/stderr of containers) on the node (/var/log/containers/*.log). Fluentbit parses these JSON formatted logs using a pre-configured docker json parser, enriches the log message with Kubernetes metadata, using a Kubernetes filter, and outputs that to an Elastic Search index, which can be visualized using Kibana. A recommended pattern is for applications to log in JSON format, so that the log messages at Elastic Search can be deserialized into separate fields, which can be queried/filtered/visualized on Kibana.

As described above, the same Fluentbit instance, running on a node, processes all container logs of all the pods running on that node. So all log messages across the pods get sent to same Elastic Search index. This may be undesirable sometimes because the index schema might need to be different for different applications/services on the Kubernetes cluster. For example, a given field may be a string in one application, and a number in another application. This can lead to parsing errors on Elastic Search when it maps that field to one type, but it receives a log message which has that field set to a different type. Another use-case could be to send log messages of namespace A to Elastic Search (on-prem) and another, say, namespace B, to AWS CloudWatch (off-prem). Yet another use-case may be to have different retention requirements for different applications. Whatever the case may be, we can leverage Fluentbit Stream Processing (https://docs.fluentbit.io/stream-processing/) to create different streams for different Kubernetes namespaces, annotations, labels etc., and forward each stream to a different endpoint configuration.

Let’s consider an example to demonstrate this. This example is implemented on a MacBook, which has Docker Desktop with Kubernetes installed (https://docs.docker.com/docker-for-mac/kubernetes/).

Create 2 pods, hello-pod (which prints “hello world” to stdout indefinitely) and bye-pod (which prints “bye world” to stdout indefinitely).

# cat hello/Dockerfile

FROM alpine:3.7

ENTRYPOINT while :; do echo "hello world"; sleep 2; done # cat bye/Dockerfile

FROM alpine:3.7

ENTRYPOINT while :; do echo "bye world"; sleep 2; done # docker build . -t hello:latest -f hello/Dockerfile # docker build . -t bye:latest -f bye/Dockerfile # cat hello.yaml

apiVersion: v1

metadata:

name: hello-pod

annotations:

purpose: greeting

spec:

containers:

- image: hello:latest

name: hello-container

imagePullPolicy: Never # cat bye.yaml

kind: Pod

apiVersion: v1

metadata:

name: bye-pod

annotations:

purpose: goodbye

spec:

containers:

- image: bye:latest

name: bye-container

imagePullPolicy: Never # kubectl apply -f pod_hello.yaml -n hello

pod/hello-pod created # kubectl apply -f pod_bye.yaml -n bye

pod/bye-pod created # kubectl get pods --all-namespaces | egrep 'hello|bye'

hello bye-pod 1/1 Running 0 81s

bye hello-pod 1/1 Running 0 88s

The pod hello-pod above is created in namespace hello, while the pod bye-pod is created in namespace bye. Each pod prints log messages to stdout as shown below:

# kubectl logs --follow hello-pod -n hello

hello world

hello world

... # kubectl logs --follow bye-pod -n bye

bye world

bye world

...

We will now create the EFK stack to process these log messages. Let’s first deploy Elastic Search and Kibana using “helm” (Installing helm is beyond the scope of this article).

Initialize “helm”:

# helm init

...

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

... # kubectl get pods -n kube-system | grep tiller

tiller-deploy-75f6c87b87-v7mkt 1/1 Running 0 23s

Add “elastic” helm repo:



"elastic" has been added to your repositories # helm repo add elastic https://Helm.elastic.co "elastic" has been added to your repositories

Setup helm charts for Elastic Search and Kibana:

# wget https://raw.githubusercontent.com/elastic/helm-charts/master/elasticsearch/examples/docker-for-mac/values.yaml # helm install --name elasticsearch elastic/elasticsearch -f ./values.yaml NAME: elasticsearch

LAST DEPLOYED: Sun May 3 17:39:07 2020

NAMESPACE: default

STATUS: DEPLOYED RESOURCES:

==> v1/Pod(related)

NAME READY STATUS RESTARTS AGE

elasticsearch-master-0 0/1 Pending 0 0s

elasticsearch-master-1 0/1 Pending 0 0s

elasticsearch-master-2 0/1 Pending 0 0s ==> v1/Service

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

elasticsearch-master ClusterIP 10.101.188.46 <none> 9200/TCP,9300/TCP 0s

elasticsearch-master-headless ClusterIP None <none> 9200/TCP,9300/TCP 0s ==> v1/StatefulSet

NAME READY AGE

elasticsearch-master 0/3 0s ==> v1beta1/PodDisruptionBudget

NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE

elasticsearch-master-pdb N/A 1 0 0s NOTES:

1. Watch all cluster members come up.

$ kubectl get pods --namespace=default -l app=elasticsearch-master -w

2. Test cluster health using Helm test.

$ helm test elasticsearch --namespace=default # helm install --name kibana elastic/kibana

NAME: kibana

LAST DEPLOYED: Sun May 3 17:41:43 2020

NAMESPACE: default

STATUS: DEPLOYED RESOURCES:

==> v1/Deployment

NAME READY UP-TO-DATE AVAILABLE AGE

kibana-kibana 0/1 1 0 0s ==> v1/Pod(related)

NAME READY STATUS RESTARTS AGE

kibana-kibana-57dfc79db8-fmq4s 0/1 ContainerCreating 0 0s ==> v1/Service

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kibana-kibana ClusterIP 10.108.57.206 <none> 5601/TCP 0s # kubectl get services | egrep 'elastic|kibana'

elasticsearch-master ClusterIP 10.101.188.46 <none> 9200/TCP,9300/TCP 158m

elasticsearch-master-headless ClusterIP None <none> 9200/TCP,9300/TCP 158m

kibana-kibana ClusterIP 10.108.57.206 <none> 5601/TCP 155m

Wait a few minutes for Elastic Search/Kibana to come up. Eventually:

# kubectl get pods

NAME READY STATUS RESTARTS AGE

elasticsearch-master-0 1/1 Running 0 4m40s

elasticsearch-master-1 1/1 Running 0 4m40s

elasticsearch-master-2 1/1 Running 0 4m40s

kibana-kibana-57dfc79db8-fmq4s 1/1 Running 0 2m4s

We will now setup Fluentbit:

# kubectl create namespace logging

namespace/logging created

serviceaccount/fluent-bit created # kubectl apply -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-service-account.yaml serviceaccount/fluent-bit created

clusterrole.rbac.authorization.k8s.io/fluent-bit-read created # kubectl apply -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role.yaml clusterrole.rbac.authorization.k8s.io/fluent-bit-read created

clusterrolebinding.rbac.authorization.k8s.io/fluent-bit-read created # kubectl apply -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role-binding.yaml clusterrolebinding.rbac.authorization.k8s.io/fluent-bit-read created

Configure the config map (https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/) used to create configuration for Fluent bit process. Setup 2 streams called hello and bye, one for namespace hello and another for namespace bye respectively. We also forward these 2 streams to different Elastic Search indices, as shown below:

# wget https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-configmap.yaml # cat fluent-bit-configmap.yaml

apiVersion: v1

kind: ConfigMap

metadata:

name: fluent-bit-config

namespace: logging

labels:

k8s-app: fluent-bit

data:

# Configuration files: server, input, filters and output

# ======================================================

fluent-bit.conf: |

[SERVICE]

Flush 1

Log_Level info

Daemon off

Parsers_File parsers.conf

Streams_File streams.conf

HTTP_Server On

HTTP_Listen 0.0.0.0

HTTP_Port 2020



@INCLUDE input-kubernetes.conf @INCLUDE filter-kubernetes.conf @INCLUDE output-elasticsearch.conf input-kubernetes.conf: |

[INPUT]

Name tail

Tag kube.*

Path /var/log/containers/*.log

Parser docker

DB /var/log/flb_kube.db

Mem_Buf_Limit 5MB

Skip_Long_Lines On

Refresh_Interval 10

[FILTER]

Name kubernetes

Match kube.*

Kube_URL

Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token

Kube_Tag_Prefix kube.var.log.containers.

Merge_Log On

Merge_Log_Key log_processed

K8S-Logging.Parser On

K8S-Logging.Exclude Off filter-kubernetes.conf: |[FILTER]Name kubernetesMatch kube.*Kube_URL https://kubernetes.default.svc:443 Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crtKube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/tokenKube_Tag_Prefix kube.var.log.containers.Merge_Log OnMerge_Log_Key log_processedK8S-Logging.Parser OnK8S-Logging.Exclude Off output-elasticsearch.conf: |

[OUTPUT]

Name es

Match hello*

Host ${FLUENT_ELASTICSEARCH_HOST}

Port ${FLUENT_ELASTICSEARCH_PORT}

Logstash_Format On

Replace_Dots On

Retry_Limit False

Logstash_Prefix hello [OUTPUT]

Name es

Match bye*

Host ${FLUENT_ELASTICSEARCH_HOST}

Port ${FLUENT_ELASTICSEARCH_PORT}

Logstash_Format On

Replace_Dots On

Retry_Limit False

Logstash_Prefix bye parsers.conf: |

[PARSER]

Name apache

Format regex

Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$

Time_Key time

Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER]

Name apache2

Format regex

Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$

Time_Key time

Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER]

Name apache_error

Format regex

Regex ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$ [PARSER]

Name nginx

Format regex

Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$

Time_Key time

Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER]

Name json

Format json

Time_Key time

Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER]

Name docker

Format json

Time_Key time

Time_Format %Y-%m-%dT%H:%M:%S.%L

Time_Keep On [PARSER]

Name syslog

Format regex

Regex ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$

Time_Key time

Time_Format %b %d %H:%M:%S streams.conf: |

[STREAM_TASK]

Name hello

Exec CREATE STREAM hello WITH (tag='hello') AS SELECT * FROM TAG:'kube.*' WHERE kubernetes['namespace_name'] = 'hello'; [STREAM_TASK]

Name bye

Exec CREATE STREAM argento WITH (tag='bye') AS SELECT * FROM TAG:'kube.*' WHERE kubernetes['namespace_name'] = 'bye';

Configure DaemonSet YAML like below to update Fluentbit version and Elastic Search host information:

# wget https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-ds.yaml # cat fluent-bit-ds.yaml

apiVersion: extensions/v1beta1

kind: DaemonSet

metadata:

name: fluent-bit

namespace: logging

labels:

k8s-app: fluent-bit-logging

version: v1

kubernetes.io/cluster-service: "true"

spec:

template:

metadata:

labels:

k8s-app: fluent-bit-logging

version: v1

kubernetes.io/cluster-service: "true"

annotations:

prometheus.io/scrape: "true"

prometheus.io/port: "2020"

prometheus.io/path: /api/v1/metrics/prometheus

spec:

containers:

- name: fluent-bit

image: fluent/fluent-bit:1.4.2

imagePullPolicy: Always

ports:

- containerPort: 2020

env:

- name: FLUENT_ELASTICSEARCH_HOST

value: "elasticsearch-master.default.svc.cluster.local"

- name: FLUENT_ELASTICSEARCH_PORT

value: "9200"

volumeMounts:

- name: varlog

mountPath: /var/log

- name: varlibdockercontainers

mountPath: /var/lib/docker/containers

readOnly: true

- name: fluent-bit-config

mountPath: /fluent-bit/etc/

terminationGracePeriodSeconds: 10

volumes:

- name: varlog

hostPath:

path: /var/log

- name: varlibdockercontainers

hostPath:

path: /var/lib/docker/containers

- name: fluent-bit-config

configMap:

name: fluent-bit-config

serviceAccountName: fluent-bit

tolerations:

- key: node-role.kubernetes.io/master

operator: Exists

effect: NoSchedule

- operator: "Exists"

effect: "NoExecute"

- operator: "Exists"

effect: "NoSchedule" # kubectl apply -f fluent-bit-configmap.yaml

configmap/fluent-bit-config created # kubectl apply -f fluent-bit-ds.yaml

daemonset.extensions/fluent-bit created

Verify that Fluentbit comes up and registers the 2 streams we created:

# kubectl get pods -n logging

NAME READY STATUS RESTARTS AGE

fluent-bit-nl267 1/1 Running 0 13s

Fluent Bit v1.4.2

* Copyright (C) 2019-2020 The Fluent Bit Authors

* Copyright (C) 2015-2018 Treasure Data

* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd

* https://fluentbit.io # kubectl logs --follow fluent-bit-t6lx7 -n loggingFluent Bit v1.4.2* Copyright (C) 2019-2020 The Fluent Bit Authors* Copyright (C) 2015-2018 Treasure Data* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd [2020/05/04 01:32:49] [ info] [storage] version=1.0.3, initializing...

[2020/05/04 01:32:49] [ info] [storage] in-memory

[2020/05/04 01:32:49] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128

[2020/05/04 01:32:49] [ info] [engine] started (pid=1)

[2020/05/04 01:32:49] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc port=443

[2020/05/04 01:32:49] [ info] [filter:kubernetes:kubernetes.0] local POD info OK

[2020/05/04 01:32:49] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server...

[2020/05/04 01:32:49] [ info] [filter:kubernetes:kubernetes.0] API server connectivity OK

[2020/05/04 01:32:49] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020

[2020/05/04 01:32:49] [ info] [sp] stream processor started

[2020/05/04 01:32:49] [ info] [sp] registered task: hello

[2020/05/04 01:32:49] [ info] [sp] registered task: bye

We can now view our pods’ logs (hello-pod and bye-pod) on Kibana:

# kubectl port-forward deployment/kibana-kibana 5601

Forwarding from 127.0.0.1:5601 -> 5601

Forwarding from [::1]:5601 -> 5601

Open browser and go to URL: http://localhost:5601

We can see below 2 indices, one for stream hello and another for stream bye:

After creating index patterns:

View log messages for each index:

As seen above, log messages from each Kubernetes namespace, hello and bye, ended up in a different Elastic Search index.

Conclusion

Stream processing in Fluentbit is pretty versatile that allows different workflows possible, based on custom requirements. Configuring stream processing is pretty easy, using familiar SQL statements that can query/filter on JSON fields. Explore more at: https://docs.fluentbit.io/manual/stream-processing/introduction