Application Logging Simplified in Kubernetes: Part 1
By: Gursimran Singh
Tags: Docker, DevOps, Open Source, Kubernetes, Logging

Application Logging Simplified in Kubernetes: Part 1

Node-Level Logging

Logging is one of the most critical parts of all production-grade applications and services.

With the rapid growth of the micro-services architecture, we have many services running in environments that add to the complexity of the collection and unification of the logs.

In this series of blog posts, we will discuss how to collect and store application logs in Kubernetes in a simple but effective way. This is a 3 part series in which we will explore different methods for logging under Kubernetes.

3 approaches can be used for log collection in Kubernetes:

Node-Level Logging Sidecar Container Direct Logging

This first blog post will focus on Node-Level Logging.

Node-Level Logging

The easiest and most embraced logging method for containerized applications is to write to the standard output and standard error streams.

Kubernetes collect all the Logs generated by the containers in the form of std-out and std-err.

Kubernetes stores logs for each container at a node level, which will then be collected by a logging agent and stored in a Logging Back-end for persistence.

Architecture

In this architecture, we will collect container logs and store those logs in Google Cloud Storage (GCS) with Fluentd as a Logging Agent.

Kubernetes cluster generally consists of more than 1 node, so we need to deploy Logging Agent at every node. This can be done with the help of DaemonSet.

A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected.

Setup

You can use the official Fluentd Docker image or create your own.

Step 1: Clone this Repository. This contains all the required files for this project.

File Structure for this repository:

https://gitlab.com/gursimran.singh1/cluster-logging cluster-logging

|

├── docker

│ ├── Dockerfile

│ ├── entrypoint.sh

│ └── fluent.conf

|

├── Kubernetes

│ ├── fluentd-configmap.yaml

│ └── fluentd-demonset.yaml

|

├── .gitignore

└── README.md

Step 2: Create a Docker image for Fluentd with all the required plugins.

Docker folder inside the repository contains 3 files:

Dockerfile : Creates a Docker image for Fluentd

: Creates a Docker image for Fluentd fluent.conf : Default configuration file used by Fluentd

: Default configuration file used by Fluentd entrypoint.sh: Configures and sets the Fluentd container at the runtime.

Build the Docker image and push that image to the Docker registry:

# Go inside docker folder

cd docker # Build docker image

docker build -t <SERVER>/<IMAGE-NAME>:<TAG> . # Push docker image

docker push <SERVER>/<IMAGE-NAME>:<TAG>

Deployment

Configurations will be created as a configmap, which will be mounted as a file in the Fluentd container.

The Fluentd application will be deployed as a Demonset so that it can run on every node.

Kubernetes folder inside the repository contains 2 files:

fluentd-configmap.yaml : Creates configmap of Fluentd configuration.

: Creates configmap of Fluentd configuration. fluentd-demonset.yaml : Deploys Fluentd as Demonset.

Change the highlighted values according to your project:

## Kubernetes/fluentd-configmap.yaml

kind: ConfigMap

metadata:

name: fluentdconf

data:

fluentd.conf: |

<source>



path /var/log/containers/*.log

pos_file /var/log/fluentd-containers.log.pos

time_format %Y-%m-%dT%H:%M:%S.%NZ

tag kubernetes.*

format json

read_from_head true

</source>

<match kubernetes.var.log.containers.**kube-system**.log>



</match>

<match kubernetes.**>



<store>



project {PROJECT_NAME}

bucket {BUCKET_NAME}

object_key_format %{path}%{time_slice}_%{index}.%{file_extension}

path logs/%Y-%m-%d/

store_as json apiVersion: v1kind: ConfigMapmetadata:name: fluentdconfdata:fluentd.conf: | @type tailpath /var/log/containers/*.logpos_file /var/log/fluentd-containers.log.postime_format %Y-%m-%dT%H:%M:%S.%NZtag kubernetes.*format jsonread_from_head true @type null @type copy @type gcsprojectbucketobject_key_format %{path}%{time_slice}_%{index}.%{file_extension}path logs/%Y-%m-%d/store_as json



path /fluentd/logs/temp

flush_mode immediate

timekey 5

time_key_wait 1

timekey_use_utc true

</buffer>

<format>



</format>

</store>

<store>

@type filepath /fluentd/logs/tempflush_mode immediatetimekey 5time_key_wait 1timekey_use_utc true @type json @type std ## Kubernetes/fluentd-demonset.yaml apiVersion: apps/v1

kind: DaemonSet

metadata:

name: fluentd-logger

labels:

k8s-app: fluentd-logging

spec:

selector:

matchLabels:

name: fluentd-logger

template:

metadata:

labels:

name: fluentd-logger

spec:

tolerations:

# this toleration is to have the daemonset runnable on master nodes

# remove it if your masters can't run pods

- key: node-role.kubernetes.io/master

effect: NoSchedule

containers:

- name: fluentd-logger

image: gursimran14/fluentd:latest

resources:

limits:

memory: 200Mi

requests:

cpu: 100m

memory: 200Mi

volumeMounts:

- name: fluentconfig

mountPath: /fluentd/etc

- name: varlog

mountPath: /var/log

- name: varlibdockercontainers

mountPath: /var/lib/docker/containers

terminationGracePeriodSeconds: 30

volumes:

- name: fluentconfig

configMap:

name: fluentdconf

- name: varlog

hostPath:

path: /var/log

- name: varlibdockercontainers

hostPath:

path: /var/lib/docker/containers

Deploy the configmap and demonset in Kubernetes.

# Go inside Kubernetes folder

cd Kubernetes # Apply configmap

kubectl apply -f fluentd-configmap.yaml # Apply Demonset

kubectl apply -f fluentd-demonset.yaml

It will start writing to your GCS Bucket as soon as the logs are generated.

Final Thoughts

This method will only work if your applications create logs in the form of standard output and standard error streams. For Cloud Storage Authentication, you can use OAuth or Service Account.

We will continue to explore approaches for log collection in Kubernetes. Keep up with this blog post series and see which approach makes the most sense for you.

Ready to Accelerate Your Digital Transformation?

At Hashmap, we work with our clients to build better together.

If you’d like additional assistance in this area, Hashmap offers a range of enablement workshops and consulting service packages as part of our consulting service offerings, and would be glad to work through your specifics.

Feel free to share on other channels, and be sure and keep up with all new content from Hashmap here. To listen in on a casual conversation about all things data engineering and the cloud, check out Hashmap’s podcast Hashmap on Tap as well on Spotify, Apple, Google, and other popular streaming apps.

Other Tools and Content You Might Like