Monitoring kubernetes with prometheus(Part-1) — 100daysoflearning
By: Saiyam Pathak
Tags: Grafana, Monitoring, Linuxacademy, Prometheus, Kubernetes

Combining DAY 11, 12 & 13

This Post is going to be interesting as I am going to talk alot about prometheus and Grafana. So recent Linuxacademy course “monitoring kubernetes with prometheus” is actually very practical course and answers for most of the questions like how to setup Prometheus for Kubernetes cluster and how to do the monitoring.

Starting off the course gives the introduction on how to setup a cluster. This is just the commands more than the explanation.Below is the list of commands copied from instructor TRAVIS this is for one master and one worker node.

“””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””

swapoff -a

====================

vi /etc/fstab

Comment out swap

#/root/swap swap swap sw 0 0

=====================

Add the Kubernetes repo

cat << EOF > /etc/yum.repos.d/kubernetes.repo

[kubernetes]

name=Kubernetes

baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64

enabled=1

gpgcheck=1

repo_gpgcheck=1

gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

exclude=kube*

EOF

=================

Disable SELinux

setenforce 0

================

Permanently disable SELinux:

vi /etc/selinux/config

Change enforcing to disabled

SELINUX=disabled

================

yum install -y kubelet-1.11.3 kubeadm-1.11.3 kubectl-1.11.3 — disableexcludes=kubernetes

================

systemctl start kubelet && systemctl enable kubelet

================

cat << EOF > /etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

EOF

================

sysctl — system

================

Create kube-config.yml:

vi kube-config.yml

Add the following to kube-config.yml:

apiVersion: kubeadm.k8s.io/v1alpha1

kind:

kubernetesVersion: “v1.11.3”

networking:

podSubnet: 10.244.0.0/16

apiServerExtraArgs:

service-node-port-range: 8000–31274

==========================

Initialize Kubernetes

kubeadm init — config kube-config.yml

==========================

Copy admin.conf to your home directory

mkdir -p $HOME/.kube

cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

chown $(id -u):$(id -g) $HOME/.kube/config

==========================

Install flannel

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml

Patch flannel

==========================

vi /etc/kubernetes/manifests/kube-controller-manager.yaml

Add the following to kube-controller-manager.yaml:

— allocate-node-cidrs=true

— cluster-cidr=10.244.0.0/16

systemctl restart kubelet

=================

Setting up the Kubernetes Worker

Do same as the master and then below command:

kubeadmin join < MASTER_IP >:6443 — token < TOKEN > — discovery-token-ca-cert-hash sha256:< HASH >

On the master node, test to see if the cluster was created properly.

================

Get a listing of the nodes:

kubectl get nodes

“””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””

So above should setup kubernetes cluster with one master and one worker node given that you have linux instance with centos and Docker installed.

Next we directly jump on Prometheus Architecture explaining about:

client libraries for application lever monitoring , exporters for application we don’t have source code for , now prometheus need to know how to access exporters and targets so we can do by prometheus config or service discovery :

service discover for kubernetes API

scraping a request is made to http end point and the result is parsed and sent to storage.

Next section we will setup prometheus on kubernetes cluster. This is a very practical lesson and you might not fully understand all the things as you see because you are expected to have good amount of kubernetes knowledge. So we go through the below steps (taken form the lecture itself):

A metrics namespace for our environment to live in

A ClusterRole to give Prometheus access to targets using Service Discovery

A ConfigMap map that will be used to generate the Prometheus config file

A Prometheus Deployment and Service

Kube State Metrics to get access to metrics on the Kubernetes API.

You can clone the YAML files form Github.

For all the yml files you need to do kubectl apply -f <filename>

For config file you need to add the ip’s of the master and the worker(private is fine).

you can access the prometheus dashboard at :Access Prometheus by visiting https://<MASTER_IP>:8080.

Next up we have the explanation for the prometheus configuration file which can seem too much to grasp at first but as you try to understand peacefully with open mind , things will start making sense. Everything can be linked wrt the documentation :Prometheus Configuration Documentation.

Next up we have the grafana setup from the same github repo mentioned earlier you can now move on to the grafana folder and start deploying the grafana files using same kubectl apply -f <filename>. If you have followed till now you have everything setup but you will be shown the node exporter as down now the below steps have to be performed in order to make the node exporter up (Content copied from the Lecture notes):

Create the Prometheus user:

adduser prometheus

Download Node Exporter.

========================================cd /home/prometheus

curl -LO “https://github.com/prometheus/node_exporter/releases/download/v0.16.0/node_exporter-0.16.0.linux-amd64.tar.gz"

tar -xvzf node_exporter-0.16.0.linux-amd64.tar.gz

mv node_exporter-0.16.0.linux-amd64 node_exporter

cd node_exporter

chown prometheus:prometheus node_exporter

vi /etc/systemd/system/node_exporter.service

/etc/systemd/system/node_exporter.service:

========================================

[Unit]

Description=Node Exporter

========================================

[Service]

User=prometheus

ExecStart=/home/prometheus/node_exporter/node_exporter

========================================

[Install]

WantedBy=default.target

Reload systemd:

========================================

systemctl daemon-reload

Enable the node_exporter service:

========================================

systemctl enable node_exporter.service

Start the node_exporter service:

========================================

systemctl start node_exporter.service

Add the following to config-map.yaml:

job_name: ‘node_exporter’

static_configs:

— targets: [‘<KUBER_MASTER_PRIVATE_IP>:9100’, <UBER_WORKER_PRIVATE_IP>:9100]

========================================

Do above on both master and worker. In order to have Grafana Dashboard you need to import the JSON from the link : Kubernetes Nodes Dashboard

and the grafana Dashboard will be working.

For now I am leaving here next few days I will complete this stuff and add screenshots and final steps and my feedback on the whole. Till now this course is really very practical with minimal theory so you need to be very focused to understand all the stuff.

Saiyam Pathak

https://www.linkedin.com/in/saiyam-pathak-97685a64/

https://twitter.com/SaiyamPathak