Deploy a Production-Ready On-Premise Kubernetes Cluster | by Joshua Yeung | Towards Data Science

Last year during the pandemic, I had a chance to deploy an on-premise Kubernetes myself. In this article, I would like to mark a few things to remind me and also tell you all that you should know when deploying an on-premise Kubernetes.
Kubespray uses Ansible playbooks as an inventory and provisioning tool to help us to deploy generic Kubernetes clusters. It also handles our configuration management tasks. It automates all the difficult stuff for you to deploy Kubernetes. All you need is just modifying the YAML file config provided by Kubespray. The following guide is copied from the documentation of Kubespray, you may check that out here:
github.com
First, you need to git clone the repository and install the Python requirements using pip.
git clone https://github.com/kubernetes-sigs/kubespray.git# Install dependencies from ``requirements.txt``sudo pip3 install -r requirements.txt
If you cannot install Ansible, check out this link:
docs.ansible.com
There is a sample inventory in the inventory folder. You need to copy that and name your whole cluster (e.g. mycluster). The repository has already provided you the inventory builder to update the Ansible inventory file.
# Copy ``inventory/sample`` as ``inventory/mycluster``cp -rfp inventory/sample inventory/mycluster# Update Ansible inventory file with inventory builderdeclare -a IPS=(10.10.1.3 10.10.1.4 10.10.1.5)CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
Next, you have to modify inventory/mycluster/hosts.yml. Below is an example, it sets three nodes to be masters and three other nodes to be workers. The etcd are located in the same nodes as the masters.
all:  hosts:    node1:      ansible_host: 192.168.0.2      ip: 192.168.0.2      access_ip: 192.168.0.2    node2:      ansible_host: 192.168.0.3      ip: 192.168.0.3      access_ip: 192.168.0.3    node3:      ansible_host: 192.168.0.4      ip: 192.168.0.4      access_ip: 192.168.0.4    node4:      ansible_host: 192.168.0.5      ip: 192.168.0.5      access_ip: 192.168.0.5    node5:      ansible_host: 192.168.0.6      ip: 192.168.0.6      access_ip: 192.168.0.6    node6:      ansible_host: 192.168.0.7      ip: 192.168.0.7      access_ip: 192.168.0.7  children:    kube-master:      hosts:        node1:        node2:        node3:    kube-node:      hosts:        node4:        node5:        node6:    etcd:      hosts:        node1:        node2:        node3:    k8s-cluster:      children:        kube-master:        kube-node:    calico-rr:      hosts: {}
Because Ansible will make an SSH connection to your hosts, your SSH key of the Ansible host must be copied to all the servers or hosts in your inventory. What you need to do is the following:
Generate an SSH Key with ssh-keygenCopy the key to a server using ssh-copy-id -i ~/.ssh/mykey user@host
Generate an SSH Key with ssh-keygen
Copy the key to a server using ssh-copy-id -i ~/.ssh/mykey user@host
Now you need to review and change the parameters of your cluster. There are some aspects you should consider.
# Review and change parameters under ``inventory/mycluster/group_vars``cat inventory/mycluster/group_vars/all/all.ymlcat inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml
The first one is to set up a load balancer for the high availability of your API servers. Load Balancer is used to prevent service failure when one of our master nodes is not working. Normally we would set up at least three masters. You may find more details in the HA endpoints for K8s section in the documentation.
github.com
For a production ready Kubernetes cluster, we need to use an external loadbalancer (LB) instead of internal LB. An external LB provides access for external clients, while the internal LB accepts client connections only to the localhost.
I used HAProxy + keepalived to configure a highly available load balancer. Two VMs are set up to perform the load balancer functionality. The reference I was referring to is from this site:
www.digitalocean.com
Although the article is a bit old, the concept is the same and the procedure is quite similar.
Given a frontend VIP address and IP1, IP2 addresses of backends, here is an example configuration for a HAProxy service acting as an external LB:
#/etc/haproxy/haproxy.cfglisten kubernetes-apiserver-httpsbind <VIP>:8383mode tcpoption log-health-checkstimeout client 3htimeout server 3hserver master1 <IP1>:6443 check check-ssl verify none inter 10000server master2 <IP2>:6443 check check-ssl verify none inter 10000balance roundrobin
And then you need to change the external load balancer config. Again, the config is copied from the HA endpoints for the K8s section in the documentation.
#kubespray/inventory/mycluster/group_vars/all/all.yml##   External LB example config##   apiserver_loadbalancer_domain_name: "elb.some.domain"#   loadbalancer_apiserver:#   address: <VIP>#   port: 8383
If your cluster cannot access the Internet directly and is behind a corporate proxy like mine, you will experience pain in the deployment process... Make sure you add your proxy setting in kubespray/inventory/mycluster/group_vars/all/all.yml
Add your proxy to http_proxy and the domain/IP that you don’t want your cluster to access through the proxy in additional_no_proxy section. (IMPORTANT!)
If your master VMs have two network interfaces and you need to access Kubernetes from another interface other than the one used by Kubernetes, you will need to add supplementary addresses in the config file:
## Supplementary addresses that can be added in kubernetes ssl keys.## That can be useful for example to setup a keepalived virtual IPsupplementary_addresses_in_ssl_keys: [10.0.0.1, 10.0.0.2, 10.0.0.3]
In order to let your user access your service which was deployed in Kubernetes, you need to expose your service. There are three ways to access from outside the cluster: Ingress, Load Balancer, and NodePort. To implement services of type LoadBalancer on-premise, we chose MetalLB to handle network load balancing functionality for us. MetalLB is a load-balancer implementation for bare metal Kubernetes clusters, using standard routing protocols.
It is quite easy to install MetalLB with Kubespray. You just need to modify the addons YAML file and also the k8s-cluster YAML file. Note that the IP range you assigned to MetalLB must be prepared by yourself beforehand.
# kubespray/inventory/mycluster/group_vars/k8s-cluster/addons.ymlmetallb_enabled: truemetallb_ip_range: - “10.5.0.50–10.5.0.99”
Remember to set parameter kube_proxy_strict_arp to be true.
# kubespray/inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml# must be set to true for MetalLB to workkube_proxy_strict_arp: true
Finally, you can deploy Kubespray with Ansible Playbook! Depending on your cluster size and network throughput, it takes around 15–30 minutes for a cluster to be ready. After that, you can now use your Kubernetes cluster!
# Deploy Kubespray with Ansible Playbook - run the playbook as root# The option `--become` is required, as for example writing SSL keys in /etc/,# installing packages and interacting with various systemd daemons.# Without --become the playbook will fail to run!ansible-playbook -i inventory/mycluster/hosts.yaml  --become --become-user=root cluster.yml
We still have a ways to go before we are able to deploy our application to the Kubernetes cluster.
Helm is a package manager for Kubernetes. From the official website of Helm, it states
Helm is the best way to find, share, and use software built for Kubernetes. Helm helps you manage Kubernetes applications — Helm Charts help you define, install, and upgrade even the most complex Kubernetes application.
To install Helm, the easiest method is to install by the script:
$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3$ chmod 700 get_helm.sh$ ./get_helm.sh
You can fetch that script and then execute it locally. It's well documented so that you can read through it and understand what it is doing before you run it.
helm.sh
In order to let your user access your service which was deployed in Kubernetes, you need to expose your service. There are three ways to access from outside the cluster: Ingress, Load Balancer, and NodePort. In production, NodePort is not recommended since it lacks availability. For Load Balancer, we have already implemented using MetalLB. Now we are going to implement Ingress. Please check the details of Ingress on this site:
kubernetes.io
From the description of the official Kubernetes documentation, Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource. An Ingress Controller controls the Ingress resource. The one I used is called the NGINX ingress controller. You may install it easily using Helm.
docs.nginx.com
helm repo add nginx-stable https://helm.nginx.com/stablehelm repo updatehelm install my-release nginx-stable/nginx-ingress
Data in the pod is ephemeral, if you want your data to be persisted, you store your data in a Persistent Volume. From the description of the official Kubernetes documentation, a PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using Storage Classes.
kubernetes.io
Each StorageClass has a provisioner that handles the volume assignment. The one I chose is Longhorn. From the description of the official Longhorn documentation, it states that
longhorn.io
Longhorn is a lightweight, reliable, and powerful distributed block storage system for Kubernetes. Longhorn implements distributed block storage using containers and microservices. It creates a dedicated storage controller for each block device volume and synchronously replicates the volume across multiple replicas stored on multiple nodes. The storage controller and replicas are themselves orchestrated using Kubernetes.
This makes our storage has no single point of failure. It has an intuitive GUI dashboard too, you can view all the details of volumes in it.
The below installation guide is copied from the official Longhorn documentation:
To Install Longhorn
1. Add the Longhorn Helm repository:
helm repo add longhorn https://charts.longhorn.io
2. Fetch the latest charts from the repository:
helm repo update
3. Install Longhorn in the longhorn-system namespace.
kubectl create namespace longhorn-systemhelm install longhorn longhorn/longhorn --namespace longhorn-system
4. To confirm that the deployment succeeded, run:
kubectl -n longhorn-system get pod
You should see all pods are running if it is ready.
You need a registry to store your developed docker image. One of the open-source choices is Harbor.
goharbor.io
Again, you can easily install harbor with Helm chart. The one I used is maintained by bitnami:
github.com
To conclude, I walk you through the whole installation process of an on-premise Kubernetes cluster. Let’s make a simple checklist:
KubesprayAnsible configurationMaster Node HA API Server Load Balancer (HAProxy + keepalived)Proxy settingConfigure kubectl access from outside the clusterService Exposure: Load Balancer and IngressPackage management: HelmStorage: LonghornInternal Image Repository: Registry
Kubespray
Ansible configuration
Master Node HA API Server Load Balancer (HAProxy + keepalived)
Proxy setting
Configure kubectl access from outside the cluster
Service Exposure: Load Balancer and Ingress
Package management: Helm
Storage: Longhorn
Internal Image Repository: Registry
It still only covers a part of a production-ready Kubernetes cluster and I am still exploring. Other important aspects are still missing in this article such as monitoring (like Prometheus & Grafana), security (RBAC/User management), CICD (Argo), etc...
Welcome to the world of Kubernetes!
If you want to know how to prepare for the Certified Kubernetes Application Developer (CKAD) examination, check out this article:
towardsdatascience.com
You may also want to check the affiliated link below.