Upgrading and Scaling Kubernetes cluster in AWS
By: Kubernetes Advocate
Tags: Startup, Kubernetes, Tech, Docker, DevOps

Upgrade K8s cluster

we are needed to upgrade the cluster to stay up with the most recent safety features and bug fixes, further as take pleasure in new options being discharged on AN on-going basis. this is often particularly necessary after we have put in the latest version (for example v1.9)

Here are the topics we will be performing to upgrade the Kubernetes cluster

In-place Upgrade

Create cluster

Edit cluster configuration

Upgrade cluster

Upgrading the Kubernetes cluster on AWS is simple with kops. during this exercise, we are going to demonstrate upgrading the Kubernetes cluster using 2 ways.

In-place Upgrade

Upgrading cluster using in-place is easy with kops. This exercise will set up a Kubernetes cluster with the 1.6.10 version and perform an automatic rolling upgrade to 1.7.2 using kops.

Create cluster

Review steps to create prereqs and make sure KOPS_STATE_STORE and AWS_AVAILABILITY_ZONES environment variables are set. More details about creating a cluster are at Create Kubernetes cluster using kops.

In this chapter, we’ll create a Kubernetes 1.6.10 version cluster as shown:

kops create cluster \

--name example.cluster.k8s.local \

--master-count 3 \

--master-zones ${AWS_AVAILABILITY_ZONES} \

--node-count 5 \

--zones ${AWS_AVAILABILITY_ZONES} \

--kubernetes-version=1.6.10 \

--yes

A multi-AZ deployed cluster ensures that your pods and services don’t see any downtime during cluster upgrades.

Validate the cluster:

$ kops validate cluster --name example.cluster.k8s.local

Validating cluster example.cluster.k8s.local INSTANCE GROUPS

NAME ROLE MACHINETYPE MIN MAX SUBNETS

master-eu-central-1a Master m3.medium 1 1 eu-central-1a

master-eu-central-1b Master m3.medium 1 1 eu-central-1b

master-eu-central-1c Master c4.large 1 1 eu-central-1c

nodes Node t2.medium 5 5 eu-central-1a,eu-central-1b,eu-central-1c NODE STATUS

NAME ROLE READY

ip-172-20-112-170.eu-central-1.compute.internal master True

ip-172-20-117-204.eu-central-1.compute.internal node True

ip-172-20-54-176.eu-central-1.compute.internal master True

ip-172-20-55-115.eu-central-1.compute.internal node True

ip-172-20-63-241.eu-central-1.compute.internal node True

ip-172-20-71-25.eu-central-1.compute.internal master True

ip-172-20-91-30.eu-central-1.compute.internal node True

ip-172-20-93-220.eu-central-1.compute.internal node True Your cluster example.cluster.k8s.local is ready

Check the version of different nodes in the cluster:

$ kubectl get nodes

NAME STATUS ROLES AGE VERSION

ip-172-20-112-170.eu-central-1.compute.internal Ready master 7m v1.6.10

ip-172-20-117-204.eu-central-1.compute.internal Ready node 6m v1.6.10

ip-172-20-54-176.eu-central-1.compute.internal Ready master 7m v1.6.10

ip-172-20-55-115.eu-central-1.compute.internal Ready node 6m v1.6.10

ip-172-20-63-241.eu-central-1.compute.internal Ready node 6m v1.6.10

ip-172-20-71-25.eu-central-1.compute.internal Ready master 7m v1.6.10

ip-172-20-91-30.eu-central-1.compute.internal Ready node 6m v1.6.10

ip-172-20-93-220.eu-central-1.compute.internal Ready node 6m v1.6.10

This shows that each node in the cluster is version 1.6.10.

Edit cluster configuration

Let’s upgrade the existing cluster of 1.6.10 to 1.7.4. Edit the cluster configuration:

kops edit cluster example.cluster.k8s.local

This will open up the cluster configuration in a text editor. Set kubernetesVersion key to the target version, 1.7.4 in our case, and save the configuration file. The previous value of:

kubernetesVersion: 1.6.10

will be changed to:

kubernetesVersion: 1.7.4

Save the changes and exit the editor. Kubernetes cluster needs to re-read the configuration. This can be done by forcing a rolling update of the cluster using the following command:

Note

This process can easily take 30–45 minutes. It is recommended to leave the cluster without any updates during that time.

Apply changes using the command:

kops update cluster example.cluster.k8s.local --yes

It shows the output:

Upgrade cluster

Determine if any of the nodes will require a restart using the command:

kops rolling-update cluster example.cluster.k8s.local

This command shows the output as shown:

NAME STATUS NEED UPDATE READY MIN MAX NODES

master-eu-central-1a NeedsUpdate 1 0 1 1 1

master-eu-central-1b NeedsUpdate 1 0 1 1 1

master-eu-central-1c NeedsUpdate 1 0 1 1 1

nodes NeedsUpdate 5 0 5 5 5 Must specify --yes to rolling-update.

The STATUS the column shows that both master and worker nodes need to be updated.

Perform the rolling update using the command shown:

kops rolling-update cluster example.cluster.k8s.local --yes

The output from this command is shown:

Validate the cluster again:

Get the list of nodes from the cluster:

Kubernetes App Auto-scaling

Table of Contents

Prerequisites

Deploy an application

Horizontal Pod Autoscaler configuration

Generate load

Stop load

Cleanup

Horizontal Pod Autoscaling (HPA) is a Kubernetes feature to dynamically increase/decrease the number of pod replicas based on resource utilization metrics.

As of k8s version 1.9, the direction for HPA is to use the Metrics Server rather than Heapster.

HPA can automatically scale pods deployed in a replication controller, deployment, or a replica set. For additional information on how HPA works, check out the Kubernetes community documentation.

Prerequisites

you’ll need to deploy configurations to a Kubernetes cluster. AWS CLI is mostly used to provision EKS based clusters. You can use kops to create a k8s cluster without using EKS

Deploy the metrics server:

Deploy an application

In this step, we deploy a simple Go web application and constrain the CPU resources just for the purposes of this test.

$ kubectl run webapp --image=trevorrobertsjr/webapp --requests=cpu=50m --expose --port=8080

service "webapp" created

deployment "webapp" created

You can see that service is published at port 8080 in the cluster.

Horizontal Pod Autoscaler configuration

Now that our application is running, we create a Horizontal Pod Autoscaler for our web app deployment.

$ kubectl autoscale deployment webapp --cpu-percent=10 --min=1 --max=10

deployment "webapp" autoscaled

This command will maintain between 1 and 10 replicas of the pod. You will see that the autoscaler will work accordingly as per CPU utilization across all pods and will increase and decrease replicas

Generate load

The simplest method to do this would be to access the application in an infinite loop similar to the example in the Kubernetes Horizontal Pod Autoscaler documentation:

First, deploy a busybox container, label it load-generator , and attach to it’s prompt:

$ kubectl run -i --tty load-generator --image=busybox /bin/sh

At the load-generator command prompt, run a continuous request of the web app

$ while true; do wget -q -O- http://webapp.default.svc.cluster.local:8080; done

If for any reason you get disconnected from the load-generator container, you can re-attach to it with the following command.

$ kubectl attach $(kubectl get pod | grep load | awk '{print $1}') -c load-generator -i -t

In a different terminal window, check the status of the Horizontal Pod Autoscaler.

$ kubectl get hpa -w

You will see output similar to the following over successive queries of the hpa resource:

$ kubectl get hpa -w

NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE

webapp Deployment/webapp 0% / 10% 1 10 1 6m

webapp Deployment/webapp 62% / 10% 1 10 1 7m

webapp Deployment/webapp 62% / 10% 1 10 4 7m

webapp Deployment/webapp 112% / 10% 1 10 4 8m

webapp Deployment/webapp 112% / 10% 1 10 4 8m

webapp Deployment/webapp 53% / 10% 1 10 4 9m

Notice that, eventually, the value in the REPLICAS the column will increase as the load generator continues to run.

Stop load

In the terminal window that is running the load generator, hit Ctrl + C to terminate the process. Again, run the kubectl get hpa -w command in your other terminal window, and you will see the number of replicas begin to decrease as the CPU load returns to 0%. It shows the output:

Note

It takes a few minutes for the number of replicas to scale down.

$ kubectl get hpa -w

NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE

webapp Deployment/webapp 51% / 10% 1 10 4 10m

webapp Deployment/webapp 51% / 10% 1 10 4 10m

webapp Deployment/webapp 27% / 10% 1 10 4 11m

webapp Deployment/webapp 27% / 10% 1 10 8 11m

webapp Deployment/webapp 0% / 10% 1 10 8 12m

webapp Deployment/webapp 0% / 10% 1 10 8 12m

webapp Deployment/webapp 0% / 10% 1 10 8 13m

webapp Deployment/webapp 0% / 10% 1 10 8 13m

webapp Deployment/webapp 0% / 10% 1 10 8 14m

webapp Deployment/webapp 0% / 10% 1 10 8 14m

webapp Deployment/webapp 0% / 10% 1 10 8 15m

webapp Deployment/webapp 0% / 10% 1 10 8 15m

webapp Deployment/webapp 0% / 10% 1 10 8 16m

webapp Deployment/webapp 0% / 10% 1 10 1 16m

webapp Deployment/webapp 0% / 10% 1 10 1 17m

Cleanup

$ kubectl delete hpa/webapp deploy/load-generator deploy/webapp

For refencer

👋 Join FAUN today and receive similar stories each week in your inbox! ️ Get your weekly dose of the must-read tech stories, news, and tutorials.

Follow us on Twitter 🐦 and Facebook 👥 and Instagram 📷 and join our Facebook and Linkedin Groups 💬