Auto Scaling Your Kubernetes Cluster in AWS — Everything You Need to Know
By: Tiexin Guo
Tags: Cloud, Kubernetes, AWS, Autoscaling, DevOps

EKS

As said, EKS is Amazon’s implementation of Kubernetes in the AWS cloud. The difference between EKS and a standard Kubernetes cluster (or what you get as a service in most other cloud providers, or what you install yourself from ground up), is that you don’t really “own” the control plane. Instead, in AWS EKS, you pay a very little fee (almost nothing compared to your EC2 nodes) and AWS manages it for you.

To setup an EKS cluster, there are multiple ways: through the web UI, using eksctl, using Terraform, etc. No matter which method you use, behind the scene, this is what happens when you create a cluster:

a control plane is created

an auto scaling group is created, min/max/desired numbers set, then EC2s are added into the auto scaling group, using a pre-built image joining the control plane

Of course there are more resources created, for example IAM roles, etc, but for the simplicity we only mentioned two things here, which are: the control plane, and the “node group”.

Managed Node Group V.S. Self-managed Node Group

Managed

With Amazon EKS managed node groups, you don’t need to separately provision or register the Amazon EC2 instances that provide compute capacity to run your Kubernetes applications. You can create, update, or terminate nodes for your cluster with a single operation. Nodes run using the latest Amazon EKS-optimized AMIs in your AWS account while node updates and terminations gracefully drain nodes to ensure that your applications stay available.

While AWS documentation tells you a lot, it also manages to tell you nothing. In essence, a “managed” node group means AWS creates the auto scaling group and manages it, for example, deciding which AMI to use, what cloudinit script to put into it so that when the EC2 is booted it knows how to and where to join the cluster.

A managed node group is the most simple way to get started, but there are a few limitations as of today. For example, if you work in a large enterprise environment, chances are, every egress traffic is regulated via an internal proxy instead of an internet gateway or a NAT gateway. But managed node groups don’t allow you to set proxy; nor can you set EKS API as a private endpoint so that you can access them without internet gateway/NAT. For another example, if you would like to use custom CNI networking, you would also need to set a few parameters in the cloudinit script, which managed node groups don’t allow you to do so.

Self-Managed

On the contrary, a self-managed node group, as the name suggests, is totally managed by yourself, thus giving you all the control over anything. While this gives you more control over it and more possibilities (as aforementioned proxy issue or custom CNI), you also need to take care of more details, like you have to prepare the cloudinit script so that the EC2 knows where to join and how many pods are allowed per node, tagging for auto scaling groups and subnets, etc. This article isn’t going to focus on self-managed node groups and how it works and how to create one so we are not going to dive too much into details here; just to make sure that we are on the same page and we know there are two types of worker node groups in the case of EKS.

Scaling

OK, after laying all the ground information about EKS, now we can talk about scaling.

You might think: if I use the “managed” node group, since it’s managed by AWS, I will get auto scaling automatically, right?

The logic behind this is intuitive, because AWS should know if I have enough resource in my cluster to spin up new pods or not so that it could scale the cluster for me, right? But unfortunately, in real life, this is not the case. Even if you use managed node groups, the cluster can’t scale. Let’s do an experiment:

Testing the Managed Node Group Scaling: Will It?

Cluster Creation

Creating a cluster using `eksctl`:

eksctl create cluster -n test —-managed

This will create a cluster named “test”, with a managed node group. What’s happening behind the scene is, creating two CloudFormation stack, one for the control plane one for the node groups.

Sample standard output:

As you can see, there are two stacks created, one being the cluster (control plane), the other being the node group. Which can also be checked in AWS console CloudFormation:

By default, in the managed node group, you get an auto scaling group of 2/2/2 (min/max/desired number of nodes) with m5.large instance, giving you total 4 vcpu and 16g mem. Now let’s update the max number of instances to 4, and try deploy something that exceeds the resources we have, and see what happens:

In AWS EKS console, update the node group size.

Deploy Workload

Now deploy some pods:

As shown above, we asked for 4 pods, each taking 4g mem and 1 vcpu. The file can be downloaded from here.

NAME READY STATUS RESTARTS AGE

test-5fbffb55d-42cw6 0/1 Pending 0 118s

test-5fbffb55d-4spwq 0/1 Pending 0 118s

test-5fbffb55d-6zqzc 1/1 Running 0 118s

test-5fbffb55d-x978b 1/1 Running 0 118s

If we try to get the pods, only two are running, the other two are pending. If you describe one of those pending pods, you will get message that says “0/2 nodes are available: 2 Insufficient cpu, 2 Insufficient memory”.

No matter how long you wait, those two pods are always waiting to be created. It seems, in the “managed” node group, auto scaling isn’t enabled by default. Doesn’t look like very “managed” to me. But if you think about it, this is normal: the cpu/mem usage isn’t actually high; what’s blocking the pod to be created is the resource requests exceeding what are available, and the auto scaling group does not have information about the allocated resources for the pods.

If you create a cluster with self-managed node group, the situation is the same for the same reason.

Manual Scaling

Of course when situation like this happens, you can always manually scale the cluster, by going in AWS console or using CLI, to update the desired number of nodes in that node group. This isn’t recommended, because:

manual means it is prone to mistakes

you must find out the auto scaling group name yourself, meaning another manual task or automated script for that, adding complexity

you will have to scale it down when the load is less, meaning you have to manually administrating the cluster, drain the node, then delete the node

Cluster Autoscaler

Enter the Cluster Autoscaler.

How It Works

I didn’t contribute to this tool actually, nor have I read the source code; but if I were to implement something that accomplish this goal, I would:

write an app in Golang (because everyone else is doing it; plus the image size is very small after docker multi stage build)

deployed as a deployment running inside the cluster node

node has IAM permission to manipulate the auto scaling groups

pod is running with a specific service account which has a role that allows it to get access to pods, messages, etc, so that it knows when there are pods pending because of not enough resource

pod calls AWS API to update the auto scaling group automatically

Prerequisites: IAM Permissions

Since the pod will manipulate the auto scaling group, it must have the following IAM permissions:

Simply go to IAM console, create a policy with above content, then attach this policy to the IAM role which your node group is using (search your cluster name in the roles, the one has “node group” as part of the name should be it, created by eksctl/CloudFormation).

Prerequisites: Auto Scaling Group Tagging

Then the pod needs to know which auto scaling group to manipulate, so you need a tag on the ASG so that pod knows which ASG belongs to the cluster it lives in. Tagging the ASG with the following key/values:

k8s.io/cluster-autoscaler/<cluster-name>: owned

k8s.io/cluster-autoscaler/enabled: true

Install

Download the file from here:

Before applying the file, open the file “cluster-autoscaler-autodiscover.yaml” with your favorite text editor, and change the placeholder “<YOUR CLUSTER NAME>” to the name of your cluster, then just “kubectl apply -f” it.

Make sure the pod is up and running in the kube-system namespace and there is no error in the log. If you experience some error logs related to IAM permission or auto scaling groups, go back and recheck the prerequisites section.

Test Scaling Up And Down

After a few minutes, you can see your cluster has 4 nodes instead of 2, and all the pods we previously deployed are already up and running.

Now let’s kill the deployment and see if it scales down:

kubectl delete deployment test

Then if you tail the log from the cluster-autoscaler pod, you will start to see something like “xxx node is unneeded since …”, “node utilization …”, and some “Scale down status …”:

After about 10 minutes, unneeded nodes will be removed, and auto scaling group desired number of instances is changed back to 2, and the logs from the cluster-autoscaler pod will show you that some instances are terminated, and auto scaling group min size is reached. You can also double check this from AWS console in the auto scaling group that the desired node number is changed, so are the instances.

Note: by default, cluster autoscaler will wait 10 minutes between scale down operations, you can adjust this using the --scale-down-delay-after-add , --scale-down-delay-after-delete , and --scale-down-delay-after-failure flag. E.g. --scale-down-delay-after-add=5m to decrease the scale down delay to 5 minutes after a node has been added.

Tear Down

eksctl delete cluster test

Conclusion

In this article we discussed Kubernetes architecture (a little bit), different types of node groups in AWS EKS, how scaling works under the hood, and how to make it automated (including scaling down without human intervention). It is highly recommended and I see this as a must in any working cluster.

If you enjoy this article, please give it a like, comment, subscribe and share it!