Baseline Application Performance In AKS Part 1
By: Chase Overmire
Tags: Performance, Aks, Kubernetes, Containers, Pods

Understanding an applications resource consumption is vital to ensuring proper performance in a Kubernetes cluster. This is part 1 of 3 and we will focus on looking at our application at rest to understand the resource impact the application has without any load

Optimizing cluster performance isn’t something that is unique to the Azure Kubernetes Service. All the concepts and ideas in this article stretch not only to every cloud provider and K8 application but all applications in general.

There are three other articles I wrote that will help you understand why these concepts are important in Kubernetes. If you have not seen them they are:

Requests and Limits

AKS Performance: Resource Requests and Limits | by Chase Overmire | Nov, 2020 | ITNEXT

Resource Quotas

AKS Performance: Resource Quotas. Limiting resource consumption for… | by Chase Overmire | Dec, 2020 | ITNEXT

Limit Ranges

AKS Performance: Limit Ranges. Control the amount of resources your… | by Chase Overmire | Dec, 2020 | ITNEXT

This article is going to be used as a jumping off point and be a little more on the simple side than the articles that will proceed this one. The idea here is to introduce you to general perf analysis and show you how you could manually accomplish it if you had to and or needed to learn how.

Why manual? I want to do it manually first just to show you how we could look at important resources. This approach isn’t what I’d suggest for production but the information will help you understand how to gather important information. This is designed to help you get in the right head space with what resources are important straight away. Speaking of resources let’s define what we really want to accomplish shall we?

For this article we have three resources we want to focus on for now. Technically Network IO we care about as well however in this article we won’t focus on that one yet.

CPU Memory Disk IO

Each of these resources plays a critical role in ensuring a cluster is actually usable. Over saturating any of these resources can often lead to awful performance from your applications and potentially negatively impact your users. We want to gather enough data to ensure we don’t over tax any of these.

I am going to create three of these articles. Application at rest (this article), application under load, application burst / exhaustion. We will use this data to determine what our request and limits should/need to be and talk about organic growth over time and how we can keep ahead of our consumption.

Getting Started

You will need a cluster

This is going to be generalized (for now) to ensure that as many people as possible can consume this. Depending on your setup you likely need to do more comprehensive testing but the concepts will apply all the same

My Cluster

I’ll be doing all of this on an older cluster, version 1.17.11. I have 2 nodes that are quite small, Standard_D2s_v3 which is 2 CPU 8GB Mem. I have the bone stock P10 500 IOP disk attached as well.

With my limited resources it’s very important for me to understand the resource impact my applications have to prevent over saturating my nodes. I have a some goals I need to accomplish by the end of the article series:

Understand the applications resting resource requirements (we learn that in this artcle) Understand the applications low/moderate/heavy resource requirements Understand the applications high spike load resource requirements Understand potential request impact

If I can check all these off by the end of the series then I will be able to properly design resource and request limits, namespace quotas and limit ranges if I so choose to utilize those tools. This information will ensure that I understand what my application is capable of and allow me to properly utilize the resources I have at my disposal.

The Application

I am going to be using a K8s pod that is used as an example for Horizontal Pod Auto Scaling in the official K8s docs. Its accessible to all and the core concepts that I want to outline are easily achievable. You don’t have to use this of course. I wanted something everyone could easily use regardless of their cluster size.

Deploy the workload

Game time. I am going to run kubectl apply -f https://k8s.io/examples/application/php-apache.yaml then wait for it to be created by using kubectl get pod --watch .

Use — watch to monitor for state changes.

Once my app is running we can start thinking about the first requirement, understand the applications resting resource requirements. There are a bunch of different ways to do this so I am going to just focus on one. Before I do that though I want to verify a few things.

Let’s start out by running exec to get into the pod and just look at things the old fashion way. I’ll run kubectl exec -it php-apache-5546ddcd97-cvkd7 bash to get myself into the pod then check to make sure the CPU and memory are what I expect them to be at.

Running cat /proc/cpuinfo | grep "model name" shows me what I expect based on my Standard B2 Sku.

2 vCPU, exactly as expected.

Next I’ll run free -m to get a feel for what the pod is thinking for memory. Something to keep in mind here is that you cannot use this command to baseline memory requirements because of the way Linux will take unused memory for disk caching. Don’t worry though this is by design.

Don’t panic! Linux believes unused ram is wasted ram. It free’s up as the applications need it.

Now that we have made sure the pod has the resources we expect let’s focus on the needs of the actual application. The point, if you are wondering, was to show the pod is currently unbound by any limitations. This unbound pod will end up being the clusters doom in a later article which is often an issue for many people run into.

Understand Resting Resource Requirements

Now it’s time to start thinking about the three resources I mentioned above.

CPU Memory Disk IO

We want to understand our applications usage patterns at rest which requires us to first gather information on each. Remember I had said we would be doing this manually. Manually gathering this information will help you understand the basics on how to gather the data itself if you’re not already aware as well as work as a way for us to think outside the box on how we could do something like this if we had no other options. We need to gather all this information and we need to be able to analyze it over time to ensure we understand the resting requirements.

Crazy people do crazy things and this is a crazy thing. In future posts we will automate this and or use awesome tools to do it for us which would be more logical but less informative. Generally speaking the idea behind this type of gather would be used for investigation in real time not gathering for historical analysis

Usually you’d collect all of this at the same time and analyze it bit by bit. We are not going to do it that way in this article because we want to do it the hard way. Instead we will gather things in groups or one offs from the hosting node and then aggregate it to be a bit friendlier for us to analyze.

Here is the game plan; I am going to make a poor mans data collector that runs every second on the hosting node and gathers the stats of the top three resources, recoding them to a file we will later consume. I will gather data for only 5 minutes or so then I’ll aggregate them together and look at the data.

The reason I am not going to gather more than 5 minutes in this article is mainly because the method we are using will get its point across without tons more data. Understand that on a real application you want a lot more data than just 5 minutes. In short, you’ll get the point with just this dataset, it’s going to be pretty flat minus one metric.

What I am going to do is gain access to my hosting node using something called node-shell and gather data directly from it.

Determine the Hosting Node and Gain Access

I’ll run kubectl get pods -o wide and see my hosting node

We can see my hosting node is aks-agentpool-19575414-vmss000000

Before I hop into my node I want to run a describe against my PHP pod so I can get the container ID. I’ll run kubectl describe pod php-apache-5546ddcd9f-cvkd7 and we can see my id.

We will look at this a bit later

Now I will go ahead and run kubectl node-shell aks-agentpool-19575414-vmss000000 to gain access to my node.

An additional pod is spun up and used to SSH into the hosting node without you needing to do anything else.

Access gained, game time. We can now find the PID of the container and start gathering our data. There is a good amount of ways to do this, I’ll be lazy and just run docker inspect -f '{{.State.Pid}}' <container id> .

Understand that on a real application you want a lot more data than just 5 minutes. In short, you’ll get the point with just this dataset.

Gathering the Data

There are a lot of ways we could manually do this. In general if you’re going to be gathering CPU, Mem and Disk you often need a bit more than single app. What do most people think about? I would bet commands like PS, TOP, HTOP, IFTOP, IOTOP and probably SAR.

What I want to do if possible is to gather data atleast every second. Technically speaking most cloud providers calculate resource throttles BELOW 1 minute, IE 10ms in Azure Disk case I believe, so we need to make sure we have atleast 1 second.

You need to make sure to be careful about anything that averages data for you as well if you are gathering data > 1 second. If you gather data say every 10 seconds and average 5 IOPS but spike to 10 in only one second the average won’t properly reflect the massive spike. We will talk about this more in a later article.

CPU + Memory

We could try batching top..

/var/lib/docker/containers# top -d 1 -b -p 26380 >> output.txt

Batch top and parse the output….

Which honestly if you said “Hey you need to make this from scratch with built in tools!” I probably wouldn’t do. Another option might be to run ps aux right? We could do something like this:

ps aux | awk -v date=”$(date +”%Y-%m-%d %r”)” ‘{print date “\t\t” $11 “\t\t” $3 “\t\t” $6/1024 “ MB\t\t” “\t” $2}’ | grep 26380

Which is going to be easier to read and parse than the top in batch mode the way I ran it. This route has the benefit of allowing me to text to columns in Excel so parsing the data would be really easy as well. You know what, for this article, I'm going with this route for CPU/Mem, it’s simple.

Note that gathering data in this manner does technically have a resource overhead, minimal as it maybe.

Disk IO

My favorite route for this would be IOTOP all day long. We know my pid 26380 is who I want to focus on. Easy enough, I can run iotop -b -q -d 1 -p 26380

Not super clean but we can work with this

There is one hiccup here though…we have no time stamp at all. We cannot awk it like we did with ps aux because we are running in batch mode, it will only show the first instance at command start…let’s rethink this. If we look at IOTOP man pages we can see there are ways to control the amount of time it will run in batch more. So let’s do this instead: iotop -q -d 1 -n 300 -p 26380

Now IOTOP will run for only 5 minutes, n being the switch and 300 being the amount of seconds. This will let us do some easier calculations…as silly as they maybe…later on. Let’s call this “okay for now” and stitch these together.

Brining it together

We know that the iotop command will run for five minutes already by design so we just need to write something to make sure the ps aux one does as well. Easy enough. I’ll just make the worlds worst script…

#!/bin/bash

n=0

iotop -q -d 1 -n 300 -p 26380 >> ioResults.txt & #background this

while [ $n -le 301 ]

do

ps aux | awk -v date=”$(date +”%Y-%m-%d %r”)” ‘{print date “\t\t” $11 “\t\t” $3 “\t\t” $6/1024 “ MB\t\t” “\t” $2}’ | grep 26380 >> cpumemResults.txt

sleep 1

n=$(( n+1 )) # increments $n

done

To the professional devs out there, yes this is hot garbage, hot garbage that will get the job done. When you start this its going to push iotop into a background process that runs while the ps aux loop runs for 5 minutes as well. Now I can just run ./MyScript.sh and wait for it to complete.

Alternative Methods

There are always lots of ways to do this and I likely choose one of the most ridiculously horrible way possible on purpose. Lets look at a couple other ways we could have done this with my specific setup in AKS.

Using Docker

Docker has a built in stats command that can gather pretty much all the important stuff we care about but there is no time element. I’ll take the container ID I gathered from my pod above and run docker stats 26bcf99507ad175af2a53772cdbe5b22b82d1ee8abebf133a7563a00cb34bb9c — format “table {{.ID}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.BlockIO}}\t{{.NetIO}}” theb watch the magic.

Delicious data!

Great! Tons of wonderful info about the containers usage…however its just refreshing to the screen over and over and it offers us no actual time frames at all which makes this great for in the moment investigations but not really over time analysis….though we could work around it if we wanted.

Let’s think about the docker stats command for a moment…its clearly pulling this data from somewhere. A little bit of investigating or knowledge as to how cgroups work can lead you the cgroup resource data. We could perhaps dream up a way of collecting the data manually…then do a bit of research to find out how the data is collected and collated then create a way to historically gather this data…

options options so many options…

The data is there and it would just be about pulling/analyzing it. A decent option to play around with if you’re at all curious. If you’re up for a challenge go find out how Kubernetes API gathers statistics, then find out how a collection app like Prometheus gathers the same statistics.

Maybe instead we could just ask the Docker API for the stats? Sure we can do that too just by curling the docker.sock.