Reduce the cost of your Kubernetes IaC infrastructure with these few steps
Tags: Cost, Kubernetes, Infrastructure, Optimisation, Finops

I vividly recall my very own Kubernetes cost-optimization exercise during exciting times working at Loveholidays. We were keen on a lean, mean Kubernetes infrastructure-as-code (DevOps) GitOps operating machine.

Cost Observability was not immediately one of them.

(Back in 2018/19) With only recent migration from the on-prem to Google Cloud Platform and a cloud-native migration at that, — we embraced Kubernetes hands-on. Now the migration was complete, it was well due to review the ever-growing Kubernetes infrastructure costs and figure a good process to keep such quite important detail under control.

Optimise your workload

At first, the cost reduction exercise was in order.

This is by no means an exhaustive list of cost optimisations, but it is the place to start, to ensure your infrastructure spending goes a long way.

These are the Kubernetes context list of activities I recommend starting with:

Assess the application workloads and services. This is more hands-on into the docker image build’s and application stack, but the infrastructure team and the application development team need to join forces and validated the assumptions, and ascertain the application runt-time requirements.

i.e. Which Apps need to be reviewed and what application processes could be optimised to fix any mem leaks.

workloads and services. This is more hands-on into the docker image build’s and application stack, but the infrastructure team and the application development team need to join forces and validated the assumptions, and ascertain the application runt-time requirements. i.e. Which Apps need to be reviewed and what application processes could be optimised to fix any mem leaks. Ensure applications can run on the smallest footprint of the docker base image. Another best practice, and worth re-iterating. Your security posture should improve as a result. But it will help reduce the overall cost for such an application container.

This workload assessment and application run-time with the load requirements should help define the Kubernetes deployment Pod Memory and CPU quotas and limits . This is to prevent the spurious resource run-aways, where existing unchecked pods will exhaust the memory on the node they run on (if without memory limits) or throttle the node CPU (if without CPU limits).

This does not just impact the one application with unchecked resource utilisation, but other containersed services running on that unfortunate node. The impact is on the other services running on that node, which may degrade your uptime and associated SLAs.

the Kubernetes deployment and and . This is to prevent the spurious resource run-aways, where existing unchecked pods will exhaust the memory on the node they run on (if without memory limits) or throttle the node CPU (if without CPU limits). This does not just impact the one application with unchecked resource utilisation, but other containersed services running on that unfortunate node. The impact is on the other services running on that node, which may degrade your uptime and associated SLAs. Taint the nodes and tolerate those taints with the workload tolerations. Ensure the right workload runs on the right machine type. I wrote about this recently here. Great graphics to highlight the why’s and the how-to’s.

the nodes and those with the workload Ensure the right workload runs on the right machine type. I wrote about this recently here. Great graphics to highlight the why’s and the how-to’s. The effect of node over-utilisation may result in the cluster node auto-scaler function to kick in. Such activity will spin up another whole node for your workload to be scheduled on, ever exacerbating your cluster costs. You may want to rely on these taints and tolerations again to help reduce such a combined impact of the points raised above and not only schedule workload on the correct CPU and Memory matched node( nodepool ) but also set the min, max values for the nodepool size.

Monitor the Costs

With the challenges of a shared, multi-tenant cluster operating environment, it is hard to assess the cost of workload/namespace effectively. This Cost Monitoring effort is not intended to be “my task routine” nor was it a platform team’s responsibility to be fair. This is an interesting area of FinOps, a shared responsibility model for the Costs of Cloud Infrastructure.

Getting started

It does not need to be too daunting, but you are required to have all your Kubernetes essentials in place, Prometheus metrics aggregator along with the Grafana dashboard UI at the minimum.

I distinctly recall the great foundations of Grafana Cluster cost dashboards here and here that were very helpful.

You can download both, they are free of course and you can adjust the cost of the Network, CPU, Storage, and Memory as well as prolonged use discounts.

Each cloud has its cost metrics to go by, update these as required

This is a recurring theme, and I see it comes as part of the client business requirement, on the microservice Kubernetes environment time and time again. Recently I was positively surprised to discover that Kubecost was of the providers of such dashboards I have used in the past has a commercial product with even more features and multi-cloud Kubernetes monitoring capabilities. Kubecost has further extended such Kubernetes workload assessments, to provide more insight into cost attirubution. One shared namespace kube-system is one such example, with a number of departments that leverage that namespace’s services. The costs of which can cost attributed/shared to the consuming teams.

Another fantastic feature was the forecast capability which can showcase such forecasting based on monthly, daily, and hourly consumption rate of current Kubernetes run-time, offering the predictable spend for the emerging services.

This is by far not the exhaustive list of features, but other notable features include

Alerting on Notification on budget spend

Infrastructure Insights

Cost Allocation

Infrastructure reliability score

When operating production workload across the Kubernetes estate the cost can spiral pretty quick. But these are all the handy features that I found to give you back the control over that spend, and with great insight and predictability.

There. I found you found it helpful.

Good luck with your cluster cost optimisation journey!

Like, share and forward the knowledge to your colleagues.

Connect on LinkedIn

Until Next time,

J