How Google Cloud Platform could milk you if you aren’t careful
By: Bragadeesh Jegannathan
Tags: Kubernetes, Google Cloud Platform, AWS, Billing, Infrastructure

Photo by Clark Young on Unsplash

GCP is right up there in the top three race to cloud infrastructure only behind Amazon Web Services and Microsoft Azure. It has almost everything you need from classical VM instances, Serverless, Hosted Kubernetes, Docker container registries, ML, Vision - almost everything!

In this article, I am going to focus on how you might get milked out with a fat invoice every month if you aren’t careful enough. We learnt the hard way shelling out thousands of dollars a month only to realize you can achieve the same with a different infrastructure provider with almost one tenth the cost.

Before I dive in, take a look at the cost chart on how our invoice went down from almost 1300€ a month down to under 10€. Of course, we did move to a different Infrastructure (DigitalOcean) but the running cost went to well under 100€ a month! Almost one order of magnitude reduction.

For the year 2020 our infrastructure cost was averaging 1300€ during the first half!

You might have learnt Kubernetes, could be an expert in Docker, very well fluent with Python, but if you are not going to be good at analyzing Google’s billing summary, you are sure in for an expensive ride.

Our biggest mistake

One of the biggest mistakes we ever did was to shift to Kubernetes. Kubernetes is an amazing orchestration framework. When you want to build an amazing, redundant, self-healing architecture with supreme monitoring capabilities and security, Kubernetes will look like a no-brainer. At least on paper. (For all the Ops engineers already starting to hate me need to understand that I am certified professional and may have more experience than you do!)

A Boeing is a complex aircraft, there are numerous moving parts, technologies that cannot fail at any part during the flight’s journey, built for scale and should be a work-horse. However, for a kid all he cares about is a structure of the airplane and needs to small enough. He does not give a damn about the complexity or engineering. On a similar note, Kubernetes is great for certain purposes, but for lightweight, not so much scale applications, this will soon be an overkill, almost in all aspects like deployment, backup, maintenance, upgrades, developer productivity, learning curve etc.,. It is a clear over engineering nightmare. And finally, the heat will be felt in the monthly bills when a “hosted” Kube environment like Google or anyone is going to present you!

How to figure out leaks

To keep the article to the point and not to divert too much, let me get back to how to identify the leaks, not in memory or space, but money.

In my case, I have selected the date range from start of year to today, use bar graph view and the important thing is to use Group by “SKU”

Grouped by SKU

This quickly reveals where your meat of the monthly bill is going to. In my case, for Jan 2020, my cost split was

The list is sorted by costly to cheaper and it is longer. For brevity, I am showing the top 10.

The predefined instance cost is 368 + 185 + 54 + 51 = 676€. All these are either independent VM instances or Nodes consumed by the Kubernetes cluster. Its showing as different line items, because they are used in different regions. This is not so surprising as one would expect, but still each of these nodes are freaking expensive when compared to a cloud provider like DigitalOcean.

Then the line item that gave us a mini heart attack is the Network Load Balancing of €136.80. Wait, what? Apparently this is something one would not expect to see in the bill, yet this is something we are using implicitly in our Kube infrastructure. And Google is not going to give this for free. There are other cost like Hosted Redis, Cloud SQL etc., but I am not going to dive deep into them as they are straightforward.

We stopped here, and asked ourselves do we need such a complex infrastructure in the first place for the scale of the App we have built. The answer to the question was still not known. Hence we ventured into a path of getting a fat machine from digital ocean with 16 GB RAM, 8 vCPUs, 6TB Bandwidth, 320 GB SSD which costed $80 a month. And we wanted to redo the same infrastructure without the hosted Redis, SQL, Kubernetes. We were able to pack all these into a single machine that our complex kube did, and still we were able to do it with ease. There were 5 micro-services with various language stack ranging from Python/Django, Rails, Machine Learning, Java and Go. Yes, it was complex, but then we realized 16GB RAM is a lot and so are 8 CPUs. The elephant in the room was over-engineering.

Problems that could be solved in a simple way was made into a monstrous nightmare in the name of Orchestration which was an overkill. Having said this, Kubernetes and Google Cloud Platform are very powerful. But if used incorrectly, they are going to milk the hell out of you and learning this the hard way is not going to be fun at all!

Are you in a similar boat? If so, it is high time to revise whether the resources that you are using are right enough for the problem that you are trying to solve!