How COVID-19 pushed us to reduce our AWS cost by half
By: Vishesh Jindal
Tags: Cloud Computing, DevOps, Aws Cost Optimization, Infrastructure, Aws Cost Reduction

Data Transfer Costs

Data transfer costs are the ones that get ignored most easily because it is hard to account for data transfer over the network as compared to compute usage. Not to forget that when you are building up your business, the network usage is very hard to estimate. The cost of network early on is usually so small that it doesn’t warrant so much attention. However, some higher level decisions such as avoiding usage of public network routes for accessing internal services or reducing data transfer due to infrastructural components (such as monitoring) can go a long way in saving money. That is as much as you can do. As you scale up, the cost of the network starts to show up.

Data transfer costs show up in EC2-Other service in AWS Cost Explorer. AWS charges you for different kinds of data transfer:

Data transfer from outside the VPC — listed as NAT Gateway, PublicIP-In, PublicIP-Out, VPCPeering-In, VPCPeering-Out in AWS Cost Explorer Cross-AZ data transfer — Listed as InterZone-In, InterZone-Out in AWS Cost Explorer

Data transfer from outside the VPC (Internet)

Our major costs here were NAT Gateway costs. NAT Gateway is used to enable instances in a private subnet to connect to the internet. AWS Cost Explorer doesn’t provide much information about what data transfer you are being charged for. This documentation by AWS recommends these solutions to reduce data transfer costs:

Setup NAT Gateway per AZ to reduce cross-AZ (or inter-AZ) data transfer. If you are using DynamoDB and S3 in the same region, setup gateway VPC endpoint. If the majority of the traffic is to AWS services supporting VPC endpoints, create one interface VPC endpoint. If the majority of traffic needs to go to the internet, set up an internet gateway.

Pro-tip: You can also set up a public instance and use that instead of a NAT gateway if you don’t want to pay for NAT Gateway running hours. This can be risky if the instance goes down, needs maintenance or your network bandwidth requirements cannot be satisfied by the network bandwidth cap of EC2, so you can probably try this in your non-prod environments. We didn’t try this but learned from people we know that some companies do this as well.

We already had a NAT Gateway per AZ so we don’t get charged for cross-AZ data transfer.

VPC endpoint is a hidden gem. Data transfer cost due to S3 is often a big contributor. As soon as you roll out the VPC endpoint, you will see a sudden drop in data transfer cost. Before we set up a VPC endpoint, we needed to know where we were making requests to get a better understanding of our costs and usage. If S3 or other AWS services show up in your analysis, you should look at deploying VPC endpoints immediately.

VPC endpoint is a hidden gem. Data transfer cost due to S3 is often a big contributor. As soon as you roll out the VPC endpoint, you will see a sudden drop in data transfer cost.

Reducing NAT Gateway charges further would require building a better understanding of the reasons for high NAT Gateway charges. To further drill down into where all we were being charged for NAT Gateway data transfer, we used VPC Flow Logs on a log group and AWS Insights to find out our biggest contributors. You can find some sample queries here and here. You can also dump VPC Flow Logs to an S3 bucket and later query them using AWS Athena to do custom analysis if your infrastructure and business setup requires you to find deeper answers about your network usage. For details on how to set that up, follow this post by AWS. If you need to identify AWS service from IPs, you can check AWS IP Ranges here.

After analyzing our VPC Flow Logs, we found out that most of our NAT Gateway cost is from:

Our applications sending requests to other applications via public load balancers Cost of of data transfer between our own EC2 and S3

Application sending requests to our own public load balancer

Identifying which applications are making requests to public load balancers can get a little tricky because access logs of your receiving load balancer will show your NAT gateway’s public IP. We had to perform the painful process of talking to all service owners who had public load balancers and set up an additional private load balancer, and then have all the applications migrate to using the private load balancer. In some cases, we had to set up two load balancers — one for public travel and another for internal traffic, because our NAT Gateway data transfer costs were higher than the cost of an additional load balancer.

Cost of of data transfer between our own EC2 and S3

We created a gateway VPC endpoint to make traffic to S3 private. After making this change, our S3 data transfer costs went significantly down, but the NAT Gateway costs due to S3 were still high. This was because we were transferring data from buckets in a different region. We had to migrate some buckets to the same region from where these buckets were being mostly accessed.

One of the big S3 data transfer expenses was caused by our internal Docker registry. After all these changes, we couldn’t reduce the cost of data transfer because our stage and CI environment is in a different region. Private endpoint was effective only for the environment where the registry’s region was the same as the environment’s region (production in our case). So we set up a docker registry with a proxy in the same region as our stage to cache docker images. We will be publishing details about how we achieved this in a separate blog post soon.

Cross-AZ Data Transfer

We had significant Cross-AZ data transfer in our production as well as stage environments. There are a few easy things you can do here — like enabling compression, reducing unnecessary data transfer like debug logs, etc. If you have done all the configuration and tuning, the next thing to look at is trying to reduce the data transfer footprint of your services. This is not easy to solve so we decided to take it up as a long term improvement. Depending on your risk appetite, you can also look at running your entire infrastructure out of a single AZ. We migrated our stage infrastructure and migrated large portions of production infrastructure to a single AZ bringing down the inter AZ cost to as less as we could. This was an interesting migration. You can read more about the migration here.

EC2 Costs

Decreasing EC2 costs depends a lot on the infrastructure you are running. There were multiple steps we took to reduce our EC2 costs.

Resource Optimization

The first step is to see which EC2 instances are costing you the most and then work your way from there. Once again our tagging strategy helped us a lot in determining this using AWS Cost Explorer. We made a list where the instance type could be changed or if an auto-scaling group’s minimum size could be reduced. For Kubernetes, we needed something similar to Cost Explorer. After some exploration, we came across kube-resource-reports. This provides the current view of costs and usage by applications, namespaces and pods. We used this to identify resources that were requesting lots of resources but not utilizing them so that we could right-sized them.

Utilization of Reserved Instances (RIs)

We had bought RIs a couple of months before COVID-19 hit us. After changing instance types and reducing auto-scaling groups’ min size, our RI Utilization went down. This was pure wastage. But fortunately, we bought Convertible RIs so we could convert them to other instance types to make sure that our RI Utilization is 100%. If you are looking to buy EC2 RIs, we would suggest looking at Savings Plan launched last year which is an offering similar to RIs but you don’t have to worry about converting RIs or purchasing them separately for every regions. One important caveat with Savings Plan is that it’s not exactly capacity reservation. It is purely a discounting construct.

Spot Instances

Spot instances are a great way to save costs. They can be as cheap as 30% of the on-demand cost. Majority of our infrastructure was running on on-demand instances. We had explored spot instances earlier but we had some concerns:

1. If the spot instance gets terminated due to price issues, how do we ensure application remains available?

Spot instances are quite easy to manage with AWS’s autoscaling groups if you are using a launch template. In a launch template, you can specify different allocation strategies to ensure high availability of your application. These strategies would ensure that if an instance type is not available, another instance type depending on your allocation strategy would be launched.

Configuration of instance allocation strategy in auto-scaling groups

Configuration of instance type mix in auto-scaling groups

2. Graceful shutdown of an application when the instance is about to terminate so that error rate doesn’t increase.

You can get to know if a spot instance is about to terminate before 2 minutes by making a simple curl request periodically (read more here). This was a little tough to solve for existing applications running on EC2 instances because that would have required changes in all of our ASGs. Another option was to use a managed service like SpotInst to do the work for you.

For Kubernetes (which is most of our production), it turns out that it’s a lot easier to do spot instances than EC2. AWS and the DevOps community has built awesome tools that make it super easy. You can use aws-node-termination-handler or kube-spot-termination-notice-handler.

It is a good idea to set up Pod Disruption Budgets for your deployments to ensure that there is least possible disruption. Today, we just change the percentage of spot instances in our node group’s ASG as and when we need to tune our spot spread.

This approach has helped us run all of our stage Kubernetes workloads and a significant portion of our production Kubernetes workloads on spot instances. In production, 99% of our workload is covered under spot, reserved instances and savings plan. Only 1% workload runs on-demand, mostly because of the difficulty of managing spot coverage to avoid RI or savings plan wastage because of our varying workloads.

Nightly shutdown of stage infrastructure

All of our developers are in the same time zone, so our stage environment is not in use at night. We were already stopping EC2 instances in the stage environment at night. On Kubernetes, we have an internal tool that helps every developer create their own namespace and set up a personal dev environment. We set replicas for all deployments and stateful sets to 0 at night. And if there hasn’t been any change in the namespace for 5 days, we delete them. This saves at least 35% of our stage costs. Also, we get a nice side effect of ensuring that our stage environments are repeatable and re-creatable through code and automation.

Nightly scale down in production

Since our operations are based only in India, we don’t receive significant traffic at night. We decrease the number of instances that we run for a service at off hours. We were already using scheduled actions on Autoscaling Groups on EC2. On Kubernetes, we needed something similar for HPA for our services running in our Kubernetes cluster. kube-schedule-scaler is capable of doing this. We use a modified fork of kube-schedule-scaler. Developers can specify the scale down schedule by just putting a simple annotation on HPA objects. When the services scale down, the cluster autoscaler takes care of reducing the size of node groups. You can also explore kube-downscaler for a similar functionality.

Here is an example of what an HPA with a scheduled action looks like:

apiVersion: autoscaling/v1

kind: HorizontalPodAutoscaler

metadata:

annotations:

kube-schedule-scaler/schedule-actions: '[

{"schedule": "0 19 * * *", "minReplicas": "7"},

{"schedule": "30 1 * * *", "minReplicas": "20"}

]'

name: my-hpa

namespace: my-namespace

spec:

maxReplicas: 100

minReplicas: 20

...

Resource optimisation for Kubernetes dev cluster

In Kubernetes, you can set requests and limits for a pod. In production, the difference between limits and requests should be small else you might see performance issues (read more on this here). But in the case of the stage environment, the workload is generally not CPU intensive. So, it doesn’t make sense to keep resource requests very high. It’s a little tough to do this for all your services, so we created a Kubernetes controller which, depending on namespace labels, decreases the requested CPU of containers but keeps the limits at the same value to allow some peaks for applications.

Reducing requests to minimum while keeping limits high enough helped us reduce number of cores by 70%.

We faced a few challenges around resource management — you can read about that here.

Reducing EBS cost

We use a mix of magnetic and GP2 volumes. We try to use magnetic volumes wherever we know that we won’t need much IO. Most of our workload using EBS volume is stateless and were used as root volumes in EC2. We didn’t have a good policy around EBS size selection while launching new EC2 infrastructure which led to a lot of wastage of EBS (most instances using as high as 20% of provisioned storage). The solution here is simple but needs a lot of manual work (some of it can be scripted out as well):

Recreate all AMIs and update the launch template to provision less only necessary EBS capacity. Fix EBS provisioning policies and set good alerts on usage to solve this in the long term.

Stage ELB costs

We were using ELB to expose the services and provide an endpoint for local testing. But this resulted in extra ELB costs and it was a big inefficiency in our setup. We decided to replace all ELB with Nginx ingress to expose services. We wrote an in-house Kubernetes controller to create a NodePort service backed by Nginx ingress for every request to create an ELB based service. This did not require any change in our service manifests and drastically reduced our ELB costs.

S3 Costs

S3 is quite cheap for data storage but without proper handle on how you are using S3, the costs can keep increasing steadily with time if you haven’t configured your lifecycle policies for your buckets. We have hundreds of buckets. To get the cost attribution report for S3, we used AWS Athena to query our CUR reports and to filter out the buckets for which we were getting charged the most for. After identifying the costliest buckets, we found a few common problems:

No lifecycle policy. We don’t retire/purge any data which adds up over months and years. If storing backups, backups were not compressed. Logs are dumped but not archived.

You can set up a lifecycle policy on your bucket to change the storage type of your files or remove them after some time. Read this post by AWS for more details.

We’d like to especially call out the intelligent tiering option which can help reduce storage costs by a great deal.

Reserved Instances

AWS offers Reserved Instances (RIs) for many of its services (like RDS, Elasticache, Redshift, etc.). If you know that a resource is going to be there for a year or more and wouldn’t require changes in its instance type, it would be better to buy a RI for that instance. AWS also supports converting RIs before expiry of purchase contract for some of their services. Explore those options before making your purchase.

Buying a lot of RIs can also be problematic if your RI Utilisation decreases because that would be pure wastage. You can use AWS Cost Explorer to track this. Some 3rd party tools can also be used for setting up alerts.