How AWS WAF saved us
By: Riad Rifai
Tags: Waf, Ec2, AWS, Load Balancer, Firewall

The official AWS WAF (Web Application Firewall) definition by AWS:

AWS WAF is a web application firewall that helps protect your web applications or APIs against common web exploits that may affect availability, compromise security, or consume excessive resources.

In other words, setting up a WAF would not only help you monitor and track the requests reaching your AWS resources, but could let you block or allow them to pass based on a set of rules you specify. That would lead to cleaner server application logs, common attacks mitigation, less traffic on the server instances and most importantly, cost cutting.

In this article, I will show you how we used AWS WAF at Interactive Life on an application that was already in production. We had discovered a bug on the frontend that was draining our EC2 instances. After research, a WAF seemed to be the most reasonable approach to implement as we needed a quick yet efficient solution that’s also cost effective. In addition to that, I will show you how you could add it to your own resources in less than 5 minutes.

If you’d like to skip the scenario we were facing at Interactive Life and get to implementing the WAF directly, kindly scroll to Finally, WAF as a solution section.

Scenario:

Note: I am only going to mention the resources that were related to the problem.

Front-end:

3 different platforms:

Angular webapp Android iOS

The applications offer many different features, but one of the main ones is watching live streams. In order to collect live streaming statistics (number of live viewers, minutes watched per user etc.) along with many different information for different features, we have HTTP polling enabled. There are many APIs used for polling and in the next section we will see how their misuse had caused the problem.

Back-end:

We have 2 EC2 instances behind a load-balancer. The instances were benchmarked and stress tested before production release, to ensure that they can handle more traffic than our actual application traffic, especially that we have short polling on the front-end.

Problem:

On a Sunday afternoon, I started receiving warning emails from AWS because requests to our EC2 instances were timing out. First thing I did was check the monitoring graphs that are offered by AWS for each EC2 instance, only to find that both instances had reached 100% of CPU Utilization.

CPU Utilization for one of the instances

It’s good to note that most of our users are active during that time of the day, when the bug was triggered; this explains why it happened that day. But however, this was still an irregular behavior, especially that the peak was sudden. After a lot of research, frontend and backend debugging and log analysis, I was able to make sense of what was going on and find the root of our problem.

Cause:

I won’t bore you with the details of the frontend bug as the goal of this article is to show how beneficial AWS WAF can be; but basically the bug was that 2 out of 3 of our frontend apps were polling the server way more times than the expected and agreed upon intervals. In addition to that, a lot of unnecessary APIs calls were being initiated repeatedly, every couple of seconds in some cases from those 2 apps.

In short, the traffic coming out of 2 of the frontend apps was unnecessarily heavy. The reason it was never that obvious before was because our user base had grown during the COVID-19 pandemic, where more users are watching the live stream instead of being on the ground of the event.

Effects:

Overload the servers with way too many requests, causing resources to be constantly locked leading to request timeouts. Blowout the log files’ size, consuming more disk storage, difficulty in going through them while debugging and even reserving resources to actually log onto the files, quickening the point mentioned above. If you have T2/T3 unlimited enabled on your instance, the CPU bursting would add extra costs after the CPU utilization exceeds a certain baseline. If we had an auto-scaling group, this would have triggered the creation of a couple of unnecessary instances and would’ve drastically increased our cost.

Finally, WAF as a solution:

To solve this, we needed to:

Obviously, fix the bug on the front-end, so that new deployments wouldn’t flood the servers anymore.

Add a shield to block all the unnecessary API calls before reaching our server (WAF).

One of the reasons why AWS WAF was a perfect solution in our case, in addition to the attacks mitigation it offers, is that it could be implemented very quickly, and we needed that since our applications were in production.

Creating the WAF:

Luckily, WAF could monitor all the web requests that are forwarded to API Gateway, an Amazon CloudFront distribution and, in our case, an Application Load Balancer.

AWS WAF Classic is the older version of AWS WAF. I recommend using AWS WAF as it has many improvements, better managed and more flexible rules. Also a more cost-effective method for pricing. You can find more information about the comparisons between the pricing and benefits in the WAF console.

Head over to the AWS console, login and search for the WAF & Shield service. Press on Create Web ACL button:

Fill in the name and description as you like.

Under the Resource Type, if you want to protect a CloudFront distribution then select it.

Under the Associated AWS resources, click Add AWS resources. Find your load balancer or API Gateway and select them.

Press on Next to add your rules and rule groups.

A rule defines attack patterns to look for in web requests and the action to take when a request matches those patterns. Rule groups are reusable collections of rules. You can use managed rule groups offered by AWS and AWS Marketplace sellers. You can also write your own rules and create your rule groups.

For this demo, we are going to add our own rules; but feel free to go through the existing managed rule groups offered by AWS and the AWS marketplace, as they are already setup to help protect against common attacks.

The rules we’re going to create will define the patterns we want to block/allow. For the simplicity of this article, we’ll add 2 rules only (these aren’t the rules implemented to fix our bug above):

Regular rule: This rule would check if the URI path contains an SQL injection. Rate-based rule: To mitigate DDOS attacks. This rule will block requests that are made from the same IP address after they exceed a certain count in less than 5 minutes.

Regular rule:

When you add your own rule, there are multiple settings you need to configure.

Rule type:

Choose Rule builder so we could create our own rule. The other 2 types have descriptions that describe them clearly, but we will not use them for the sake of this demo.

Next, we will set the pattern to match for blocking a request.

There is a lot of flexibility when it comes to creating the pattern, as you can match using logical operators instead of matching one statement only.

For the Statement I chose to inspect the URI path to check if it contains an SQL injection statement. However, it’s very flexible and you can create almost any condition as you have access to most of the request details (source, IP address, Headers, Body, URI path etc..)

Under Match type you can create the condition that has to be matched in order to take action. In this case it should contain an SQL injection attack; but it could be as simple as matching a URI that contains the path /fishing for example, in this case, all API calls with the URI https://yourdomain.com/fishing/... will match the condition.

Add text transformation if you would like to customize the text received by the inspection you chose, before evaluating the condition.

Finally, add the action to apply in case the condition you set for the rule was matched:

In this case, if the condition was matched, the API call will be blocked. You could choose however to allow requests only if they match your condition, or count if you just want to register how many times an API was triggered.

Rate-based rule:

Everything is similar to what we’ve already done, but when choosing the Type under the rule section, choose rate-based rule.

After that, set the number of requests allowed to be received from the same IP address in less than 5 minutes. This totally depends on your application and use case, if you expect more than 150 requests from the same IP in less than 5 minutes then increase the limit to suit your need.

If you want apply the rate-limit based on a condition to instead of all the received requests, then select the second point showed in the image above; you will see a condition creation interface just like the one we’ve seen in the first rule so you can manage from there.

One final thing to note is Web ACL rule capacity units; each ACL has a limit of 1500 capacity units. With each rule and condition you add, more capacity units are used.

The last steps after you’ve set up your rules and conditions are simply to choose the order in which the rules will be evaluated in, and set up a simple logging metric.

Let’s talk numbers

After creating the ACL, you will see a useful overview and a graph that would display information about the requests you receive. You could filter the requests by rule, date and even whether the API was allowed or blocked.

After creating our set of rules and conditions at Interactive Life, we were able to cut the requests that were received on our servers by almost 40%, in a quick, reliable and cost-effective method thanks to AWS WAF.

As you can see, for almost 10,000,000 requests per month, it cost us 6.30$ to protect our app from various possible attacks, and to work around the critical bug we had discovered. It’s good to note that the price includes 5 Web ACL rules.

Thanks for reading!

Feel free to follow me on Github and LinkedIn!