[25-Nov-2020] Amazon Kinesis Outage — What Can We Learn?
By: Bideep Bezbaruah
Tags: Outage, Cloud, Amazon Web Services, Distributed Systems, AWS

My Key Takeaways

#1. Identify The Fire Before Picking The Right Fire Extinguisher

It’s quite a common tendency for engineers to dive into action of putting out the fire when systems fail and pagers go off. But, it’s critical to know the type of fire before dealing with it.

In the paragraph starting “At 5:15 AM PST, the first alarms began firing for errors on putting and getting Kinesis records…..”, there are multiple probable options mentioned which could have caused the issue, and hence there were multiple options to mitigate/resolve the issue.

AWS mentioned —

While the leading candidate (an issue that seemed to be creating memory pressure) looked promising, if we were wrong, we would double the recovery time as we would need to apply a second fix and restart again.

It’s at the crossroads like these when we need deeper understanding of the root cause to pick the right path of resolution.

Once the root cause was identified to the thread capacity issue and not related to memory pressure as was initially thought of, the resolution path became quite clear.

AWS’s ability to add additional configuration to the fleet of front-end servers to gather additional insights quickly and in the middle of a crisis is exemplary. This is a system-state we should strive for when designing our systems.

Without such precise insights we would risk much longer outages and recovery times. It’s critical to know the type of fire before picking the right fire extinguisher for the job.

#2. Rollback Plan Does Not Always Go As Planned

Based on the fact that the root cause was identified to be related to thread capacity limits, I would assume such cases would not have been considered as part of the rollback plan scenarios. I could be wrong though!

Nonetheless, during an incident rollback plans may not work out as planned. It’s critical that we remain nimble to innovate and devise alternate plans on the fly. This requires deep understanding of the systems we manage and run.

Point is, expect all that is unexpected during a service disruption and be prepared to handle it quickly and effectively.

#3. Keep An Open Mind For Anti-Patterns

One of the design principles that we come across regarding compute (read EC2) scaling capacity is to have a larger fleet of EC2 instances with smaller configurations (as feasible) than having a smaller fleet with larger configurations. This is primarily to minimize the blast radius and the impact on available capacity in case of disasters.

As part of the resolution strategy during the Kinesis outage the below anti-pattern was adopted in the short term, which helped control the chaos.

In the very short term, we will be moving to larger CPU and memory servers, reducing the total number of servers and, hence, threads required by each server to communicate across the fleet.

Although this looks like an anti-pattern, the point is, we need to adapt and be open to implement strategies that work well given a particular situation at hand.

#4. The Importance of Cell-Based Architectures

Cell-based architecture is a key design strategy to minimize blast radius.

Citing from the report —

We are moving the front-end server cache to a dedicated fleet. We will also move a few large AWS services, like CloudWatch, to a separate, partitioned front- end fleet. In the medium term, we will greatly accelerate the cellularization of the front-end fleet to match what we’ve done with the back-end. Cellularization is an approach we use to isolate the effects of failure within a service, and to keep the components of the service (in this case, the shard-map cache) operating within a previously tested and operated range. This had been under way for the front-end fleet in Kinesis, but unfortunately the work is significant and had not yet been completed. In addition to allowing us to operate the front-end in a consistent and well-tested range of total threads consumed, cellularization will provide better protection against any future unknown scaling limit.

This was a bit surprising to see AWS was still working on cellularizing the architecture considering the advocacy around cell-based architectures. But, the good news is, changes are already in-flight.

If you are into distributed system architectures, listen to Peter Vosshall’s re:Invent 2018 talk if you have not already done so. I am sure you’ll love it!

This talk provides insights into how AWS uses cell-based architectures to contain blast radius of failure.

Another interesting related topic is ‘shuffle sharding’. I would highly recommend going through this post https://aws.amazon.com/builders-library/workload-isolation-using-shuffle-sharding/.

#5. Multi-Cloud Is NOT Answer 42

Whenever there is a cloud service disruption, the concern around putting all the eggs in one basket grows; tweets and posts pop up everywhere advocating multi-cloud strategy.

While it can help address a lot of other concerns, but would it really help in scenarios like the Kinesis outage?

Most likely, it won’t. Even if it does in some cases, it may not be cost-effective to manage such an architecture.

Imagine multi-cloud failover in case there is a Kinesis or S3 outage. Now, let’s say we want to fail over and have Google Cloud Pub/Sub or Azure Event Hub start taking in the data when Kinesis goes out of service.

What does it take to achieve it? I’ll leave it to your imagination the complexity of implementation, starting from the compatibility issues, data lakes, capacity, throughput differences, and so on. And, this is just one of the services!

#6. Try Best To Avoid Cascading Failures

This outage impacted multiple AWS services — CloudWatch, Cognito, AutoScaling, Lambda, EventBridge,Elastic Container Service (ECS) and Elastic Kubernetes Service (EKS), and Service Health Dashboard.

Cascading failures can quickly go out of control in distributed systems where there are multiple upstream/downstream dependencies. Cascading failures can bring down distributed systems to a grinding halt unless they are designed to withstand such failures.

To exacerbate an outage situation, unforeseen issues can crop up, like latent bugs coming to life, tuning configurations not being good enough, thresholds becoming bane than boon, so on and so forth.

We need to be wary of such potential risks when designing our systems and do the best to incorporate all the possible best practices. Some of them can be circuit breakers, jitters, exponential back-off, rate limits, etc.

#7. Keep All The Fail-Safes Functional

Another interesting citation from the report is as below —

While this worked as expected, we encountered several delays during the earlier part of the event in posting to the Service Health Dashboard with this tool, as it is a more manual and less familiar tool for our support operators.

This shows the importance of drills even for the seemingly trivial exercises.

In this case, Service Health Dashboard status update delays might not have been critical compared to the overall outage.

But, it is crucial to have all the manual steps documented in runbooks and exercised at regular intervals. Albeit manual, it’s necessary to keep the skills sharp and not get rusty.

It’s imperative that we keep all the fail-safes functional!