by thevenincloud | 2025-10-23 14:15:03+00:00
"It's always DNS" the phrase that comes up from sysadmin and DevOps alike.

And there are reasons for this common saying, according to The Uptime Institute's 2022 Outage Analysis Report the most common reasons behind a network-related outage are a tie between configuration/change management errors and a third-party network provider failure. DNS failures often fall into these categories.

This was the case of last AWS us-east-1 outage on 20th October . An issue with DNS prevented applications from finding the correct address for AWS's DynamoDB API, a cloud database that stores user information and other critical data. Now this DNS issue happened to an infra giant like AWS and frankly it could happen to any of us, but are there methods to make our system resilient against this?

Can we avoid DNS issues increasing TTL? The thing is IPs are meant to change. When we are hitting one API we are usually not hitting one server, but a collection of servers with different IPs. Even if we were to hit only one server it is extremely likely the IP of it will change on rollout, scaling, update, maintenance and many different events that happen in daily operations.

Can we be reliant against DNS issues using a DNS Backup Server? In this case in particular it wouldn't have been helpful to remediate the AWS outage, since most of the time spent on the outage was on Root Cause Analysis and that usually applies to any incidence in most companies. So even if you do the DNS server switch you already had all that outage time realizing it was dns.

What about NodeLocal DNSCache?

A NodeLocal functions just like any other DNS cache. Its primary job is to hold onto a DNS record for the duration of its Time-to-Live (TTL).

However the serve_stale CoreDNS option is the one key feature that could have made a difference, depending on its configuration. NodeLocal DNSCache can be set up with a serve_stale option.

If this feature is enabled, when the TTL expires and the cache fails to get a new record from the upstream server, it can be instructed to return the old, expired ("stale") record anyway. This allows applications to continue functioning on the last known IP.

Even if there are risks associated with the IP change this method helps with the retry storm.

All of the methods above could make some system resilient regarding DNS issues. But in the specific case of the AWS outage new info shows that all DNS records were deleted by an automated system:

"The root cause of this issue was a latent race condition in the DynamoDB DNS management system that resulted in an incorrect empty DNS record for the service’s regional endpoint (dynamodb.us-east-1.amazonaws.com) that the automation failed to repair. " AWS RCA

A Kubernetes Operator is a specialized, automated administrator that lives inside your cluster. Its purpose is to capture the complex, application-specific knowledge of an Operations administrator and run it 24/7, think it like an automated SRE. While Kubernetes is great at managing simple applications, an Operator teaches it how to manage complex resources like DNS.

The DNS Management System failed because a delayed process (Enactor 1) overwrote new data. In Kubernetes, this is prevented by etcd's atomic "compare-and-swap" mechanism. Every resource has a resourceVersion. If an Operator tries to update a resource using an old version, the API server rejects the write. This natively prevents a stale process from overwriting a newer state.

The entire concept of the DynamoDB DNS Management System, one Enactor applying an old operations plan while another cleans it up is prone to crate concurrency issues. In any system, there should be only one desired state. Kubernetes Operators always try to reconcile toward that one state being based on traditional Control Systems.

I wrote up a more detailed analysis on: https://docs.thevenin.io/blog/aws-dns-outage