The day Facebook, Instagram & Whatsapp were wiped off the face of the world (How did Meta’s global outage actually took place ? )
By: Aayush Shukla
Tags: Networking, Technology, Facebook, WhatsApp, Instagram

The day Facebook, Instagram & Whatsapp were wiped off the face of the world (How did Meta’s global outage actually took place ? ) Aayush Shukla Dec 11, 2021·5 min read

Remember Facebook, Whatsapp & Instagram outage that occurred on 4th October 2021 ? As a developer what surprised me was how can a company as big as Facebook ( Or as we call it now “Meta” ) goes down and that too for more than 6 hours! I tried looking down for the reason behind it on the internet but only answer i found was it happened due to a DNS issue. Now do you feel it is plausible that one of the biggest tech company in the world faces 6 hour global outage because of “some” DNS issue ? I didn’t. I decided to dig up further & found this!

Prerequisites:

Before i go on further with the chain of events that happened, I wanted to list down some networking abbreviations which even i forgot the next day after my computer networking final exam.

D NS — Domain Name System ( The only one i remembered out of these ) ANS — Authoritative Name Server / Authoritative DNS / Name Server RNS — Root Name Server BGP — Border Gateway Protocol

Domain Name System (DNS) — Think of DNS like a phone directory or like contacts app on your smartphone ( if you are born after 90’s ). It basically eliminates the need to memorise an IP address, all you need to know is the domain name like facebook.com. DNS will translate domain name to IP address, from which browsers can load the requested resources. The 4 main DNS servers are DNS Recursor, RNS, TLD Name server & ANS.

Root Name Server ( RNS ) — The job of RNS is to narrow down the search of IP addresses using the given domain name by user. It does that by using the “.com” in the “facebook.com”. Basically all the domain name & IP are stored in an hierarchy based on “.com” , “.co.in”, “.org” and so on.

Authoritative Name Server (ANS) — ANS is that one guy who actually has the working knowledge of the project in the entire team. ANS is the final holder of domain name requested by the user. It returns the IP address to the DNS Recursor , which is the starting point of a DNS.

Border Gateway Protocol (BGP) — It’s a mechanism to exchange routing information between autonomous systems (AS) on the Internet. Confused right ? Think of BGP like an Indian aunty who knows what all is happening in each and every house hold in her locality. All the ANS are real time listed over the BGP so that anyone from anywhere can access it.

If you wish to know more about DNS do comment down below.

What is Cloud ?

So for someone not familiar with how cloud works, it is a bunch of servers located in different part of world which are connected with each other with ultra high bandwidth connection enabling them to be connected and transfer data between each other at high speed.

Image from AWS

The chain of events that took place on 4th Oct 2021

1. The Human Blunder

“Humans make mistakes, so let computer take care of such things while you hold its beer” — Aayush

On 4th October 2021 someone from Facebook was given the task to update a single node ( server in single location of entire cloud cluster ) but the person accidentally took down all the nodes in the entire infrastructure. How it happened ? maybe he was sleep deprived or playing COD while working. Who knows. Anyway this was the first event that took place which let to the following domino effect.

2. The Human Blunder 2.0

Question: What more worse can happen if your parachute doesn't open while you jump off a place ? Answer: Your backup parachute also fails to open.

For a company as big as Facebook its sounds odd right that someone can take down the entire network with a single command ? Yes. So Facebook has a auditing program in place to prevent such things to happen at first place but the auditing program itself had a bug in it which let the command take down all nodes at once.

3. The only right thing that happened that day

Once a node goes down in the Facebook’s cloud infrastructure, it automatically stops broadcasting its ANS to the BGP, disabling the users to reach out to Facebook.com. Now you might think why is it right to take off the ANS from BGP right ? If the ANS is not able to give a response because it has been taken down, it doesn't really make sense to broadcast it over to the BGP. That is what happened here, since all nodes were taken down Facebook.com took out its domain broadcasting to BGP. This is the point from when Facebook, Instagram and Whatsapp were taken off the face of the world.

4. Checkmate ( DNS failure & physical security )

The failure of Cloud Infrastructure resulted in disconnection between all nodes and failure of intranet, eventually which led to DNS failure. Now since all nodes were disconnected and were on their own, it was impossible to SSH in between them to restart other nodes. So now comes the final hurdle, because the servers can’t be reached remotely, only option left was to access to in person. All the Meta’s data centre's are highly secure and require a person to go through multiple checkpoints which made this who process even more time consuming.

This is my first medium post. Please comment down below if you have any doubts or suggestions for me. The failure of DNS itself is a lengthy part, if you would like to more about it, hit me up. Let’s discuss!