Why Netflix Kills Instances of Their Own Services
By: Shubham Pathania
Tags: Software Engineering, System, Testing, Netflix, Developerscoffee

Necessity is the mother of invention

In early 2010, when Netflix shifted from providing its services via physical servers to cloud computing, they had a couple of challenges on their way.

Cloud computing offers new challenges to software teams. Computers are linked via network connections and there is less control over the cloud-based computers.

Computers can also be of many different types. They can have different operating systems, memory capacities, computer processing speeds, graphics processing speeds, and graphics memories.

When testing the system’s resiliency, chaos engineering asks, If this computer goes down, can we move the task it was handling to another computer? And what another computer? Finally, how well does a system handle moving tasks from one computer to another?

This resulted in the invention of a tool by the engineering team of Netflix to test the system’s resiliency in cloud infrastructure.

The New Tool — Chaos Monkey

Chaos Monkey is a tool invented in 2011 by Netflix to test the resilience of its IT infrastructure. It works by intentionally disabling computers in Netflix’s production network to test how remaining systems respond to the outage.

Chaos Monkey is developed to see how well a system shifts its resources when there is a computer outage. It enforces proactive engineering practices by assuming that failure will and does occur.

The tool gets its name by envisioning a monkey running through a data center. Randomly pressing buttons, removing plugs and cables for multiple servers. It is bringing down the whole infrastructure by making such chaos.

How can an engineer design the system to work regardless of what the monkey does?

Getting Started with Chaos Monkey

According to the Chaos Monkey GitHub, “Chaos Monkey randomly terminates virtual machine instances and containers that run inside of your production environment. Exposing engineers to failures more frequently incentivizes them to build resilient services.”

Chaos Monkey is built upon another Netflix-made tool, Spinnaker, which is an open-source, multi-cloud continuous delivery platform. It works with all major providers:

Amazon Web Services

Azure

Cloud Factory

Google Compute Engine

Kubernetes V2 (manifest based)

Oracle

Once you’ve installed Spinnaker, you can install Chaos Monkey. For a closer look at how to use Chaos Monkey, see this page of the documentation.

The Simian Army

Chaos Monkey is not the only tool in their pocket. Since it was developed and its popularity rose, a whole bunch of tools has been developed to simulate outages and test system response times. Chaos engineers now have the whole Simian Army:

Chaos Kong — for AWS Region.

— for AWS Region. Chaos Gorilla — for AWS Availability Zone.

— for AWS Availability Zone. Latency Monkey — simulates network outages or delays.

— simulates network outages or delays. Doctor Monkey — performs health checks.

— performs health checks. Janitor Monkey — identifies unused resources.

— identifies unused resources. Conformity Monkey — identifies non-conforming instances.

— identifies non-conforming instances. Security Monkey — tests for known vulnerabilities. Outdated now.

— tests for known vulnerabilities. Outdated now. 10–18 Monkey — detects problems in instances in multiple geographic regions.

Pros and Cons of Chaos Monkey

As they say, everything comes with its benefits as well as challenges. The same holds for Chaos Monkey. Let’s take a look at both sides of the coin —

Pros:

Prepares the system for random instance failures.

Encourages redundant backup that might save one day.

Enables cross-cloud compatibility.

Cons: