A Design Analysis of Cloud-based Microservices Architecture at Netflix
By: Cao Duc Nguyen
Tags: Cloud Computing, Streaming, Netflix, Microservices, Software Architecture

4. Design Goals

In previous sections, I have described in detail the cloud architecture and its components powering Netflix’s video streaming business. In this section and the subsequent sections, I would like to go deeper into analyzing this design architecture. I start with the list of most important design goals as follows:

Ensure high availability for streaming services at global scale.

for streaming services at global scale. Tackle network failures and system outages by resilience .

. Minimize streaming latency for every supported device under different network conditions.

for every supported device under different network conditions. Support scalability upon high request volume.

In the subsections, I am going to analyze the availability of the streaming service and its corresponding optimal latency. Section 6 looks at more in depth analysis about resilience mechanisms such as Chaos Engineering while Section 7 covers scalability of the streaming services.

4.1 High Availability

By definition, availability of a system is measured in terms of how many times a response would be fulfilled for a request within a period of time, without guarantee that it contains the most recent version of the information. In our system design, the availability of streaming services depends on both the availability of Backend services and OCAs servers keeping the streaming video files.

The goal of Backend services is to get the list of most healthy OCAs proximity to a specific client, either from cache or by execution of some microservices. Therefore, its availability depends on different components involving the Playback request: load balancers (AWS ELB), proxy servers (API Gateway Service), Play API, execution of microservices, cache stores (EVCache) and data stores (Cassandra):

Load balancers can improve the availability by routing traffic to different proxy servers to help prevent overloading workloads. Play API controls the execution of microservices with timeout via Hystrix commands which could help to prevent cascading failures to further services. Microservices can respond to Play AI with data in cache in case the call to outside services or data stores takes more time than expected. Cache is replicated for faster access.

When receiving the list of OCAs servers from Backend, the client probes the network to these OCAs and chooses the best OCAs to connect to. If that OCA is overloaded or failed during the streaming process, then the client switches to another good one or the Platform SDK would request other OCAs. Therefore its availability is highly correlated with the availability of all OCAs available in its ISPs or IXPs.

The high availability of Netflix streaming services comes at the cost of complex multi-region AWS operations and services as well as the redundancy of OCAs servers.

4.2 Low Latency

The latency of streaming services depends mostly on how fast Play API can resolve the list of healthy OCAs and how well the connection of a client to the chosen OCA server.

As I have described in the Application API component section, Play API does not wait for a microservice’s execution forever since it uses Hystrix commands to control how long it would like to wait for the result before it gets the not-up-to date data from the cache. Doing so could control the acceptable latency as well as stop the cascading failures to further services.

The client would immediately switch to other nearby OCAs servers with most reliable network connection if there is a network failure to the current selected OCA server or that server is overloaded. It can also lower the video quality to match with the network quality in case it finds out a degradation in network connection.

5. Tradeoffs

In the above described system design, there are two prominent trade-offs have been carefully implemented:

Low latency over consistency

High availability over consistency

Latency over Consistency trade-off is built into the architecture design of Backend services. Play API can get stale data from EVCache stores or from eventually consistent data stores like Cassandra.

Similarly, Availability over Consistency trade-off would prefer constructing responses in acceptable latency without requiring the execution of microservices on latest data in data stores like Cassandra.

There is also a not-quite-relevant trade-off between Scalability and Performance ([21]). In this trade-off, improving scalability by increasing the number instances to process more workloads may cause the system running under its expected increasing performance. This could be a problem with those design architectures in which the workloads are not load well balanced among available workers. However, Netflix has resolved this trade-off with AWS auto scaling. We come back to this resolve in more detail in Section 7.

6. Resilience

Designing a cloud system capable of self-recover from failures or outages has been the long goal at Netflix from the start day of migration to AWS cloud. Some common failures that the system have been addressed as follows:

A failure in resolving service dependencies. A failure of executing a microservice would cause cascading failures to other services. A failure of connecting to an API due to overloading. A failure of connecting to an instances or servers such as OCAs.

To detect and resolve these failures, the API Gateway Service Zuul ([20]) has built-in features such as adaptive retries, limiting concurrent calls to Application API. In return, the Application API uses Hystrix commands to time-out calls to microservices, to stop cascading failures and isolate points of failures from others.

Netflix technical teams are also famous for their chaos engineering practices. The idea is to inject pseudo-randomly errors into production environments and build solutions to automatically detect, isolate and recover from such failures. The errors can be adding delays to responses of executing microservices, killing services, stopping servers or instances, and even bringing down the whole infrastructure of a region([5]). By purposefully introducing realistic production failures into a monitored environment with tools to detect and resolve such failures, Netflix can uncover such weaknesses quickly before they cause bigger problems.

7. Scalability

In this section, I will analyze the scalability of Netflix streaming services by covering horizontal scaling, parallel execution and database partitioning. The other parts such as caching and load balancing also help increasing scalability have been mentioned in Section 4.

First, the horizontal scaling of EC2 instances at Netflix is provided by AWS Auto Scaling Service. This AWS service automatically spins up more elastic instances if the request volume increases and turns off unused ones. More specifically, on top of thousands of these instances, Netflix has built Titus ([17]), an open source container management platform, to run about 3 million containers per week. Also, any component of our architecture Figure 2 can be deployed inside a container. Moreover, Titus allows containers to run on multi-regions across different continents around the world.

Second, the implementation of an Application API or a microservice in Section 3.2.2 also increases the scalability by allowing parallel execution of tasks on the Network Event Loop and the asynchronous Outgoing Event Loop.

Lastly, the wide column stores such as Cassandra and key-value object stores like ElasticSearch also offer high availability and high scalability with no single point of failure.