Microservices at Netflix Scale

First Principles, Tradeoffs, Lessons Learned
Ruslan Meshenberg@rusmeshenberg

Microservices:
all benefits, no costs?

Netflix is the world’s leading Internet television networkwith over 81 million members in over 190 countries
enjoying more than 125 million hours of TV shows andmovies per day, including original series, documentaries
and feature films.

Ruslan Meshenberg
Director, Platform Engineering•

Runtime Systems•

Container Runtime•

Persistence and Databases•

Real Time Data InfrastructureNetflix runs on microservices

Netflix journey
to microservices

Our journey took 7 years

https://media.netflix.com/en/company-blog/completing-the-netflix-cloud-migration

Data Center - Monolith

RDBMSAugust 2008

First Principles

Buy vs. Build● Use or contribute to OSS technologies first● Only build what you have toServices should be stateless*● Must not rely on sticky sessions● Prove by Chaos testing*Except the Persistence / Caching layers

Scale out vs. scale up● If you keep scaling up, you’ll hit a limit● Horizontal scaling gives you a longer runway

Redundancy and Isolation
For Resiliency● Make more than one of anything● Isolate the blast radius for any given failure

Automate destructive testing● Simian Army
● Started with Chaos Monkey

First Principles
In Action

Stateless services

Service A

Service B

Service B

Service B

Service B

Service B

Verify statelessData – from RDBMS to Cassandra● NoSQL at scale● Open Source● Multi-Regional● Multi-directional● Available● Partition Tolerance● Tunable Consistency*

Multi-Regional Replication
Zone
B
Zone
A

Zone
B

500ms

Zone
B

Zone
A
ClientClient

Zone
A

Local Quorum(Typical)

Zone
CZone
A

Zone
CZone
CZone
CRegion AZone
BBi-directional
Nightly compare & repair

Region B

Last, but not least - BillingMicroservices –BenefitsOur Priorities
3. Efficiency

1. Innovation

2. Reliability

Innovation:tight coupling doesn’t work

Develop• Team A• Team B• Team C• …

Test

ReleaseInnovation: Loose coupling

Team A

Develop,
Test, Deploy,
SupportTeam B

Develop,
Test, Deploy,
SupportTeam C

Develop,
Test, Deploy,
SupportSupport

Architect

Run

DesignEnd-endownership
DeployDevelop

Test

Review

End-end ownership + velocity
SupportSupportArchitect

Architect

SupportRun

DesignRun

DesignDeployDevelop

DeployDevelop

Test

Review

Test

ReviewSupport

Architect

SupportArchitect

Run

DesignRun

DesignDeployDevelop

DeployDevelop

Test

Review

Test

Review

SupportArchitect

SupportArchitect

Run

DesignRun

DesignDeployDevelop

DeployDevelop

Test

Review

Test

Review

SupportArchitect

Architect

Run

DesignRun

DesignDeployDevelop

DeployDevelop

Test

Review

Test

Review

SupportArchitect

SupportArchitect

Run

DesignRun

DesignDeployDevelop

DeployDevelop

Test

Review

Test

Review

SupportArchitect

SupportArchitect

Run

DesignRun

DesignDeployDevelop

DeployDevelop

Test

Review

Test

Review

Leverage

Separation of concerns
UI

Feature A

Personalization

Feature D

Mid-tier

Infrastructure

Feature B

Feature C

A/B Test EA/B
Test F

Feature H

Availability

Scalability

Security

Microservices –Costs

Microservices
Is an org change!

Org changes are hard!

Evolving the organization

Central infrastructure
investment

Migration doesn’t happenovernight● Living in the hybrid world● Supporting 2 tech stacks● Double the maintenance● Multi-master data replication

Microservices Lessons Learned

IPC is crucial
for loose coupling● Common language between the services● Establishes the contract of interaction

Caching to protect DBs
Client Application

Request Cache

Client Library
EVCache Client Service Client...

S
1. Read from Cache
2. On cache miss call service
3. Service calls DB and responds
4. Service updates the cache

SDBSDBDB...

...

SDB

Operational visibility matters

If you can’t see it, you can’t improve it

Will your Telemetry scale?
Observe

OrientActDecide

EdgeMiddle Tier & Platform
Zuul

EVCache
ELBAPI
Cassandra
Playback

Reliability Matters● We strive for 4 9’s of availability● That leaves only 52 minutes of downtime per YEAR
● Netflix outages lead to…

Disappointment

OutrageWithdrawal

HumorCascading failures…
99% availability

99%500= 0.0657%

99% availability

99% availability

Microservice failure

FIT

Fault-Injection
Test Framework

Regional fail-overxx

Regional fail-overA word on containers● Containers change the level of encapsulationfrom VM to process● Containers can help deliver great developer
experience● To run containers in production at scale…

Requires something like this:Cassandra

ZookeeperDockerDockerRegistry
DockerRegistry
Registry

Titus UI
Titus UI
Titus UI

Titus Master
Titus Agent
Job Management &Scheduler

RheaAPI
Titus
Rhea

Fenzo

container
container
container
container
container
containerdocker

Mesos Masterzfs docker

EC2 Autocaling
APICI/CDS3

mesos agent

metrics agent
Titus executor
logging agent
VPC networking
driverAWS container Integration
metadata proxyAmazon VM’s50

Microservices Resources

http://netflix.github.com

http://netflix.github.com

http://netflix.github.com

http://netflix.github.com

http://netflix.github.com

http://netflix.github.com

Wrap up

Microservices bring great value to development
velocity, availability and other dimensions

Microservices at scale require organizationalchange and centralized infrastructure investment

Be aware of your situation and what works for youQuestions?

Ruslan Meshenberg@rusmeshenberg