Buffer sizing and VideoQoE Measurements atNetflix
Bruce Spang, Brady Walsh, Te-Yuan Huang,
Tom Rusnock, Joe Lawrence, Nick McKeown
February 10, 2020

What are we talking about?

What are we talking about?
BufferServer 1
Server 2…

ISP

How big should a buffer be?
Too big: packets wait for too long
Too small: too many packets thrown away

“A buffer should be at least one BDP” [Villamizar, Song 1994]

“A buffer should be at least one BDP” [Villamizar, Song 1994]
BDP=Bandwidth x Delay# of packets in a link for full utilization

“A buffer should be at least one BDP” [Villamizar, Song 1994]
BDP=Bandwidth x Delay# of packets in a link for full utilization

Congestion
WindowTime

“A buffer should be at least one BDP” [Villamizar, Song 1994]
BDP=Bandwidth x Delay# of packets in a link for full utilization

Congestion
Window

Loss happens when link andbuffer are full

BDP + BTime

“A buffer should be at least one BDP” [Villamizar, Song 1994]
BDP=Bandwidth x Delay# of packets in a link for full utilization

Congestion
Window

Loss happens when link andbuffer are full

BDP + B½(BDP + B)
TCP stops sending until ½(BDP+B) packets received

Time“A buffer should be at least one BDP” [Villamizar, Song 1994]
BDP=Bandwidth x Delay# of packets in a link for full utilization

Congestion
Window
BDP + B½(BDP + B)

Loss happens when link andbuffer are full}

Buffer needs to hold thismany packets

TCP stops sending until ½(BDP+B) packets received

TimeHow big should a buffer be?

BDP: Villamizar and Song 1994
BDP/√n: Appenzeller, McKeown, Keslassy 2004
O(n): Dhamdhere, Jiang, Dovrolis 2005
O(1): Enachescu, Ganjali, Goel, McKeown, Roughgarden 2006

Which is correct?

It’s complicated1. TCP New Reno (mostly) behaves
as expected
2. Video performance varies
3. Real routers complicate this story

Our Experiment

Catalog servers
Uses spinning disks, cheaply
stores entire catalog

Offload servers
Use SSDs to serve top ~30%of content faster

These three racks are calleda stack

Make thisbuffer small…

…and thisone large

1. TCP New Reno (mostly) behaves
as expected
2. Video performance varies
3. Real routers complicate this story

Large buffer has higher latency
during congested hour

Sometimes the large buffer has
much higher latency

Large buffer has lower loss
during congested hour

1. TCP New Reno (mostly) behaves
as expected
2. Video performance varies
3. Real routers complicate this story

Good buffer size:+ Fewer rebuffers+ Better video quality+ Videos start fasterBad buffer size:- More rebuffers- Worse video quality- Videos start slower

Good buffer size:+ Fewer rebuffers+ Better video quality+ Videos start fasterBad buffer size:- More rebuffers- Worse video quality- Videos start slower}

This happens
when buffer is
too large or
too small.

Site #2: A smaller buffer is better
Reducing the buffer from 500MB to 25MB
-15.6% decrease in sessions with a rebuffer-5.3% decrease in low quality video-13.5% decrease in play delay

Site #3: A smaller buffer is better
Reducing the buffer from 500MB to 50MB
-22.1% decrease in sessions with a rebuffer-7.0% decrease in low quality video-14.8% decrease in play delay

Site #1: A smaller buffer is worseReducing the buffer from 500MB to 50MB
+46.3% increase in sessions with a rebuffer+5.7% increase in low quality video-5.9% decrease in play delay

1. TCP New Reno (mostly) behaves
as expected
2. Video performance varies
3. Real routers complicate this story

Large buffer has higher latency
during congested hourRemember how the large buffer
has much higher latency…

Servers have different very latency distributions

Min RTT (ms)What I imagined
BufferServer 1
Server 2…

ISP

What I imagined

LIES!
BufferServer 1
Server 2…

ISP

Line card #1Line card #2
Line card #3
Line card #4

VOQ #1

VOQ #2

VOQ #3

VOQ #4

VOQ #5

VOQ #6

VOQ #7

VOQ #8

Bufferarchitecture“Offload” VOQ
Server #1
2/3

Server #2100Gbps
ISP“Catalog” VOQ
Server #3

1/3

Traffic is fairly split whenload is equal“Offload” VOQ
40 Gbps
40 Gbps

67 Gbps
100Gbps
ISP“Catalog” VOQ
40 Gbps

33 GbpsWhen one VOQ offers less than its
fair share, it sees no congestion“Offload” VOQ
50 Gbps
50 Gbps

90 Gbps
100Gbps
ISP“Catalog” VOQ
10 GbpsNo delay!

10 GbpsVOQs explain the RTT differences

This VOQ isserved faster
This VOQ isserved slower

This VOQ is allover the place
Min RTT (ms)

Switches prioritize long-tail content

Switches prioritize long-tail content
Same latency during
uncongested hours

Switches prioritize long-tail content
Same latency during
uncongested hours

Long-tail content
not congested

Popular content
is congested

New schedulingalgorithm!

“Offload” VOQ
Server #1
Load-dependent

Server #2

100Gbps
ISP“Catalog” VOQ
Server #3

Load-dependentNew schedulingalgorithm is moreconsistent

Default schedulingalgorithm

1. TCP New Reno (mostly) behaves
as expected
2. Video performance varies
3. Real routers complicate this story

How big should abuffer be?

Thanks!
For more details, please see:https://brucespang.com/papers/netflix-buffer-sizing.pdf