ClickOS and the Art of Network Function
Virtualization(NSDI 2014 Paper)
Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto Bifulco*,
Vladimir Olteanu§, Michio Honda*, Felipe Huici** NEC Labs Europe, Heidelberg, Germany§ University Politehnica of Bucharest
firstname.lastname@neclab.eu, firstname.lastname@cs.pub.roThe Idealized Network

Application

Application

Transport

Transport

Network

Page 2

Network

Network

Datalink

Datalink

Datalink

Datalink

PhysicalPhysicalPhysicalPhysicalA Middlebox Worldad insertion

WAN accelerator
BRAS

carrier-grade NATtranscoder

DDoS protection
firewallQoE monitor

Page 3

IDS

session bordercontrollerload balancer

DPIHardware Middleboxes - Drawbacks▐ Expensive equipment/power costs▐ Diﬃcult to add new features (vendor lock-in)▐ Diﬃcult to manage▐ Cannot be scaled on demand (peak planning)

Page 4

Shifting Middlebox Processing to Software▐ Can share the same hardware across multiple users/tenants▐ Reduced equipment/power costs through consolidation▐ Safe to try new features on a operational network/platform▐ But can it be built using commodity hardware while stillachieving high performance?
▐ ClickOS: tiny Xen-based virtual machine that runs Click

Page 5

From Thought to Reality - Requirements
ClickOS▐ Fast Instantiation

30 msec boot times▐ Small footprint

5MB when running▐ Isolation▐ Performance▐ Flexibility

provided by Xen
10Gb/s line rate*
45 µsec delay
provided by Click* for most packet sizes

Page 6

What's ClickOS ?
domU

ClickOS

apps

Click

guest
OS

mini
OS

paravirt

paravirt▐ Work consisted of:l Build system to create ClickOS images (5 MB in size)l Emulating a Click control plane over MiniOS/Xenl Reducing boot times (roughly 30 milliseconds)l Optimizations to the data plane (10 Gb/s for almost all pkt sizes)l Implementation of a wide range of middleboxes

Page 7

Performance analysis

Driver Domain (or Dom 0)
netback
NW driver

OVS

vif

packet size
(bytes)

10 Gbit/s
rate

64

14.88 Mp/s

128

8.4 Mp/s

256

4.5 Mp/s

512

2.3 Mp/s

1024

1.2 Mp/s

1500

810 Kp/s

Xen bus/store

ClickOS Domain

netfront

Click
FromDevice

Event channel

ToDevice
Xen ring API(data)

300* Kp/s

350 Kp/s

225 Kp/s* - maximum-sized packets

Page 8Performance analysis
ClickOS Domain

Driver Domain (or Dom 0)
netback
NW driver

OVS

vif

Xen bus/store
Event channel~600 ns (2)772 ns (1)▐ Copying packets between guests greatly
aﬀects packet I/O (1)▐ Packet metadata allocations (2)▐ Backend switch is slow (3)▐ MiniOS netfront not as good as Linux

Page 9Click
FromDevice
ToDevice

Xen ring API~3.4 us (3)netfrontOptimizing Network I/O – Backend Switch
ClickOS Domain

Driver Domain (or Dom 0)
NW driver(netmap mode)

netback

netfront

Click

Xen bus/store

VALE
OVS

port

Event channel

FromDevice
ToDevice

Xen ring API(data)▐ Reuse Xen page permissions (frontend)▐ Introduce VALE[1] as the backend switch▐ Increase I/O requests batch size

[1] VALE, a switched ethernet for virtual machines, ACM CoNEXT'2012
Luigi Rizzo, Giuseppe Lettieri
Universita di Pisa
Page 10

Optimizing Network I/O
ClickOS Domain

Driver Domain (or Dom 0)
netback netback
NW driver

VALE

Xen bus/store

netfront

Click

Event channel

FromDevice

port

ToDevice

Xen ringAPI
API
Netmap(data)(data)▐ Minimal memory requirements– For max. throughput a guest only needs 2 MB ofmemory▐ Breaks other (non-MiniOS) guests– But we have implemented Linux netfront driverPage 11

slots

KB
(per
ring)

# grants(per ring)

64

135

33

128

266

65

256

528

130

512

1056

259

1024

2117

516

2048

4231

1033

ClickOS Prototype Overview▐ Click changes are minimal ~600 LoC▐ New toolstack for fast boot times▐ Cross compile toolchain for MiniOS-based apps
▐ netback changes comprise ~500 LoC▐ netfront (Linux/MiniOS) around ~600 LoC▐ VALE switch extended to:–Connect NIC ports and modular switching
Page 12EVALUATION

Page 13

Experiments▐ClickOS Instantiation▐State reading/insertion performance▐Delay compared with other systems▐Memory footprint▐Switch performance for 1+ NICs
▐ClickOS/MiniOS performance▐Chaining experiments▐Scalability over multiple guests▐Scalability over multiple NICs
▐Implementation and evaluation of middleboxes▐Linux Performance

Page 14

ClickOS Base Performance

Measurement Box

ClickOS
10Gb/s direct cableIntel Xeon E1220 4-core 3.2GHz (Sandy bridge)
16GB RAM, 1x Intel x520 10Gb/s NIC.
One CPU core assigned to VMs, the rest to the Domain-0
Linux 3.6.10

Page 15

ClickOS Base TX Performance

Page 16

ClickOS (virtualized) Middlebox Performance

10Gb/s direct cable

10Gb/s direct cable

Host 1

ClickOS

Host 2

Intel Xeon E1220 4-core 3.2GHz (Sandy bridge)
16GB RAM, 2x Intel x520 10Gb/s NIC.
One CPU core assigned to Vms, 3 CPU cores Domain-0
Linux 3.6.10

Page 17

ClickOS (virtualized) Middlebox Performance

Page 18

Linux Guest Performance▐ Note that our Linux optimizations apply only to netmap-based applications
Page 19

Conclusions▐ Virtual machines can do flexible high speed networking▐ ClickOS: Tailor-made operating system for network processingl Smaller is better: Low footprint is the key to heavy consolidationl Memory footprint: 5MBl Boot time: 30ms

▐ Future work:l Massive consolidation of VMs (thousands)l Improved Inter-VM communication for service chainingl Reactive VMs (e.g., per-flow)

Page 20

ClickOS Boot times

220 milliseconds

30 milliseconds

Page 21

Scaling out – Multiple NICs/VMs

Host 2

ClickOS

Host 1
6x 10Gb/s direct cable

6x 10Gb/s direct cableIntel Xeon E1650 6-core 3.2GHz, 16GB RAM, dual-port Intel x520 10Gb/s NIC.
3 cores assigned to VMs, 3 cores for dom0
Page 22

Scaling out – 100 VMs Aggregate Throughput

Intel Xeon E1650 6-core 3.2GHz, 16GB RAM, dual-port Intel x520 10Gb/s NIC.
3 cores assigned to VMs, 3 cores for dom0
Page 23

ClickOS Delay vs. Other Systems

Page 24

Towards Massive Server Consolidation

Filipe Manco, João Martins, Felipe Huici{filipe.manco,joao.martins,felipe.huici}@neclab.eu

NEC Europe Ltd.

Xen Developer Summit 2014

The Super Fluid Cloud●

Target: remove barriers in current cloud deployments
–
–
–

●

This would allow new use cases
–
–
–
–

26

Extremely flexible infrastructure
Milliseconds instantiation and migration of resources
Thousands of concurrent units running
On the fly deployment of middleboxes
Flash crowds
Energy consumption reduction
Your use case here...

18 August 2014

Recent trend: specialized guests●

ClickOS, OSv, Mirage, Erlang on Xen, etc
–
–
–

●

Our work focuses on ClickOS–

27

Small memory footprints
Relatively fast boot times
Provide the basic functionality to make use cases a reality

Targets network processing using the Click modular router software

18 August 2014

Wouldn't it be Nice if...
●Thousands of guests on a single server––●

Extremely fast domain creation, destruction and migration––

28

Short-term target: 10K
Medium-term target: 100K
Tens of milliseconds
Constant as number of guests increases

18 August 2014

Experiment Setup●

Freshly installed Xen/Debian system
–
–
–

●

Commodity server––29

Xen 4.2
Linux 3.6.10
Debian squeeze
64 Cores @ 2.1GHz [4 x AMD Opteron 6376]
128GB RAM DDR3 @ 1333MHz

18 August 2014

Xen and ClickOS Architecture
Dom0 (Linux)

ClickOS VM 1

ClickOS VM 2

ClickOS VM n

TOOLSTACK

SWIG

xl

cosmosnetfront

Xen
bus

Click
MiniOS

Xen
bus

ClickOSControl

vif
netbackkernelnw
driverXen
store

ToNetfrontsw switch

FromNetfront

Xen UI libraries

middlebox
configuser-space

Pythontool

netfront

Xenbus

netfront

Xen
bus

Xen

NIC

© NEC Corporation 2009

Baseline Test
Boot as many guests as possible before system breaks●

Using ClickOS guests––●

Guests are mostly idle––

31

8 MB of RAM
1 VIF
Running arp responder configuration
Only arping guests to check they're working

18 August 2014

Didn't Work Quite Well...
●

Stopped test after 4K guests––●

All the domains were running, but:–●

Only first ~300 guests fully functional

System got extremely slow–

32

Took ~ 5 days
Up to ~ 100 seconds for creation of last guest (normally ClickOS
boots in ~30 milliseconds)

Dom0 unusable

18 August 2014

Domain Creation Time

33

18 August 2014

Domain Creation Time

92 s

34

18 August 2014

Two Types of Problems●

Hard limitations––●

Performance limitations––

35

Prevent guests from booting correctly
Only ~300 guests fully usable
Decreasing system performance
System unusable after just a few hundred guests

18 August 2014

HARD LIMITATIONS© NEC Corporation 2009

Issues●Cannot access guests' console–●

Guests' VIF is not created–●

Only first ~1300 guests have access to it

The back-end switch doesn't provide enough ports–

37

Only first ~1300 guests have usable VIF

Guests cannot access the Xenstore–●

Only first ~300 guests have accessible console

Only 1024 available

18 August 2014

Number of File Descriptors●

xenconsoled opens 3 FD per guest
–

●

Fix
–
–

38

/dev/xenbus; /dev/ptmx; /dev/pts/<id>;Linux can easily handle > 300K FD
Tune fs.file-max; nofile ulimit;

18 August 2014

Number of PTYs●●

xenconsoled opens 1 PTY per guest
Fix
–
–

●

Future––39

Linux can easily handle > 100K PTY
Tune kernel.tty.max
Only create PTY when user connects to console
This also reduces number of FD to 1 per guest

18 August 2014

Number of Event Channels●

3 Interdomain evtchn per guest
–
–
–

●

xenstore; console; VIF
64bit Dom0: NR_EVTCHNS == 4096
Dom0 runs out after ~1300 guests

Fix–

Upgrade to Xen 4.4 + Linux 3.14:
●

–

40

NR_EVTCHNS == 128K

Split services into stub domains

18 August 2014

Number of IRQs●

Linux runs out of IRQs to map evtchn
–

●

Fix
–
–

41

Limited by NR_CPUS
Build with: MAXSMP=y; NR_CPUS=4096
NR_IRQS == 256K

18 August 2014

vSwitch Ports●

Currently back-end switch supports up to few thousand ports––●

Workaround–●

Create multiple bridges

Longer-term fix–

42

Linux bridge: 1K
Open vSwitch: 64K

Develop a purpose-built back-end switch

18 August 2014

Summarizing●

Xen 4.4; Linux 3.14
fs.file-max; nofile ulimit
kernel.tty.max
MAXSMP=y; NR_CPUS=4096●

Not yet fixed:

●
●
●

–

43

Back-end switch ports

18 August 2014

PERFORMANCE LIMITATIONS© NEC Corporation 2009

Issues●

Overall system becomes too slow–

oxenstored●–

Xenconsoled●●

CPU limited after ~ 2K guests

Domain creation takes too long–45

CPU fully utilized after a few dozen guests

Affects migration too

18 August 2014

“Blind” optimizations●

4 Core Dom0
–
–
–

●
●
●

46

1 core for oxenstored
1 core for xenconsoled
2 cores for remaining processes

Pin all vCPUs to pCPUs
Round robin remaining 60 cores for guests
Put everything in a ramfs

18 August 2014

Tools' Optimizations●

xl toolstack–––

Disable xl background process (xl create -e)
Disable memory ballooning on Dom0
Never use domain name
●

–
–
●

Use specialized VIF hotplug script
Don't retrieve domain list on creation [PATCH]

oxenstored–

Use more recent version of Xenstore from:•47

This causes xl to retrieve all guest names from the Xenstore

https://github.com/mirage/ocaml-xenstore

18 August 2014

Creation Times with Optimizations

48

18 August 2014

Creation Times with Optimizations

2.3 s

49

18 August 2014

How much better is it?

50

18 August 2014

With Optimizations●

Improvement: system is still usable after 10K guests–●

However...
––51

Although domain creation time is far from ideal

xenstored still CPU heavy
xenconsoled still CPU heavy

18 August 2014

xenconsoled●

Two major optimizations––

Move from poll to epoll
On INTRODUCE_DOMAIN, search from last domid
●

●

52

Avoid listing all existing domains

CPU usage down to ~ 10% max.

18 August 2014

What Bottlenecks Remain?

53

18 August 2014

Domain Creation Breakdown

54

18 August 2014

Let's Look at the Toolstack Again●

The domain creation process is too complex for our specialized
VMs
–
–

●

Some checks take a lot of time–

55

Also makes the profiling really difficult and inaccurate
A lot of unnecessary Xenstore entries
Mainly checking for duplicate names

18 August 2014

xcl: XenCtrl Light●●

A very simplified toolstack
Small abstraction on top of libxc (~600 LOC)–

Optimized for our use case●●–

Reduced Xenstore usage
●
●

–

56

Only boots PV and PVH domains
Only supports VIFs
From 37 to 17 entries per guest
Less Xenstore operations

Doesn't check domain name18 August 2014

xl vs xcl

57

18 August 2014

xl vs xcl

2.3 s

0.1 s

58

18 August 2014

With xcl●●

Much better
But reducing the number of Xenstore entries is only a palliative– Eventually the issue will come back as we increase thenumber of guests●

59

Xenstore remains a major bottleneck

18 August 2014

lixs: LIghtweight XenStore
●
●
●
●

60

Work in progress (about 1 month)
Written from scratch but compatible with the Xenstore protocol
Currently ~1800 LOC
C++

18 August 2014

lixs vs oxenstored

Cumulative time: ~ 11 min

Cumulative time: ~ 8

61

18 August 2014

Breakdown with lixs

62

18 August 2014

lixs: Future Work●

Optimize protocol––●

Make Xenstore more specialized
Avoid all possible listing operations

Optimize implementation––

Remove unix sockets
Generic storage backend
●
●
●

63

std::map; noSQL DB; <your backend here>;
10K guests with std::map took 10m 3s
10K guests with boost::unordered_map took 7m 54s

18 August 2014

Where are we?
●●

Usable system running 10K guests
10K guests actually working–●

Lower domain creation times–––

First domain: < 10ms
With 10K domains: < 100ms

Recent test: 1,000 VMs running ICMP responder configuration,plus one running content cache (Minicache)–64

Although idle most of the timeAll 1,001 VMs work as expected!

18 August 2014

Will it work? Can we reach 100K?
●

There are no fundamental issues with Xen–●

Xenstore protocol needs work––

65

But we only tested it up to 10K guests
Make Xenstore more specialized
With 10K+ guests we need to avoid listings

18 August 2014

Future work●●●●

Improve lixs and Xenstore protocol
Multi thousand-port vSwitch
Have guests doing useful work
Scheduling––●

Reducing Memory Usage––

66

Number of guests much bigger than number of coresWith that many guests we'll have scheduling issues
Smaller image sizes
Share memory between guests booting same image

18 August 2014

Xenstore Entries: xl vs xcl
XL

XCL

1 = ""
vm = "/vm/2baefa82-612c-4e5b-a52d-396a91d5ad7b"name = "proxy"
cpu = ""

1 = ""

0 = ""

control = ""availability = "online"
memory = ""shutdown = ""static-max = "8192"

vm = "/vm/4c3f2a04-e39f-4ad8-9d7f-1b5556f02b34"

target = "8193"

name = "proxy"

videoram = "-1"device = ""domid = "48"suspend = ""console = ""event-channel = ""port = "2"

vif = ""
0 = ""

ring-ref = "3157830"

backend = "/local/domain/0/backend/vif/46/0"

type = "xenconsoled"backend-id = "0"state = "1"

tty = "/dev/pts/1"

handle = "0"

device = ""mac = "00:16:3e:32:ca:23"

vif = ""

1 = ""
backend = "/local/domain/0/backend/vif/46/1"

0 = ""

backend-id = "0"

backend = "/local/domain/0/backend/vif/48/0"

state = "1"

backend-id = "0"

handle = "1"
mac = "00:16:3e:2e:22:7c"

state = "1"

control = ""handle = "0"

shutdown = ""platform-feature-multiprocessor-suspend = "1"

mac = "00:00:00:00:00:00"

platform-feature-xs_reset_watches = "1"

1 = ""

data = ""backend = "/local/domain/0/backend/vif/48/1"

domid = "46"store = ""backend-id = "0"

port = "1"state = "1"

ring-ref = "3188551"

handle = "1"

console = ""backend = "/local/domain/0/backend/console/46/0"

mac = "00:00:00:00:00:00"

backend-id = "0"limit = "1048576"
type = "xenconsoled"output = "pty"tty = "/dev/pts/1"
port = "2"ring-ref = "3188550"

67

18 August 2014

Number of grants●

2 grants per domain–––

xenstore; xenconsole;With v1: 512 grants per frame
DEFAULT_MAX_NR_GRANT_FRAMES == 32
●

●

Fix
–

Boot xen with max_nr_grant_frames=512●

68

Maximum of (512 * 32) / 2 == 8K

Up to 128K domains

18 August 2014

It's Open Source!

Checkoutl ClickOS, Backend Switch, Xen optimizations and more!
l Tutorialsl Better performance than listed in the papers!

Page 69

We are always looking for…
Interns
Visiting researchers
Collaborations(and often full-time researchers)

Interested? felipe.huici@neclab.eu© NEC Corporation 2009