Containers Do Not Need Network Stacks
Ryo Nakamura
iijlab seminar 2018/10/16
Based on Ryo Nakamura, Yuji Sekiya, and Hajime Tazaki. 2018. “Grafting Sockets for
Fast Container Networking”. In ANCS ’18: Symposium on Architectures for Networking
and Communications Systems, July 23–24, 2018, Ithaca, NY, USA.

Containers• A package of an application execution environment• version-controllable• portable• lightweightContainer • Microservice architecture • An application (service) runs on a container• Multiple containers comprise a systemSystemContainer

ContainerContainer

Container

Container        ...

  2

The beginning of container networking• A container is a separated namespace in a host OS• Containers need to connect to other containers, host, and external networks• The conventional approach: Adapters and Links• Virtual NICs (veth interface in Linux)

Guest OS

Guest OS

Hardware
Virtualization
Host Kernel

Native HostContainerContainer

Namespace

Host Kernel

VM Network Stack Architecture

Host Kernel

Container Network Stack Architecture3

Overhead of container networking• Container involves• virtual NIC (veth)• virtual bridge and NAT (docker0) in the host network stack• Network performance degradation

Container• degrade throughput by 50%• increase latency by 25%
3530

latency (usec)

Throughput (Gbps)

40

2520
151050

Host Container

10987654
32
10Overhead
Bridge/NAT

TCP/IP Network Stack

Host Container

4

The long data path• from an application in container to NIC

App
TCP/UDP

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0IP

Container

Ethernet
vNIC

7060time (us)

CDF

• Time to transmit a packet increasesØThroughput and latency are degraded

3080 120
depth

Host10040

App4020

HostContainer
0vNIC

50160

Depth of called functions
from udp_sendmsg()

Ethernet

BridgeTCP/UDPIPIP

Ethernet

Ethernet

NIC

NIC

NAT

Host Container

Elapsed time in
udp_sendmsg()

5

State-of-the-art container networking
1. Interface Virtualization• Directly attaching interfaces to containers (bypassing host network stack)• macvlan, SR-IOV

2. Optimized Network Stacks• Reinventing the entire or a part of network stacks• FreeFlow[1], Cilium[2][1] Tianlong Yu, et al., “FreeFlow: High Performance Container Networking”. HotNets’16[2] Cilium, https://cilium.io/6

State-of-the-art: Interface Virtualization
App• Bypassing the host network stack

TCP/UDP• macvlan achieves comparable
network performance with native host[3]IPEthernet• Complicating management

vNIC• Outer networks must manage container networks• addressing, tenant separation,access control, etc• NAT conceals container networks
from outer networks and infrastructures

ContainerApp
TCP/UDPContainervNIC
Ethernet

BridgeIP
Ethernet

IPvNIC

Ethernet

NICNIC[3] Yang, et al, “Performance of Container Networking Technologies”, HotConNet’17

NAT7State-of-the-art: Optimized network stacks
Reinvent• Using high-speed packet I/O techniques• FreeFlow uses DPDK and RDMA
• Cilium uses XDP (eBPF)

Container

BridgeNATAppApp

TCP/UDPTCP/UDPIP

IP

Ethernet

Ethernet

vNIC

vNIC

vNIC

vNIC

Ethernet

Ethernet

IP

IP

Ethernet

Ethernet

NIC

NIC

Container

BridgeNAT

8

State-of-the-art: Optimized network stacks• Using high-speed packet I/O techniques• FreeFlow uses DPDK and RDMA
• Cilium uses XDP (eBPF)

App
TCP/UDPContainer• The long data path will be
the next bottleneck• It will be more significant bottleneckin comparison with native hosts

there is still thesame overhead dueto the architecture!

EthernetvNIC• Protocol processing cost do not disappear• In Arrakis OS[4], network protocol
processing occupies 100% ofprocessing cost on a simple
UDP echo server

IP

BridgevNIC
App

Ethernet

TCP/UDP

NAT

IP

IP

Ethernet

Ethernet

NIC

NIC[4] Simon, et al, “Arrakis: The Operating System is the Control Plane”, OSDI’14

Host9

The third approach:Bypassing container network stacks• A container isApp
TCP/UDP• just an application execution environment• not interested in how packets are delivered

IPContainer

Ethernet
vNIC

Then, we can bypass
container network stacks
to mitigate the overhead?

ContainerAppvNIC
Ethernet

BridgeTCP/UDP

HostIP

IP

Ethernet

Ethernet

NIC

NIC

NAT10

A question: Do containers really need network stacks?
Container

Port forward
docker -p 80:80

HostTo Global Addr
in Host Stack

User Client

Container has
network stack
The Internet

HostContainer

To Global Addr
in Host Stack

User Client

Container does nothave network stack
The Internet11

A question: Do containers really need network stacks?
Container

Port forward
docker -p 80:80

To Global Addr
in Host Stack

HostUser Client

Container has
network stack
The InternetNodifferences
HostContainer

To Global Addr
in Host Stack

User Client

Container does nothave network stack
The Internet12

The third approach:Bypassing container network stacks, cont’d• A container isApp• just an application execution environment• not interested in how packets are delivered• Then, we can bypass container network stacks

TCP/UDP• Network stack separation should be retainedvNIC• docker run --net=host can cause
unintended or malicious resource usesIPEthernetContainer• connecting App on a container to the host• with proper access control

AppvNIC
Ethernet• address, port, protocol, etc• A new mechanism is needed

Container

BridgeTCP/UDP

HostIPIP

Ethernet

Ethernet

NIC

NIC

NAT13

The third approach:Bypassing container network stacks, cont’d• A container isApp• just an application execution environment• not interested in how packets are delivered• Then, we can bypass container network stacks

TCP/UDP• Network stack separation should be retainedvNIC• docker run --net=host can cause
unintended or malicious resource uses
• address, port, protocol, etc• A new mechanism is needed

SocketLayer!

• connecting App on a container to the host
• with proper access controlIP

Container

Ethernet

Container

AppvNIC
Ethernet

BridgeTCP/UDP

HostIP

IP

Ethernet

Ethernet

NIC

NIC

NAT14

Approach: Socket-Grafting• Grafting sockets in containers onto sockets in hosts• A socket-layer communication channel design• graft =üOne Network stack on the data pathüIndependent from network stack
implementations

Container

Host

Application

TCP/UDPTCP/UDPIPIP

Ethernet

Ethernet

Virtual NIC

Physical NIC-15

Mechanism: AF_GRAFT• A new address family for grafting sockets• Applications in containers create AF_GRAFT sockets• AF_GRAFT sockets are grafted onto other AF sockets across the network
namespace boundarywrite(), send()read(), recv()

Container
Hostwrite(), send()

TX

read(), recv()

RX16

Graft endpoint• Names for AF_GRAFT sockets in the bind() semantics-:.3 -• Arbitrary strings• GRAFT <-> Host endpoint mapping• AF_GRAFT manages themapping table per container• preventing misuse of
the host namespace
Graft endpoint
ep-httpep4
ep6
ep-un

Host endpoint
10.0.0.1:80
10.0.0.1:8080[2001:db8::beef]:8080/tmp/un-sk

-:01

G C
01

610

G C

F HG F
GF
HH

F I

GF

GF

HA H

F I

HHGF

H

H

GF

F I

HHGF

H F/G812AN

GIC 4

EH F C
C-4N

E

1017

AF_GRAFT Socket API/* Structure describing a graft socket address (endpoint) */struct sockaddr_gr {__kernel_sa_family_t sgr_family;char sgr_epname[AF_GRAFT_EPNAME_MAX];};
int sock;struct sockaddr_gr saddr_gr;
sock = socket(AF_GRAFT, SOCK_STREAM, IPPROTO_TCP);saddr_gr.sgr_family = AF_GRAFT;
strncpy(saddr_gr.sgr_epname, “ep-http”, 7);bind(sock, (struct sockaddr *)&saddr_gr, sizeof(saddr_gr));/* Then, you can use sock as usual TCP sockets */18

Outbound connections• Dynamic-port graft endpoint• It uses randomly selected port numbers == typical client sockets• For example, mapping ep-out on X.X.X.X:random
sock = socket(AF_GRAFT, SOCK_STREAM, IPPROTO_TCP);
saddr_gr.sgr_family = AF_GRAFT;
strncpy(saddr_gr.sgr_epname, “ep-out”, 7);bind(sock, (struct sockaddr *)&saddr_gr, sizeof(saddr_gr));/* Then sock is grafted onto source IP:RandomPort socket*/connect(sock, (struct sockaddr *)&dst, sizeof(dst));19

Implementation• https://github.com/upa/af-graft, AF_GRAFT kernel module• no kernel patches (but overwriting an existing AF number, AF_IPX )• Grafting is implemented as function call
• no buffering, queueing, messaging => minimal overhead!

• A few socket options for practical uses
• A modified iproute2 for configuring the mapping table$ ip graft add ep-http type ipv4 addr 10.0.0.1 port 80$ ip graft add ep-out type ipv4 addr 10.0.0.2 port dynamic$ ip graft del ep-un$ ip graft show

20

Existing application with AF_GRAFT• AF_GRAFT is a new address family• Applications need source code modifications• It is easy because of the familiar socket API, but difficult to deployØOverriding system calls by the LD_PRELOAD trick• $ LD_PRELOAD libgraft-hijack.so app• hijacking functions in shared library• Hijacking:1.
2.

getaddrinfo()
socket(), bind(), and connect()• to convert address family-dependent socket operations into AF_GRAFTcapable ones
21

getaddrinfo()• It was carefully designed to achieve AF-independent codes• Our modified getaddrinfo() can return AF_GRAFT and sockaddr_gr• However, unfortunately, this is not the case in practical applications...
/* IPv4 */
if (server_res->ai_family == AF_INET) {... make ipv4 socket ...
}/* IPv6 */else if (server_res->ai_family == AF_INET6) {... make ipv6 socket ...
}/* Unknown protocol */else {errno = EAFNOSUPPORT;return -1;}!

from iperf3[5] Jun-ichiro itojun Itoh, KAME Project, “Implementing AF-independent application”, http://www.kame.net/newsletter/19980604/

22

Hijacking socket() and bind()• Hijacked socket()• returns AF_GRAFT sockets instead of AF_INET/INET6• Hijacked bind()• uses sockaddr_gr instead of sockaddr_in/in6• An env variable specifies which sockaddr convert to which sockaddr_gr• GRAFT_CONV_PAIRS=“0.0.0.0:80=ep-http”23----A--()---()-24

bind() before connect() for outbound connections
1. connect() does not need to call bind()
2. But, AF_GRAFT requires bind() to determine host sockets
ü The hijacked connect() calls bind before connect()• sendto() and sendmsg() are also hijacked in the same manner25

Evaluation• Throughput and latency• iperf3 and sockperf

Baseline performance• HTTP server• NGINX and siege• Message Queue

docker0 or
AF_GRAFT

MicroserviceArchitecture• Zero MQ• Networking• native host• docker0 (NAT)• AF_GRAFT• with libgraft-hijack.so40Gbps link
Host:
Linux 4.4.0, Intel Core i7-3770K 3.5GHz CPU,
32GB memory, Mellanox ConnectX-4 LX 40Gbps NIC
26

Throughput
Receive
40

80

35

3570

30
25
2015
10

Throughput (Gbps)

40
Throughput (Gbps)

Throughput (Gbps)

Transmit

30
25
20
15106050
40
30
205510000

Host GRAFT NAT

Host GRAFT NAT

Container to Containeron the same hostGRAFT• AF_GRAFT successfully mitigates the degradation• Container to container communication via AF_GRAFT is the same as thecommunication via the loopback interface

Bridge27

Latency
109876543210

UDP

latency (usec)

latency (usec)

TCP

Host GRAFT NAT

109876543210

Host GRAFT NAT• As well as the throughput test, AF_GRAFT also mitigates degradation fromthe latency perspective
28

HTTP server benchmark
Transaction rate (Ktps)

50 concurrent sessions
30
25
20
15
10
50

Increasedby 40%GRAFT
NAT

HTTP GET

64 256 1K 4K 16K 64K 256K 1M 4M 16M 64M
File size (byte)
29

Message Queue benchmark
approximately
doubledThroughput (Gbps)

4035RX andmeasured

GRAFT
NAT30TX

252015105064

128 256 512

1K 2K 4K 8K 16K 32K 64K 128K 256K
Message size (Byte)

30

Limitations• The LD_PRELOAD trick is not applicable to• Statically linked libraries• Golang that implements syscall without libc

• AF_GRAFT does not improve network stack performance• It never outperforms the performance of native hosts• Network-sensitive applications• e.g., Container-based NFV

31

Conclusion• Socket-Grafting• Containers with network-insensitive applications do not need network stacks• Bypassing container’s network stack by exploiting the socket layer• A new address family, called AF_GRAFT, as a practical mechanism for grafting• The evaluation results demonstrated• Mitigating the network performance degradation due to the long data path
• HTTP: 10-40% throughput improvement• ZeroMQ: up to doubled the throughput and 30% shorter latenct

32

ToDo
• Integrating AF_GRAFT into Docker• Docker network driver plugin?
• Option like -p?
• We need comments or partners implementing such plugins ;)• Integrating AF_GRAFT into Kubernetes• More complicated due to the service IP abstraction and load balancing• The Container Network Interface (CNI) focuses on the traditional abstraction (?)• Go Go Netdev 0x13!

33