Part1: PASTE:
Network Stacks Must Integrate with NVMM
Abstractions
Part2: Report on ACM HotNets 2016
Michio Honda (NEC Laboratories Europe)With acknowledge to Lars Eggert and Douglas Santry
IIJ-II Seminar
December 26th, Tokyo, Japan*work done

Motivation• Non-Volatile Main Memories (NVMMs)• Persistent• Byte-addressable• Low latency
https://www.hpe.com/us/en/servers• 10s-1000s of ns/persistent-memory.html• Shift from block- to byte-granularity persistency• OS abstractions• Direct access to mmap()-ed files• Data structures• Filesystems and databases

What are implications for networking?

Case Study: Write-Ahead Logging• Persist client’s request prior to acknowledgment• Durably store data into a log file to mask overhead ofupdating primary database read()
write()/fsync() ormemcpy()/msync()(e.g. B-tree) to the clientApp• 1KB commit

(3)

(4)

Networkstack

Storage
stack

(2)
(5)

client(1)• 2030 us• Networking takes 40 us

NIC

DRAM

SSD/DIskCase Study: Write-Ahead Logging• Persist client’s request prior to acknowledgment• Durably store data into a log file to mask overhead of
updating primary database read()
write()/fsync() ormemcpy()/msync()(e.g. B-tree) to the clientApp• 1KB commit

(3)

(4)

NetworkstackStoragestack

(2)

DRAM

NVMM

NIC

Emulated using areserved region ofDRAM(5)client(1)• 2000 42 us• Networking takes 40 us• This 2 us is not small

Case Study: Write-Ahead Logging• Parallel requests are serialized on each core

33 % throughput decrease, 50 % latency increaseData Copies Matter• Cache Misses• Copy to tmp buffer (e.g., read()) is cheap• Logging always happens to a different destination
memcpy()read()

kernel buffer

app buffer

log file (mmap()-ed)

Overall cache misses Largest Contributor
Networking only

0.0004 %net_rx_action() (84%)

Networking + NVMM
4.4121 %(read() + memcpy() + msync())

memcpy() (98%)

Networking + NVMM
(read()+msync())

sys_read() (99 %)8.3451 %We must avoid data copy for logging!

Packet Store (PASTE) Overview• Static packet buffers on a named NVMM region• DMA to NVMM
• Zero-copy APIs
• Fast loggingmetadata only
(e.g., buffer index)(3)

AppNetwork
stackNVMM
clientStorage
stack/mnt/nvmm/pktbufs
(2)

(1)

(4)

NIC/mnt/pmem/appmdFast Logging with PASTE
Application(1) Read data (zero copy)(2) Write metadata entry(3) Flush (buffer and metadata)

User

netmap API

Kernelmmap()
metadata header

TCP/IP
input
andoutput/mnt/nvmm/pktbufs
buf_ofs: 123

metadata entries
buf_idx off
0100
1100
3100

len
1135
932
1024/mnt/nvmm/myapp_metadata

packet buffers(static)

NIC ring/mnt/nvmm/pktbufs

Fast Logging with PASTE
UnreadRead orwritten

Application(1) Read data (zero copy)(2) Write metadata entry(3) Flush (buffer and metadata)

User

netmap API

Kernelmmap()
metadata header

TCP/IP
input
andoutput/mnt/nvmm/pktbufs
buf_ofs: 123

metadata entries
buf_idx off
0100
1100
3100

len
1135
932
1024

/mnt/nvmm/myapp_metadata

packet buffers(static)

NIC ring/mnt/nvmm/pktbufs

Fast Logging with PASTE
UnreadRead orwritten

Application(1) Read data (zero copy)(2) Write metadata entry(3) Flush (buffer and metadata)

User

netmap APIKernel

mmap()
metadata header

TCP/IPinput
andoutput/mnt/nvmm/pktbufs
buf_ofs: 123

metadata entries
buf_idx off
0100
1100
3100

len
1135
932
1024/mnt/nvmm/myapp_metadata

packet buffers(static)

NIC ring/mnt/nvmm/pktbufs

Fast Logging with PASTE
UnreadRead orwritten

Application(1) Read data (zero copy)(2) Write metadata entry(3) Flush (buffer and metadata)

User

netmap API

Kernelmmap()
metadata header

TCP/IP
input
andoutput/mnt/nvmm/pktbufs
buf_ofs: 123

metadata entries
buf_idx off
0100
1100
3100

len
1135
932
1024/mnt/nvmm/myapp_metadata

packet buffers(static)

NIC ring/mnt/nvmm/pktbufs

Fast Logging with PASTE
UnreadRead orwritten
FlushedApplication(1) Read data (zero copy)(2) Write metadata entry(3) Flush (buffer and metadata)

User

netmap API

Kernelmmap()
metadata header

TCP/IP
input
andoutput/mnt/nvmm/pktbufs
buf_ofs: 123

metadata entries
buf_idx off
01001100
3100

len
1135
932
1024/mnt/nvmm/myapp_metadata

packet buffers(static)

NIC ring/mnt/nvmm/pktbufs

DMA is performed to L3 cache (DDIO)

Fast Logging with PASTE
UnreadRead orwritten
FlushedApplication(1) Read data (zero copy)(2) Write metadata entry(3) Flush (buffer and metadata)

User

netmap API

Kernelmmap()
metadata header

TCP/IP
input
andoutput/mnt/nvmm/pktbufs
buf_ofs: 123

metadata entries
buf_idx off
0100
1100
3100

len
1135
932
1024/mnt/nvmm/myapp_metadatapacket buffers(static)

NIC ring/mnt/nvmm/pktbufs

DMA is performed to L3 cache (DDIO)

Fast Logging with PASTE
UnreadRead orwritten
FlushedApplication(1) Read data (zero copy)(2) Write metadata entry(3) Flush (buffer and metadata)

User

netmap API

Kernelmmap()
metadata header

TCP/IP
input
andoutput/mnt/nvmm/pktbufs
buf_ofs: 123

metadata entries
buf_idx off
0100
1100
3100

len
1135
932
1024/mnt/nvmm/myapp_metadata

packet buffers(static)

NIC ring/mnt/nvmm/pktbufs

DMA is performed to L3 cache (DDIO)

Fast Logging with PASTE
UnreadRead orwritten
FlushedApplication(1) Read data (zero copy)(2) Write metadata entry(3) Flush (buffer and metadata)

User

netmap API

Kernelmmap()
metadata header

TCP/IP
input
andoutput/mnt/nvmm/pktbufs
buf_ofs: 123

metadata entries
buf_idx off
0100
1100
3100

len
1135
932
1024

/mnt/nvmm/myapp_metadata

packet buffers(static)

NIC ring/mnt/nvmm/pktbufs

DMA is performed to L3 cache (DDIO)

Fast Logging with PASTE
UnreadRead orwritten
FlushedApplication(1) Read data (zero copy)(2) Write metadata entry(3) Flush (buffer and metadata)

User

netmap APIKernel

mmap()
metadata header

TCP/IPinput
andoutput/mnt/nvmm/pktbufs
buf_ofs: 123

metadata entries
buf_idx off
0100
1100
3100

len
1135
932
1024/mnt/nvmm/myapp_metadata

packet buffers(static)

NIC ring/mnt/nvmm/pktbufs

DMA is performed to L3 cache (DDIO)

Fast Logging with PASTE
Idempotent
Application(1) Read data (zero copy) request(2) Write metadata entry(3) Flush (buffer and metadata)

UnreadRead orwritten
Flushedmmap()

netmap API

metadata header

metadata entries
len1135
932
1024/mnt/nvmm/myapp_metadata

Kernel
TCP/IP
input
andoutput/mnt/nvmm/pktbufs
buf_ofs: 123

buf_idx off
0100
1100
3100

User

packet buffers(static)

NIC ring/mnt/nvmm/pktbufs

DMA is performed to L3 cache (DDIO)
Unnecessary data is not flushed to DIMM

Fast Logging with PASTE
UnreadRead orwrittenFlushedApplication(1) Read data (zero copy)(2) Write metadata entry(3) Flush (buffer and metadata)

User

netmap APIKernelmmap()
metadata header

TCP/IPinput
andoutput/mnt/nvmm/pktbufs
buf_ofs: 123

metadata entries
buf_idx off
0100
1100
3100

len
1135
932
1024/mnt/nvmm/myapp_metadata

packet buffers(static)

NIC ring/mnt/nvmm/pktbufs

DMA is performed to L3 cache (DDIO)
Unnecessary data is not flushed to DIMM

Fast Logging with PASTE
UnreadRead orwrittenFlushedApplication(1) Read data (zero copy)(2) Write metadata entry(3) Flush (buffer and metadata)

User

netmap APIKernelmmap()
metadata header

TCP/IPinput
andoutput/mnt/nvmm/pktbufs
buf_ofs: 123

metadata entries
buf_idx off
0100
1100
3100

len
1135
932
1024/mnt/nvmm/myapp_metadata

packet buffers(static)

NIC ring/mnt/nvmm/pktbufs

DMA is performed to L3 cache (DDIO)
Unnecessary data is not flushed to DIMM

Fast Logging with PASTE
UnreadRead orwritten
FlushedApplication(1) Read data (zero copy)(2) Write metadata entry(3) Flush (buffer and metadata)

User

netmap APIKernelmmap()
metadata header

TCP/IPinput
andoutput/mnt/nvmm/pktbufs
buf_ofs: 123

metadata entries
buf_idx off
0100
1100
3100

len1135
932
1024/mnt/nvmm/myapp_metadata

packet buffers(static)

NIC ring/mnt/nvmm/pktbufs

DMA is performed to L3 cache (DDIO)
Unnecessary data is not flushed to DIMM

Implementation• Extension to netmap memory allocator• Claim packet buffers from a given file backed by NVMM
• e.g., pkt-gen -i eth1@/mnt/pmem/bufs -f rx• Server app using the netmap API can easily implement
logging

2016-9-12© 2016 NetApp, Inc. All rights reserved.

21

Preliminary Results• Implementation• Extend the netmap framework• Stackmap for TCP/IP

10-88 % throughput increase, 9-46 % latency reduction

Related Work

• Enhanced network stacks• MegaPipe (OSDI’12), Stackmap (ATC’16), Fastsocket(ASPLOS’16)• IX and Arrakis (OSDI’14), mTCP (NSDI’13), Sandstorm(SIGCOMM’14), MICA (NSDI’14)
No NVMM aware• NVMM filesystems• BPFS (SOSP’09), NOVA (FAST’15)• NVMM databases• NVWAL (ASPLOS’15), REWIND (VLDB’15), NV-Tree(FAST’15)
No networking aware

Conclusion
PASTE: Fast logging withnamed packet buffers on NVMM and zero-copy API• Implications• Network stacks are now a bottleneck fordurably storing data
• Improving network and storage stacks inisolation is not enough• We need new stacks design

HotNets 2016 Reports• ACM Workshop on Hot Topics in Networks• Focus on new ideas and future directions in networking• November 9-10 2016 @ Atlanta• ~90 attendees (Invitation only)• 1 author per paper• Invited people in the community• Lottery2016-9-12© 2016 NetApp, Inc. All rights reserved.

25

Submission and Reviews• 30 papers (out of 108 submitted)• 3-6 reviews• 48 papers on PC discussion• We submitted 2 papers• 1 rejected (received 4 reviews)• 3 weak rejects + 1 weak accept• 1 accepted (received 6 reviews)• 1 accept + 4 weak accepts + 1 weak reject

2016-9-12© 2016 NetApp, Inc. All rights reserved.

26

Workshop Format• Format was awesome• Very friendly – like a university internal workshop• Young people can be active• Senior people (incl. TPCs) comment and/or raise discussion

2016-9-12© 2016 NetApp, Inc. All rights reserved.

27

Workshop Topics•ISPs••

Resource allocation••

Topology, deadlockes in RDMA networks, debugging

Mobile••SaaS, recommendation etc

Datacenters••

RDMA-based interfaces

Social networks and clouds••

ML for cluster scheduling etc, blockchain for the Internet (BGP, DNS)

Container networking••

Frontier networks, Traffic engineering using MPTCP, monitoring

Low-energy consumption network stack, network personalization

Network monitoring and analysis•

Using programmable switches•Wireless (MIT)•NF modeling verification•DDoS

2016-9-12© 2016 NetApp, Inc. All rights reserved.

28