PASTE: A Network Programming Interface
for Non-Volatile Main Memory
Michio Honda (NEC Laboratories Europe)
Giuseppe Lettieri (Università di Pisa)
Lars Eggert and Douglas Santry (NetApp)
IIJ lab Seminar, May 2nd 2018, Tokyo, Japan
The paper was presented at NSDI’18

Summary● PASTE is a network programming interface that:○ Enables zero copy to NVMM
○ Helps apps organize persistent data structures on NVMM
○ Lets apps use modern TCP/IP and be protected○ Offers high-performance network stack even w/o NVMM

https://github.com/luigirizzo/netmap/tree/paste

Review: Memory HierarchySlow, block-oriented persistence
Byte access w/
load/store

CPUCaches
Main Memory

Block access w/
system calls

HDD / SSD

5-50 ns
70 ns
100-1000s us

Review: Memory Hierarchy
Fast, byte-addressable persistence
CPUCaches
Byte access w/
load/store
Block access w/
system calls

5-50 ns

70 ns -1000s ns
Main Memory

HDD / SSD

100-1000s us

Networking is faster than disks/SSDs
1.2KB durable write over TCP/HTTP
Cables, NICs, TCP/IP,
socket API
ClientSyscall, PCIe bus,
physical media
Server23us

SSD
1300usNetworking is slower than NVMM
1.2KB durable write over TCP/HTTP
Cables, NICs, TCP/IP,socket API
ClientMemcpy, memory bus,
physical media
Server

23us

NVMM
2us

Networking is slower than NVMM
1.2KB durable write over TCP/HTTP
Cables, NICs, TCP/IP,
socket API
ClientClientClientMemcpy, memory bus,
physical media
ServerNVMM

nevts = epoll_wait(fds)for (i =0; i < nevts; i++) {read(fds[i], buf);...
memcpy(nvmm, buf);...
write(fds[i], reply)}

Innovations at both stacks

Network stack

Storage stack

MegaPipe [OSDI’12]
SeastarmTCP [NSDI’14]
IX [OSDI’14]
Stackmap [ATC’16]

NVTree [FAST’15]
NVWal [ASPLOS’16]
NOVA [FAST’16]
Decibel [NSDI’17]
LSNVMM [ATC’17]

Stacks are isolated

Network stack
MegaPipe [OSDI’12]
SeastarmTCP [NSDI’14]
IX [OSDI’14]
Stackmap [ATC’16]

Costs ofmoving data

Storage stack
NVTree [FAST’15]
NVWal [ASPLOS’16]
NOVA [FAST’16]
Decibel [NSDI’17]
LSNVMM [ATC’17]

Bridging the gap

Network stack

Storage stack

MegaPipe [OSDI’12]
SeastarmTCP [NSDI’14]
IX [OSDI’14]
Stackmap [ATC’16]

NVTree [FAST’15]
NVWal [ASPLOS’16]
NOVA [FAST’16]
Decibel [NSDI’17]
LSNVMM [ATC’17]

PASTE

PASTE Design Goals● Durable zero copy
○ DMA to NVMM

● Selective persistence○ Exploit modern NIC’s DMA to L3 cache● Persistent data structures○ Indexed, named packet buffers backed fy a file

● Generality and safety○ TCP/IP in the kernel and netmap API● Best practices from modern network stacks○ Run-to-completion, blocking, busy-polling, batching etc

PASTE in Action
Plog/mnt/pm/plog
pbuf off lenZero copy

App threadPringslot [0] [7]cur20 2726212225
23 24

Ppool (shared memory)/mnt/pm/pp
[0]
[4]
[8]

Pbufs

user
kernelFile system/mnt/pm
TCP/IPNIC

PASTE in Action
Plog/mnt/pm/plog
pbuf off len

Zero copy

App threadPringslot [0] [7]cur20 27
26
21
22
25
23 24

Ppool (shared memory)/mnt/pm/pp
[0]
[4]
[8]

Pbufs

user
kernelFile system/mnt/pm

TCP/IPNIC

PASTE in Action
1. Run NIC I/O and TCP/IP

Zero copy

App threadPlog/mnt/pm/plog
pbuf off len

Pring
slot [0] [7]cur20 2726212225
23 24

Ppool (shared memory)/mnt/pm/pp
[0]
[4]
[8]

Pbufs

user
kernelFile system/mnt/pm
TCP/IP
NIC● poll() system call

PASTE in Action
1. Run NIC I/O and TCP/IP

Zero copy

App threadPlog/mnt/pm/plog
pbuf off len

Pring
slot [0] [7]
cur
20 27
26
21
22
25
23 24

Ppool (shared memory)/mnt/pm/pp
[0]
[4]
[8]

Pbufs

user
kernelFile system/mnt/pm
TCP/IP
NIC● poll() system call
○ Got 7 in-order TCP
segments

PASTE in Action
1. Run NIC I/O and TCP/IPApp threadPlog/mnt/pm/plog
pbuf off len

slots

Zero copyPringslot [0] [7]cur0 27 tail61253 4● poll() system call
○ They are set to Pring

Ppool (shared memory)/mnt/pm/pp
[0]
[4]
[8]

Pbufs

user
kernelFile system/mnt/pm
TCP/IPNIC

PASTE in Action
1. Run NIC I/O and TCP/IP

Zero copy

App threadPlog/mnt/pm/plog
pbuf off len

Pring
slot [0] [7]cur0 27 tail61253 4

Ppool (shared memory)/mnt/pm/pp[0][4][8]Pbufs

userkernelFile system/mnt/pm
TCP/IPNIC● Return from poll()

PASTE in Action
1. Run NIC I/O and TCP/IP
2. Read data on Pring

Zero copy

App threadPlog/mnt/pm/plog
pbuf off len

Pring
slot [0] [7]cur
0 27 tail61253 4

Ppool (shared memory)/mnt/pm/pp
[0]
[4]
[8]

Pbufs

user
kernelFile system/mnt/pm
TCP/IPNIC

PASTE in Action
Plog/mnt/pm/plog
pbuf off len

Pring
slot [0] [7]cur
0 27 tail61253 4● flush Pbuf data fromCPU cache to DIMM
○ clflush(opt) instruction

Zero copy

App thread

1. Run NIC I/O and TCP/IP
2. Read data on Pring
3. Flush Pbuf(s)Ppool (shared memory)/mnt/pm/pp
[0]
[4]
[8]

Pbufs

userkernel

File system/mnt/pm
TCP/IPNIC

PASTE in Action

Zero copy

App thread

1. Run NIC I/O and TCP/IP
2. Read data on Pring
3. Flush Pbuf(s)4. Flush Plog entry(ies)Pringslot [0] [7]cur
0 27 tail61253 4

Ppool (shared memory)/mnt/pm/pp
[0]
[4]
[8]

Pbufs

Plog/mnt/pm/plog
pbuf off len
1 96 120

userkernel

File system/mnt/pm
TCP/IPNIC● Pbuf is persistent
data representation○ Base address is statici.e., file (/mnt/pm/pp)○ Buffers can berecovered after reboot

PASTE in Action

Zero copy

App thread

1. Run NIC I/O and TCP/IP
2. Read data on Pring
3. Flush Pbuf(s)4. Flush Plog entry(ies)
5. Swap out Pbuf(s)Pringslot [0] [7]cur
0 27 tail68253 4

Ppool (shared memory)/mnt/pm/pp[0][4][8]Pbufs

Plog/mnt/pm/plog
pbuf off len
1 96 120user
kernelFile system/mnt/pm
TCP/IPNIC● Prevent the kernel
from recycling thebuffer

PASTE in Action

Zero copy

App thread

1. Run NIC I/O and TCP/IP
2. Read data on Pring
3. Flush Pbuf(s)4. Flush Plog entry(ies)
5. Swap out Pbuf(s)Pringslot [0] [7]cur
0 27 tail
10895
3 4

Ppool (shared memory)/mnt/pm/pp[0][4][8]

Pbufs

Plog/mnt/pm/plog
pbuf off len
1 96 120
2 96 768
6 96 987
user
kernelFile system/mnt/pm
TCP/IPNIC● Same for Pbuf 2 and 6

PASTE in Action

Zero copy

App thread

1. Run NIC I/O and TCP/IP
2. Read data on Pring
3. Flush Pbuf(s)4. Flush Plog entry(ies)
5. Swap out Pbuf(s)6. Update Pring

Pring Ppool (shared memory)/mnt/pm/pp[7]
0 27 tail10 cur [0]
8[4]
Pbufs
95[8]
3 4

slot [0]Plog/mnt/pm/plog
pbuf off len
1 96 120
2 96 768
6 96 987
userkernel

File system/mnt/pm
TCP/IPNIC● Advance cur○ Return buffers in slot
0-6 to the kernel atnext poll()

PASTE in Action

Zero copy

App thread

1. Run NIC I/O and TCP/IP
2. Read data on Pring
3. Flush Pbuf(s)4. Flush Plog entry(ies)
5. Swap out Pbuf(s)6. Update Pring

Pring Ppool (shared memory)/mnt/pm/pp[7]
0 27 tail10 cur [0]
8[4]
Pbufs
95[8]
3 4slot [0]Plog/mnt/pm/plog
pbuf off len
1 96 120
2 96 768
6 96 987
userkernel

File system/mnt/pm
TCP/IP
NIC

Write-Ahead Logs

PASTE in Action

Zero copy

App thread

1. Run NIC I/O and TCP/IP
2. Read data on Pring
3. Flush Pbuf(s)4. Flush Plog entry(ies)
5. Swap out Pbuf(s)6. Update Pring

Pring Ppool (shared memory)/mnt/pm/pp[7]
0 27 tail10 cur [0]8[4]Pbufs
95[8]3 4slot [0]3Plog/mnt/pm/plog

055

7

B+tree(1, 96, 120)
(2, 96, 987)
(6, 96, 512)
user
kernelFile system/mnt/pm
TCP/IPNIC● We can organize various
data structures in Plog

Evaluation
1. How does PASTE outperform existing systems?
2. Is PASTE applicable to existing applications?
3. Is PASTE useful for systems other than file/DB storage?

How does PASTE outperform existing systems?

64B

What if we use more
complex data structures?
1280B

WAL

B+tree (all writes)

How does PASTE outperform existing systems?

64B

1280B

WAL

B+tree (all writes)Is PASTE applicable to existing applications?
● RedisYCSB (read mostly) YCSB (update heavy)Is PASTE useful for systems other than DB/filestorage?
● Packet logging prior to forwarding○ Fault-tolerant middlebox [Sigcomm’15]○ Traffic recording● Extend mSwitch [SOSR’15]○ Scalable NFV backend switch

Multicore Scalability● WAL throughput

Conclusion● PASTE is a network programming interface that:○ Enables durable zero copy to NVMM
○ Helps apps organize persistent data structures on NVMM
○ Lets apps use modern TCP/IP and be protected○ Offers high-performance network stack even w/o NVMM

https://github.com/luigirizzo/netmap/tree/paste
micchie@sfc.wide.ad.jp or @michiohFurther Opportunity with Co-designed Stacks● What if we use higher access latency NVMM?
○ e.g., 3D-Xpoint● Overlap flushes and processing with clflushopt andmfence before system call (triggers packet I/O)
○ See the paper for results
Systemcall
Receive new
requests

Examinerequest clflushopt

Examinerequest clflushopt

mfenceWait forflushes done

Systemcall
Send
responses

timeExperiment Setup●●●●Intel Xeon E5-2640v4 (2.4 Ghz)
HPE 8GB NVDIMM (NVDIMM-N)Intel X540 10 GbE NIC
Comparison○ Linux and Stackmap [ATC’15] (current state-of-the art)
○ Fair to use the same kernel TCP/IP implementation

PASTE Concept● DMA to NVMM, just flush if necessaryclflush

App
Socket / FIle API

TCP/IP

File Systems

Memory mgmt.
fileNICs
DMA write

NVMM absts.

NVMMmmap()ed

Netmap API

Discussion and Future Work● Encrypted data
○ Modern/Smart NICs

● Impact on NVMM wear
● In the paper○ How to cope with higher NVMM access latency■e.g., Intel 3D-Xpoint

PASTE Concept● DMA to NVMM, just flush if necessaryread() (write() later)

memcpy()/clflush

App

User

TCP/IP

File Systems

Memory mgmt.
HWDRAM

NVMM absts.
NVMM

file

Netmap API

Socket / FIle API

TCP/IP

File Systems

Memory mgmt.
fileNICsNICs

DMA write

DMA write

LegacyApp

NVMM absts.

NVMM

PASTEmmap()edKernelmmap()ed

Socket / File API

clflushPASTE Concept● DMA to NVMM, just flush if necessary
clflushAppSocket / FIle API

TCP/IP

File Systems

Memory mgmt.
file

NVMM absts.

NVMM

NICs
DMA write

Write-ahead logging performance for 1.2 KB data

PASTEmmap()ed

Netmap API

Is PASTE useful to systems other than DB/filestorage?
● Packet logging prior to forwarding○ Fault-tolerant middlebox [Sigcomm’15]○ Traffic recording● Extend mSwitch [SOSR’15]