Pluginizing QUICQuentin De Coninck, François Michel, Maxime Piraux, Florentin Rochet,Thomas Given-Wilson, Axel Legay, Olivier Pereira, Olivier Bonaventure
SIGCOMM 2019UCLouvain, Belgium

1

QUIC
● Currently under standardization at IETF
HTTP/2● Providing TCP/TLS1.3 service atop UDP
TLS○ (Nearly) everything is encrypted
TCP○ Reliable, in-order, multistream data transfer
IP
H FFF● Frame-based protocol● Large companies are deploying both client and server sides○ Limited interoperability today

HTTP/3
TLS QUICUDPIP2

Application Requirements EvolveSimple reliable file transfer

Mobility, (live) video streaming

Latency critical, partially unreliable

Transport protocols need to adapt!

3

Negotiating Extensions: The Human Protocol
How could I
learn Japanese?

Hello, I speak [French, English, Dutch]
se, English]
enapa[Jdntarsedn
Hi, I uOk, let’s speak English!

Require support from both sides: delayed deployment

4Adding Monitoring to an Implementation

Clients seemto have poor
performance

A Client AppAB QUIC lib● High-performance, native server implementation● Client side based on open-source 3rd-party library○ A opens Pull Request to extend the library

Activatemonitoring withspecific filters■ Get monitoring information for some connections○ But not merged by B
No, thanks.

BA
5

What If We Could Have...
A base, simple implementation enabling
Personalized Transport Functionalities
Per-Connection Deployment

6Agenda●●●●Motivations
Pluginized QUIC Design
Evaluating PQUIC with Use Cases
Conclusion and Future Works

7

Pluginized QUIC
● Revisit the structure of protocol implementations● Transport protocol = set of basic functions (protocol operations)Add newfunctions○ Plugin: new set of protocol functions (modified or added)
Changealgorithms

RTO
RTO
Computation
Computation

HeaderPreparation

Add to sendbufferTransport

Unreliable
Message...

● Dynamic per-connection customization
Different
algorithms

C1

RTO
RTO
Computation
Computation

HeaderPreparation

Add to sendbufferC2

RTO
Computation

HeaderPreparation

Add to sendbuffer...
Unreliable
Message...

Isolation
betweenconnections
8

Let’s monitor
the client state.

Exchanging Plugins: 1st Connection
Let’s request
monitoring
Initial: Client Hello - “Hey, I support multipath”
Initial: Server Hello - “I want to inject monitoring”

Encrypted - PLUGIN_REQUEST(monitoring)
Encrypted - PLUGIN(monitoring)...
Bytecode9

Exchanging Plugins: Next Connections
Let’s usemonitoring
Initial: Client Hello - “Hey, I support multipath and monitoring”
Initial: Server Hello - “Let’s use monitoring”

Encrypted - STREAM, STAT(info about RTT, reordering,...)
Encrypted - STREAM...

Added by themonitoringplugin10

Revisiting Protocol Implementation Design
What do we need?
● Running plugins in a safe environment● Identifying protocol operations● Providing an API to the protocol plugins

11Running Plugins● Plugin = set of bytecodes○ 1 bytecode → 1 function○ Hardware/OS independent○ Kept under control● Rely on a virtual machine○ Isolate bytecode from host implementation

VirtualMachine
01011
10010

12

eBPF Virtual Machine● Lightweight, integrated in Linux kernel since 2014
○ RISC instruction set (~100)■ ALU, memory and branch purposes● Bytecode recompiled to native architecture● Verifier to ensure programs safety● Dedicated, isolated stack memory

01011
10010x86_64○ But no persistence● Rely on a user-space implementation○ With relaxed verifier○ With persistent heap memory13

Identifying Protocol Operations
Flags

Connection ID

PacketNumberCleartext

Process
incoming
packetACK 4

STREAM 4 “Some text”

Encrypted

Parse
packetheader

Process
ACK
frame
ComputeRetransmit
Timer

Each operation has its own
built-in implementation
Process
STREAMframe

14

Modifying/Adding Protocol Operations
Flags

Connection ID

PacketNumberSTREAM 4 “Some text”

Encrypted
Encrypted

Cleartext

Process
incoming
packet

ACK 4Parse
packetheader

Process
ACK
frame

VM
01011
10010

ComputeRetransmit
Timer

STAT (RTT info,...)

Each operation has its own
built-in implementation
Process
STREAMframe

VM

Process
STAT
frame

01100
10101
15

Changing the Retransmission Timerint process_ack_frame(args) {// Some processing
ret = compute_retransmission_timer(args);

int compute_retransmission_timer(args) {REPLACE
rto = srtt
4 * rttvar;
rto =+ srtt
+ 3 * min_rtt;

VM

return rto;
return rto;// Some processing}return val;}

Exclusive full-access toconnection state16

Inserting Monitoring Plugin
int process_ack_frame(args) {// Some processingret = compute_retransmission_timer(args);

int compute_retransmission_timer(args) {PREVM

01100
10101

VM

rto = srtt + 4 * rttvar;
return rto;// Some processing}

POST VM

11111
00000

Read-only
access toconnection
state

11100
10000

return val;}
17

Linking Plugins to the Core Implementation
Exposing connection fields
Offering persistent heap memory
Retrieving data in plugin memory
Calling other protocol operations
CallVM’
stack

10110
00101

heapCall●●●●built_in()

PQUIC core

Get specific plugin dataVMstack

11110
00001

heapPlugin
Memoryget() / set()

Connection
State18

Communication between Plugin Bytecodes
parse_frame[ACK]built_in()write_frame[STAT]

replacecompute_retransmission_timer

post

bbr_congestion_control

replace

VM
stack

11110
00001

heap

Plugin A
Memory

VM’
stack

01011
10010

heap

VM”
stack

10110
00101

heap

Plugin B
Memory19

Agenda●●●●Motivations
Pluginized QUIC Design
Evaluating PQUIC with Use Cases
Conclusion and Future Works20

https://github.com/private-octopus/picoquic

Evaluation Overview● Implementation based on picoquic○ IETF QUIC in C language● Evaluation in our lab○ Use NetEm and HTB to configure links{d, bw}w}{d, b● Experimental design○ 139 points○ Consider median over 9 runs
MetricMinimumMaximum

One-way delay (ms)2.525

Bandwidth (Mbps)

550

21

Exploring Use Cases● Monitoring● A QUIC VPN● Multipath● Forward Erasure Correction
PluginLines of C Code

Number of bytecodes

Monitoring

500

14

QUIC VPN

500

11Multipath

2600

32

Forward Erasure Correction

2500

5122

Exploring Use Cases● Monitoring● A QUIC VPN● Multipath● Forward Erasure Correction

Read paper for otheruse cases

PluginLines of C Code

Number of bytecodes

Monitoring

500

14

QUIC VPN

500

11Multipath

2600

32

Forward Erasure Correction

2500

51

23

Alternative to DTLS, IPsec

1st Plugin: QUIC VPN
IPTCP
DATA

VIPIP

QUIC

QUIC

IPTCP
DATA

QUIC
Connection

IP

VTCP

IP
TCP
DATA

DATA

→ But QUIC provides reliable, in-order data delivery● Datagram plugin○ Unreliable data delivery○ Application API■ Protocol operation

VPN Application
send_datagram()
Datagram
PluginQUIC
24QUIC VPN in Action
Compare download time for a single file transfer using TCPCubic over 1 pathPlain
Faster TCP
in VPN Faster

Overhead of≊7%
Processing
time + MTU
difference25

2nd Plugin: Multipath QUIC
Implement the IETF Multipath draft● Link paths to specific networks○ Connection not bound to 4-tuple● Exchange IP addresses● Create paths over pair of IP addresses○ Full-mesh● Schedule packets over paths

Corefunctionalities
MultipathPlugin

ModularAlgorithms○ Round-robin fashionhttps://datatracker.ietf.org/doc/draft-deconinck-quic-multipath/

26

Multipath QUIC in Action
GET request over a dedicated stream● Consider 2-path network with symmetric characteristics

Aggregate both paths!
Equivalent to MPTCP!

27

Combining Plugins into a Multipath QUIC VPN
Datagram
PluginMultipath
PluginQUICConnection
28

Combining Plugins into a Multipath QUIC VPN
Datagram
Plugin

VMultipath
PluginQUIC
Connection

V
29

Combining Plugins into a Multipath QUIC VPN
Datagram
PluginMultipath
Plugin+V

QUIC
ConnectionV
30

Multipath QUIC VPN in Action
Compare download time for a single file transfer using TCPCubic● 2 symmetric network paths

VPN enables TCP
to use both paths!

32

Agenda●●●●MotivationsPluginized QUIC Design
Evaluating PQUIC with Use Cases
Conclusion and Future Works

33

Conclusion and Future Works

0101110010● PQUIC: Dynamically extends protocol implementations● Common pluginizable client, specialized server implementation○ Possible through secure plugin exchange● Four different use-cases fully implemented with plugins○ Monitoring, VPN, Multipath, Forward Erasure Correction● Future works○ Approach applicable to many other protocols○ How can we have independent, interoperable PQUIC implementations?
■ Specification of the VM, protocol operations, API,...

https://pquic.org34

Thanks for your attention!
https://pquic.org
35Extending/Customizing TCP Is Hard
● TCP Option Negotiation● Extension deployment challenges○ Specification process (IETF)○ Implementation puzzle

SYN [SACK, W

SCALE, TSTA

M P]

SCALE]
SYN/ACK [SACK, W■ Client waits for server support■ Server waits for client support○ Middlebox interferences● Coarse protocol tuning○ Global parameters (sysctl,...)○ Very specific socket options36

Processing New STAT frames
Type of the frame=Parameter to a generic protocol operationSTATint process_frame(args) {Frame Fields (ex. used for RTT estimation)

int process_stat_frame(args) {PREfor each frame {REPLACE

ret = process_frame_param(frame.type, args);// Code processing STAT

VM// Some processing}return val;}

return ret;

POST}

37

Trusting Plugins
Developer
blisu
P

PluginRepository

hCheck own bindings

Verifier A

Verifier B

Pull and build Merkel Tree

VerifierC...

Lookup for proof

Ask plugin and proof from B
ClientServer
Give plugin and B authentication path38

Proof of Consistency
Compare with storedverifier Root0Root = H(h0 || h1)01

h0 = H(h00 || h01)

h1

...

1h00

h01 = H(h010 ||...

h011)

001011h010 = H(‘multipath’||multipath bytecode)
10010

RecomputedH(‘multipath’) = ‘010’011010...

1h011

...
Provided
Authentication
Path39Plugin OverheadIntel Xeon E5-2640 v310 Gbps

PluginMean Goodput

PQUIC, no plugin

1104 Mbps

Monitoring1037 Mbps

Multipath 1 path757 Mbps

Monitoring + Multipath 1 path

714 Mbps

JITed eBPF ~2x slower than native code
Get/set interface ~5x slower than direct access-7% goodput+8% CPU instructions

40

Verifying the Termination of Plugins with T2

PluginLines of C Code

# Bytecodes

Proven terminating

Monitoring

500

1413VPN

500

118

Multipath

2600

32

29

FEC

2500

513741