Oko: Extending Open vSwitch with Stateful Filters
Paul Chaignon

Orange Labs
Inria Nancy Grand Est
paul.chaignon@orange.com

Kahina Lazri

Orange Labs
kahina.lazri@orange.com

Thibault DelmasJérôme François

Inria Nancy Grand Est
jerome.francois@inria.fr

Olivier Festor

Telecom ParisTech
thibault.delmas@telecom-paristech.fr

Inria Nancy Grand Est
Telecom Nancy
University of Lorraine
olivier.festor@inria.fr

ABSTRACTCCS CONCEPTS

With the Software-Defined Networking paradigm, software
switches emerged as the new edge of datacenter networks.
The widely adopted Open vSwitch implements the OpenFlow forwarding model; its simple match-action abstraction
eases network management, while providing enough flexibility to define complex forwarding pipelines. OpenFlow,however, cannot express the many packets processing algorithms required for traffic measurement, network security, orcongestion diagnosis, as it lacks a persistent state and basicarithmetic and logic operations.
This paper presents Oko, an extension of Open vSwitchthat enables runtime integration of stateful filtering andmonitoring functionalities based on Berkeley Packet Filter(BPF) programs into the OpenFlow pipeline. BPF programs
attached to OpenFlow rules act as intelligent filters over packets, while leaving the packets unmodified. This approachenables the transparent extension of Open vSwitch’s flow
caching architecture, retaining its high-performance benefits. Furthermore, the use of BPF allows for safe runtime
extension and prevention of switch failures due to faulty
programs.
We compare our implementation based on Open vSwitchDPDK to existing approaches with comparable isolation
properties and measure a near 2x improvement of performance.

• Networks → Programmable networks;

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights
for components of this work owned by others than the author(s) must
be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee. Request permissions from permissions@acm.org.
SOSR ’18, March 28–29, 2018, Los Angeles, CA, USA© 2018 Copyright held by the owner/author(s). Publication rights licensed
to Association for Computing Machinery.
ACM ISBN 978-1-4503-5664-0/18/03. . . $15.00
https://doi.org/10.1145/3185467.3185496

KEYWORDS
Software-Defined Networking; Programmable Networks; Datacenter Networks
ACM Reference Format:
Paul Chaignon, Kahina Lazri, Jérôme François, Thibault Delmas,
and Olivier Festor. 2018. Oko: Extending Open vSwitch with Stateful
Filters. In SOSR ’18: SOSR ’18: Symposium on SDN Research, March 28–
29, 2018, Los Angeles, CA, USA. ACM, New York, NY, USA, 13 pages.
https://doi.org/10.1145/3185467.3185496

1

INTRODUCTION

Software switches are taking an increasingly important roleas the edge of datacenter networks. Not only do they switch
packets between virtual machines (VMs) and physical interfaces, but they also often terminate overlay tunnels andenforce ACLs and QoS policies [22]. The short development
cycle of software switches—compared to their hardware
counterparts—and their ubiquity in datacenter networks
make them an ideal place to implement new network services.
There is, additionally, growing pressure on performancefor software switches. Datacenters are moving to 40 Gbps
and 100 Gbps physical interfaces, and network IO-boundworkloads are becoming more common. Furthermore, software switches must expose programming abstractions simple
enough to enable centralized control at the scale of datacenter networks. These abstractions need to express diverse
forwarding policies, yet allow for efficient implementation.
Open vSwitch, a software switch widely used in virtualization platforms [3, 22], meets these goals through a careful
implementation of the stateless match-action forwarding
pipeline of OpenFlow [31]. Although this stateless design
enables flow caching optimizations, it limits the application
scope of Open vSwitch. Open vSwitch cannot, for example,

SOSR ’18, March 28–29, 2018, Los Angeles, CA, USAaggregate measurements in the dataplane or maintain persistent information on flows and connections. Even simple
arithmetic and logic operations beyond bit matching require
control plane applications—possibly with a collocated controller [26, 38]—or middleboxes and VMs [17, 29], at the costof performance.
Recent advances in programmable hardware motivate
the need for new programming languages and abstractions,such as P4 [8] and Domino [36], which can express complex
operations and access persistent memories in the switch.
With these new abstractions, network operators can program switches to debug TCP connections [13], aggregate
measurements at the switch-level [37], or mitigate DDoS
attacks [32].
Despite running on flexible commodity hardware, OpenvSwitch did not benefit significantly from this new trend;
it still exposes a stateless match-action pipeline. Building
a stateful extension to Open vSwitch is difficult because it
needs to address the two following design challenges.
1. Prevent switch failures during updates. The pipeline
may receive frequent updates in response to attacks, or to
debug transient network problems. The pipeline thus needs
to be updated without interruption of forwarding, as withclassic OpenFlow rules. Without proper verification, however, loading unsafe programs at runtime increases the riskof failure. A fault in the new features, whether it may be aninvalid operation or a subtler denial of service (e.g., a memory leak or an infinite loop), may cause the switch to crash,
thereby disconnecting virtual machines from the network.
The switch must therefore provide a mean to prevent thesefailures.
2. Preserve caching mechanisms. Open vSwitch implements several flow caching mechanisms, fundamental toits high performance, in particular for the long forwarding
pipelines typical of multi-tenant networks [22, 30, 31]. Thesemechanisms are built on the assumption that forwarding
rules change infrequently compared to the rate of packet
arrival. Therefore, stateless rules stay valid for a long enoughtime to be worth caching. Conversely, with stateful rules,
switches may have to reconstruct cache entries after eachstate change, thus eliminating any caching benefit.
Berkeley Packet Filter (BPF) [24] is a known solution toaddress the first design challenge. In Linux-based operating
systems, BPF allows userspace applications to extend thekernel at runtime, without interruption. The kernel relies ona bytecode interpreter and a verifier to execute BPF programs
if they are deemed safe. BPF received a lot of attention in theLinux kernel [10, 11, 40].
The second design challenge received fewer attention. In[18], E. J. Jackson et al. propose SoftFlow, an extension ofOpen vSwitch that can execute arbitrary programs as actions

P. Chaignon et al.
in the OpenFlow pipeline. Program developers can tell SoftFlow when rule lookups are unnecessary after a program
execution to skip them and fully leverage Open vSwitch’s
caches.
In this paper, we present Oko, an stateful software switchthat addresses the above two challenges. Oko is based on
Open vSwitch and executes BPF bytecode programs as partof the OpenFlow pipeline. These programs can access persistent data structures to perform stateful operations such as
tracking the state of TCP connections or aggregating traffic
measurements. Although Oko does not use the in-kernel BPF
infrastructure, it relies on BPF’s bytecode and security model
for its own specialized BPF infrastructure.
Compared to SoftFlow, Oko explores a new point in thedesign space of Open vSwitch extensions. Oko programs act
as filters over packets, in the same manner as conventional
OpenFlow match fields. Because its programs never modify
packets, Oko always caches them without requiring further
information from developers. This design choice effectively
limits programs to filtering and monitoring applications. In
this sense, Oko is complementary to the SoftFlow approachsince programs requiring write access to packets could be
implemented as actions.
In summary, we make the following contributions:• The design of Oko, an software switch based on OpenvSwitch that supports stateful filtering and monitoring
programs as part of its OpenFlow pipeline.
• An implementation of Oko based on Open vSwitchDPDK. Thanks to a careful extension, Oko preserves
Open vSwitch’s cache design despite its original stateless assumption (Section 2.3 and 2.4). Because we leverage the high-performance version of Open vSwitch1 ,we cannot directly reuse the Linux BPF infrastructure.
Instead, our implementation relies on a userspace BPF
implementation based on uBPF [23] (Section 2.2).
• An evaluation of our implementation and a comparison to state-of-the-art approaches to extend software
switches (Section 4), with three security and network
diagnosis programs (Section 3). We show that Oko
provides a near 2x improvement of performance compared to a process using shared memory to exchange
packets with the switch.
Oko is open source and available at https://github.com/
Orange-OpenSource/oko.

2

DESIGNWe first describe the overall workflow of Oko. We then elaborate on our specialized BPF infrastructure and our extension
to Open vSwitch’s caching mechanisms.
1 The high-performance Open vSwitch-DPDK helped us uncover several

performance bottlenecks in our initial design.

Oko: Extending Open vSwitch with Stateful Filters

2.1

Oko Workflow

In this section, we describe our extensions to OpenFlow aswell as the Oko workflow, from the compilation of programs
at the controller to their execution in switches.
Oko extends match-action tables of OpenFlow with an
optional match field referencing a filter program, a statefulpacket matching program. If all other fields match the packet
headers, the filter program is executed, with the packet as its
sole argument. Filter programs have a binary result: if they
match, their corresponding action is executed; if they do notmatch, the lookup continues with rules of lower priority.
Each filter program may read and write to its maps, persistent data structures allocated on the switch, with some
restrictions detailed in Section 2.2. Since they are embedded
as match fields in flow tables, filter programs only impact
whether or not actions of a rule are executed; they cannot
define new actions.
Filter programs are written in higher-level languages suchas C or Lua and compiled to BPF bytecode. This compilation
step happens once, at the controller. The controller thensends the program to switches as an object file embedded
into LOAD_FILTER_PROGRAM OpenFlow messages. Oko
switches load the filter program into a filter program instance,a memory object that contains both the relocated bytecodeand its allocated persistent memory. At this point, the verifier
performs a series of checks to ensure that the program is safe(see Section 2.2) and returns an OpenFlow error message if
it is not.
Match-action rules may then reference filter program instances. In particular, several rules may share the same filter
program instance, allowing different rules to share and update the same maps. A filter program may be instantiatedseveral times to dispose of several different memories. For
example, when implementing a stateful firewall, each ACL
rule may be associated with its own record of establishedconnections (similar to conntrack zones). In our current implementation, to create a new filter program instance, thecontroller must send the program to the switch a secondtime.
We extended the OpenFlow protocol with new messages
and fields to load filter programs, retrieve maps, and change
OpenFlow rules with filter programs attached. These extensions to the OpenFlow protocol are summarized in Table
1. We used OpenFlow vendor extensions for new messages,
but had to modify the OpenFlow protocol itself for the newmatch field, since such vendor extensions are not available.
We added support for these extensions in both Open vSwitch
and OpenDaylight [4].
The astute reader might notice that Oko does not provide
an OpenFlow message for the controller to proactively retrieve the content of maps. To retrieve information collected

SOSR ’18, March 28–29, 2018, Los Angeles, CA, USA
by filter programs, the controller defines a SEND_MAPS action, which the filter program can then trigger (an example
is given in Section 3). While developing Oko filter programs,
we have not found a need for such an OpenFlow message.
Filter programs are often in a better position to decide when
the content of their maps should be sent to the controller:
they can trigger the SEND_MAPS action when their maps
reach a threshold size, or after a specific network event. Nevertheless, if required, the addition of a controller to switch
SEND_MAPS message would be trivial to implement.

2.2

Specialized BPF Infrastructure

BPF was originally designed as a minimalist bytecode andan in-kernel infrastructure to filter packets destined to auserspace capture application [24]. In the Linux kernel, it
evolved into a general purpose infrastructure [10], also referred to as extended BPF or eBPF. Its current applications include tracing [14], firewalling [7], and container networking[5]. In this section, we (1) provide the necessary backgroundon the security model of BPF and (2) describe the design andimplementation of our userspace BPF infrastructure.
Background on the BPF security model. In the Linuxkernel, the BPF virtual machine consists of a stack and a setof 64-bit registers. The bytecode it recognizes was designed
to closely match hardware instruction set architectures. Bytecode programs can call external functions, implemented outside the VM, to perform operations or retrieve information
not available within the VM.
In-kernel interpreters2 impose limits on the number of
instructions and the size of the stack and reject programswith out-of-bounds memory accesses, jumps to non-existing
instructions, or null accesses. In addition, in-kernel interpreters expose a BPF machine abstraction with a computational power equivalent to that of a Decider [35], a type ofTuring machine that always halts. In the Linux kernel, thiscomputational power is enforced in a strict way by rejecting
all jumps to previously visited instructions. This approachprohibits cycles when interpreting the bytecode and, at ahigher level, makes long loops difficult to implement3 .
The jump restriction, however, does not apply to externalfunctions; data structures are therefore implemented outside the BPF VM. For example, the Linux BPF infrastructure
contains a hash table implementation with linked lists forcollision resolution. It is exposed to BPF programs throughthree external functions: bpf_map_lookup_elem,
bpf_map_update_elem, and bpf_map_delete_elem to lookup,
update, or delete an element respectively.
2 Both the Linux kernel and the BSD kernel have a form of BPF in-kernelinterpreter.
3 Short loops can be unrolled during compilation to bytecode.

SOSR ’18, March 28–29, 2018, Los Angeles, CA, USA
Name
LOAD_FILTER_PROG
FLOW_MODSEND_MAPSSEND_MAPS

Type
Controller to switch
message
Controller to switch
message
ActionSwitch to controller
message

P. Chaignon et al.

Description
Contains an ID and a filter program as an object file. The switch loads the filter program andassigns it the given ID.
Modified to include a new filter_prog field with a filter program ID. Used to add, remove, or modify
OpenFlow rules.
Instructs the switch to send the content of data structures given in argument to the controller.
The argument is a bit array where each bit corresponds to a data structure referenced in the filter
program attached to the rule.
Contains the binary content of data structures.

Table 1: Oko’s extensions to the OpenFlow protocol.

• Three functions to lookup, update, and delete elementsfrom data structures.
• ubpf_time to read the current time—used to implement traffic policing algorithms and to periodicallysend measurements to the controller.
• ubpf_hash to compute a 32-bit hash of a variablelength input. Together with the array data structure,ubpf_hash can be used to implement probabilistic
data structures common in network monitoring applications [41], such as Bloom filters and Count-Min
sketches.

First
packetDatapath

Oko’s userspace BPF infrastructure. We retain the BPF
bytecode and its abstract machine, but we re-implement a
set of external functions and a verifier tailored to Oko.
Our BPF infrastructure relies on a userspace implementation of the BPF VM [23] and extends it to support maps
(allocation and external functions). When Oko receives BPF
programs from the control plane, it loads them in BPF VMs.
To this end, it first allocates memory for maps and relocatesaddresses in the bytecode. A just-in-time compiler then translates the BPF bytecode into machine code.
Oko supports two data structures outside the BPF VM: ahash table and a simple array. The hash table implementation
uses Bob Jenkins’ lookup3 hash function [19] and resolves
hash collisions with linked lists.
We implemented five external functions filter programs
can call:Slow path

Although external functions can iterate through dynamic
data structures, they do not increase the computational powerof the BPF machine abstraction. Indeed, external data structures have a bounded size, fixed by program developers, toensure termination and limit memory consumption.
Compared to other bytecodes [1, 27], the BPF bytecodeis easier to verify: it has a minimalist instruction set andexternal functions with well-defined interfaces (expected
types, such as potential null pointer or pointer to the packet,
for arguments and returned values).

Table 0...

Table nMegaflow cache

Microflow cache

Figure 1: Open vSwitch caching architecture.

For the verifier, we implemented a depth-first traversal of
the control flow graph (CFG) of BPF programs, to detect andreject programs with cycles (back-edges in the CFG). In asecond traversal of the CFG, the verifier tracks the state ofregisters to determine if they contain, for example, a constant,a pointer to the packet, or a potential null pointer. This
information is then used to reject programs with potential
invalid operations such as null memory accesses or writes
to the packet.
In summary, our verifier implements the same safety checks
as the Linux verifier, except that it only supports the specific needs of Oko. We do not, however, implement some ofthe optimizations the Linux verifier performs (such as state
pruning).

2.3Filter Program Caching

Open vSwitch relies on a hierarchy of caches to achieve highperformance without loss of generality [31]. In this section,we (1) give the necessary background on Open vSwitch’s
caching mechanism and (2) elaborate on our design to preserve the performance of Open vSwitch and execute filter
programs during the cache lookup.

Oko: Extending Open vSwitch with Stateful Filters
Open vSwitch’s forwarding pipeline. Figure 1 illustrates
the caching architecture of Open vSwitch. The full matchaction pipeline of the OpenFlow forwarding model is implemented in the slow path, in userspace. This userspacecomponent hence consists of a collection of match-action
tables, where each table contains a number of rules, withmatch fields, a priority, and corresponding actions. When apacket arrives at the slow path, Open vSwitch executes theaction of the matching rule with the highest priority. Processing of the packet may continue with subsequent tables
depending on the action applied.
In addition, Open vSwitch includes a number of datapath
implementations for different environments: kernel modules for the default Linux and Windows datapaths and an
additional userspace implementation based on DPDK (Data
Plane Development Kit) for Linux. Our implementation of
Oko uses the DPDK userspace datapath as it achieves thehighest performance.
Open vSwitch’s caching mechanisms. The datapath issometimes referred to as the fast path of Open vSwitch. It
implements a number of flow caching mechanisms and only
forwards a packet to the slow path if no cached rules arematched. The DPDK datapath implementation includes twocache levels illustrated in Figure 1: the megaflow cache andthe microflow cache.
The megaflow cache is a simplified implementation of
the OpenFlow match-action pipeline: it contains a single
OpenFlow table with no priority. The slow path component
can thus only install disjoint rules into this cache. These
megaflow rules are built as an aggregation of rules matchedduring the slow path lookup. To this end, and in order tobuild only disjoint megaflow rules, Open vSwitch keeps trackof which fields were used during the slow path lookup. The
megaflow rule installed afterwards matches only on the field
used; other match fields are wildcarded. Open vSwitch uses
a number of additional techniques during the constructionof megaflow rules to improve the cache hit rate.
Because even the simplified megaflow table has a highlookup cost, Open vSwitch includes a second cache, the microflow cache, implemented as an exact-match table. As any
packet header change results in a cache miss, the microflow
cache does not perform well with many traffic patterns suchas short lived flows, in which case Open vSwitch must fallback to the megaflow cache.
Oko’s filter program chains. To maintain the performanceof Open vSwitch, filter programs need to be cached and executed in the datapath. The caching mechanisms of OpenvSwitch, however, are tailored for its stateless OpenFlow
pipeline. Introducing stateful programs in caches raises animportant challenge.

SOSR ’18, March 28–29, 2018, Los Angeles, CA, USA
Priority Source Destination Filter prog. Action
100*
10.0.0.1:80
a
drop
10
*
*:80
b
port 1
1**
drop
Table 2: Example of Oko match-action table. If packets
do not match a they will inevitably run against b.

The result from the execution of a filter program depends
on the full content of the packet and the memory associatedwith the program. Conversely, in the OpenFlow forwarding
model, the actions depend only on the packet’s headers. Thus,
two packets with the same header will always be processed
identically in an OpenFlow pipeline, whereas they may resultin different actions taken when processed by Oko.
Open vSwitch relies on this determinism to aggregate asuccession of OpenFlow rules into a single megaflow rule.
With Oko, the path through the OpenFlow pipeline, andthus the megaflow aggregated rule, depends on the resultsfrom executed filter programs, which may change from one
execution to the next.
To overcome this challenge, we introduce the conceptof filter program chains for caches. A filter program chain
consists of an ordered list of filter programs, each with an
expected result. Each cache entry contains a filter program
chain, assembled during the slow path lookup as the concatenation of executed filter programs with their result. By
construction, the filter program chain preserves the order inwhich filter programs were executed.
Table 3 illustrates a plausible content of the megaflow
cache after Oko processed two packets through the slow pathtable defined in Table 2. Program a matched the first packet—
and the filter program chain [a:1] was installed in the cache—but not the second packet. The slow path lookup continued
for the second packet with rules of lower priority. Programb matched the second packet, resulting in the installation of
the filter program chain [a:0, b:1] in the cache.
The datapath executes filter program chains by iterating
through them and executing each program until it finds one
that does not return the expected value. A filter program
chain matches a packet if all its filter programs return their
expected results. In the caches’ classifiers, several rules withthe same stateless match fields but different filter program
chains are referenced under the same hash. Oko iterates
through them until it finds a matching filter program chain.
If none of the cached rules match, the traditional behaviorof Open vSwitch is preserved and a slow path lookup is
performed.
For a given set of packet headers, the possible paths throughthe OpenFlow pipeline can be viewed as a binary tree whose
nodes are filter programs. Each slow path lookup with thisSOSR ’18, March 28–29, 2018, Los Angeles, CA, USA
Source Destination Filter program chain Action*
10.0.0.1:80
[a:1]
drop
*
10.0.0.1:80[a:0, b:1]
port 1
Table 3: Example of megaflow cache in Oko after slow
path lookups for 2 packets toward 10.0.0.1:80. Only
one of the packets matched a, the other matched b.

a

drop

b

port 1drop

Figure 2: Tree of filter programs from Table 2 for flow
ip_dst=10.0.0.1,port=80. Only branch a->b->drop has
not been cached yet.
same set of headers results in a new path through the tree
being cached, i.e. a new result of a filter program of the chain
is reached. Filter program chains encode these paths. Figure 2 represents the filter program tree of Table 2 for flow
ip_dst=10.0.0.1, port=80, with the paths already cached (listed
in Table 3) in bold.
Although filter programs are still not deterministic fromthe caches’ perspective, encoding their results into cachedrules enables the aggregation of several stateful rules into asingle disjoint rule.
Filter program chains, however, do not guarantee on theirown that a program is not executed twice for the same packet.
Two filter program chains may share some filter programs
that we do not want executed twice for a same packet. To prevent this, a history of executed filter programs is saved alongwith their results for each processed packet. This history isthen checked before executing any filter program.
If filter programs had write access to packets, our filter
program chain extension would not be possible. In this case,the path through the OpenFlow pipeline would also dependon the packet modifications, which would have to be encodedin cache entries as well. Encoding all packet modifications,in addition to the program results, is prohibitively expensive.

2.4

Cache Invalidation

Because cached rules depend on their aggregated OpenFlow
rules and on higher priority rules, their invalidation is difficult to compute. Therefore, rather than trying to track whichcache entries need to be updated after an OpenFlow update,Open vSwitch adopts a brute-force approach and proactively
revalidates the whole cache.

P. Chaignon et al.
To this end, it runs the set of packet headers that generated
each cached rule through the slow path again. The new rule
generated and its actions are compared against the installedcached rule. Depending on the result, cached rules may be
updated, removed from the cache, or kept as they are.
Preserving this cache revalidation mechanism is challenging for filter programs because it presumes that a rule canbe matched against several times and give the same result.
Unfortunately, executing filter programs during cache revalidation would update their internal state and so may change
the expected result of a filter program for future incoming
packets. To avoid any execution of filter programs during
cache revalidation, we use the filter program chain attached
to cached rules. During the slow path lookup for cache revalidation, we compare the identifier of selected filter programwith those from the cached filter program chain. A cachedrule is deemed valid only if the same filter programs are
selected, in the same order.

3

FILTER PROGRAM EXAMPLESIn this section, we describe three example use cases for Oko: astateful firewall, a TCP performance diagnosis system, and astateless filtering application. Each use case requires a single
filter program written in C and compiled to BPF bytecode.
We evaluate these filter programs in Section 4.
p0f signature filtering. We describe a simple stateless filtering application of Oko, to emphasize its extended matching capabilities. We implemented a TCP SYN DoS mitigation
application based on p0f signatures [42]. p0f is a passive fingerprinting tool that can identify the system from which a
packet originated based on pre-established signatures. p0f
also includes signatures to discriminate SYN flood packets
from legitimate traffic [6].
Matching p0f signatures against packets requires a few
arithmetic and bitwise operations (e.g., subtractions and bit
shifts) on TCP options, and as such, could not be expressedwith OpenFlow rules, even if new fields were added. Conversely, with Oko, the filter program extends the OpenFlow
table and performs the operations required to match p0f signatures against packets. The first rule in Table 4a illustrates
the use of p0f signatures to filter packets at the beginning of
the pipeline (with the highest priority). The filter program
matches and drops packets that match its p0f signatures.
Our application relies on the OpenDaylight REST API tocontrol the switch. It first generates a p0f signature fromsampled TCP packets (mirrored to the controller). The p0f
signature is then compiled to BPF and loaded in the switch
to block attacks directly in the dataplane.

Oko: Extending Open vSwitch with Stateful Filters
Prio.
12
11
10
10
1

Source***10.0.0.1*

Dest.
Filter prog. Actions
10.0.0.1 anti-ddos
drop10.0.0.1 firewall
table 210.0.0.1 table 2*
firewalltable 2*drop

SOSR ’18, March 28–29, 2018, Los Angeles, CA, USA
Prio.
10
10
1
1

Source10.0.0.1*
10.0.0.1*

Dest.
Filter prog.
*
Dapper
10.0.0.1 Dapper*
10.0.0.1 -Actions
send_maps, port 1
send_maps, port 2
port 1
port 2(b) Table 2(a) Table 1

Table 4: OpenFlow tables to illustrate example filter programs.

Stateful firewall. As a descriptive example of Oko’s workflow, we implemented a stateful firewall using a filter program as the connection tracker and OpenFlow rules for the
ACL rules. As illustrated in Table 4a, the same program, denoted as firewall, tracks connections in both directions inorder to protect a web server hosted at 10.0.0.1. It matches
packets if they are part of an established connection. It isonly used to filter packets from the server (fourth rule) since
only packets from established connections are authorized inthat direction. Packets destined to the server are forwarded
to the second table (second and third rules), regardless ofwhether they are from an established connection.
The stateful firewall maintains the state of each connection, identified by their 5-tuple (protocol, source and destination IP addresses and ports), in a hash table. The filter
program only needs to implement the TCP state machine andaccesses the hash table through the three external functions
defined in Section 2.2.
Dapper: TCP performance analysis. Dapper [13] is a system to analyze TCP connections in real-time near end-hosts.
Dapper collects a set of information on each TCP connection (e.g., flight size, MSS, sender’s reaction time) at the edge
switch. A controller can then retrieve this information todetermine the limiting factor of a connection among thesender, the receiver, and the network.
An example integration of Dapper in the OpenFlow pipeline
is given in Table 4b. Our Dapper implementation analyzes
all traffic to and from the server and stores information on
TCP connections in a hash table. By default, the Dapper filter
program returns false and packets therefore match the last
two rules from the OpenFlow table. The program returning true triggers the SEND_MAPS action: statistics on all
connections are sent to the controller. This action allows
the program to send its collected statistics in batches to thecontroller when it reaches a given number of connections,after a given amount of time, or when it detects a troubled
connection.

4

EVALUATION

Our evaluations aim to answer the following questions:• How does Oko compare to other methods to extend
software switches?
• How much overhead do Oko’s extensions introduce
in Open vSwitch? In particular, how does it impact
traditional stateless forwarding pipelines?
• How efficient is our filter program chain extension forOpen vSwitch’s caching mechanisms?
First, we describe our evaluation environment. Next, througha set of microbenchmarks, we measure the overhead andefficiency of our extensions to Open vSwitch. Finally, we
compare, for the three use cases described in Section 3, theperformance of Open vSwitch to that of two alternate solutions to extend software switches: (1) a KVM virtual machineusing a vhost-user interface and running a DPDK application,and (2) a zero-copy DPDK application running as a process.

4.1

Evaluation Environment

Our testbed consists of two servers directly connected with
Mellanox 40 Gbps NICs. The two NICs use firmware 2.40.700
and are configured with a single queue. The Device Under
Test (DUT) hosting the switch (Oko or vanilla Open vSwitch)
has an Intel Xeon E5-2640 2.6 Ghz with 20 MB of L3 cacheand 16 GB of DDR4 memory at 2133 MHz and runs Linux
4.4.0.
To avoid Linux’s I/O interfaces being the main bottleneck,and to get a clear view of the performance limitations of
our own modifications, we based our prototype on the highperformance, userspace datapath of Open vSwitch.
In all experiments, the switch runs on a single core, isolated from the Linux scheduler, with hyper threading disabled. All cores used in experiments are on the same NUMA
node, to which the NIC is connected. Unless stated otherwise, we use Open vSwitch 2.5.0, on which Oko is based,with DPDK 2.2 for all comparisons. The switch is configured
with a single poll-mode thread and receive checksum offload
disabled. The second server runs MoonGen [12] to send minimum size 64 bytes frames and replay packet captures.

SOSR ’18, March 28–29, 2018, Los Angeles, CA, USA

P. Chaignon et al.

In several of the following evaluations, the packet generator replays a CAIDA packet trace [2] of 34 minutes. The
trace contains 87 million packets forming 7 million L4 conversations. On average, L4 conversations last 5.6 seconds andcontain 11.7 packets. The maximum number of packets in asingle conversation is 11k and the median is 4 packets.
Each experiment lasts 5 minutes and we report the mean
and the standard deviation over 10 runs.

4.2

Microbenchmarks

We first measure the overhead introduced in Open vSwitch
by Oko’s extensions through a set of microbenchmarks.
Setup. Because Oko extends only the packet classification
algorithms of Open vSwitch and does not impact the OpenFlow actions, we configure the switch to drop received packets after classification. The switch therefore acts as a packet
classifier in all comparisons to vanilla Open vSwitch. We
perform end to end evaluations in Section 4.3.
Because of the two-level caching architecture, the performance of Open vSwitch greatly depends on the traffic
patterns and the cache hit rate of both its microflow andmegaflow caches. Therefore, in all experiments, we provide
results under three switch setups: the first setup is the default
switch configuration with all caches; in the second setup,the microflow cache is disabled and packets directly hit themegaflow cache with its single OpenFlow table; in the last
setup, both caches are disabled and all packets go throughthe full OpenFlow pipeline in the slow path.
Scenarios. In the following microbenchmarks, we configure
the switch and the MoonGen packet generator according to
two scenarios.
In the synthetic scenario, a single OpenFlow rule matching all incoming packets is installed while the packet generator sends a single flow of minimum sized (64 bytes) UDP
packets. The rule has a baseline filter program attached, witha single instruction to return 1, and therefore matches all
packets. This scenario is an ideal case for Open vSwitch’s performance. Once the OpenFlow rule is cached, Open vSwitch
requires few cycles to process subsequent packets, and any
overhead added by Oko is therefore accentuated.
Conversely, the realistic scenario is designed to stressthe switch datapath. It consists of a two-stage pipeline, withan L3 table over 170 IP prefixes and an ACL table consistingof 500 rules over destination ports.
We choose the L3 and ACL tables so as to produce a highnumber of cached rules at runtime. The L3 table matches the
101 /8 IP prefixes from the trace, as well as the 69 most used
/16 IP prefixes, whereas the ACL table matches the 500 mostused destination ports. As shown in Figure 3, thanks to thisFigure 3: Cumulative distribution of packets over
OpenFlow rules.
setup, packets are well distributed over all OpenFlow rules.
The two ACL rules matching a larger number of packets are
the default deny rule and the port 80 rule. After a brief ramp
up period, this pipeline results in an average of 170k distinct
rules installed in the datapath4 .
The short duration of conversations and the high distribution of packets over OpenFlow rules significantly stresses
the switch caches. Indeed, as can be observed when comparing figures 4a and 4b (OVS bar), the efficiency of the two
datapath caches is greatly reduced in the realistic scenario
compared to the synthetic scenario (x5 and x3.6 difference
for the microflow and megaflow caches respectively). In particular, because of the short duration of conversations, themicroflow cache brings little improvement (3%) over themegaflow cache under the realistic scenario.
Overhead Evaluation. We next measure the performanceof Open vSwitch, Oko without filter programs, and Oko witha single baseline filter program. Figure 4a shows the packet
classification performance for each cache level under thesynthetic scenario.
Without filter programs in the forwarding pipeline, Oko
has a small performance overhead over Open vSwitch (nonoticeable overhead with all caches, 6% with the microflow
cache disabled) largely due to the initialization of several
additional packet fields to store the history of executed filter programs. With a baseline filter program, the overhead
grows to 12.6% when all cache levels are enabled, and 11.3%
when the microflow cache is disabled. This evaluation provides us with an estimated upper bound of the performance
4 We increase the maximum number of rules for the DPDK datapath to 200k.

Oko: Extending Open vSwitch with Stateful Filters
overhead. Under the synthetic scenario, because of the single
OpenFlow rule, the two caches require very few instructions
to classify each packet. Therefore any additional processing (fields initialization, filter program chain iteration, etc.)
appears accentuated. Under the realistic scenario, however,Oko adds very little overhead (no noticeable overhead withall caches, 2% with the microflow cache disabled) to OpenvSwitch and the overhead added by a baseline filter program
decreases to 5.5% with all caches (5.6% with the microflow
cache disabled).
Figures 4a and 4b also highlight the need for our filter
program chain extension to Open vSwitch’s caching mechanisms. Without filter program chains, the performance of
Oko with a filter program would be that of the slow pathsince the caches would be stateless and all packets would
go through the slow path. Filter program chains therefore
provide an estimated x5 improvement of performance under
the realistic scenario (x18 for the synthetic scenario).
Filter program chains. OpenFlow rules are installed in
the datapath caches after a packet passed through the full
OpenFlow pipeline in the slow path. If filter programs were
executed during the slow path lookup, the cached rule has
a non-empty filter program chain attached. The size of thefilter program chain is equal to the number of filter programs
executed during the slow path lookup.
To obtain a chain of n filter programs in the datapath, we
modify the synthetic scenario and create a pipeline of n tablesin the slow path. Each table contains a single OpenFlow rulewith a baseline filter program attached and forwards packets
to the next table. Baseline filter programs match all packets.
Figure 5 shows the packet classification performance forthese pipelines. Each new filter program results in about 4%
additional overhead. The packet classification performance isalmost halved (-43.8%) with 10 filter programs in the pipeline.
These numbers are in line with what others have reported
for similar chains of programs executed in Open vSwitch’s
datapath [18].
Filter Programs Evaluation. To provide an estimate of
the cost of running filter programs as part of the forwarding pipeline, we integrate and evaluate each filter program
example from Section 3 under the realistic scenario.
To evaluate the p0f signature filtering program, an additional table is added at the beginning of the realistic scenario’s pipeline. This first table has two rules to (1) drop all
packets that match the p0f signature, and (2) forward otherpackets to the second table. In addition, at the packet generator, we inject both legitimate traffic (from the realistic packet
trace) and a flood of TCP SYN packets matching the p0f signature. As in previous tests, the packet trace is replayed witha single core; 3 additional cores participate in the TCP SYNSOSR ’18, March 28–29, 2018, Los Angeles, CA, USA
flood, resulting in approximately one on four packets being
legitimate.
The results in Figure 6 count all packets successfully classified by Oko. The Baseline bar reports the performance of
Oko when there are no filter programs in the pipeline, forcomparison. With the p0f filter program, Oko achieves better
performance than the baseline case because it drops illegitimate packets at the beginning of the pipeline, before theL3 and ACL lookups. As highlighted by this p0f example,
Oko can help filter illegitimate or malformed packets at the
beginning of the pipeline to avoid unnecessary processing
steps.
To evaluate our Dapper implementation, tables are addedat the end of the realistic pipeline with corresponding filter
programs. We use the same packet trace as in the realistic
scenario.
The initial implementation of Dapper [13] relies on a fixed
size data structure to store statistics on TCP connections. For
this reason, it can miss some connections if the number of
TCP connections grows larger than the size of the data structure. With Oko we have the choice to implement a similararray-based version of Dapper or rely on a hash table to store
statistics. We implement both options and compare their performance in Figure 6. Under our test scenario, the hash table
implementation has a relatively low cost compared to thearray implementation. Oko can therefore benefit from thehigher flexibility offered by CPUs to implement algorithms
impractical with specialized hardware.
Since its 2.6.1 version, Open vSwitch can act as a stateful
firewall thanks to a connection tracking module (conntrack)
implemented in the DPDK datapath. We compare this implementation against our Oko stateful firewall under therealistic scenario, with additional rules to distinguish established TCP connections from new connections. The OpenvSwitch firewall uses Open vSwitch 2.6.1 and DPDK 16.07.
As shown in Figure 6, the two stateful firewalls perform
comparably. Our implementation achieves higher packet
classification performance (18% higher with all caches), but
contrary to Open vSwitch’s connection tracking module,
it does not track UDP conversations. With Oko, however,network operators can customize the connection tracking
program at runtime to fit a particular application’s needs.
In summary, each filter program adds only a small overhead to the baseline pipeline, with the exception of the p0f filtering program that improves performance when the switch
is under attack. Notice that Dapper and our stateful firewall
implementations achieve very close performance results, despite being completely different applications. An in-depth
analysis of the CPU consumption reveals that, in both cases,
the hash table lookup dominates the running time with amajority of CPU cycles spent on the 5-tuple hash computation.

SOSR ’18, March 28–29, 2018, Los Angeles, CA, USA(a) Synthetic scenario, with a standard deviation below 0.30
for all measurements.

P. Chaignon et al.

(b) Realistic scenario, with a standard deviation below 0.06
for all measurements.

Figure 4: Packet classification performance between Open vSwitch and Oko.

Figure 5: Packet classification performance for different filter program chain lengths, with a standard deviation below 0.30 for all measurements.

4.3

End-to-End Comparisons

Setups. For each of the three use case examples, we compare
Oko to two existing solutions to extend software switches.
• A DPDK application running inside a KVM virtual machine. The VM is connected to Open vSwitch using
the vhost-user virtual NIC. Packets are copied fromthe switch memory space to the VM memory space

Figure 6: Performance evaluation (on a log scale) for
the three Oko use cases, with a standard deviation below 0.08 for all measurements.

through a shared memory. vhost-user is the recommended virtual NIC for use with Open vSwitch-DPDK
because it achieves the highest performance by opening a direct channel between the host userspace andthe guest memory space.
• A DPDK application running as a single process on thehost. By using the DPDK Ring Port type, the process
shares the memory space of Open vSwitch, enablingzero-copy packet processing.

Oko: Extending Open vSwitch with Stateful Filters
VM memory
space

DPDK
vhost-user

Switchmemory space

SwitchSOSR ’18, March 28–29, 2018, Los Angeles, CA, USA

VM
Process
Oko

DPDK

NIC

Figure 7: The three evaluation setups for the end-toend performance comparison. Packet copies are only
necessary when crossing memory space boundaries.
Figure 8: Comparison of performance for the three use
cases, with Oko, a vhost-user KVM virtual machine,
and a DPDK Ring Port process. The standard deviation
is below 0.10 for all measurements.

The switch (both Oko and Open vSwitch), the VM, and theprocess each have their own dedicated core, isolated fromthe Linux scheduler. In each setup, the switch has only two
rules to send packets through the application.
Even though the VM and the process setups each have
two dedicated cores, they use comparable amount of CPU
resources. In both cases, the core dedicated to the switch
does minimal work: it only forwards packets from the NIC
to the virtual port and vice versa.
Figure 7 illustrates our three test setups. When using Oko,a single core receives packets from the NIC, executes thefilter program on them and sends them back to the NIC, as
per the run-to-completion model of Open vSwitch. Under
the Process setup, however, the addition of a second process
to run the application breaks the run-to-completion model.
Packets are first forwarded by the switch, then processed by
the DPDK application, and finally forwarded back to the NIC
by the switch. Finally, the VM setup entails two additional
packet copies: from the switch to the VM and back.
For the purpose of this comparison, we implemented thethree use cases from Section 3 as DPDK applications. The
two filtering applications, the stateful firewall and the p0f application, drop filtered packets in the VM and in the process,without requiring an additional transfer to the switch.
The packet generator replays the same CAIDA trace aspreviously. For the evaluation of the p0f application only, we
configure 3 additional cores at the packet generator to senda continuous stream of TCP SYN packets.

There is a large body of work on packet processing in software [15, 18, 28, 29, 33, 34, 38]. We focus here on (1) propositions to extend software switches, (2) alternatives to BPF forthe isolation of packet processing programs, and (3) emerging usages of BPF in software switches.

Results. Figure 8 presents the results of our comparison
evaluation. As expected, Oko outperforms the VM applications by 2-3x. The difference is less pronounced for the p0f
filtering application because illegitimate packets (3 out of 4
packets) are dropped at the VM, requiring one less copy.

Extending Software Switches. Several recent works addressed the issue of extending software switches to execute
arbitrary packet processing [18, 25, 26, 38].
OFX [38] extends the OpenFlow API to allow SDN applications to load programs into a switch agent. This agent acts

Although VMs are increasingly considered as potential
targets to deploy packet processing application, it is not afair comparison to Oko as they provide stronger isolation.
Contrary to Oko, VMs (1) provide resource isolation between
applications, and (2) leverage hardware support for memory
isolation. Using our userspace BPF verifier, Oko provides
memory isolation between filter programs and between theswitch and filter programs, and prevents faults in filter programs from crashing the switch. These guarantees are closer
to that of processes, which also isolate faults from the switch
and separate each application into its own memory space.
When compared to the Process setup, Oko provides a 1.71.9x improvement of performance. Thanks to the run-tocompletion model of Open vSwitch, Oko benefits from fewer
cache misses, consolidated processing steps (packet parsing
and classification are performed once), and the lack of IPC.

5RELATED WORKSOSR ’18, March 28–29, 2018, Los Angeles, CA, USA

P. Chaignon et al.

as a local controller for a few tables at the beginning of theWith some adjustments, these systems could be used inswitch’s OpenFlow pipeline. [26] presents the design of aplace of BPF in Oko. We chose to use BPF because it has
similar extension, except the local controller intercepts packbeen tailored for packet processing [24] and provides theets by modifying table-miss OpenFlow rules. As shown inminimum functionalities required for Oko.
Section 4.3, because it breaks the switch’s run-to-completion
BPF for software switching. In a recent work, C.-C. Tu etmodel—packets are processed by the local controller in aal. [39] detail the design of an BPF-based datapath for Open
different execution context than the forwarding pipeline—,vSwitch, to replace the Linux kernel module datapath. Their
this approach cannot offer the same performance as Oko.
work focuses on implementing the two caches in BPF andWith NEWS [25], the authors of [26] propose an improvedoes not expose BPF capabilities through the switch API.
ment of their design and integrate the local controller in
S. Jouet et al. propose in [20] to use stateless BPF programs
Open vSwitch. Although NEWS preserves the run-to-completion
as OXM match fields to build a protocol independent switch.
model of Open vSwitch, it runs in the slow path as the auIn [21], they propose a packet processing framework basedthors do not extend caching mechanisms. As hinted in Secon BPF, upon which a software switch can be developed. Intion 4.2, extending Open vSwitch’s caching mechanisms is aboth [20] and [21], the prototypes do not implement any flow
required step to achieve high performance.
caching mechanisms, which eases the design but severely
SoftFlow [18] is probably the work closest to ours. Basedlimits performance for long forwarding pipelines.
on Open vSwitch, SoftFlow preserves the run-to-completion
Finally, it is also possible to place BPF hooks in the Linuxmodel and runs arbitrary programs in the datapath as Openkernel networking stack (e.g., at the traffic classifier or in
Flow actions. SoftFlow, however, sacrifices isolation to attainthe driver). Such programs, however, cannot benefit fromhigh-performance; as a result, a faulty program may crash
the ease and performance of Open vSwitch’s forwarding
the switch. Thanks to BPF, Oko is immune to such failures.
pipeline; they would have to reimplement flow tables andBecause SoftFlow supports arbitrary programs, it relies on
their caching optimizations from scratch. To benefit fromtheir developers to tell the switch when a program does notboth Open vSwitch and the Linux BPF infrastructure, a new
mandate a new rule lookup after its execution. In contrast, inhook point and an extended Open vSwitch kernel module
Oko, programs are integrated and act as match fields; in parwould
be required. We leave this in-kernel implementation
ticular, they cannot write to packets. This restriction makesofOko
to our future work.
the filter program chain extension possible since new rule
lookups are never required after a filter program execution.
To support programs that require write access to packets, a
6 CONCLUSION
future version of Oko could adopt SoftFlow’s approach and
Because of their stateless match-action abstractions, software
implement those programs as OpenFlow actions. This design
switches do not leverage the flexibility offered by the comwould provide a clear separation between filter programs
modity hardware on which they run. Due to their unique pothat can be chained and action programs that require write
sition at the edge of datacenter networks, software switches
access to packets.
are critical pieces of software, difficult to extend.
In this paper, we introduced Oko, a software switch based
Software Isolation. Besides BPF, several other systems
on Open vSwitch that can be extended at runtime with stateaddress the issue of providing memory isolation without theful programs. Oko protects itself against faulty programs
significant cost of context switches.
using the same security model as the BPF in-kernel interIn particular, Singularity [16] explores the use of type-safe
preter. As our evaluations demonstrate, Oko can preservelanguages and memory-safe runtimes to provide memory
the high performance of Open vSwitch, while providing a
isolation in place of the hardware memory management unit.
near 2x improvement over existing approaches to extend
NetBricks [29] applies this approach to packet processing
software switches. Thanks to Oko, SDN applications canand relies on LLVM as the runtime and Rust as the language.
load programs in the dataplane to mitigate attacks, diagIn particular, NetBricks leverages a new compile-time mechnose performance drops, or customize network services atanism, called Zero-Copy Software Isolation, to ensure packet
runtime.
isolation without the overhead of packet copies.
Closer to BPF, DTrace [9] is a dynamic tracing infrastrucACKNOWLEDGMENTS
ture for Solaris. DTrace’s bytecode is verified by the kernel
We thank our shepherd, Petr Lapukhov, and the anonymous
before execution. Safety checks are similar to that of BPF:
SOSR reviewers for their valuable comments that helped
invalid operations and forward jumps are prohibited, illegal
improve the quality of this paper. We also thank Alex Palememory accesses are intercepted at runtime.
sandro, Xiao Han, and Diane Adjavon for their helpful input.

Oko: Extending Open vSwitch with Stateful Filters

REFERENCES[1] 2005. The LuaJIT Project. (2005). Retrieved Feb. 16, 2017 from http://luajit.org[2] 2012. The CAIDA anonymized OC48 Internet traces 2002-2003
dataset. (2012). Retrieved Apr., 2017 from http://data.caida.org/
datasets/passive/passive-oc48[3] 2012.
What is Open vSwitch (OVS)?
(2012).
Retrieved
Feb. 9, 2018 from https://www.sdxcentral.com/cloud/open-source/
definitions/what-is-open-vswitch[4] 2013. OpenDaylight project. (Feb. 2013). Retrieved Feb. 9, 2018 fromhttps://www.opendaylight.org[5] 2015. Linux native, HTTP aware network security for containers. (Dec.
2015). Retrieved Feb. 9, 2018 from https://github.com/cilium/cilium[6] G. Bertin. 2016.
Introducing the p0f BPF compiler.
(Aug.
2016). Retrieved Feb. 9, 2018 from https://blog.cloudflare.com/
introducing-the-p0f-bpf-compiler[7] D. Borkmann. 2018. net: add bpfilter. (Feb. 2018). Retrieved Feb. 27,
2018 from https://lwn.net/Articles/747504[8] P. Bosshart, D. Daly, G. Gibb, M. Izzard, N. McKeown, J. Rexford,
C. Schlesinger, D. Talayco, A. Vahdat, G. Varghese, and D. Walker.
2014. P4: Programming protocol-independent packet processors. ACM
SIGCOMM Comput. Commun. Rev. 44, 3 (Jul. 2014).
[9] B. M. Cantrill, M. W. Shapiro, and A. H. Leventhal. 2004. Dynamic
instrumentation of production systems. In Proc. USENIX ATC.
[10] J. Corbet. 2014. BPF: The universal in-kernel virtual machine. (May2014). Retrieved Feb. 9, 2018 from https://lwn.net/Articles/599755[11] J. Corbet. 2016. Early packet drop—and more—with BPF. (Apr. 2016).
Retrieved Feb. 9, 2018 from https://lwn.net/Articles/682538[12] P. Emmerich, S. Gallenmüller, D. Raumer, F. Wohlfart, and G. Carle.
2015. MoonGen: A scriptable high-speed packet generator. In Proc.
ACM IMC.
[13] M. Ghasemi, T. Benson, and J. Rexford. 2017. Dapper: Data plane
performance diagnosis of TCP. In Proc. ACM SOSR.
[14] B. Gregg. 2016. Linux 4.X tracing tools: Using BPF superpowers.
USENIX LISA.
[15] S. Han, K. Jang, A. Panda, S. Palkar, D. Han, and S. Ratnasamy.
2015. SoftNIC: A software NIC to augment hardware. Technical Report UCB/EECS-2015-155. EECS Department, University of California, Berkeley. http://www2.eecs.berkeley.edu/Pubs/TechRpts/2015/
EECS-2015-155.html[16] G. C. Hunt and J. R. Larus. 2007. Singularity: Rethinking the software
stack. ACM SIGOPS Oper. Syst. Rev. 41, 2 (Apr. 2007).
[17] J. Hwang, K. K. Ramakrishnan, and T. Wood. 2014. NetVM: High performance and flexible networking using virtualization on commodity
platforms. In Proc. USENIX NSDI.
[18] E. J. Jackson, M. Walls, A. Panda, J. Pettit, B. Pfaff, J. Rajahalme, T.
Koponen, and S. Shenker. 2016. SoftFlow: A middlebox architecture
for Open vSwitch. In Proc. USENIX ATC.
[19] B. Jenkins. 2016. A hash function for hash table lookup. (2016). Retrieved Feb. 9, 2018 from http://burtleburtle.net/bob/hash/doobs.html[20] S. Jouet, R. Cziva, and D. Pezaros. 2016. Programmable dataplane for next generation networks.
(Mar. 2016).
Retrieved Feb. 9, 2018 from https://netlab.dcs.gla.ac.uk/uploads/files/
d99abd5bbadbed8c0f29808ee812bd26.pdf[21] S. Jouet and D. P. Pezaros. 2017. BPFabric: Data plane programmability
for software defined networks. In Proc. IEEE ANCS.
[22] T. Koponen, K. Amidon, P. Balland, M. Casado, A. Chanda, B. Fulton,
I. Ganichev, J. Gross, N. Gude, P. Ingram, E. Jackson, A. Lambeth, R.
Lenglet, S.-H. Li, A. Padmanabhan, J. Pettit, B. Pfaff, R. Ramanathan, S.
Shenker, A. Shieh, J. Stribling, P. Thakkar, D. Wendlandt, A. Yip, andR. Zhang. 2014. Network virtualization in multi-tenant datacenters. In

SOSR ’18, March 28–29, 2018, Los Angeles, CA, USA
Proc. USENIX NSDI.
[23] R. Lane. 2015. Userspace eBPF VM. (Aug. 2015). Retrieved Feb. 9, 2018from https://github.com/iovisor/ubpf[24] S. Mccanne and V. Jacobson. 1993. The BSD packet filter: A newarchitecture for user-level packet capture. In Proc. USENIX WinterConf.
[25] H. Mekky, F. Hao, S. Mukherjee, T. V. Lakshman, and Z.-L. Zhang. 2017.
Network function virtualization enablement within SDN data plane.
In IEEE INFOCOM.
[26] H. Mekky, F. Hao, S. Mukherjee, Z.-L. Zhang, and T. V. Lakshman.
2014. Application-aware data plane processing in SDN. In Proc. ACM
SIGCOMM HotSDN.
[27] J. Meyer and T. Downing. 1997. Java Virtual Machine. O’Reilly &Associates, Inc.
[28] R. Morris, E. Kohler, J. Jannotti, and M. F. Kaashoek. 1999. The Click
modular router. In Proc. ACM SOSP.
[29] A. Panda, S. Han, K. Jang, M. Walls, S. Ratnasamy, and S. Shenker. 2016.
NetBricks: Taking the V out of NFV. In Proc. USENIX OSDI.
[30] B. Pfaff. 2016. Converging approaches in software switches. ACM
APSys.
[31] B. Pfaff, J. Pettit, T. Koponen, E. Jackson, A. Zhou, J. Rajahalme, J. Gross,
A. Wang, J. Stringer, P. Shelar, K. Amidon, and M. Casado. 2015. The
design and implementation of Open vSwitch. In Proc. USENIX NSDI.
[32] V. Puš, J. Kučera, M. Žádník, and J. Kořenek. 2016. FPGA-based 100
Gbps DDoS protector. TNC17. https://tnc17.geant.org/core/event/31[33] L. Rizzo. 2012. Netmap: A novel framework for fast packet I/O. In Proc.
USENIX ATC.
[34] M. Shahbaz, S. Choi, B. Pfaff, C. Kim, N. Feamster, N. McKeown, and
J. Rexford. 2016. PISCES: A programmable, protocol-independent
software switch. In Proc. ACM SIGCOMM.
[35] M. Sipser. 1996. Introduction to the Theory of Computation (1st ed.).
International Thomson Publishing.
[36] A. Sivaraman, A. Cheung, M. Budiu, C. Kim, M. Alizadeh, H. Balakrishnan, G. Varghese, N. McKeown, and S. Licking. 2016. Packet
Transactions: High-level programming for line-rate switches. In Proc.
ACM SIGCOMM.
[37] V. Sivaraman, S. Narayana, O. Rottenstreich, S. Muthukrishnan, and
J. Rexford. 2017. Heavy-hitter detection entirely in the data plane. In
Proc. ACM SOSR.
[38] J. Sonchack, J. M. Smith, A. J. Aviv, and E. Keller. 2016. Enabling
practical software-defined networking security applications with OFX.
In NDSS.
[39] C.-C. Tu, J. Stringer, and J. Pettit. 2017. Building an extensible OpenvSwitch datapath. ACM SIGOPS Oper. Syst. Rev. 51, 1 (Aug. 2017).
[40] X. Wang, D. Lazar, N. Zeldovich, A. Chlipala, and Z. Tatlock. 2014. Jitk:
A trustworthy in-kernel interpreter infrastructure. In Proc. USENIX
OSDI.
[41] M. Yu, L. Jose, and R. Miao. 2013. Software defined traffic measurementwith OpenSketch. In Proc. USENIX NSDI.
[42] M. Zalewski. 2012. p0f v3. (2012). Retrieved Feb. 9, 2018 from http://lcamtuf.coredump.cx/p0f3