E2: A Framework for NFV Applications
Shoumik Palkar ∗Chang Lan∗

Sangjin Han

UC Berkeley
sppalkar@berkeley.edu

UC Berkeley
clan@eecs.berkeley.edu

UC Berkeley
sangjin@eecs.berkeley.edu

Keon Jang

Aurojit Panda

Sylvia Ratnasamy

Intel Labs
keon.jang@intel.com

UC Berkeley
apanda@cs.berkeley.edu

UC Berkeley
sylvia@eecs.berkeley.edu

Luigi Rizzo

Scott Shenker

Università di Pisa
rizzo@iet.unipi.it

UC Berkeley and ICSI
shenker@icsi.berkeley.edu

Abstractboxes into virtualized software applications that can be run
on commodity, general purpose processors. NFV has quickly
gained signiﬁcant momentum with over 220 industry participants, multiple proof-of-concept prototypes, and a numberof emerging product offerings [2, 9].
While this momentum is encouraging, a closer look “under the hood” reveals a less rosy picture: NFV products andprototypes tend to be merely virtualized software implementations of products that were previously offered as dedicatedhardware appliances. Thus, NFV is currently replacing, on a
one-to-one basis, monolithic hardware with monolithic software. While this is a valuable ﬁrst step – as it is expected
to lower capital costs and deployment barriers – it fails toprovide a coherent management solution for middleboxes.
Each software middlebox still comes as a closed implementation bundled with a custom management solution that addresses issues such as overload detection, load balancing,elastic scaling, and fault-tolerance for that particular NF.
This leads to two problems. First, the operator must copewith many NF-speciﬁc management systems. Second, NF
developers must invent their own solutions to common but
non-trivial problems such as dynamic scaling and fault tolerance; in the worst case this results in inadequate solutions(e.g., solutions that do not scale well) and in the best caseresults in vendors constantly reinventing the wheel.
Inspired by the success of data analytic frameworks (e.g.,
MapReduce, Hadoop and Spark), we argue that NFV needs
a framework, by which we mean a software environment
for packet-processing applications that implements generaltechniques for common issues. Such issues include: placement (which NF runs where), elastic scaling (adapting thenumber of NF instances and balancing load across them),
service composition, resource isolation, fault-tolerance, en-

By moving network appliance functionality from proprietary hardware to software, Network Function Virtualization promises to bring the advantages of cloud computing to
network packet processing. However, the evolution of cloudcomputing (particularly for data analytics) has greatly beneﬁted from application-independent methods for scaling andplacement that achieve high efﬁciency while relieving programmers of these burdens. NFV has no such general management solutions. In this paper, we present a scalable andapplication-agnostic scheduling framework for packet processing, and compare its performance to current approaches.

1.

Introduction

The proliferation of network processing appliances (“middleboxes”) has been accompanied by a growing recognitionof the problems they bring, including expensive hardwareand complex management. This recognition led the networking industry to launch a concerted effort towards Network
Function Virtualization (NFV) with the goal of bringing
greater openness and agility to network dataplanes [8]. Inspired by the beneﬁts of cloud computing, NFV advocatesmoving Network Functions (NFs) out of dedicated physical∗ Joint ﬁrst authors

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor proﬁt or commercial advantage and that copies bear this notice and the full citationon the ﬁrst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
SOSP’15, October 04-07, 2015, Monterey, CA.
Copyright is held by the owner/author(s).
ACM 978-1-4503-3834-9/15/10.
http://dx.doi.org/10.1145/2815400.2815423

121

2.

ergy management, monitoring, and so forth. Although weare focusing on packet-processing applications, the above
are all systems issues, with some aiding NF development(e.g., fault-tolerance), some NF management (e.g., dynamic
scaling) and others orchestration across NFs (e.g., placement, service interconnection).
In this paper, we report on our efforts to build such aframework, which we call Elastic Edge (E2). From a practical perspective, E2 brings two beneﬁts: (i) it allows developers to rely on external framework-based mechanisms forcommon tasks, freeing them to focus on their core application logic and (ii) it simpliﬁes the operator’s responsibilities,
as it both automates and consolidates common managementtasks. To our knowledge, no such framework for NFV exists
today, although several efforts explore individual aspects ofthe problem (as we discuss in §9).
From a conceptual perspective, our contributions are also
twofold. First, we describe algorithms to automate the common tasks of placement, service interconnection, and dynamic scaling. In other work, we also address the issue offault-tolerance [46], with other issues such as performanceisolation, energy management and monitoring left for future
work. Second, we present a system architecture that simpliﬁes building, deploying and managing NFs. Our architecture departs from the prevailing wisdom in that it blurs the
traditional distinction between applications and the network.
Typically one thinks of applications as having fully generalprogramming abstractions while the network has very limited abstractions (essentially that of a switch); this constrains
how functionality is partitioned between application and network (even when network processing is implemented at endhosts [31, 39]) and encourages separate management mechanisms for each. In contrast, because we focus on more limited packet-processing applications and fully embrace software switches, we can push richer programming abstractions
into the network layer.
More concretely, because of the above reasoning, we eschew the dominant software switch, OVS, in favor of a more
modular design inspired by Click [30]. We also depart fromthe traditional SDN/NFV separation of concerns that uses
SDN to route packets between NFs and separately lets NFVmanage those NFs [17, 21, 40]; instead, in E2, a single controller handles both the management and interconnection of
NFs based on a global system view that spans applicationand network resources (e.g., core occupancy and numberof switch rules available). We show that E2’s ﬂexibility together with its coordinated approach to management enables
signiﬁcant performance optimizations; e.g., offering a 2541% reduction in CPU use through ﬂexible system abstractions (§7.1) and a 1.5-4.5x improvement in overall systemthroughput through better management decisions (§7.2).

Context and Assumptions

We now provide a motivating context for the deploymentof a framework such as E2, describe the form of hardware
infrastructure we assume, and brieﬂy sketch the E2 design.
2.1

Motivation: A Scale-Out Central Ofﬁce

We present a concrete deployment context that carriers cite
as an attractive target for NFV: a carrier network’s broadband and cellular edge, as embodied in their Central Ofﬁces (COs) [1]. A CO is a facility commonly located ina metropolitan area to which residential and business lines
connect. Carriers hope to use NFV to transform their COs
to more closely resemble modern datacenters so they canachieve: a uniform architecture based on commodity hardware, efﬁciency through statistical multiplexing, centralized
management across CO locations, and the ﬂexibility andportability of software services. Carriers cite two reasons foroverhauling CO designs [1].
First, the capital and operational expenses incurred by acarrier’s COs are very high. This is because there are manyCOs, each of non-trivial scale; e.g., AT&T reports 5,000
CO locations in the US alone, with 10-100K subscribers per
CO. These COs contain specialized devices such as Broadband Network Gateways (BNGs) [3, 4] that connect broadband users to the carrier’s IP backbone, and Evolved Packet
Core (EPC) gateways that connect cellular users to the IP
backbone. These are standalone devices with proprietary internals and vendor-speciﬁc management APIs.1 NFV-based
COs would enable operators to utilize commodity hardware
while a framework such as E2 would provide a uniﬁed management system.
Secondly, carriers are seeking new business models basedon opening up their infrastructure to 3rd party services.
Hosting services in their COs would enable carriers to exploit their physical proximity to users, but this is difﬁcult
when new features require custom hardware; an NFV-based
CO design would address this difﬁculty. In fact, if carriers
succeed in opening up their infrastructure, then one mightview the network as simply an extension (closer to the user)of existing cloud infrastructure in which case the transition
to NFV becomes necessary for portability between cloudand network infrastructures.
Carrier incentives aside, we note that a CO’s workload
is ideally suited to NFV’s software-centric approach. A perusal of broadband standards [7] and BNG datasheets [4] reveals that COs currently support a range of higher-level trafﬁc processing functions – e.g., content caching, Deep Packet
Inspection (DPI), parental controls, WAN and application
acceleration, trafﬁc scrubbing for DDoS prevention and encryption – in addition to traditional functions for ﬁrewalls,
IPTV multicast, DHCP, VPN, Hierarchical QoS, and NAT.
1 Standardization efforts such as OpenFlow target L2 and L3 forwarding devices and do not address the complexity of managing these specialized systems or middleboxes more generally [44, 47].

122

K external ports

2.3

N-K internal ports

Design Overview

Before presenting E2 in detail in the following sections, weﬁrst provide a brief overview.
E2 Context. We assume that COs reside within an overall
network architecture in which a global SDN controller isgiven (by the operator) a set of network-wide policies toimplement. The SDN controller is responsible for translatingthese network-wide policies into instructions for each CO,
and the E2 cluster within each CO is responsible for carryingout these instructions.The E2 cluster is managed by an E2
Manager, which is responsible for communicating with theglobal SDN controller.

Servers
N-portswitchFigure 1: Hardware infrastructure that E2 manages. We show three
examples of possible forwarding paths through the cluster, including one that involves no server.

E2 Interface. Akin to several recent network management
systems [12, 15–17, 20, 37, 49], E2 provides a declarativeinterface through which the global SDN controller tells each
E2 cluster how trafﬁc should be processed. It does so by
specifying a set of policy statements that we call pipelets.
Each pipelet deﬁnes a trafﬁc class and a corresponding directed acyclic graph (DAG) that captures how this trafﬁcclass should be processed by NFs. A trafﬁc class here refers
to a subset of the input trafﬁc; the DAG is composed of nodes
which represent NFs (or external ports of the switch) andedges which describe the type of trafﬁc (e.g., ‘port 80’) thatshould reach the downstream NF. Figure 2 shows a simpliﬁed example of a pipelet.
Thus, the global SDN controller hands the E2 Manager a
set of pipelets. The E2 Manager is responsible for executing
these pipelets on the E2 cluster as described below, while
communicating status information – e.g., overall load or
hardware failure – back to the global controller.
In addition to policy, E2 takes two forms of external
input: (i) a NF description enumerating any NF-speciﬁc
constraints (e.g., whether the NF can be replicated acrossservers), conﬁguration directives (e.g., number and type ofports), resource requirements (e.g., per-core throughput),and (ii) a hardware description that enumerates switch andserver capabilities (e.g. number of cores, ﬂow table size).

As CO workloads grow in complexity and diversity, so do
the beneﬁts of transitioning to general-purpose infrastructure, and the need for a uniﬁed and application-independent
approach to dealing with common management tasks.
Thus, E2 addresses the question of how you efﬁciently
manage a diverse set of packet processing applications without knowing much about their internal implementation. “Efﬁcient” here means both that the management system introduces little additional overhead, and that it enables high utilization of system resources.
2.2

Hardware Infrastructure

E2 is designed for a hardware infrastructure composed ofgeneral-purpose servers (residing in racks) interconnectedby commodity switches. As shown in Figure 1, we assume
a fabric of commodity switches with N ports, of which K
are dedicated to be ‘externally’ facing (i.e., carrying trafﬁc
to/from the E2 cluster) while the remaining N -K interconnect the servers running NFV services. This switch fabriccan be a single switch, or multiple switches interconnectedwith standard non-blocking topologies. Our prototype uses
a single switch but we expect our design to scale to larger
fabrics.
E2 is responsible for managing system resources andhence we brieﬂy elaborate on the main hardware constraints
it must accommodate. First, E2 must avoid over-booking the
CPU and NIC resources at the servers. Second, E2 must
avoid overloading the switch capacity by unnecessarily placing functions on different servers; e.g., a ﬂow processed byfunctions running at two servers will consume 50% more
switching capacity than if the two functions were placed on
the same server (Figure 1). Third, since commodity switchesoffer relatively small ﬂow tables that can be slow to update,
E2 must avoid excessive use of the ﬂow table at the switch(see §5.3).
Our current prototype has only a single rack. We presume,
based on current packet processing rates and CO trafﬁc volumes, that a CO can be serviced by relatively small cluster sizes (1-10 racks); while we believe that our architecture
will easily scale to such numbers, we leave an experimental
demonstration of this to future work.

E2 Internal Operation. Pipelets dictate what trafﬁc should
be processed by which NFs, but not where or how this processing occurs on the physical cluster. E2 must implement
the policy directives expressed by the pipelets while respecting NF and hardware constraints and capabilities, andit does so with three components, activated in response to
conﬁguration requests or overload indications. (i) The scaling component (§5.3) computes the number of NF instances
needed to handle the estimated trafﬁc demand, and then dynamically adapts this number in response to varying trafﬁcload. It generates an instance graph, or iGraph, reﬂecting
the actual number of instances required for each NF mentioned in the set of pipelets, and how trafﬁc is spread acrossthese instances. (ii) The placement component (§5.1) translates the iGraph into an assignment of NF instances to speciﬁc servers. (iii) The interconnection component (§5.2) con-123

thus be viewed as composed of general attribute-value pairs,
where attributes can be direct (deﬁned on a packet’s contents) or derived (capturing higher-level semantics exposedby network applications). A packet follows an edge only if
it matches all of the trafﬁc ﬁlters on the edge. Note that a
trafﬁc ﬁlter only deﬁnes which trafﬁc ﬂows between functions; E2’s interconnection component (§5.2) addresses how
this trafﬁc is identiﬁed and forwarded across NF ports.
In addition to trafﬁc ﬁlters, an edge is optionally annotated with an estimate of the expected rate of such trafﬁc.
E2’s placement function uses this rate estimate to derive its
initial allocation of resources; this estimate can be approximate or even absent because E2’s dynamic scaling techniques will dynamically adapt resource allocations to varying load.

Web Cache
IDS

IDS.safe && !(dst port 80)

Network
MonitorTraffic
Normalizer

Figure 2: An example pipelet. Input trafﬁc is ﬁrst sent to an IDS;
trafﬁc deemed safe by the IDS is passed to a Web Cache if it’s
destined for TCP port 80 and to a Network Monitor otherwise.
Trafﬁc that the IDS ﬁnds unsafe is passed to a Trafﬁc Normalizer;
all trafﬁc leaving the Trafﬁc Normalizer or the Web Cache are also
passed to the Network Monitor.

ﬁgures the network (including network components at theservers) to steer trafﬁc across appropriate NF instances.
In the following sections we describe E2’s system architecture (§3), its dataplane design (§4), and its control plane
design (§5). We present the implementation (§6) and evaluation (§7) of our E2 prototype then discuss related work (§8)
before concluding in §9.

3.

3.2

In addition to pipelets, E2 takes an NF description thatguides the framework in conﬁguring each NF, and a hardware description that tells the framework what hardware resources are available for use. We describe each in turn.
NF descriptions. E2 uses the following pieces of information for each NF. We envisage that this information (except
the last one) will be provided by NF developers.

E2 System Architecture

We now describe E2’s API, inputs, and system components.
3.1

System Inputs(1) Native vs. Legacy. E2 exports an optional API that allow NFs to leverage performance optimizations (§4). NFs
that use this API are considered “native”, in contrast to unmodiﬁed “legacy” NFs running on the raw socket interface
provided by the OS; we discuss the native API further in §7.

System API

As mentioned in §2, an operator expresses her policies via acollection of pipelets, each describing how a particular trafﬁcclass should be processed. This formulation is declarative, so
operators can generate pipelets without detailed knowledgeof per-site infrastructure or NF implementations. The necessary details will instead be captured in the NF and hardware
descriptions. We now elaborate on how we express pipelets.
Additional detail on the policy description language we use
to express pipelets can be found in the Appendix.
Each pipelet deﬁnes a trafﬁc class and a corresponding
directed acyclic graph (DAG) that captures how this trafﬁcclass should be processed by NFs. In our current implementation, we deﬁne trafﬁc classes in terms of packet headerﬁelds and physical ports on the switch; for example, one
might identify trafﬁc from a particular subscriber via thephysical port, or trafﬁc destined for another provider throughaddress preﬁxes.
A node in the pipelet’s DAG represents a NF or a physicalport on the switch, and edges describe the trafﬁc between
nodes. Edges may be annotated with one or more trafﬁcﬁlters. A ﬁlter is a boolean expression that deﬁnes what
subset of the trafﬁc from the source node should reach thedestination node.
Filters can refer to both, the contents of the packet itself (e.g., header ﬁelds) and to semantic information associated with the packet. For example, the characterization of
trafﬁc as “safe” or “unsafe” in Figure 2 represents semantic information inferred by the upstream IDS NF. Filters can(2) Attribute-Method bindings. Each derived attribute has an
associated method for associating packets with their attributevalues. Our E2 prototype supports two forms of methods:
ports and per-packet metadata (§4).
With the port method, all trafﬁc with an attribute value
will be seen through a particular (virtual or physical) port.
Since a port is associated with a speciﬁc value for an attribute, ports are well-suited for “coarse-grained” attributes
that take on a small number of well-known values. E.g., in
Figure 2, if the IDS deﬁnes the method associated with the“safe” attribute to be “port,” all safe trafﬁc exits the IDS
through one virtual port, and all unsafe trafﬁc through another. Legacy applications that cannot leverage the metadata
method described below ﬁt nicely into this model.
The metadata method is available as a native API. Conceptually, one can think of metadata as a per-packet annotation [30] or tag [17] that stores the attribute-value pair; §4
describes how our system implements metadata using a custom header. Metadata is well-suited for attributes that takemany possible values; e.g., tagging packets with the URL
associated with a ﬂow (versus using a port per unique URL).
(3) Scaling constraints tell E2 whether the application can bescaled across servers/cores or not, thus allowing the framework to react appropriately on overload (§5.3).

124

4.1

           







Our E2D implementation is based on SoftNIC [23], a highperformance, programmable software switch that allows arbitrary packet processing modules to be dynamically conﬁgured as a data ﬂow graph, in a similar manner to the Click
modular router [30].
While the Click-style approach is widely used in various
academic and commercial contexts, the de-facto approach
to trafﬁc management on servers uses the Open vSwitch(OVS) and the OpenFlow interface it exports. OVS is built
on the abstraction of a conventional hardware switch: it isinternally organized as a pipeline of tables that store ‘matchaction’ rules with matches deﬁned on packet header ﬁelds
plus some limited support for counters and internal state.
Given the widespread adoption of OVS, it is reasonable toask why we adopt a different approach. In a nutshell, it isbecause NFV does not share many of the design considerations that (at least historically) have driven the architectureof OVS/Openﬂow and hence the latter may be unnecessarily
restrictive or even at odds with our needs.
More speciﬁcally, OVS evolved to support “network
virtualization platforms” (NVPs) in multi-tenant datacenters [31]. Datacenter operators use NVPs to create multiple
virtual networks, each with independent topologies and addressing architectures, over the same physical network; thisenables (for example) tenants to ‘cut-paste’ a network conﬁguration from their local enterprise to a cloud environment.
The primary operation that NVPs require on the dataplaneis the emulation of a packet’s traversal through a series ofswitches in the virtual topology, and thus OVS has focusedon fast lookups on OpenFlow tables; e.g., using multiple layers of caching internally [39] and limited actions.
NFV does not face this challenge. Instead, since mostcycles will likely be consumed in NFs, we are more interested in performance optimizations that improve the efﬁciency of NFs (e.g., our native APIs below). Thus, rather
than work to adapt OVS to NFV contexts, we chose to explore a Click-inspired dataﬂow design more suited to ourneeds. This choice allowed us to easily implement various
performance optimizations (§7) and functions in support ofdynamic scaling (§5.3) and service interconnection (§5.2).



Figure 3: The overall E2 system architecture.

(4) Afﬁnity constraints. For NFs that scale across servers,
the afﬁnity constraints tell the framework how to split trafﬁc across NF instances. Many NFs perform stateful operations on individual ﬂows and ﬂow aggregates. The afﬁnity
constraints deﬁne the trafﬁc aggregates the NF acts on (e.g.,“all packets with a particular TCP port,” or “all packets ina ﬂow”), and the framework ensures that packets belonging
to the same aggregate are consistently delivered to the same
NF instance. Our prototype accepts afﬁnity constraints deﬁned in terms of the 5-tuple with wildcards.
(5) NF performance. This is an estimate of the per-core, perGHz trafﬁc rate that the NF can sustain2 . This is optional
information that E2’s placement function uses to derive acloser-to-target initial allocation of cores per NF.
Hardware description. In our current prototype, the hardware constraints that E2 considers when making operational
decisions include: (1) the number of cores (and speed) andthe network I/O bandwidth per server, (2) the number ofswitch ports, (3) the number of entries in the switch ﬂow table, and (4) the set of available switch actions. Our hardware
description thus includes this information. We leave to future work the question of whether and how to exploit richer
models – e.g., that consider resources such as the memoryor CPU cache at servers, availability of GPUs or specialized
accelerators [24], programmable switches [15], and so forth.
3.3

System Components

Figure 3 shows the three main system components in E2:
the E2 Manager orchestrates overall operation of the cluster,a Server Agent manages operation within each server, andthe E2 Dataplane (E2D) acts as a software trafﬁc processing
layer that underlies the NFs at each server. The E2 Manager
interfaces with the hardware switch(es) through standard
switch APIs [6, 14, 36] and with the Server Agents.

4.

Rationale

4.2

SoftNICSoftNIC exposes virtual NIC ports (vports) to NF instances;
vports virtualize the hardware NIC ports (pports) for virtualized NFs. Between vports and pports, SoftNIC allows
arbitrary packet processing modules to be conﬁgured as adata ﬂow graph, in a manner similar to the Click modularrouter [30]. This modularity and extensibility differentiate
SoftNIC from OVS, where expressiveness and functionality
are limited by the ﬂow-table semantics and predeﬁned actions of OpenFlow.
SoftNIC achieves high performance by building on recent
techniques for efﬁcient software packet processing. Speciﬁ-

The E2 Dataplane, E2D

In the following subsections we describe the design of the E2
Dataplane (E2D). The goal of E2D is to provide ﬂexible yet
efﬁcient “plumbing” across the NF instances in the pGraph.
2 Since the performance of NFs vary based on server hardware andtrafﬁc characteristics, we expect these estimates will be provided by the
network operator (based on proﬁling the NF in their environment) rather
than by the NF vendor.

125

cally: SoftNIC uses Intel DPDK [27] for low-overhead I/O tohardware NICs and uses pervasive batch processing within
the pipeline to amortize per-packet processing costs. In addition, SoftNIC runs on a small number of dedicated processor cores for high throughput (by better utilizing the
CPU cache) and sub-microsecond latency/jitter (by eliminating context switching cost). The SoftNIC core(s) continuously polls each physical and virtual port for packets.
Packets are processed from one NF to another using a pushto-completion model; once a packet is read from a port, it
is run through a series of modules (e.g. classiﬁcation, rate
limiting, etc.) until it reaches a destination port.
In our experiments with the E2 prototype (§7), we dedicate only one core to E2D/SoftNIC as we ﬁnd a single core
was sufﬁcient to handle the network capacity of our testbed;[23] demonstrates SoftNIC’s scalability to 40 Gbps per core.
4.3

B
A

A

B
B
B

(a) Original pGraph

A
A
A
A

B

B

(c) iGraph with split NF A and B(b) iGraph with split NF B

A
A
A
A

B

B

(d) Optimized iGraph

Figure 4: Transformations of a pGraph (a) into an iGraph (b, c, d).

up and conﬁguring the interconnections between NFs), (iii)
scaling (dynamically adapting the placement decisions depending on load variations), and (iv) ensuring afﬁnity constraints of NFs.

Extending SoftNIC for E2D

We extend SoftNIC in the following three ways. First, we
implement a number of modules tailored for E2D including modules for load monitoring, ﬂow tracking, load balancing, packet classiﬁcation, and tunneling across NFs. These
modules are utilized to implement E2’s components for NF
placement, interconnection, and dynamic scaling, as will be
discussed in the rest of this paper.
Second, as mentioned earlier, E2D provides a native API
that NFs can leverage to achieve better system-wide performance and modularity. This native API provides support for:zero-copy packet transfer over vports for high throughput
communication between E2D and NFs, and rich message abstractions which allow NFs to go beyond traditional packetbased communication. Examples of rich messages include:(i) reconstructed TCP bytestreams (to avoid the redundant
overhead at each NF), (ii) per-packet metadata tags that accompany the packet even across NF boundaries, and (iii)inter-NF signals (e.g., a notiﬁcation to block trafﬁc from an
IPS to a ﬁrewall).
The richer cross-NF communication enables not only various performance optimizations but also better NF design by
allowing modular functions – rather than full-blown NFs–
from different vendors to be combined and reused in a ﬂexible yet efﬁcient manner. We discuss and evaluate the native
API further in §7.
Lastly, E2D extends SoftNIC with a control API exposed
to E2’s Server Agent, allowing it to: (i) dynamically create/destroy vports for NF instances, (ii) add/remove modulesin E2D’s packet processing pipeline, stitching NFs together
both within and across servers, and (iii) receive notiﬁcationsof NF overload or failure from the E2D (potentially triggering scaling or recovery mechanisms).

5.

B

5.1

NF Placement

The initial placement of NFs involves ﬁve steps:
Step 1: Merging pipelets into a single policy graph. E2
ﬁrst combines the set of input pipelets into a single policy
graph, or pGraph; the pGraph is simply the union of theindividual pipelets with one node for each NF and edges
copied from the individual pipelets.
Step 2: Sizing. Next, E2 uses the initial estimate of theload on a NF (sum of all incoming trafﬁc streams), andits per-core capacity from the NF description, to determine
how many instances (running on separate cores) should beallocated to it. The load and capacity estimates need not be
accurate; our goal is merely to ﬁnd a reasonable starting
point for system bootstrapping. Dynamically adapting toactual load is discussed later in this section.
Step 3: Converting the pGraph to an iGraph. This steptransforms the pGraph into the “instance” graph, or iGraph,in which each node represents an instance of a NF. Splitting a node involves rewiring its input and output edges andFigure 4 shows some possible cases. In the general case, as
shown in Figure 4(b) and 4(c), splitting a node requires distributing the input trafﬁc across all its instances in a mannerthat respects all afﬁnity constraints and generating the corresponding edge ﬁlters. As an example, NF B in Figure 4(b)might require trafﬁc with the same 5-tuple go to the same
instance, hence E2 inserts a ﬁlter that hashes trafﬁc from A
on the 5-tuple and splits it evenly towards B’s instances.
When splitting multiple adjacent nodes, the afﬁnity constraints may permit optimizations in the distribute stages, as
depicted in Figure 4(d). In this case, node B from the previous example is preceded by node A that groups trafﬁc bysource IP addresses. If the afﬁnity constraint for A already

The E2 Control Plane

The E2 control plane is in charge of (i) placement (instantiating the pipelets on servers), (ii) interconnection (setting126satisﬁes the afﬁnity constraint for B, E2 does not need to reclassify the outputs from A’s instances, and instead can create direct connections as in Figure 4(d). By minimizing thenumber of edges between NF instances, instance placement
becomes more efﬁcient, as we explain below.

   





 ≠     

Step 4: Instance placement. The next step is to map each
NF instance to a particular server. The goal is to minimize
inter-server trafﬁc for two reasons: (i) software forwarding within a single server incurs lower delay and consumes
fewer processor cycles than going through the NICs [19, 43]
and (ii) the link bandwidth between servers and the switch
is a limited resource. Hence, we treat instance placement as
an optimization problem to minimize the amount of trafﬁc
traversing the switch. This can be modeled as a graph partition problem which is NP-hard and hence we resort to an
iterative local searching algorithm, in a modiﬁed form of theclassic Kernighan-Lin heuristic [28].
The algorithm works as follows: we begin with a valid
solution that is obtained by bin-packing vertices into partitions (servers) based on a depth-ﬁrst search on the iGraph;then in each iteration, we swap a pair of vertices from two
different partitions. The pair selected for a swap is the onethat leads to the greatest reduction in cross-partition trafﬁc.
These iterations continue until no further improvement canbe made. This provides an initial placement of NF instancesin O(n2 lg n) time where n is the number of NF instances.
In addition, we must consider incremental placement as
NF instances are added to the iGraph. While the above algorithm is already incremental in nature, our strategy of migration avoidance (§5.4) imposes that we do not swap anexisting NF instance with a new one. Hence, the incremental
placement is much simpler: we consider all possible partitions where the new instance may be placed, and choose theone that will incur the least cross-partition trafﬁc by simply
enumerating all the neighboring instances of the new NF instance. Thus the complexity of our incremental placement
algorithm is O(n), where n is the number of NF instances.

   "#   !!   

Figure 5: E2 converts edge annotations on an iGraph (a) into
output ports (b) that the applications write to, and then adds trafﬁcﬁlters that the E2D implements (c).

implement NFs, though we have not explored the issue inthe current prototype.
5.2

Service Interconnection

Recall that edges in the pGraph (and by extension, iGraph)
are annotated with ﬁlters. Service interconnection uses theseannotations to steer trafﬁc between NF instances in threestages.
Instantiating NFs’ ports. The NF description speciﬁes how
many output ports are used by a NF and which trafﬁc
attributes are associated with each port. E2D instantiates
vports accordingly as per the NF description and the iGraph.
For example, Fig. 5(b) shows an IDS instance with two
vports, which output “safe” and “unsafe” trafﬁc respectively.
Adding trafﬁc ﬁlters. An edge may require (as speciﬁed by
the edge’s ﬁlters) only a subset of the trafﬁc generated by the
NF instance it is attached to. In this case, E2 will insert an
additional classiﬁcation stage, implemented by the E2D, toensure that the edge only receives trafﬁc matching the edge
ﬁlters. Figure 5(c) illustrates an example where “safe” trafﬁc is further classiﬁed based on the destination port number.
While E2’s classiﬁer currently implements BPF ﬁltering [35]on packet header ﬁelds and metadata tags, we note that it
can be extended beyond traditional ﬁltering to (for example)
ﬁlter packets based on CPU utilization or the active/standby
status of NF instances. To disambiguate trafﬁc leaving ‘mangling’ NFs that rewrite key header ﬁelds (e.g., NAT), theE2D layer dynamically creates disambiguating packet steering rules based on the remaining header ﬁelds. 3

Step 5: Ofﬂoading to the hardware switch. Today’s commodity switch ASICs implement various low-level features, such as L2/L3-forwarding, VLAN/tunneling, and QoS
packet scheduling. This opens the possibility of ofﬂoadingthese functions to hardware when they appear on the policy graph, similar to Dragonet [48] which ofﬂoads functions
from the end-host network stack to NIC hardware). On theother hand, ofﬂoading requires that trafﬁc traverse physicallinks and consume other hardware resources (table entries,
switch ports, queues) that are also limited, so ofﬂoading is
not always possible. To reduce complexity in the placement
decisions, E2 uses an opportunistic approach: a NF is considered as a candidate for ofﬂoading to the switch only if,
at the end of the placement, that NFs is adjacent to a switch
port, and the switch has available resources to run it. E2 does
not preclude the use of specialized hardware accelerators to

Conﬁguring the switch and the E2D. After these steps,
E2 must conﬁgure the switch and E2D to attach NF ports
to edges and instantiate the necessary ﬁlters. Edges that are
local to one server are implemented by the E2D alone. Edges
between servers also ﬂow through the E2D which routes
them to physical NICs, possibly using tunneling to multiplex
several edges into available NICs. Packet encapsulation fortunneling does not cause MTU issues, as commodity NICs
and switches already support jumbo frames.
3 Our approach to handling mangling NFs is enabled by the ability to
inject code inline in the E2D layer. This allows us to avoid the complexity
and inefﬁciency of solutions based on legacy virtual switches such as OVS;
these prior solutions involve creating multiple instances of the mangling
NF, one for each downstream path in the policy graph [20] and invoke thecentral controller for each new ﬂow arrival [17, 20].

127

5.3

Dynamic Scaling

The initial placement decisions are based on estimates of
trafﬁc and per-core performance, both of which are imperfect and may change over time. Hence, we need solutions
for dynamically scaling in the face of changing loads; inparticular we must ﬁnd ways to split the load over several
NF instances when a single instance is no longer sufﬁcient.
We do not present the methods for contraction when underloaded, but these are similar in spirit. We provide hooks for
NFs to report on their instantaneous load, and the E2D itself
detects overloads based on queues and processing delays.
We say we split an instance when we redistribute its load
to two or more instances (one of which is the previous instance) in response to an overload. This involves placing the
new instances, setting up new interconnection state (as described previously in this section), and must consider the
afﬁnity requirements of ﬂows (discussed later in this section), so it is not to be done lightly.
To implement splitting, when a node signals overload
the Server Agent notiﬁes the E2 Manager, which uses theincremental algorithm described in §5.1 to place the NF
instances. The remaining step is to correctly split incoming
trafﬁc across the new and old instances; we address this next.

" 

" 

" 

" 

" 

" 

      

(a)

(b)

(c)

Figure 6: (a) Flows enter a single NF instance. (b) Migration
avoidance partitions the range of Flow IDs and punts new ﬂows
to a new replica using the E2D. Existing ﬂows are routed to thesame instance. (c) Once enough ﬂows expire, E2 installs steeringrules in the switch.
• the E2D on each server can track individual, active ﬂows

that each NF is currently handling. 4 We call Fold (A) the
current set of ﬂows handled by some NF A.
When an iGraph is initially mapped onto E2, each NF instance A may have a corresponding range ﬁlter [X, Y ) → A
installed in the E2D layer or in the hardware switch. When
splitting A into A and A’, we must partition the range [X, Y ),but keep sending ﬂows in Fold (A) to A until they naturally
terminate.
Strawman approach. This can be achieved by replacing theﬁlter [X, Y ) → A with two ﬁlters

5.4[X, M ) → A, [M, Y ) → AMigration Avoidance for Flow Afﬁnity

Most middleboxes are stateful and require afﬁnity, where
trafﬁc for a given ﬂow must reach the instance that holds
that ﬂow’s state. In such cases, splitting a NF’s instance(and correspondingly, input trafﬁc) requires extra measures
to preserve afﬁnity.
Prior solutions that maintain afﬁnity either depend on
state migration techniques (moving the relevant state from
one instance to another), which is both expensive and incompatible with legacy applications [21], or require large
rule sets in hardware switches [41]; we discuss these solutions later in §7.
We instead develop a novel migration avoidance strategy
in which the hardware and software switch act in concert tomaintain afﬁnity. Our scheme does not require state migration, is designed to minimize the number of ﬂow table entries
used on the hardware switch to pass trafﬁc to NF instances,and is based on the following assumptions:and higher priority ﬁlters (“exceptions”) to preserve afﬁnity:∀f : f ∈ Fold (A) ∧ H(f ) ∈ [M, Y ) : f → A
The number of exceptions can be very large. If the switch has
small ﬁltering tables (hardware switches typically have only
a few thousand entries), we can reduce the range [M, Y )to keep the number of exceptions small, but this causes anuneven trafﬁc split. This problem arises when the ﬁlters must
be installed on a hardware switch, and A and A’ reside on
different servers.
Our solution To handle this case efﬁciently, our migration
avoidance algorithm uses the following strategy (illustrated
in Figure 6) :• Opon splitting, the range ﬁlter [X, Y ) on the hardware

switch is initially unchanged, and the new ﬁlters (twonew ranges plus exceptions) are installed in the E2D ofthe server that hosts A;• As ﬂows in Fold (A) gradually terminate, the corresponding exception rules can be removed;• each ﬂow f can be mapped (for instance, through a hashfunction applied to relevant header ﬁelds) to a ﬂow ID
H(f ), deﬁned as an integer in the interval R = [0, 2N );• the hardware switch can compute the ﬂow ID, and canmatch arbitrary ranges in R with a modest numberof rules. Even TCAM-based switches, without a native
range ﬁlter, require fewer than 2N rules for this;• When the number of exceptions drops below somethreshold, the new ranges and remaining exceptions
are pushed to the switch, replacing the original rule
[X, Y ) → A.

• each NF instance is associated with one subrange of the

4 The NF description indicates how to aggregate trafﬁc into ﬂows (i.e.,the same subset of header ﬁelds used to compute the ﬂow ID).

interval R;128

By temporarily leveraging the capabilities of the E2D, migration avoidance achieves load distribution without thecomplexity of state migration and with efﬁcient use ofswitch resources. The trade-off is the additional latency to
new ﬂows being punted between servers (but this overhead
is small and for a short period of time) and some additionalswitch bandwidth (again, for a short duration) – we quantify
these overheads in §7.

Experimental Setup. We evaluate our design choices using
the above E2 prototype. We connect a trafﬁc generator toexternal ports on our switch with four 10 G links. We use
a server with four 10G NICs and two Intel Xeon E5-2680
v2 CPUs as a trafﬁc generator. We implemented the trafﬁc
generator to act as the trafﬁc source and sink. Unless statedotherwise, we present results for a trafﬁc workload of all
minimum-sized 60B Ethernet packets.

6.

7.1

Prototype Implementation

We show that E2D introduces little overhead and that its
native APIs enable valuable performance improvements.

Our E2 implementation consists of the E2 Manager, theServer Agent, and the E2D. The E2 Manager is implementedin F# and connects to the switch and each server using an
out-of-band control network. It interfaces with the switch
via an OpenFlow-like API to program the ﬂow table, which
is used to load balance trafﬁc and route packets between
servers. The E2 Manager runs our placement algorithm (§5)
and accordingly allocates a subset of nodes (i.e., NF instances) from the iGraph to each server and instructs theServer Agent to allocate cores for the NFs it has been assigned, to execute the the NFs, to interface with the E2D
to allocate ports, create and compose processing modules in
SoftNIC, and to set up paths between NFs.
The Server Agent is implemented in Python and runs as a
Python daemon on each server in the E2 cluster. The Server
Agent acts as a shim layer between the E2 Manager and its
local E2D, and it simply executes the instructions passed by
the E2 Manager.
The E2D is built on SoftNIC (§4). Our E2D contains several SoftNIC modules which the Server Agent conﬁgures forservice interconnection and load balancing. Speciﬁcally, we
have implemented a match/action module for packet metadata, a module for tagging and untagging packets with tunneling headers to route between servers, and a steering module which implements E2D’s part in migration avoidance.
The E2D implements the native API discussed in §4.3; forlegacy NFs, E2D creates regular Linux network devices.

7.

E2D: Data Plane Performance

E2D Overhead. We evaluate the overhead that E2D introduces with a simple forwarding test, where packets are generated by an NF and ‘looped back’ by the switch to the same
NF. In this setup, packets traverse the E2D layer twice (NF→ switch and switch → NF directions). We record an average latency of 4.91 μs.
We compare this result with a scenario where the NF is
directly implemented with DPDK (recall that SoftNIC andhence E2D build on top of DPDK), in order to rule out
the overhead of E2D. In this case the average latency was
4.61 μs, indicating that E2D incurs 0.3 μs delay (or 0.15 μs
for each direction). Given that a typical end-to-end latency
requirement within a CO is 1 ms5 , we believe that this latency overhead is insigniﬁcant.
In terms of throughput, forwarding through E2D on asingle core fully saturates the server’s 10 Gbps link as expected [23, 27].
The low latency and high throughput that E2D achieves
is thanks to its use of SoftNIC/DPDK. Our results merely
show that the baseline overhead that E2D/SoftNIC adds
to its underlying DPDK is minimal; more complex packet
processing at NFs would, of course, result in proportionallyhigher delays and lower throughput.
E2D Native API. Recall that E2’s native API enables performance optimizations through its support for zero-copy
vports and rich messages. We use the latter to implement
two optimizations: (i) bytestream vports that allow the costof TCP session reconstruction to be amortized across NFs
and, (ii) packet metadata tags that eliminate redundant workby allowing semantic information computed by one NF to
be shared with the E2D or other NFs. We now quantify theperformance beneﬁts due to these optimizations: zero-copy
vports, bytestream vports, metadata.
Zero-copy vports. We measure the latency and throughput
between two NFs on a single server (since the native API
does nothing to optimize communication between servers).
Table 1 compares the average latency and throughput of thelegacy and native APIs along this NF → E2D → NF path.
We see that our native API reduces the latency of NF-toNF communication by over 2.5x on average and increases

Evaluation

Prototype. Our E2 prototype uses an Intel FM6000 Seacliff
Trail Switch with 48 10 Gbps ports and 2,048 ﬂow table
entries. We connect four servers to the switch, each with one
10 Gbps link. One server uses the Intel Xeon E5-2680 v2
CPU with 10 cores in each of 2 sockets and the remaining
use the Intel Xeon E5-2650 v2 CPU with 8 cores in eachof 2 sockets, for a total of 68 cores running at 2.6 GHz. On
each server, we dedicate one core to run the E2D layer. The
E2 Manager runs on a standalone server that connects toeach server and to the management port of the switch on
a separate 1 Gbps control network.
We start with microbenchmarks that evaluate E2’s data
plane (§7.1), then evaluate E2’s control plane techniques(§7.2) and ﬁnally evaluate overall system performance with
E2 (§7.3).

5 From discussion with carriers and NF vendors.

129

Path
NF → E2D→ NF
Legacy API
Native Zero-Copy API

Latency
(μs)
3.2
1.214

Gbps
(1500B)
7.437
187.515

Mpps
(64B)
0.929
15.24

Path
NF → E2D→ NF
Header-Match
Metadata-Match

Table 1: Latency and throughput between NFs on a single serverusing E2’s legacy vs. native API. Legacy NFs use the Linux raw
socket interface.
TCP

SIG

HTTP

Latency
(μs)
1.56
1.695

Gbps
(1500B)
152.515145.826

Mpps
(64B)
12.76
11.96

Table 2: Latency and throughput between NFs on a single serverwith and without metadata tags.
<==*(*#;9';
ADF(57#99%:%';

RE>?:$+5(*@$(:';J!,(M#$3O,'/%3#$'/(57#99%:%';
Library#$%&'()*+


ByteStream



!

  
0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

"



Figure 8: Comparison of CPU cycles between using URL metadata
and a dedicated HTTP parser

GHz per Gb traffic

Figure 7: Comparison of CPU cycles for three DPI NFs, without
and with bytestream vports. The both cases use the native API.

To measure the overhead, we measure the inter-NF
throughput using our zero-copy native API under two scenarios. In Header-Match, the E2D simply checks a particular
header ﬁeld against a conﬁgured value; no metadata tags areattached to packets. In Metadata-Match, the source NF creates a metadata tag for each packet which is set to the valueof a bit ﬁeld in the payload; the E2D then checks the tag
against a conﬁgured value. Table 2 shows our results. We seethat Metadata-Match achieves a throughput of 11.96 mpps,
compared to 12.7 for Header-Match. Thus adding metadata
lowers throughput by 5.7%.
We demonstrate the performance beneﬁts of metadatatags using a pipeline in which packets leaving an upstream
HTTP Logger NF are forwarded to a CDN NF based on
the value of the URL associated with their session. Since
Logger implements HTTP parsing, a native implementationof the Logger NF can tag packets with their associated URL
and the E2D layer will steer packets based on this metadataﬁeld. Without native metadata tags, we need to insert astandalone ‘URL-Classiﬁer’ NF in the pipeline between the
Logger and CDN NFs to create equivalent information. In
this case, trafﬁc ﬂows as Logger→ E2D → URL-Classiﬁer→ E2D→CDN. As shown in Figure 8, the additional NF
and E2D traversal (in bold) increase the processing load by
41% compared to the use of native tags.

throughput by over 26x; this improvement is largely due tozero-copy vports (§4) and the fact that legacy NFs incur OSinduced overheads due to packet copies and interrupts. Ournative APIs matches the performance of frameworks such as
DPDK [27] and netmap [42].
Bytestream vports. TCP session reconstruction, which involves packet parsing, ﬂow state tracking, and TCP segment reassembly, is a common operation required by most
DPI-based NFs. Hence, when there are multiple DPI NFs
in a pipeline, repeatedly performing TCP reconstruction canwaste processing cycles.
We evaluate the performance beneﬁts of bytestream
vports using a pipeline of three simple DPI NFs: (i) SIG
implements signature matching with the Aho-Corasickalgorithm, (ii) HTTP implements an HTTP parser, and(iii) RE implements redundancy elimination using Rabin
ﬁngerprinting. These represent an IDS, URL-based ﬁltering,
and a WAN optimizer, respectively. The Library case in
Fig. 7 represents a baseline, where each NF independently
performs TCP reconstruction over received packets with ourcommon TCP library. In the ByteStream case, we dedicate
a separate NF (TCP) to perform TCP reconstruction andproduce metadata (TCP state changes and reassembly
anomalies) and reassembled bytestream messages for thethree downstream NFs to reuse. E2D guarantees reliabletransfer of all messages between NFs that use bytestream
vports, with much less overhead than full TCP. The results
show that bytestream vports can save 25% of processing
cycles, for the same amount of input trafﬁc.

7.2

E2 Control Plane Performance

We now evaluate our control plane solutions for NF placement, interconnection, and dynamic scaling, showing thatour placement approach achieves better efﬁciency than two
strawmen solutions and that our migration-avoidance design
is better than two natural alternatives.

Metadata Tags. Tags can carry information along with packets and save repeated work in the applications; having theE2D manage tags is both a convenience and potentially alsoa performance beneﬁt for application writers. The following
two experiments quantify the overhead and potential performance beneﬁts due to tagging packets with metadata.

NF Placement. E2 aims to maximize cluster-wide throughput by placing NFs in a manner that minimizes use of the
hardware switch capacity. We evaluate this strategy by simulating the maximum cluster-wide throughput that a rackscale E2 system (i.e., with 24 servers and 24 external ports)

130

RandomLinearRandomized
0

20
40
60
Aggregate throughput (Gbps)

80









(
(



Figure 10: Trafﬁc load at the original and new NF instance with
migration avoidance; original NF splits at 2s.

(μFigure 9: Maximum cluster throughput with different placement
solutions, with two different pGraphs.

could sustain before any component – cores, server links, or
switch capacity – of the system is saturated. We compare oursolution to two strawmen: “Random” that places nodes onservers at random, and “Packing” that greedily packs nodes
onto servers while traversing the iGraph depth-ﬁrst. We consider two iGraphs: a linear chain with 5 nodes, and a more
realistic random graph with 10 nodes.
Figure 9 shows that our approach outperforms the strawmen in all cases. We achieve 2.25-2.59× higher throughput
compared to random placement; bin-packing does well ona simple chain but only achieves 0.78× lower throughput
for more general graphs. Thus we see that our placement
heuristic can improve the overall cluster throughput over thebaseline bin-packing algorithm.
Finally, we measure the controller’s time to computeplacements. Our controller implementation takes 14.6ms tocompute an initial placement for a 100-node iGraph and has
a response time of 1.76ms when handling 68 split requests
per second (which represents the aggressive case of one split
request per core per second). We conclude that a centralized
controller is unlikely to be a performance bottleneck in thesystem.
Updating Service Interconnection. We now look at thetime the control plane takes to update interconnection paths.
In our experiments, the time to update a single rule in theswitch varies between 1-8ms with an average of 3ms (thedatasheet suggests 1.8ms as the expected time); the switch
API only supports one update at a time. In contrast, the perrule update latency in E2D is only 20 μs, which can be further amortized with a batch of multiple rules. The relatively
long time it takes to update the hardware switch (as compared to the software switch) reinforces our conservative useof switch rule entries in migration avoidance. Reconﬁguring
the E2D after creating a new replica takes roughly 15ms, including the time to coordinate with the E2 Manager and to
invoke a new instance.
Dynamic Scaling. We start by evaluating migration avoidance for the simple scenario of a single NF instance thatsplits in two; the NF requires ﬂow-level afﬁnity. We drive
the NF with 1 Gbps of input trafﬁc, with 2,000 new ﬂows
arriving each second on average and ﬂow length distributions drawn from published measurement studies [32]. This








( 
( 











( Packing( 

E2 Heurstic(

Figure 11: Latency and bandwidth overheads of migration avoidance (the splitting phase is from 1.8s to 7.4s).

results in a total load of approximately 10,000 active concurrent ﬂows and hence dynamic scaling (effectively) requires‘shifting’ load equivalent to 5,000 ﬂows off the original NF.
Fig. 10 shows the trafﬁc load on the original and new NF
instances over time; migration avoidance is triggered close
to the 2 second mark. We see that our prototype is effective
at balancing load: once the system converges, the imbalancein trafﬁc load on the two instances is less than 10%.
We also look at how active ﬂows are impacted during theprocess of splitting. Fig. 11 shows the corresponding packet
latency and switch bandwidth consumption over time. We
see that packet latency and bandwidth consumption increaseduring the splitting phase (roughly between the two andeight second markers) as would be expected given we ‘detour’ trafﬁc through the E2D layer at the original instance.
However this degradation is low: in this experiment, latency
increases by less than 10μsecs on average, while switch
bandwidth increases by 0.5Gbps in the worst case, for asmall period of time; the former overhead is a ﬁxed cost, thelatter depends on the arrival rate of new ﬂows which is relatively high in our experiment. In summary: migration avoidance balances load evenly (within 10% of ideal) and withina reasonable time frame (shifting load equivalent to roughly
5,000 ﬂows in 5.6 seconds) and does so with minimal impact to active ﬂows (adding less than 10μseconds to packet
latencies) and highly scalable use of the switch ﬂow table.
We brieﬂy compare to two natural strawmen. An “always
migrate” approach, as explored in [41] and used in [21], migrates half the active ﬂows to the new NF instance. This approach achieves an ideal balance of load but is complex6 and
6 For example, [41] reroutes trafﬁc to an SDN controller while migration is in progress while [21] requires a two-phase commit between the131CPU (cores)

Optimal CPU
Switch Trafficload. As points of comparison, we also plot the input trafﬁcload (a lower bound on the switch bandwidth usage) and acomputed value of the optimal number of cores. We derived
the optimal number of cores by summing up the optimal
number of NF instances for each NF in the pipeline. To
derive the optimal number of NF instances for a NF, we
multiply the cycles per unit of trafﬁc that the NF consumes
when running in isolation by the total input trafﬁc to the NF,
then we divide it by the cycle frequency of a CPU core.
We observe that E2’s resource consumption (both CPU
and switch bandwidth) scales dynamically to track the trend
in input load. At its maximum scaling point, the system
consumed up to 60 cores, running an iGraph of 56 vertices(i.e., 56 NF instances) and approximately 600 edges. We
also observe the gap between actual vs. optimal resource
usage in terms of both CPU cores (22.7% on average) andthe switch bandwidth (16.4% on average). We note that
our measured CPU usage does include the cores dedicatedto running SoftNIC (which was always one per server inour experiments) while these cores are not included in the
optimal core count. Thus the overheads that E2 incurs appear
reasonable given that our lower bounds ignore a range ofsystem overheads around forwarding packets between NFs,
the NIC and the switch, as also the overheads around running
multiple NFs on a shared core or CPU (cache effects, etc.),
and the efﬁciencies that result from avoiding migration andincremental placement as trafﬁc load varies. Finally, we notethat our NFs do not use our bytestream and metadata APIs
which we expect could yield further improvements.

40

60

30

40

20

20

10

0

Throughput (Gbps)

CPU Usage
Input Traffic

80

0
0

20

40
60
Time (s)80100Figure 12: E2 under dynamic workload.










Figure 13: Pipeline used for the evaluation

requires non-trivial code modiﬁcations to support surgicalmigration of per-ﬂow state. In addition, the disruption due
to migration is non-trivial: the authors of [41] report taking
5ms to migrate a single ﬂow during which time trafﬁc mustbe “paused”; the authors do not report performance when
migrating more than a single ﬂow.
A “never migrate” approach that does not leverage software switches avoids migrating ﬂows by pushing exception
ﬁlters to the hardware switch. This approach is simple andavoids the overhead of detouring trafﬁc that we incur. However, this approach scales poorly; e.g., running the above experiment with never-migrate resulted in a 80% imbalance
while consuming all 2,048 rules on the switch.7 Not only
was the asymptotic result poor, but convergence was slow
because the switch takes over 1ms to add a single rule andwe needed to add close to 2,000 rules.
7.3

8.

Related WorkWe elaborate on related efforts beyond the work mentioned
inline throughout this paper.

E2 Whole-System Performance

Industry efforts. Within industry, ETSI operates as the standards body for NFV and OPNFV is a nascent open-source
effort with many of the same participants. While “orchestration” ﬁgures prominently in their white papers, discussions with participants in these efforts revealed that few
demonstrated solutions exist as yet. Components of E2 havebeen approved as an informative standard at ETSI to ﬁll thisrole [10].

To test overall system performance for more realistic NF
workloads, we derived a policy graph based on carrier guidelines [7] and BNG router datasheets [4] with 4 NFs: a NAT, aﬁrewall, an IDS and a VPN, as shown in Figure 13. All NFs
are implemented in C over our zero-copy native API.
We use our prototype with the server and switch conﬁguration described earlier. As in prior work on scaling middleboxes [21], we generate trafﬁc to match the ﬂow-size distribution observed in real-world measurements [13].
We begin the experiment with an input load of 7.2 Gbps
and the optimal placement of NFs. Over the course of theexperiment, we then vary the input load dynamically up to a
maximum of 12.3 Gbps and measure the CPU utilization andswitch bandwidth used by E2. Figure 12 plots this measured
CPU and switch resource consumption under varying input

In terms of academic projects, there are various systems that
address individual aspects of E2’s functionality, such as loadbalancing [18, 38], interconnection [17], declarative policies[49], migration [21, 41], but do not provide an overall framework and the system optimizations this enables.
End-to-end NF management. Closest to E2 is Stratos [20],an orchestration layer for NFs deployed in clouds. E2 andStratos share similar goals but differ signiﬁcantly in theirdesign details. At a high level, we believe these differences
stem from E2’s decision to deviate from the canonical SDN
architecture. In canonical SDN, the virtual switch at servers

controller and switches; the crux of the problem here is the need for close
coordination between trafﬁc and state migration.
7 The “always migrate” prototype in [41] also uses per-ﬂow rules inswitches but this does not appear fundamental to their approach.

132

offers only the limited processing logic associated with the
OpenFlow standard. Instead, E2’s control solutions exploit
the ability to inject ‘non standard’ processing logic on thedata path and this allows us to devise simpler and morescalable solutions for tasks such as service interconnection,
overload detection, and dynamic scaling.

do not address the consistency issues that arise when global
or aggregate state is spread across multiple NF instances.
This is a difﬁcult problem that is addressed in the SplitMerge [41] and OpenNF [21] systems. These systems require that NF vendors adopt a new programming model [41]or add a non-trivial amount of code to existing NF implementations [21]. To support NFs that adopt the Split-Merge
or OpenNF architecture, we could extend the E2 controller
to implement their corresponding control mechanisms.

NF interconnection. Our use of metadata tags (§3) takes
inspiration from prior work on FlowTags [17] but with a few
simplifying differences: (1) we do not require metadata toaccommodate ‘mangling’ NFs, such as NAT (a key focus in[17]); and (2) metadata tags are only declared and conﬁgured
at the time of system initialization, and at runtime the E2
datapath does not require reactive involvement of the SDN
controller on a per-ﬂow basis, as in FlowTags. Once again,
we believe these differences follow from our decision toembed rich programmability in the E2D layer. Our metadatatags are also inspired by the concept of packet annotations inClick [30] and ‘network service headers’ as deﬁned by the
IETF’s Service Function Chaining [45].

9.

Conclusion

In this paper we have presented E2, a framework for NFV
packet processing. It provides the operator with a single coherent system for managing NFs, while relieving developersfrom having to develop per-NF solutions for placement, scaling, fault-tolerance, and other functionality. We hope that an
open-source framework such as E2 will enable potential NF
vendors to focus on implementing interesting new NFs while
network operators (and the open-source community) can focus on improving the common management framework.
We veriﬁed that E2 did not impose undue overheads, andenabled ﬂexible and efﬁcient interconnection of NFs. We
also demonstrated that our placement algorithm performed
substantially better than random placement and bin-packing,and our approach to splitting NFs with afﬁnity constraints
was superior to the competing approaches.

Hardware platform. E2’s platform of choice is a combination of commodity servers and merchant switch silicon. The
ServerSwitch system [33] and some commercial “service
routers” and appliances [5] also combine x86 and switching hardware; however, in these designs, the two are tightly
integrated, locking operators into a ﬁxed ratio of switching
to compute capacity. E2 is instead based on loose integration: operators can mix-and-match components from different vendors and can reconﬁgure (even in the ﬁeld) theamount of switching and compute capacity based on their
evolving needs. Greenhalgh et al. [22] describe their visionof a “ﬂowstream” platform that combines switch and x86
hardware in a manner similar to E2 but we are unaware ofa detailed design or implementation based on the proposed
platform, nor do they articulate the need for a framework of
the form E2 aims to provide.

Acknowledgements
We thank our shepherd, Jeff Mogul, and the anonymous reviewers of the SOSP program committee, for their thoughtful feedback which greatly improved this paper. Ethan Jackson, Kay Ousterhout, and Joshua Reich offered feedback onearly drafts of this paper and Maziar Manesh helped us deﬁne our lab environment. We are indebted to Tom Anschutz(AT&T), Wenjing Chu (Dell), Yunsong Lu (Huawei), Christian Maciocco (Intel), Pranav Mehta (Intel), Andrew Randall (Metaswitch), Attila Takacs (Ericsson), Percy Tarapore(AT&T), Martin Taylor (Metaswitch) and Venky Venkatesan(Intel) for discussions on NFV efforts in industry.

Data plane components. Multiple recent efforts provide specialized platforms in support of efﬁcient software
packet processing. Frameworks such as DPDK [27] andnetmap [42] are well established tools for high performance packet I/O over commodity NICs. Other systems
address efﬁcient packet transfer between VMs in a single
server [19, 25, 26, 43], still others explore the trade-offs inhosting NFs in processes, containers, or VMs [29, 34]. All
these systems address issues that are complementary but orthogonal to E2: these systems do not address the end-to-end
NFV orchestration that E2 does (e.g., placement, scaling),
but E2 (and the NFs that E2 supports) can leverage ideas
developed in these systems for improved performance.

Appendix: E2 Policy Language
In E2, the operator writes a collection of policy statements,
each of which is represented as a directed acyclic graphwhich we call a ‘pipelet’. Nodes in a pipelet correspond to
Network Functions (NFs) which receive packets on inboundedges and output packets to outbound edges. Edges are associated with ﬁlters which we elaborate on shortly.
NFs are instantiated from speciﬁc application types,
much like objects are instantiated from classes in objectoriented programming. In addition to user-deﬁned NFs,
there are predeﬁned ones such as Port which denotes a porton the switch, Drop, which discards packets, and Tee which
creates multiple copies of a packet.

Dynamic scaling with cross-NF state. Our migration
avoidance scheme (§5.4) avoids the complexity of state migration for NFs that operate on state that is easily partitionedor replicated across NF instances; e.g., per-ﬂow counters,
per-ﬂow state machines and forwarding tables. However, we

133

// First, instantiate NFs from application types.
Proxy p;
NAT
n;
FW
f;
Port<0-7> int; // internal customer-facing ports
Port<8-15> ext; // external WAN-facing ports

dst port 80

ext

int

p

!(dst port 80)

fFW.safe &&(dst ip wan_net)!(dst ip my_net)n// subnet declarations, to simplify filter writing
Address my_net 10.12.30.0/24; // private IP addr
Address wan_net 131.114.88.92/30; // public IP addrn!(src port 80) &&(dst ip my_net)fpipelet { // outbound traffic
int [dst port 80] -> p;
int [!(dst port 80)] -> n;p [!(dst ip my_net)] -> n;n -> f;f [FW.safe && !(dst ip wan_net)] -> ext;
f [!FW.safe] -> Drop;}

FW.safe &&!(dst ip wan_net)
ext!FW.safeDrop(a) pipelet for outbound trafficint

src port 80 &&dst ip my_netpdst ip my_net(b) Pipelet for outbound traffic

Figure 15: Graphical representation of the pipelets in Figure 14.

pipelet { // inbound traffic
ext -> f;
f [FW.safe && (dst ip wan_net)] -> n;n [(src port 80) && (dst ip my_net)] -> p;n [!(src port 80) && (dst ip my_net)] -> int;
p [dst ip my_net] -> int;}

RateLimiter r;
Port<0-7> int;// two input ports, vp0 and vp1// internal customer-facing ports

pipelet { // outbound traffic
int [tos 0] -> [prio0] r;
int [tos 1] -> [prio1] r;
r -> ...
}

Figure 14: An example speciﬁcation of a policy. (Certain drop
actions are omitted for clarity.)

Figure 16: An example of a policy that uses ﬁlter in annotations.

Figure 14 shows an example of a policy with two pipelets,
each of which represents a subset of the possible paths forforward and reverse trafﬁc respectively. Figure 15 shows
the same example in graph form. The ﬁrst ﬁve lines deﬁne
the nodes in the graph. Following this are two pipelets,
each deﬁning a graph for a speciﬁc trafﬁc class. Within thepipelet, we list all the edges forming a policy graph for that
trafﬁc class. An edge is described using the simple syntax:

the operator to be agnostic to how the rate-limiter identiﬁes
prio0 vs. prio1 trafﬁc. For example, if this is done using
metadata (a native rate limiter), E2 will automatically conﬁgure the E2D layer to add the appropriate metadata tags; if
instead the rate-limiter offers different input ports for eachpriority class, then the E2D layer will simply steer trafﬁc tothe appropriate vport (with no tags, as in our example).
E2 merges all pipelets into a single policy graph, termeda pGraph. During this process, E2 checks that each packet
coming from a node has exactly one destination. This is veriﬁed by checking that the ﬁlters on every pair of edges coming out from a NF has an empty intersection, and that theunion of all ﬁlters attached to a NF’s output evaluates totrue. If a packet has more than one possible destination, E2
ﬁrst attempts to remedy this by adding ﬁlters to ambiguousedges, specifying the trafﬁc class of the pipelet corresponding to that edge. E.g., simply merging the two pipelets in
Figure 15 results in ambiguity for packets leaving the NAT.
E2 will thus add a ﬁlter that limits the edge for the ‘n -> f’rule to trafﬁc arriving from an internal port. If E2 is unable
to automatically resolve ambiguity, it returns an error (thisis possible if the operator writes a malformed pipelet); E2
also returns an error if a packet coming from a node has no
destination.

src [filter_out] -> (bw) [filter_in] dst;

where all three annotations—ﬁlter out, bw, and ﬁlter in—are optional. Filters are boolean expressions computed overpacket header ﬁelds, physical/virtual ports, or metadata tags,
and are written in the libpcap-ﬁlter syntax [11]. The ﬁlter out annotations specify which packets generated from a
NF should enter the edge and implicitly deﬁne the trafﬁcclass; ﬁlter in annotations capture requirements on incoming trafﬁc that are imposed by the downstream NF (example
below) and bw denotes the expected amount of trafﬁc on theedge, at system bootup.
Figure 16 shows a partial example of the use of ﬁlter inannotations. Here outbound trafﬁc is passed through a ratelimiter with two input vports; high priority trafﬁc must arrive on one vport and low priority trafﬁc on the other. Thus
trafﬁc must be ﬁltered prior to entering the downstream NF;
we use ﬁlter in annotations to capture such requirements. In
this example, prio0 and prio1 are NF-speciﬁc metadata that,in this case, are resolved to ports vp0 and vp1 at compile
time based on information in the rate-limiter’s NF description (see §2). In some sense, these are ‘placeholder’ metadata
in that they serve as a level of indirection between policy
and mechanism but may not appear at runtime. This allows

134

References

ing with Hardware and Software. In Proc. ACM SIGCOMM(2014).

[1] AT&T Domain 2.0 Vision White Paper.
https://www.att.com/Common/about_us/pdf/AT&T%
20Domain%202.0%20Vision%20White%20Paper.pdf.

[19] G ARZARELLA , S., L ETTIERI , G., AND R IZZO , L. Virtual
Device Passthrough for High Speed VM Networking. In Proc.
ANCS (2015).

[2] Brocade Vyatta 5400 vRouter. http://www.brocade.com/
products/all/network-functions-virtualization/
product-details/5400-vrouter/index.page.

[20] G EMBER , A., K RISHNAMURTHY, A., J OHN , S. S.,
G RANDL , R., G AO , X., A NAND , A., B ENSON , T., A KELLA ,A., AND S EKAR , V. Stratos: A Network-Aware Orchestration
Layer for Middleboxes in the Cloud. CoRR abs/1305.0209(2013).

[3] Ericsson SE Family.
http://www.ericsson.com/
ourportfolio/products/se-family.
[4] Evolution of the Broadband Network Gateway. http://
resources.alcatel-lucent.com/?cid=157553.

[21] G EMBER -JACOBSON , A., V ISWANATHAN , R., P RAKASH ,C., G RANDL , R., K HALID , J., DAS , S., AND A KELLA , A.
OpenNF: Enabling Innovation in Network Function Control.
In Proc. ACM SIGCOMM (2014).

[5] Evolved Packet Core Solution. http://lte.alcatellucent.com/locale/en_us/downloads/wp_mobile_core_technical_innovation.pdf.

[22] G REENHALGH , A., H UICI , F., H OERDT, M., PAPADIM ITRIOU , P., H ANDLEY, M., AND M ATHY, L. Flow Processing and the Rise of Commodity Network Hardware. ACM
SIGCOMM Computer Communications Review 39, 2 (2009),
20–26.

[6] Intel Ethernet Switch FM6000 Series - Software Deﬁned Networking.
http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/ethernetswitch-fm6000-sdn-paper.pdf.
[7] Migration to Ethernet-Based Broadband Aggregation.
http://www.broadband-forum.org/technical/
download/TR-101_Issue-2.pdf.

[23] H AN , S., JANG , K., PANDA , A., PALKAR , S., H AN , D.,
AND R ATNASAMY, S. SoftNIC: A Software NIC to Augment
Hardware. UCB Technical Report No. UCB/EECS-2015-155(2015).

[8] Network Functions Virtualisation. http://www.etsi.org/
technologies-clusters/technologies/nfv.
[9] NFV Proofs of Concept.
http://www.etsi.org/
technologies-clusters/technologies/nfv/nfv-poc.

[24] H AN , S., JANG , K., PARK , K., AND M OON , S. PacketShader: a GPU-Accelerated Software Router. In Proc. ACM
SIGCOMM (2010).

[10] REL002: Scalable Architecture for Reliability (work inprogress).
http://docbox.etsi.org/ISG/NFV/Open/
Drafts/.

[25] H ONDA , M., H UICI , F., L ETTIERI , G., AND R IZZO , L.
mSwitch: A Highly-Scalable, Modular Software Switch. In
Proc. SOSR (2015).

[11] pcap-ﬁlter(7) FreeBSD Man Pages, Jan 2008.

[26] H WANG , J., R AMAKRISHNAN , K. K., AND W OOD , T.
NetVM: High Performance and Flexible Networking Using
Virtualization on Commodity Platforms. IEEE Transactions
on Network and Service Management 12, 1 (2015), 34–47.

[12] A NDERSON , C. J., F OSTER , N., G UHA , A., J EANNIN , J.B., KOZEN , D., S CHLESINGER , C., AND WALKER , D.
NetKAT: Semantic Foundations for Networks. In Proc. ACM
POPL (2014).

[27] Intel Data Plane Development Kit. http://dpdk.org.

[13] B ENSON , T., A KELLA , A., AND M ALTZ , D. Network Trafﬁc
Characteristics of Data Centers in the Wild. In Proc. Internet
Measurement Conference (2010).

[28] K ERNIGHAN , B., AND L IN , S. An Efﬁcient Heuristic Procedure for Partitioning Graphs. Bell System Technical Journal
49, 2 (February 1970).

[14] B OSSHART, P., DALY, D., I ZZARD , M., M C K EOWN , N.,
R EXFORD , J., TALAYCO , D., VAHDAT, A., VARGHESE ,G., AND WALKER , D. Programming Protocol-Independent
Packet Processors. CoRR abs/1312.1719 (2013).

[29] K IVITY, A., L AOR , D., C OSTA , G., E NBERG , P., H AR ’E L ,N., M ARTI , D., AND Z OLOTAROV, V. OSv—Optimizing the
Operating System for Virtual Machines. In Proc. USENIX
ATC (2014).

[15] B OSSHART, P., G IBB , G., K IM , H.-S., VARGHESE ,G., M C K EOWN , N., I ZZARD , M., M UJICA , F., AND
H OROWITZ , M. Forwarding Metamorphosis: Fast Programmable Match-Action Processing in Hardware for SDN.
In Proc. ACM SIGCOMM (2013).

[30] KOHLER , E., M ORRIS , R., C HEN , B., JANNOTTI , J., ANDK AASHOEK , M. F. The Click Modular Router. ACM Transactions on Computer Systems 18, 3 (August 2000), 263–297.
[31] KOPONEN , T., A MIDON , K., BALLAND , P., C ASADO , M.,
C HANDA , A., F ULTON , B., G ANICHEV, I., G ROSS , J., I N GRAM , P., JACKSON , E., L AMBETH , A., L ENGLET, R., L I ,S.-H., PADMANABHAN , A., P ETTIT, J., P FAFF , B., R A MANATHAN , R., S HENKER , S., S HIEH , A., S TRIBLING , J.,
T HAKKAR , P., W ENDLANDT, D., Y IP, A., AND Z HANG , R.
Network Virtualization in Multi-tenant Datacenters. In Proc.
USENIX NSDI (2014).

[16] C ASADO , M., F REEDMAN , M. J., P ETTIT, J., L UO , J.,
M C K EOWN , N., AND S HENKER , S. Ethane: Taking Controlof the Enterprise. In Proc. ACM SIGCOMM (2007).
[17] FAYAZBAKHSH , S., C HIANG , L., S EKAR , V., Y U , M., AND
M OGUL , J. FlowTags: Enforcing Network-Wide Policies in
the Face of Dynamic Middlebox Actions. In Proc. USENIXNSDI (2014).

[32] L EE , D., AND B ROWNLEE , N. Passive Measurement of
One-way and Two-way Flow Lifetimes. ACM SIGCOMM
Computer Communications Review 37, 3 (November 2007).

[18] G ANDHI , R., L IU , H. H., H U , Y. C., L U , G., PADHYE , J.,
Y UAN , L., AND Z HANG , M. Duet: Cloud Scale Load Balanc-135

[33] L U , G., G UO , C., L I , Y., Z HOU , Z., Y UAN , T., W U , H.,
X IONG , Y., G AO , R., AND Z HANG , Y. ServerSwitch: A Programmable and High Performance Platform for Data Center
Networks. In Proc. USENIX NSDI (2011).

[41] R AJAGOPALAN , S., W ILLIAMS , D., JAMJOOM , H., AND
WARFIELD , A. Split/Merge: System Support for Elastic Execution in Virtual Middleboxes. In Proc. USENIX NSDI
(2013).

[34] M ARTINS , J., A HMED , M., R AICIU , C., O LTEANU , V.,
H ONDA , M., B IFULCO , R., AND H UICI , F. ClickOS and
the Art of Network Function Virtualization. In Proc. USENIXNSDI (2014).

[42] R IZZO , L. netmap: A Novel Framework for Fast Packet I/O.
In Proc. USENIX ATC (2012).
[43] R IZZO , L., AND L ETTIERI , G. VALE: A Switched Ethernet
for Virtual Machines. In Proc. ACM CoNEXT (2012).

[35] M C C ANNE , S., AND JACOBSON , V. The BSD Packet Filter:
A New Architecture for User-level Packet Capture. In Proc.
USENIX Winter (1993).

[44] S EKAR , V., R ATNASAMY, S., R EITER , M. K., E GI , N., AND
S HI , G. The Middlebox Manifesto: Enabling Innovation in
Middlebox Deployment. In Proc. ACM HotNets (2011).

[36] M C K EOWN , N., A NDERSON , T., BALAKRISHNAN , H.,
PARULKAR , G., P ETERSON , L., R EXFORD , J., S HENKER ,S., AND T URNER , J. OpenFlow: Enabling Innovation in
Campus Networks. ACM SIGCOMM Computer Communications Review 38, 2 (2008), 69–74.

[45] Network Service Header.
https://tools.ietf.org/
html/draft-quinn-nsh-00.
[46] S HERRY, J., G AO , P., BASU , S., PANDA , A., K RISH NAMURTHY, A., M ACCIOCCO , C., M ANESH , M., M AR TINS , J., R ATNASAMY, S., R IZZO , L., AND S HENKER , S.
Rollback-Recovery for Middleboxes. In Proc. ACM SIGCOMM (2015).

[37] M ONSANTO , C., R EICH , J., F OSTER , N., R EXFORD , J.,
AND WALKER , D. Composing Software-Deﬁned Networks.
In Proc. USENIX NSDI (2013).
[38] PATEL , P., BANSAL , D., Y UAN , L., M URTHY, A., G REEN BERG , A., M ALTZ , D. A., K ERN , R., K UMAR , H., Z IKOS ,M., W U , H., K IM , C., AND K ARRI , N. Ananta: Cloud Scale
Load Balancing. In Proc. ACM SIGCOMM (2013).

[47] S HERRY, J., H ASAN , S., S COTT, C., K RISHNAMURTHY, A.,
R ATNASAMY, S., AND S EKAR , V. Making Middleboxes
Someone Else’s Problem: Network Processing as a Cloud
Service. In Proc. ACM SIGCOMM (2012).

[39] P FAFF , B., P ETTIT, J., KOPONEN , T., C ASADO , M., AND
S HENKER , S. Extending Networking into the Virtualization
Layer. In Proc. ACM HotNets (2009).

[48] S HINDE , P., K AUFMANN , A., ROSCOE , T., AND K AESTLE ,S. We Need to Talk About NICs.

[40] Q AZI , Z., T U , C., C HIANG , L., M IAO , R., V YAS , S., AND
Y U , M. SIMPLE-fying Middlebox Policy Enforcement Using
SDN. In Proc. ACM SIGCOMM (2013).

[49] S OUL É , R., BASU , S., K LEINBERG , R., S IRER , E. G., AND
F OSTER , N. Managing the Network with Merlin. In Proc.
ACM HotNets (2013).

136