Leveraging Programmable Dataplanes for a High
Performance 5G User Plane Function
Abhik Bose∗

Diptyaroop Maji†

Prateek Agarwal

Department of Computer Science& Engineering Indian Institute ofTechnology Bombay, India
abhik@cse.iitb.ac.in

Department of Computer Science& Engineering Indian Institute ofTechnology Bombay, India
diptyaroop@cse.iitb.ac.in

Department of Computer Science& Engineering Indian Institute ofTechnology Bombay, India
prateekag@cse.iitb.ac.in

Nilesh Unhale

Rinku Shah

Mythili Vutukuru‡

Department of Computer Science& Engineering Indian Institute ofTechnology Bombay, India
nileshunhale@cse.iitb.ac.in

Department of Computer Science& Engineering Indian Institute ofTechnology Bombay, India
rinku@cse.iitb.ac.in

Department of Computer Science& Engineering Indian Institute ofTechnology Bombay, India
mythili@cse.iitb.ac.in

ABSTRACT

work presents a preliminary implementation towards a comprehensive programmable dataplane-accelerated 5G UPF.

Emerging 5G applications require a dataplane that has a highforwarding throughput and low processing latency, in addition to low cost and power consumption. To meet theserequirements, the state-of-the-art 5G User Plane Functions(UPFs) are built over high performance packet I/O mechanisms like the Data Plane Development Kit (DPDK), andfurther offload some functionality to programmable dataplane hardware. In this paper, we design and implement
several standards-compliant UPF prototypes, beginning witha software-only DPDK-based UPF, progressing to designs
which offload different functions to programmable hardware.
We evaluate and compare the performance of these designs,
to highlight the costs and benefits of these offloads. Our results show that offload techniques employed in prior work
help improve performance in certain scenarios, but also havetheir limitations. Overcoming these limitations and fully
realizing the power of programmable hardware requires offloading more complex functionality than is done today. Our

CCS CONCEPTS• Networks → In-network processing; Programmablenetworks; Network performance analysis; Mobile networks.

KEYWORDS
5G core, cellular networks, programmable networks, DPDKACM Reference Format:
Abhik Bose, Diptyaroop Maji, Prateek Agarwal, Nilesh Unhale,
Rinku Shah, and Mythili Vutukuru. 2021. Leveraging Programmable
Dataplanes for a High Performance 5G User Plane Function. In 5th
Asia-Pacific Workshop on Networking (APNet 2021) (APNet 2021),
June 24–25, 2021, Shenzhen, China. ACM, New York, NY, USA,
8 pages. https://doi.org/10.1145/3469393.3469400

1

INTRODUCTION

The growth in mobile services and subscribers has resulted
in an exponential increase in mobile signaling and data traffic [3, 41, 47, 59]. The upcoming 5G standards aim to support
applications with diverse traffic characteristics and requirements like enhanced mobile broadband, dense deploymentsof IoT devices, self-driving cars, and AR/VR [34, 54]. These
applications require high throughput, very low processing latencies, and stringent Quality-of-Service (QoS) enforcement.
The mobile packet core, which connects the radio accessnetwork to external networks, comprises of control planecomponents that process signaling messages and a dataplanethat forwards user traffic. The User Plane Function (UPF)
is the main entity in the dataplane of the future 5G mobile
packet core, and has a significant impact on the performancethat users are going to perceive with 5G.

∗ Abhik

Bose and Diptyaroop Maji are student authors with equal
contribution.
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. Copyrights for componentsof this work owned by others than ACM must be honored. Abstracting withcredit is permitted. To copy otherwise, or republish, to post on servers or toredistribute to lists, requires prior specific permission and/or a fee. Request
permissions from permissions@acm.org.
APNet 2021, June 24–25, 2021, Shenzhen, China© 2021 Association for Computing Machinery.
ACM ISBN 978-1-4503-8587-9/21/06. . . $15.00
https://doi.org/10.1145/3469393.3469400
1

APNet 2021, June 24–25, 2021, Shenzhen, China

Abhik Bose, et al.

Most state-of-the-art UPFs are built as multicore-scalable
software packet processing appliances running over commodity servers, and process traffic using a high performancepacket I/O mechanism like the Data Plane Development Kit(DPDK) [20]. However, given the stringent performance requirements of 5G networks [34, 54], and ever increasing
network speeds running into hundreds of Gbps, offloadingsome UPF packet processing to programmable dataplane
hardware can lead to improved performance, along with costand power savings. We establish this benefit of offload in
Table 1, which shows the data forwarding capacity per unit
cost/power for various programmable hardware platforms
as well as a general purpose server CPU core that all run
a UPF. We obtain this table as follows: the UPF throughput
values for the first two columns (single core server and Agilio
CX 2x10GbE) were measured using our standards-compliant
UPF implementations (§3), while we assumed that the otherhardware platforms were capable of handling offloaded UPF
processing at linerate (a reasonable assumption from ourexperience with one platform). The cost and power consumption were obtained from the hardware specifications.
We see from the table that offloading UPF processing to programmable hardware can result in significant cost and powersavings across a wide variety of programmable dataplane
platforms. The idea of accelerating UPF using programmablehardware is not new—prior work has proposed offloading
the logic of steering packets to multiple CPU cores of the
UPF [10, 12, 15, 42], as well as the dataplane forwarding itself [9, 12, 15, 24, 26]. However, to the best of our knowledge,
none of the existing works systematically enumerates all thepossible ways in which UPF functionality can be offloaded
to programmable hardware, nor do they precisely quantify
the costs and benefits of such offloads.
In this work, we begin with an industry-grade software
based UPF built over DPDK, and progressively offload functions to programmable hardware, to come up with several different UPF prototypes (§3). We then measure the throughput
and latency characteristics of such UPFs to analyze the pros
and cons of offloading UPF functionality to programmablehardware (§4). For example, we find that offloading the logic
of steering packets to the multiple CPU cores reduces UPF
processing latency by up to 37% and increases throughput by
45%. However, performing packet steering in hardware is lessflexible than doing so in software, and performs badly during
scenarios involving dynamic scaling and skewed traffic distribution across users. Another interesting observation is that,while forwarding data directly from the programmable dataplane hardware (rather than via userspace) leads up to 24%
lower latency in the dataplane as expected, it significantly
worsens the control plane performance. This is because the
UPF continues to process signaling messages (from the control plane that configures forwarding rules) within userspace

Server[22, 23]

Agilio CX SmartNICs [19, 55]
[56]

[57]

[58]

Tofino switch[21]

Mpps per USD

0.03

0.33

0.28

0.24

0.4

Mpps per Watt

0.14

2.96

2.96

2.96

8.52

Table 1: Performance per unit cost and power.

Figure 1: 5G Architecture.
itself, and in this split architecture, the communication between the UPF software and the programmable hardware
becomes the bottleneck.
In this paper, we posit that truly leveraging the performance of programmable hardware to accelerate the 5G dataplane requires the UPF to process and respond to signaling messages from the hardware itself, in order not be limited by the hardware configuring capacity of userspace software. This is challenging to do for several reasons, including
the variable-sized message formats of UPF signaling messages. This paper provides a preliminary implementationthat solves some of these challenges, and provides initial results that show the promise of offloading these complex UPF
functions to programmable hardware. Our work paves theway towards a high-performance 5G UPF that fully leverages
the power of programmable dataplanes.

2

BACKGROUND & RELATED WORK

5G architecture. Figure 1 shows a high-level overview of
the 5G architecture [5]. A 5G network consists of the wireless radio access network (RAN), which includes the User
Equipment (UE) and the Base Station/g-NodeB (gNB), andthe wired packet core network. In the control plane of the
5G core, the Access and Mobility Function (AMF) deals withregistration and mobility management, and the Session Management Function (SMF) manages the UE’s data sessions.
The data plane comprises of one or more User Plane Functions (UPFs) that forward user data through the packet core.
The control and data plane components communicate using
Packet Forwarding Control Protocol (PFCP) messages thatare exchanged between the SMF and UPF over UDP [4]. In
the dataplane, a UE’s IP datagram packets are encapsulated
in GPRS Tunnelling Protocol (GTP) headers when transiting through the packet core; tunnelling aids easy mobility
among other things. The GTP-encapsulated IP datagrams aretransmitted over a UDP link between the gNB and the UPF.
2

Leveraging Programmable Dataplanes for a High Performance 5G UPF

APNet 2021, June 24–25, 2021, Shenzhen, China

The UPF is responsible for encapsulating downlink packets
entering the core with GTP headers and correspondingly decapsulating uplink packets leaving the core. The UPF is also
responsible for usage reporting and charging, and enforcing
Quality-of-service (QoS), e.g., via rate limiting.
This paper deals with UPF implementation, so we describe
the internal state and data structures at the UPF in moredetail. A UE sets up one or more “sessions” to forward data
through the core, and each data session will cause the SMF
to install one or more Packet Detection Rules (PDRs) at the
UPF via PFCP messages. A PDR helps associate an incoming
data packet to a session based on its GTP/IP packet headers.
A PDR has several types of actions associated with it, that
instruct the UPF on how to handle the packet. For example, the Forward Action Rules (FARs) specify the forwarding
behavior of the packet (e.g., GTP tunnel identifiers for encap), and the QoS Enforcement Rules (QERs) specify the QoS
processing to be performed. An example of a field in the
QER is the Aggregate Maximum Bit-Rate (AMBR) of the session. On receiving a PFCP message from the SMF, the UPF
creates/updates the required rules, and sends an acknowledgement back to the SMF. On receiving a data packet, theUPF first identifies the matching PDR. For packets belonging
to a valid session, the UPF executes forwarding and QoS
behavior specified by the FARs and QERs linked to that PDR,
in addition to updating usage and charging counters. Packets
belonging to “oversubscribed” sessions that exceed the rate
limit specified in the QoS rules are buffered for an appropriate duration, and scheduled for transmission suitably.
Programmable dataplanes. Programmable dataplanes allow networking hardware to be easily programmed to perform complex functions, via code written in a high-level language like P4 [25]. Packet processing pipeline specifications
written in P4 can be compiled to a variety of programmabledataplanes, e.g., programmable hardware ASICs [1, 2, 46, 48],
NPUs [8, 56], and FPGAs [7, 11]. The programmable hardware platforms have several limitations put in place, in orderto ensure linerate processing. They have limited expressiveness in terms of the supported instruction set and programming constructs, and lack dynamic data-structures. The packets cannot stall during the switch pipeline processing—they
have to be either forwarded or dropped. The amount of onboard memory on such hardware is limited (∼few tens ofMBs). Despite these limitations, researchers have observed
substantial performance benefits by offloading applications
to programmable hardware, via creative solutions that address the hardware limitations [27, 29–33, 36, 37, 39, 40, 43–
45, 51, 53, 61].
State-of-the-art UPFs. Most production grade UPFs arebuilt over kernel-bypass techniques like DPDK to achieve
high dataplane throughput in software. Metaswitch [10] uses

Figure 2: 5G User Plane Function designs.
a specialized processing engine (CNAP) in the software itself to achieve high throughput. Some UPFs also use programmable hardware or specialized processing engines tooffload some part of the UPF processing to hardware. Few
proposals [12, 15, 24, 26] offload the GTP encap/decap based
forwarding to hardware, while some [42] offload packet steering to cores via deep packet inspection (DPI) of the inner
IP header. Kaloom [9] offloads a subset of QoS processing(bit rate policing) along with GTP processing to the programmable hardware. Our work explores more offload based
designs than those considered in prior work, and systematically analyzes the costs and benefits of the offloads. TurboEPC [52] offloads the subset of 4G core signaling messages
to the programmable hardware, while our work explores thesignaling message offload problem in the context of 5G which
has a different architecture.

3

DESIGN & IMPLEMENTATIONThis section describes the various UPF prototypes comparedin this paper, beginning with a software-based UPF, andprogressively moving towards UPFs that offload more functionality to programmable hardware, as shown in Figure 2.

3.1

DPDK-based software UPF

We begin with describing our purely software DPDK-based
UPF (Figure 2a) that is representative of the most common
UPF design used in production networks today. Our implementation is based on a fully standards-compliant UPF obtained from [16, 17]. Our UPF supports GTP-based forwarding and AMBR-based QoS enforcement (using a variant of
the algorithm in [50]), among other features. Our implementation spans 6.5K lines of code. Our UPF has a pipelinebased design, with multiple master and worker threads, eachpinned to separate CPU cores. The master threads receive
packets from the NIC via the polling-based DPDK APIs, anddistribute them to the worker threads for further processing.
PFCP packets and dataplane packets are processed by separate worker cores. The inter-core communication between
3

APNet 2021, June 24–25, 2021, Shenzhen, China

Abhik Bose, et al.

the master and workers is done via the lockless shared rings
provided by DPDK, for efficiency and high performance.
The worker threads continuously poll the shared rings forreceived packets, process them, and transmit the output.
When steering packets to worker cores, we would liketo ensure that the traffic of a UE is processed by the same
worker, in order to avoid splitting the state of a particular
UE (forwarding rules, buffered packets, and so on) acrossmultiple workers. This steering is achieved by using the hashover the TCP/IP headers of the“inner” IP datagram (the datagram originated by the UE, which has been encapsulated
within GTP for transit through the core) to partition traffic
to worker cores. Note that we cannot simply use a hash overthe “outer” UDP/IP header fields because dataplane traffic of
all UEs between a gNB-UPF pair arrives on the same UDPlink, so the outer IP header fields cannot be used to differentiate UEs. Most modern NICs have the capability to distribute
packets to multiple CPU cores via RSS [60]; however, theset of header fields used for RSS is restricted to the outer
IP header fields. Therefore, a purely software-based DPDK
design cannot rely on the NIC to perform packet steering
based on the inner IP header fields, and must perform thissteering in software. (RSS based on outer header fields isstill useful to distribute traffic to multiple master cores for
performance scaling.) Packet steering in software also allows
the UPF to efficiently rebalance load across worker cores incase some worker cores are more overloaded than others,
and to dynamically scale to more worker cores quickly on
demand. In our design, the master cores periodically monitor the queue lengths of the lockless rings shared with theworker cores, and reassigns UEs across workers if it findsthat a worker core is overloaded (as inferred from a persistently high queue length), and another underloaded. If
all worker cores are overloaded, the master can spawn aworker on new CPU core (when available). We have only
implemented a simple load balancing algorithm, but more
complex algorithms that assign special classes of UEs (e.g.,high priority UEs) to specific worker cores are also possible.

3.2

from their corresponding queues. The worker threads thenprocess the received packets in a run-to-completion model.
The control plane traffic is redirected to (and processed on)
dedicated cores as before. This design represents the simplest
possible offload that can be done to programmable hardware,and is used by state-of-the-art UPFs [10, 12, 15, 42].
Offloading packet steering to hardware provides higher
throughput and lower latency, due to minimal inter-corecommunication in software and faster hash computation in
hardware. However, this design is also less flexible as we havelesser control on assigning UEs to cores. For example, theDPDK i40e poll mode driver [13] does not support dynamic
load balancing by remapping queues to cores based on load.
Further, dynamic scaling is also complicated by the fact that
one needs to stop and reconfigure the port in order to change
the number of receive queues.

3.3

Dataplane offload

Our next design (Figure 2c) offloads the complete dataplane
processing of the UPF to a programmable hardware-based
NIC or switch that can intercept packets destined to the software UPF. Without loss of generality, we assume that the
dataplane is offloaded to a programmable NIC; our current
implementation uses the Agilio CX 2x10GbE smartNIC [56].
In this design, the on-NIC programmable parser extracts 5G
header fields from incoming packets. The control plane PFCP
messages are directed to the software UPF and processedby the control plane worker thread. After processing the
PFCP messages, the worker thread communicates with the
NIC firmware to install/update session forwarding rules in
the match-action tables. The worker-NIC communication
happens via our custom C++ library that invokes the NIC’s
Thrift API [18] to install hardware rules. The incoming dataplane packets are matched against these rules for suitableforwarding. Note that the number of concurrent active user
sessions that can be supported by such a system is inherently limited by the amount of memory available to store
forwarding rules in the hardware; our current prototype canstore the forwarding state of only 10K users but higher-end
programmable hardware [46] can do more.
Prior work has also proposed similar designs that offload
GTP-based forwarding to programmable hardware [9, 15, 24,26], but does not provide much detail on how rate limiting
for QoS is handled in the offloaded design. In our design,
all traffic that is within the QoS-prescribed rate limit is forwarded directly from hardware, and traffic that cannot betransmitted immediately is sent to the userspace for buffering. Our implementation presently supports enforcement ofsession-wide AMBR (aggregate maximum bit rate), which
requires us to rate limit the aggregate traffic of a user’s data
session (across all flows) to a maximum value and queue
up the traffic that exceeds this limit. One key challenge we

Packet steering offload

In our next design (Figure 2b), the steering of UE traffic tocores happens not in software but within the NIC itself. This
design relies on advanced NICs that allow RSS based on inner
IP headers. For example, Intel NICs support the Dynamic
Device Personalization [49] feature on 40Gbps+ NICs, whichenables parsing of the GTP header and inner TCP/IP header
fields for hash computation. Because the input to the hashfunction now contains the UE’s IP address and GTP tunnel
identifier, the packets of a UE are redirected to the same
receive queue and CPU core via RSS. The multiple worker
threads of the DPDK-based software UPF are assigned dedicated hardware receive queues and directly receive traffic4Leveraging Programmable Dataplanes for a High Performance 5G UPF

APNet 2021, June 24–25, 2021, Shenzhen, China

had to overcome in this implementation was to identify the
packets that exceeded a session’s rate limit. Our initial implementation started with using P4 register arrays (stateful
memory available for packet processing in programmablehardware) to track incoming rates of various sessions. Because these hardware registers can be accessed concurrently
by multiple packets across different stages of the packet processing pipeline, we must use mutual exclusion in accessing
and updating these registers. Unfortunately, we found that
updating P4 registers under mutual exclusion significantly
impacted the performance of our UPF (dropping to 350 Mbps
from the linerate of 10Gbps). To overcome this limitation,
we moved to using the P4 meter primitive [28, 38] to identify
packets exceeding the rate limit. P4 meter is implemented asa device specific extern, that sets the “color” of a packet in its
metadata to different values based on whether the session’srate exceeds a pre-defined rate limit. While there was noslowdown due to P4 meters, we found that the hardware dataplane can no longer compute the time that a packet needs
to be buffered for correctly, because the meter interface does
not expose any rate calculations beyond the color. Therefore, once a session exceeds its rate limit, we forward all
subsequent packets of that session to userspace, and let thesoftware handle the session’s dataplane processing.

3.4

with fixed structure that fits within the hardware memory,whereas in practice, the PFCP header comprises of a variable
length component that consists of recursive PDR, FAR, andQER (§2) headers. We defer the comprehensive handling ofcomplex PFCP messages to future work.

4

EVALUATION

We now evaluate our various UPF designs and quantify theperformance gains of offloading UPF functionality.
Experiment Setup. All our experiments run on three servers
with Intel Xeon processors (2.2Ghz, 24 cores) and 128GB
RAM. The first server runs a RAN emulator/load generator
that generates emulated traffic to the UPF, comprising of control plane PFCP messages and uplink/downlink dataplane
traffic from a large number of UEs. The RAN emulator runs
over DPDK to generate traffic at a high rate and we ensurethat it can generate enough load to saturate the control/data
plane of the UPF in all experiments. The second server in our
setup runs one of the versions of our UPF. The third server
runs a sink application that generates downlink traffic andconsumes uplink traffic from the RAN emulator. Our RAN
emulator and sink span 12K lines of code. We use two sets
of NICs in our experiments. For experiments which evaluate the impact of offloading packet steering, we connect theservers using Intel XL710 i40e 40Gbps NICs that are capableof offloading packet steering. For the rest of the experiments,
we use Agilio CX 2x10GbE programmable NICs [56].
Parameters and metrics. We generate different traffic mixes
for dataplane experiments by varying the packet sizes, across
64B, 1400B and a typical traffic mix (“IMIX”) found in realuser traffic [14]. We use an uplink to downlink traffic ratio of 1:2 [6, 35]. Control plane traffic consists of sets ofsession creation, modification, and deletion PFCP messages
processed one after the other—we refer to this set as a control
plane “procedure”. All results reported are for an experimentconducted for 300 seconds, unless mentioned otherwise. We
show error-bars that denote the maximum and minimum values where applicable. The performance metrics measured are
the dataplane throughput (bps or pps), control plane throughput (procedures/sec), and average end-to-end response latency (RTT, where the response is mirrored by sink for data
packets, and is generated by the control plane processing forsignaling packets), as measured at UPF saturation. For experiments evaluating the effect of offloading packet steering,
we send data from 65K concurrent users. For experiments
involving dataplane and control plane offload to smartNICs,
we emulate 1K concurrent users.
Packet Steering Offload. We begin by evaluating the impact of packet steering to programmable NICs, and comparea pure software-based UPF (§3.1) with a UPF that offloads
packet steering to programmable hardware (§3.2). Figure 3
shows the throughput and latency of both the designs whenControl plane offload

In the dataplane offload design (Figure 2c), signaling messages that configure the dataplane are still handled in userspace.
As a result, a workload with a large number of signaling
messages may slow down the hardware dataplane, due to abottleneck at the software that configures hardware rules. To
overcome this limitation, we implement a UPF design which
offloads signaling message processing also to programmablehardware (Figure 2d). In this design, we move away fromstoring forwarding/QoS rules of a session in match-action
tables (which need to be updated via the controller runningin userspace), and store them instead in P4 register arrays.
The register data structure supports update operations toregister array state within the dataplane pipeline at linerate,
but cannot perform key-based lookup. When a control plane
PFCP message for a particular session arrives, we map the64-bit session identifier to a 24-bit index into the register
array. We can then access or manipulate the corresponding
session state directly from within the dataplane, without
requiring the intervention of the userspace controller. Since
our index calculation is (currently) not collision-free, westore additional state in the register array to validate if weare accessing the correct session. In case of a collision or
session mismatch, the control plane message is forwarded tothe userspace for processing. We also make a few simplifying
assumptions when parsing PFCP messages in our current
implementation. For example, we assume a PFCP header
5

APNet 2021, June 24–25, 2021, Shenzhen, China

Abhik Bose, et al.

176

294

DPOffload

130

140

222

4000
3000
2000
1000
0
59.8

Table 2: SoftUPF vs. DPOffload: Dataplane latency.

60.8

138

5000

60.6

SoftUPF

Incoming Load
SoftUPF
SteerOffload

6000

60.4

1400B

60.2

IMIX

60

64B

Throughput (Mbps)

Latency (in µs ) with packet size

UPF design

Time (in secs)

58.5

300
250
200
150
100

Time (in secs)

Figure 5: SoftUPF vs. SteerOffload: Heavy hitters.
Control Plane Offload. Next, we compare the control plane
performance of three designs: the pure software UPF (§3.1),
the design that offloads only the dataplane forwarding toprogrammable hardware but processes control plane packets
in userspace (§3.3), and our preliminary implementation ofa UPF that offloads even control plane processing to programmable hardware (§3.4). Table 3 shows the average rateof processing signaling messages, and the processing latency,
for all three designs. We observe that only offloading dataplane processing to hardware has a significant adverse impact on control plane performance, because the performanceis limited by the overheads of control plane message processing in software and the subsequent communication with NIC
firmware. Therefore, the design that offloads only dataplane
processing has 86% lower control plane throughput and 15×
higher control plane latency, as compared to the pure software UPF. However, the UPF that offloads control plane processing has 402× higher throughput as compared to the pure
software design, and 3K× higher throughput as compared tothe dataplane offload design. It also has 77% latency reduction compared to the pure software design, and 98% latency
reduction compared to the dataplane offload design. This
result points to the conclusion that offloading GTP-based
dataplane processing alone to programmable hardware is
likely to cause additional performance concerns for signaling message processing, especially for future 5G networks
that expect to have frequent signaling with IoT devices. A
comprehensive programmable dataplane accelerated UPF must
offload signaling message processing to hardware as well, in
addition to offloading dataplane processing.

50
0
64B

IMIX
Payload size

SoftUPF
SteerOffload

65

SoftUPF latency
SteerOffload latency

Latency (µs)

Throughput (Mpps)

SoftUPF Throughput
SteerOffload Throughput
8
7
6
5
4
3
2
1
0

Traffic burst

64.5

Table 3: Control plane performance.

800
700
600
500
400
300
200
100
0

64

26

63

1646

63.5

113

62

Latency ( µs )Figure 4: SoftUPF vs. SteerOffload: Dynamic scaling.

62.5

2.05M

61.5

666

61

5.1K

60.5

Throughput (messages/sec)

60

CPOffload

59.5

DPOffload

59

SoftUPF

Latency (µs)

Performance metric

1400B

Figure 3: SoftUPF vs. SteerOffload: Performance.
we dedicate 2 cores for dataplane processing—one for uplinktraffic and another for downlink traffic. As expected, we findthat offloading packet steering leads to 45% higher throughput and up to 37% lower latency (except when both designs
saturate linerate at maximum packet size), due to lesser overhead of intercore communication and packet steering in theoffloaded design. However, the performance gains due tothis offload also come with a loss in flexibility. We consider
a scenario where a UPF has to dynamically scale the numberof cores it is running on due to an increase in incoming traffic. Figure 4 shows the UPF throughput during scaleup, andwe see the adverse impact of having to restart the port andreconfigure hardware queues in the offload design. Next, we
configure our RAN emulator in such a way that 20% of theUEs processed by a single worker core generate a sudden
burst of traffic, and Figure 5 shows how the offload design
suffers high latency due to its inability to balance load acrosscores in the presence of such “heavy-hitter” UEs. In summary, we find that offloading packet steering to NICs leads tosignificant performance gains, but comes at a loss of flexibility
that can hurt operators that expect to perform dynamic scaling
frequently or see skewed traffic across users.
Dataplane Offload. Next, we evaluate the benefits of offloading dataplane processing to programmable hardware
by comparing the performance of the pure software UPF(§3.1) with one that offloads dataplane processing to a programmable NIC (§3.3). We find that both designs saturate the
10Gbps linerate in our setup, and Table 2 shows end-to-end
latency measurements for the various traffic scenarios. As
expected, dataplane offload improves latency by up to 24%.

5

CONCLUSION AND FUTURE WORK

Our work evaluates several designs of the 5G user plane function (UPF) that leverage programmable dataplane hardware
in different ways to improve performance. Our evaluation
6

Leveraging Programmable Dataplanes for a High Performance 5G UPF

APNet 2021, June 24–25, 2021, Shenzhen, China

highlights the costs and benefits of each offload strategy, andprovides lessons for future research in this area. We find that
UPF offload designs proposed in prior work improve performance, but hurt flexibility in dynamic workload scenarios,and cannot process signaling traffic efficiently. To fully realize the power of programmable hardware offload for UPFs,
we need to overcome the technical challenge of being able
to parse complex 5G signaling messages in hardware. Our
work is a first step towards using programmable dataplanes
to realize the vision of 5G.

[21] Edgecore Networks AS9516-32D (Tofino-2).
(2021).
https://stordirect.com/shop/switches/400g-switches/edgecore-networksas9516-32d/[22] Intel Xeon E5-2670 V3 Dodeca-core (12 Core) 2.30 Ghz Processor.
(2021). https://www.amazon.com/Intel-Xeon-E5-2670-Dodeca-coreProcessor/dp/B00NFA7ILQ#HLCXComparisonWidgetf eatured iv[23] Intel XL710-BM2 Dual-Port 40G QSFP+ PCIe 3.0 x8, Ethernet Network
Interface Card. (2021). https://www.fs.com/products/75604.html[24] Ashkan Aghdai et al. 2018. Transparent Edge Gateway for Mobile
Networks. In IEEE 26th International Conference on Network Protocols(ICNP).
[25] Pat Bosshart, Dan Daly, Glen Gibb, Martin Izzard, Nick McKeown,
Jennifer Rexford, Cole Schlesinger, Dan Talayco, Amin Vahdat, George
Varghese, and David Walker. 2014. P4: Programming Protocolindependent Packet Processors. SIGCOMM Computer Communication
Review 44 (2014).
[26] Carmelo Cascone and Uyen Chau. 2018. Offloading VNFs to programmable switches using P4. In ONS North America.
[27] Eyal Cidon, Sean Choi, Sachin Katti, and Nick McKeown. 2017. AppSwitch: Application-layer Load Balancing Within a Software Switch.
In Proceedings of the Asia-Pacific Workshop on Networking (APNet).
[28] Edgar Costa. P4 Meter Application. (2020). https://github.com/nsgethz/p4-learning/tree/master/examples/meter[29] Huynh Tu Dang et al. 2018. Consensus for Non-Volatile Main Memory.
In IEEE 26th International Conference on Network Protocols (ICNP).
[30] Huynh Tu Dang, Daniele Sciascia, Marco Canini, Fernando Pedone,
and Robert Soulé. 2015. NetPaxos: Consensus at Network Speed. In
Proceedings of the the Symposium on SDN Research (SOSR).
[31] Hans Giesen, Lei Shi, John Sonchack, Anirudh Chelluri, NishanthPrabhu, Nik Sultana, Latha Kant, Anthony J McAuley, Alexander
Poylisher, André DeHon, et al. 2018. In-network computing to therescue of faulty links. In Proceedings of the 2018 Morning Workshop on
In-Network Computing (NetCompute).
[32] Rob Harrison, Qizhe Cai, Arpit Gupta, and Jennifer Rexford. 2018.
Network-wide heavy hitter detection with commodity switches. In
Proceedings of the Symposium on SDN Research (SOSR).
[33] Rob Harrison, Shir Landau Feibish, Arpit Gupta, Ross Teixeira, S
Muthukrishnan, and Jennifer Rexford. 2020. Carpe Elephants: Seize
the Global Heavy Hitters. In Proceedings of the Workshop on Secure
Programmable Network Infrastructure (SPIN).
[34] R. E. Hattachi.
Next Generation Mobile Networks, NGMN.
(2015).
https://www.ngmn.org/wp-content/uploads/
NGMN5 GW hiteP aperV 10 .pdf[35] Harri Holma and Antti Toskala. LTE Advanced: 3GPP Solution for
IMT-Advanced. (2012). https://www.oreilly.com/library/view/lteadvanced-3gpp/9781118399422/c01anchor-3.html[36] Thomas Holterbach, Edgar Costa Molero, Maria Apostolaki, Alberto
Dainotti, Stefano Vissicchio, and Laurent Vanbever. 2019. Blink: Fast
Connectivity Recovery Entirely in the Data Plane. In 16th USENIX
Symposium on Networked Systems Design and Implementation (NSDI).
[37] Xin Jin, Xiaozhou Li, Haoyu Zhang, Robert Soulé, Jeongkeun Lee, Nate
Foster, Changhoon Kim, and Ion Stoica. 2017. NetCache: Balancing
Key-Value Stores with Fast In-Network Caching. In Proceedings of the
Symposium on Operating Systems Principles (SOSP).
[38] Jaco Joubert. P4 Loadbalancer and Metering. (2017). https://github.com/open-nfpsw/p4b asicl bm eteringn ic[39] Naga Katta, Mukesh Hira, Changhoon Kim, Anirudh Sivaraman, andJennifer Rexford. 2016. HULA: Scalable Load Balancing Using Programmable Data Planes. In Proceedings of the the Symposium on SDN
Research (SOSR).

ACKNOWLEDGMENTS
We thank the 5G testbed project, funded by the Departmentof Telecommunications, Govt. of India, for access to 5G UPF.

REFERENCES[1] Cisco highlights next big switch.
(2013).
https://www.biztechafrica.com/article/cisco-announces-next-bigswitch/5448/[2] Cavium Xpliant ethernet switch product line. (2015). https://people.ucsc.edu/~warner/Bufs/Xpliant-cavium.pdf[3] Ericsson Mobility Report. (2016). http://mb.cision.com/Main/15448/
2245189/661253.pdf[4] 3GPP Ref #: 29.244. System architecture for the 5G System (5GS). (2017).
https://www.3gpp.org/ftp/Specs/archive/29s eries/29.244/[5] 3GPP Ref #:23.501. System architecture for the 5G System (5GS). (2017).
https://www.3gpp.org/ftp/Specs/archive/23s eries/23.501/[6] Minimum requirements related to technical performance for IMT-2020
radio interface(s). (2017). https://www.itu.int/dmsp ub/itu-r/opb/rep/
R-REP-M.2410-2017-PDF-E.pdf[7] Altera. (2019). https://www.mouser.in/manufacturer/altera/[8] EZchip. (2019). https://www.radisys.com/partners/ez-chip(2019).
https:[9] The Kaloom 5G User Plane Function (UPF).
//www.mbuzzeurope.com/wp-content/uploads/2020/02/ProductBrief-Kaloom-5G-UPF-v1.0.pdf[10] Lighting Up the 5G Core with a High-Speed User Plane on Intel
Architecture. (2019). https://builders.intel.com/docs/networkbuilders/lighting-up-the-5g-core-with-a-high-speed-user-plane-on-intelarchitecture.pdf[11] Xilinx. (2019). https://www.xilinx.com/[12] 5G User Plane Function (UPF) - Performance with ASTRI.
(2020). https://networkbuilders.intel.com/solutionslibrary/5g-userplane-function-upf-performance-with-astri-solution-brief[13] I40E Poll Mode Driver. (2020). https://doc.dpdk.org/guides/nics/
i40e.html[14] Internet Mix (IMIX) Traffic. (2020). https://en.wikipedia.org/wiki/
InternetM ix[15] Optimizing UPF performance using SmartNIC offload.
(2020).
https://mavenir.com/wp-content/uploads/2020/11/
MavenirU PFS olutionB rief .pdf
[16] 5G testbed at IIT Bombay. (2021). https://www.cse.iitb.ac.in/~5gtestbed/[17] 5G testbed, DoT, Govt. of India. (2021). https://5gtestbed.in/[18] Apache Thrift - Home. (2021). https://thrift.apache.org/[19] Cost of Agilio CX SmartNICs. (2021). https://colfaxdirect.com/store/
pc/showsearchresults.asp?IDBrand=38&iPageSize=50[20] DPDK Overview. (2021). https://doc.dpdk.org/guides/progg uide/
overview.html
7

APNet 2021, June 24–25, 2021, Shenzhen, China

Abhik Bose, et al.

[40] Changhoon Kim, Anirudh Sivaraman, Naga Katta, Antonin Bas, Advait Dixit, and Lawrence J Wobker. 2015. In-band network telemetry
via programmable dataplanes. In ACM Special Interest Group on Data
Communication (SIGCOMM).
[41] Dr. Kim. 5G stats. (2017). https://techneconomyblog.com/tag/economics/[42] DongJin Lee, JongHan Park, Chetan Hiremath, John Mangan,
and Michael Lynch. Towards achieving high performance in
5G mobile packet core’s user plane function. (2018). https://builders.intel.com/docs/networkbuilders/towards-achieving-highperformance-in-5g-mobile-packet-cores-user-plane-function.pdf[43] Bojie Li, Zhenyuan Ruan, Wencong Xiao, Yuanwei Lu, Yongqiang
Xiong, Andrew Putnam, Enhong Chen, and Lintao Zhang. 2017. KVDirect: High-Performance In-Memory Key-Value Store with Programmable NIC. In Proceedings of the Symposium on Operating Systems
Principles (SOSP).
[44] Rui Miao, Hongyi Zeng, Changhoon Kim, Jeongkeun Lee, and Minlan
Yu. 2017. SilkRoad: Making Stateful Layer-4 Load Balancing Fast andCheap Using Switching ASICs. In Proceedings of the the ACM Special
Interest Group on Data Communication (SIGCOMM).
[45] YoungGyoun Moon, SeungEon Lee, Muhammad Asim Jamshed, andKyoungSoo Park. 2020. AccelTCP: Accelerating Network Applications
with Stateful TCP Offloading. In 17th USENIX Symposium on Networked
Systems Design and Implementation (NSDI).
[46] Barefoot networks. NoviWare 400.5 for Barefoot Tofino chipset.
(2018). https://noviflow.com/wp-content/uploads/NoviWare-TofinoDatasheet.pdf[47] David Nowoswiat. Managing LTE Core Network Signaling Traffic. (2013). https://www.nokia.com/eni nt/blog/managing-lte-corenetwork-signaling-traffic[48] Recep Ozdag. Intel Ethernet Switch FM6000 Series - Software Defined
Networking. (2019). https://people.ucsc.edu/~warner/Bufs/ethernetswitch-fm6000-sdn-paper.pdf[49] Brian Johnson Robin Giller, Andrey Chilikin. Intel® Ethernet
Controller 700 Series GTPv1 - Dynamic Device Personalization. (2018).
https://builders.intel.com/docs/networkbuilders/intel-ethernetcontroller-700-series-gtpv1-dynamic-device-personalization.pdf/[50] Ahmed Saeed, Nandita Dukkipati, Vytautas Valancius, Vinh The Lam,
Carlo Contavalli, and Amin Vahdat. 2017. Carousel: Scalable traffic
shaping at end hosts. In Proceedings of the Conference of the ACM
Special Interest Group on Data Communication (SIGCOMM).
[51] Amedeo Sapio, Marco Canini, Chen-Yu Ho, Jacob Nelson, Panos Kalnis,
Changhoon Kim, Arvind Krishnamurthy, Masoud Moshref, Dan R. K.
Ports, and Peter Richtárik. Scaling Distributed Machine Learning with
In-Network Aggregation. (2020). arXiv:cs.DC/1903.06701[52] Rinku Shah, Vikas Kumar, Mythili Vutukuru, and Purushottam Kulkarni. 2020. TurboEPC: Leveraging Dataplane Programmability to Accelerate the Mobile Packet Core. In Proceedings of the Symposium on SDN
Research (SOSR).
[53] Vibhaalakshmi Sivaraman, Srinivas Narayana, Ori Rottenstreich, S.
Muthukrishnan, and Jennifer Rexford. 2017. Heavy-Hitter Detection
Entirely in the Data Plane. In Proceedings of the the Symposium on SDN
Research (SOSR).
[54] Gábor Soós, Ferenc Nándor Janky, and Pál Varga. 2019. Distinguishing
5G IoT Use-Cases through Analyzing Signaling Traffic Characteristics.
In 2019 42nd International Conference on Telecommunications and Signal
Processing (TSP).
[55] Netronome systems. Agilio CX SmartNICs. (2018). https://www.netronome.com/products/agilio-cx/[56] Netronome systems. Agilio CX 2x10GbE SmartNIC. (2020). https://www.netronome.com/media/documents/PBA gilioC X2 x10GbE-720.pdf[57] Netronome systems.
Agilio CX 2x25GbE SmartNIC.
(2020).
https://colfaxdirect.com/store/pc/viewPrd.asp?idproduct=
3144&idcategory=0[58] Netronome systems. Agilio CX 2x40GbE SmartNIC. (2020). https://colfaxdirect.com/store/pc/viewPrd.asp?idproduct=2871[59] Sami Tabbane.
Core network and transmission dimensioning.
(2016).
https://www.itu.int/en/ITU-D/RegionalPresence/AsiaPacific/SiteAssets/Pages/Events/2016/Aug-WBBIran/Wirelessbroadband/core%20network%20dimensioning.pdf[60] Amy Viviano, David Coulter, Nick Schonning, Duncan
MacMichael, and Bill Latimer.
Receive Side Scaling(RSS).
(2017).
https://docs.microsoft.com/en-us/windowshardware/drivers/network/introduction-to-receive-side-scaling[61] Zhaoqi Xiong and Noa Zilberman. 2019. Do Switches Dream of Machine Learning? Toward In-Network Classification. In Proceedings ofthe 18th ACM Workshop on Hot Topics in Networks (HotNets).

8