Network Stacks for the
Virtualized Mobile Packet Core
Ashwin Kumar*, Priyanka Naik+, Sahil Patki*, Pranav Chaudhary*, Mythili Vutukuru*
Indian Institute of Technology, Bombay*

IBM Research, India +Network Function Virtualization (NFV)
Why NFV?
● Cost effective● Easy to scale● Easy to roll out
updates
NFV is great
BUT
Harder to ensure Service
Level Objectives (SLOs) inthe NFV case

Traditional
network
appliance
approach

Network function
virtualization
approach

NFs have to make use ofabstractions provided byalready existing software
Linux kernel network stack
one such existing software!!Why Kernel Bypass?
Connection Locality
CORE YTCB
Tug of war!!

NIC

TX packet
processing

RX packet
processing

CORE XVFS overheads

System calloverheads

UserApplication

User
Application

1 SYSTEMCALLVFS
redirection

KERNEL
Socket

●
●

●
●
●

1 I/O
OPERATION

Memory allocationand deallocationand more ……...

Linux kernel was not built for being on the data path of 40 Gbps and 100 Gbps network
cards.
Difficult to fix the kernel, so bypass it.
The era of Kernel bypass … DPDK and netmap

TAS[3]Novel Network Stacks

User space Networking stack[2]IXUser space networking stack

Data plane Operating System

Core local datastructures….

●Network thread●Borrows the core local
data structure designfrom mTCP
The network stack runson the same thread as
the application as a dataplane OS
Run to completion
instead of batching

Batched I/O toamortize cost ofcontext switch….
●Good breadth of various user
space network stack designs!!
CORE X[1] mTCP: a Highly Scalable User-level TCP Stack for Multicore Systems, NSDI ‘14[2] IX: A Protected Dataplane Operating System for High Throughput and Low Latency, OSDI ‘14[3] TAS: TCP Acceleration as an OS Service, EuroSys ‘19

CORE Y….

Network thread

Application thread●CORE X
Application threadmTCP[1]●Deviates fromco-locating the network
stack and theapplicationPrefetching5G Use Case●
●Access Mobility Function (AMF) → eachsignalling message goes through this NF, andhence current user state maintained here
Authentication Service Function (AUSF) →responsible for user authentication andsecurity setup during user registration

Telco embracing NFV, fast!!
Network functions in the 5G mobile packet core being standardized
as VNFs[1]Mobile packet core connects mobile devices and basestations to external networks

UserAMF

AUSF

SMF

CONTROL
PLANE

OtherBaseStationUPF

Internet[1] 3GPP. 2020. 3GPP Specification Set: 5G. https://www.3gpp.org/dynareport/SpecList.htm?release=Rel-15&tech=4.

Control plane performs
subscriber registration,authentication, and othersignalling messages

We will be focussing on theuser registration procedure
to figure out effective
network stack designs for
AMF and AUSF

Complex 5G NFs●

Novel research network stacks have mostly looked at I/O intensive networkfunctions●But, the 5G use case is different○Control plane consists of CPU intensive network functions which terminate TCP endpoints○AUSF takes more than 300,000 CPU cycles to service some requests as opposed tosome load balancers[1] which take ~500 cycles to service a request.

○●AMF maintains and manages user state

Our goal is to look at effective network stack designs in the CPU
intensive NFs on the control plane[1] Stateless Datacenter Load-balancing with Beamer, NSDI ‘18

Experimental Setup
We focus on the user registration procedure in the 5G packet core

User + Basestation Emulator

AMF

SCTP

We benchmark the user registration
procedure to judge network stackperformance

Option 3: Dummy AUSF withshort connections

XAUSF

Option 2: PersistentConnections

HTTP overTCP

X
Option 1: Short Connections

NetworkStacks
1.
Kernel
2.
mTCP
3.
TAS
4.
IX

Dummy AUSF with short connections●●AUSF just responds with dummy data without doing any
processing
Emulates previously looked at I/O intensive NFs●Novel network stacks
like IX outperform thekernel network stack bya factor of 5

●TAS lags behind onshort connection
workloads.
○

Lot ofPrefetching!!

X axis → Simultaneous users
sending registration requests
Y axis → Throughput

AUSF with short connections●Connection between AMF and AUSF terminated after eachrequest/response cycle●

Throughput of the Linuxkernel stack is only 23%
lower than IX atsaturation.

●

Major overhead in the
kernel stack ofmanaging connectionstate

X axis → Simultaneous users
sending registration requests
Y axis → Throughput

AUSF with persistent connections●Single connection between AMF and AUSF to service eachrequest/response cycle●

TAS does a lot better inthis case.
○Prefetching makes
sense here

●Kernel does better
because of lesser state
management overheads.

X axis → Simultaneous users
sending registration requests
Y axis → Throughput

Key Takeaways●

Run to Completion policy works best for AUSF○●

IX doing a little better than the other network stacks

Linux networking stack not a bad option○

CPU intensive applications don’t benefit enough from the optimizations done in the data pathof the networking stack.

●○Alternate network stacks hard to set up○Protocols like SCTP not supported by these novel network stacks○Stacks other than the kernel use lightweight versions of the TCP/IP networking stackNFs on the 5G control plane just not CPU intensive○

AMF maintains and manages user state. Why is it relevant?

User State Aware Steering
NFs like load balancer maintain state at
the granularity of TCP flows
CORE
W

CORE
X

CORE
Y

Stateforcore W

Stateforcore X

Stateforcore YAMF maintains and manages
subscriber (user) context
CORE
W

COREX

Shared user stateNo Locking
neededCORE
Y

Still locking
requiredbecause userstate
information notencapsulated intransport
protocol header

Application
AMF NetworkFunction

NetworkstackNetworkstack

Packet redirectedbased on TCP/IP
headers

Packet redirectedbased on TCP/IP
headers

NICNICWhat if we had user state aware steering?
Emulated user state aware steering to enable a lock free AMF●

Lock free design
outperforms thelocking design by~67%●

Worth building UE
aware steering
into the network
stackSummary●

Network stack design pivotal in NF’s performance●Best performing network stack outperform the kernel network stack by at leasta factor of 5 on I/O intensive workloads●The gap between the kernel and kernel bypass stacks drop significantly (just23%) when we move to more CPU intensive workloads.

●Some NFs in the 5G packet core would significantly benefit if application stateaware packet steering is incorporated into the network stack.

Thank You! Questions?
You can also reach out to me at ashkumar@cse.iitb.ac.in