⾼速ソフトウェアルータ(Kamuee)による
GTP-U/SRv6のStateless変換の遅延評価
MPLS JAPAN 2020
Chunghan Lee1), Naoyuki Mori2) Yasuhiro Ohara3)
Tetsuya Murakami4), Shogo Asaba5), Satoru Matsushima6)
Nov.9th.2020
Toyota Motor Corp.1), Intel K.K. 2), NTT Communications Corporation3)
Arrcus, Inc.4), Fujitsu Laboratories Limited5), SoftBank Corp.4)

Introduction• Segment routing IPv6 (SRv6)• Encode the path in each packet

1

SRv6 based on source routing has many advantages: statelesstraffic steering, state reduction, network
programming and so onWhy the translation is required? – (1)• The data plane of mobile
network has multiple stackingheaders• Stacking multiple small IDs tofulfill requirements of reliability,VPNs, etc.,• After network failure,
the efficient network pathwould not be selected
due to the state of stacking
headers

2IPv6 as user
PDN protocol
GTPv1U as
mobile userplane protocol
L3 VPN for
mobile coreand back-haul
L2 VPN for
virtual networks
LSP for high quality
and reliability

Why the translation is required? – (2)• Consolidate all layer headers into one single IPv6 layer

512bits
Segment-ID [1]* (128 bits)572bits

Segment-ID [0]* (128 bits)
IPv6 SA (128 bits)
IPv6 DA (128 bits)

3*Exist in Segment Routing Extension Header (SRH)GTP-U/SRv6 translation for mobile network• Mobile devices are only communicated with UPF (GTP-U) on LTE network• Using Segment routing gateway (SRGW), the stacking headers can beembedded to SRH and the efficient path can be selected<Current LTE >4<LTE + SRGW>What is GTP-U and SRv6 StatelessTranslation?
• All identifiers of a GTP-U tunnel (IPv4 info. and TEID) can beembedded as an argument of a Segment ID (SID)5Current state of translation method• The method (SRv6 for Mobile UserPlane) has been proposed inIETF• https://tools.ietf.org/html/draft-ietf-dmm-srv6-mobile-uplane-08

6However, there are no suitable measurement platform andperformance evaluation results between GTP-U and SRv6

Our previous work1)…
• We implemented the translation functions on the H/W P4
switch and measured the latency on the H/W P4 switch
However, the SRv6 Mobile UserPlane with the P4 switch
would be hard to deploy Edge/MEC or local 5G network< SRv6 functions on P4 switch >7< Normalized latency at SR functions >1) "Performance Evaluation of GTP-U and SRv6 Stateless Translation", 2nd Workshop on Segment Routing and Service Function Chaining (SFC 2019)

Research goal
• Evaluate the quantitative performance of SRv6 Mobile User
Plane on DPDK-based software router (Kamuee)• Observe the latency characteristics on the software router(Kamuee)

8How to measure the precise latency
on the software router?
1. How can we preciously measure latency when a receiving
packet type is changed from the corresponding packet typesent?
➞ Use nano scale timestamp injection in the P4 switch
as telemetry with a pair of unique inner headers

2. What kind of latency characteristics do we observe on thesoftware router?
➞ Analyze the latency with rte_eth_tx_burst and an outgoingpacket interval on Kamuee
9Overview

Experiments on software router• SRv6 translation functions are used for the latency measurement• DUT : DPDK-based software router (Kamuee) is used
• Traffic generators• IXIA : a hardware-based commercial traffic generator product• TRex : an DPDK based open source traffic generator• P4 switch : Inject timestamp as telemetry to source mac address at mirrored packets
Incoming packets

Trafficgenerator(IXIA, TRex)
Packet capture
on IXIA

10P4 switch fortimestamp
Outgoingpackets…

Mirroring withnano scale timestamp

Timestamp point for incoming packet
Timestamp point for outgoing packet

DUT forSRv6 StatelessTranslation(Kamuee)
Server
Incoming Packet
Outgoing Packet

Overview of software router

Kamuee – (1)• A simple DPDK router that evolved from the l3fwd example applicationof the DPDK
• Kamuee successfully increases its performance as the number of CPUcores increase (input throughput : 360 [Gbps])11< Performance by adding CPU cores2) >2) Kamuee: An IP Packet Forwarding Engine for Multi-Hundred-Gigabit Software-based Networks プロシーディング集 Internet Conference (IC) 2018

Overview of software router

Kamuee – (2)• Kamuee processes on a packet-by-packet basis that rather reflects
an input traffic pattern• GTP-U and SRv6 stateless translation is occurred at tunnel vport (11)12

GTP-U/SRv6 translation

Precise latency measurement

Nano scale timestamp using P4 switch• During the packet mirroring, the timestamp is stored in the source MACaddress field at the outer header• This function overwrites and abandons the original source MAC address

14

Traffic generators (IXIA and TRex)

CDF of incoming packet interval• The CDF curves of IXIA and TRex are different, and the curve of TRex is
widely fluctuated15

Configured
[ns]

Min
[ns]

50%tile
[ns]

Mean
[ns]

Max
[ns]

IXIA

500

484

499

500

537TRex500

72450088,291

Cumulative distribution function (CDF)• The traffic generated by TRex is more bursty in constant traffic load (2 [Mpps])
10.80.6

0.4

0.20IXIATRex

101001000

Packet interval [ns]10000Measurement scenarios• Focus on the precise latency measurement on Kamuee• To achieve it, we only used one CPU core for packet processing threadwith one NIC port at constant Mpps (2 Mpps)• The input load (2 Mpps) is close to maximum throughput (2.5 Mpps)• Two traffic gen. (IXIA and TREX) are used
• Short packet size : 94 bytes (GTP-U)/98 bytes (SRv6)• Long packet size : 976 bytes (GTP-U)/980 bytes (SRv6)
Packetsize

Input
Load [Mpps]

NumberOf trials

Capturedpackets

GTP-U→SRv6

SRv6→GTP-U

RX [Gbps]

TX [Gbps]

RX [Gbps]

TX [Gbps]

Short

2

5100,000

1.50

1.57

1.57

1.50

Long

2

5100,000

15.55

15.68

15.68

15.55

16

Measurement results on Kamuee

Evaluation results• Statistics of translation latency• The latency at all conditions is smaller than 70 μs• We cannot observe significant difference between the packet sizes in comparison
with the generators• The latency with TREX is widely fluctuated• Major cause of fluctuated latency is a wide range of input packet interval• The interval is affected to the latency measurement→ It would be hard to understand the latency measurement results

SR function

TrafficGen.

GTP-U→SRv6
SRv6→GTP-U
17

Short packet size

Long packet size

Min[us]50%tile[us]

Mean[us]

Max[us]Min[us]50%tile[us]

Mean Max
[us]
[us]

IXIA

49919

510

1018

TRex

3172051

4141658

IXIA

3771948816TRex

3131666

3161768Measurement results on Kamuee

CDF of translation latency• Observe a difference between the generators• There are ripple points on the short packet in the CDF curve (IXIA)< Short >Ripplepoints
18< Long >Why the ripple points are occurred?

The impact of rte_eth_tx_burst• What is rte_eth_tx_burst?
• The packets are sent in a bulk on Kamuee after the translation• Observe the impact of rte_eth_tx_burst to the latency measurement• IXIA is only used for the precise latency measurement• The following tx_burst values are used
rte_eth_tx_burst

19416 (default)

32

Why the ripple points are occurred?

CDF of translation latency (short packet)• Under the input load (2Mpps), the small size of tx_burst contributes tothe short latency• As the tx_burst size is changed, the latency is proportionately changed• The tx_burst size is the major cause of ripple points

Input load
[2 Mpps]
GTP-U→SRv6

SRv6→GTP-U

20

tx_burst

Min[us]50%tile[us]

Mean[us]

Max[us]4355111649919325151528423311163771932

3121225< GTP-U→SRv6 >Why the ripple points are occurred?

CDF of outgoing packet interval (short packet)• Approximately 80% of interval is in the bulk
• The tail interval (i.e. 99%tile) is increased due to the tx_burst size

Input load
[2 Mpps]
GTP-U→SRv6

SRv6→GTP-U

21tx_burst

Min
[ns]

50%tile
[ns]

Mean
[ns]

Max
[ns]

4

7185007,793

1671850013,4103271850018,9794721500

8,461

166

16500

15,3513261350018,872< GTP-U→SRv6 >Conclusion and future work
• Conclusion• We evaluated the latency of translation functions on Kamuee using
nano-scale measurement with P4 switch• The latency at one CPU core is in the range (roughly 3–70 μs)• TREX• The latency is widely distributed by the wide range of incoming packet
interval• IXIA• There are ripple points on the short packet• The major cause of ripple points is the tx_burst size
• When the tx_burst size is changed, the latency is proportionately changed• Future work22• Measure the latency on more complicated cases, such as multiple
routing entries and multiple CPU cores at maximum performance• Evaluate the latency on other software routers

Backup slideL3 forwarding vs Translation

Comparison of maximum throughput [Mpps]• Measure the maximum throughput (L3 forwarding [IPv4] vs Translation)• There is NO packet loss and the single CPU core is used
• L3 forwarding [IPv4]: 5.9 Mpps
• GTP-U/SRv6 Translation : 2.5 Mpps

42.3%

Latency measurement setup (server)• H/W specification• One CPU core is used for latencymeasurement• S/W configuration• C state/P state are disable• CPU frequency is fixed (2.6 GHz)• Intel TurboBoost is also disable

CPUIntel Xeon Gold 6126
(2.6 GHz) [19.25 MB L3 cache]

Memory384 GB

NIC

Mellanox ConnectX-4

OS

Ubuntu 18.04.2 LTS

Bandwidth40 Gbps

Kernel

4.15.0-45

DPDK

18.11.2 (stable)• rte_eth_tx_burst on Kamuee• default : 16

How to generate unique packets forlatency measurement?
• We fixed outer headers and changed unique inner packet headers
randomly and sequentiallyOuter headers

Inner headersWhy the ripple points are occurred?

CDF of translation latency (long packet)• As the tx_burst size is changed, the latency is proportionately changed• The tx_burst size is the major cause of ripple points

GTP-U→SRv6

SRv6→GTP-U

tx_burst

Min[us]50%tile[us]

Mean[us]

Max[us]

44

6

6

1316

5

101018325151526

445

5

1116

4881632

4141426< GTP-U→SRv6 >Why the ripple points are occurred?

CDF of translation latency (long packet)• As the tx_burst size is changed, the latency is proportionately changed• The tx_burst size is the major cause of ripple points

GTP-U→SRv6

SRv6→GTP-U

tx_burst

Min[us]50%tile[us]Mean[us]

Max[us]

4

4

6

6

1316

510

1018

32

5151526

445511164881632

4141426< SRv6→GTP-U>