Optimizing Packet Accesses for aDomain Specific Language on
Network Processors
Tao Liu1, Xiao Feng Li2, Lixia Liu2,Chengyong Wu1, Roy Ju3

1. Institute of Computing Technology, China Academy of Sciences
2. Intel China Research Center
3. Microprocessor Technology Labs, Intel Corporation

LCPC 20051

Outliney Motivationy System Overview
y Packet accesses optimizationsy Experimental results

LCPC 2005

2

Network Processors
y Advantages– More flexible than ASICs/custom design– Higher performance on packet processing– Lower development costy Unfortunately, difficult to program– Complicated hardware– Limited resources– Low level programming languages

LCPC 2005

3

Domain specific language:Baker
y Handle programming challenges in compiler– Eliminate need for assembly programming– Automate resource management– Perform domain-specific optimizations in compiler

y Assist portable application development– Protocol stack component modularity– Abstracted programming model hiding underlying
hardware details– Build-in language types and libraries for network
applications– Packet typey Big headache: still achieve high performance
LCPC 2005

4

Packet accesses critical toperformancey Key factors of performance– Strict instructions budget per packet– 700 cycles on IXP2400– Constrained memory bandwidth– 2 DRAM accesses on IXP2400

y Characteristics of packet accesses– Consist of dozens of instructions– Need memory reference per access– Occur frequently in applications

LCPC 2005

5

Outliney Motivationy System overview
y Packet accesses optimizationsy Experimental results

LCPC 2005

6

Intel IXP2400 network
processor
External Media
External MediaSRAM
SRAM
SRAM
SRAM

Network Interface

SRAM Controller 0packet
DRAM
DRAM

DRAM• Controller
0Multi-processorand Multi-threading

SRAM Controller 1

Data memory• 640 words of local memory• No cache• Explicit memory instructions

LCPC 2005

Scratch Memory

ME 0:1

ME 0:2

ME 1:1ME 1:2

Hash Unit

ME 0:3

ME 0:4

ME 1:3

ME 1:4

Intel XScale®

7

L3-Switch written in theBaker

l3_switch.l2_clsfr.ppf( ether_pkt_t * pkt )l3_switch_module {int is_arp = ( pkt->type == ETH_TYPE_ARP );int forward
pkt->dst ==lpm_lookup = ( l3_fwdr_module
mac_addrs[pkt->metadata.rx.port] );l3_clsif(is_arp ){icmp_proc
channel_put( arp_cc, packet_copy( pkt ));}
if( forward ){options_proc
ipv4_pkt_t* ipkt = packet_decap( pkt );channel_put( l3_forward_cc,encap ipkt );}Txelse{arp
channel_put( l2_bridge_cc,pkt );}eth_encap_modulel2_clsRx}

bridgel2_bridge_module

LCPC 20058

Outliney Motivationy System overview
y Packet accesses optimizationsy Experimental results

LCPC 2005

9

Packet primitives in Baker
and implementationy Protocol constructy Packet handley Packet accessy Decap/Encap

IPv4 overEthernet
packetethernet
ethernet
headerheader

ipv4ipv4header
header

14B

20B

ipv4
ipv4
payloadpayloadpacket_decap
headpointer + offset

Metadata

Head pointerHead pointerTail pointerTail pointerUser-defined meta data
User-defined meta data

Packet handle

protocol ether {dst : 48;
src : 48;type : 16;
demux{ 14 };};LCPC 2005

void A.process(ether_packet_t* in_pkt){
ipv4_packet_t* p;
mac_addr_t mac;
mac = in_pkt->dst;
if(fwd){p = packet_decap(in_pkt);channel_put(l3_fwdr_chnl,p);}}

10Packet access combiningcombined
packet read
readA…

…

packet write

decap

…

…

packet read B

read register
packetread CLCPC 2005y Assumptions– HW can perform very
wide memory accesses– Packet pointers areuniquey Algorithm– Select the best
candidate to combine– Keep cached data inregisters– Ensure data
dependence by data
flow analysis

11Compiler-generated packet
caching (static)
IPv4 over
Ethernetpacketethernet
ethernet
headerheaderipv4
ipv4
headerheaderipv4
ipv4
payloadpayloady Packet flow analysis– Inter-procedural and Intertasks analysis– Estimate the cache range– Annotate info onto packet
primitives

y Code generator

Cached packet
fragmentin ME local memoryp->dst_ip
p->src_ip
p->checksump->ttl– Preload & write back cache– Generate efficient packet
access code according toannotations– Remove unneeded packet
primitivesp->srcLCPC 200512

ethernet
ethernet
headerheader

...
...

mpls
mpls
label
label

MPLS over
Ethernetpacket

mpls
mpls
label
label

Compiler-generated packet
caching (dynamic)
ipv4
ipv4
headerheader

ipv4ipv4
payloadpayloadpacket_decap()
phPacketheaderphphCache (local memory)

fall through tooriginal packet
primitives

offset+y Packet primitives dynamically resolve field offsets and alignmentsy Packet flow analysis– Estimates the cache range with profilingy Code generator– Variable and run-time instructions to resolve offset and alignments
dynamically– Run-time offset check to guarantee correctness
LCPC 2005

13

Outliney Motivationy System overview
y Packet accesses optimizationsy Experimental results

LCPC 200514

Experimental setup
y Radisys ENP2611* boardRadisys
ENP2611*2 x 1Gbps
optical links

IXIA packetgenerator– IXP2400– 8MB SRAM, 64MB DRAM
– 3 x 1Gbps optical ports

y IXP2400 runtime system– Linux on Intel XScale®
– Language runtime system

y Benchmarks– L3-Switch† – L2 bridge & L3 routing using dest IP– MPLS† – Fast routing using label stack– Firewall – WAN / LAN isolation† Evaluated using Network Processor Forum traces

LCPC 2005* Third party brands/names property of their respective owners15

L3-Switch forwarding rate *ForwardingRate(2Gbps)

100%Max performance
limited by DRAM
bandwidth

80%
60%

BASE (L3-switch)

PAC (L3-switch)

40%

CGPC (L3-switch)

Single ME improvementshows reduced dynamic
instruction count

20%
0%
1

2

* min-sized 64B packets

LCPC 2005

3 ME

45

6

Performance tests and ratings are measured using specific computer systems and/or
components and reflect the approximate performance of Intel products as measured bythose tests. Any difference in system hardware or software design or configuration may
affect actual performance.

16

Forwarding Rate(2Gbps)

MPLS & Firewall Performance
100%
80%

BASE (MPLS)

60%PAC (MPLS)

40%CGPC (MPLS)
20%0%

Forwarding Rate(2Gbps)

13 ME 4

56

100%
80%

BASE (Firewall)60%

PAC (Firewall)

40%

CGPC (Firewall)

20%0%
1

LCPC 2005

223 ME 45617

Packet access count andaggregate access size
DRAM
AccessCount
L3Switch

BASE
PAC
CGPC

MPLS

BASE
PAC
CGPC

FirewallLCPC 2005

BASE

Aggregate
Access Size(Bytes)

Instruction
Count

29

696

2033

13200

1190

PAC & CGPC has approximate
272
instruction countdue to MPLS
consisting of many dynamic
16384
packetaccesses
PAC& CGPC
significantly
212
reduce packet9DRAM
accesses, aggregate memory248
access size and instruction
count
24.2
580

770
1851
1428
1495
1742

PAC

4.4

140572

CGPC

13237518

Performance summaryy All benchmarks exhibit similar trends andperformance curves– CGPC shows 5.8x performance speedup

y PAC & CGPC can efficiently reduce aggregate
memory access size and instruction count– PAC: reduce 70% memory access size
– CGPC: Reduce 90% memory access sizey CGPC is also effective to reduce dynamic packet
accesses

LCPC 2005

19Conclusions
y Packet access optimizations are critical to theperformance of high-level programming
environments– Performance is limited by instruction count and memory
bandwidth– Efficiently relieve memory bandwidth contention andreduce instruction county PAC and CGPC are effective on performance
improvement– Reduce aggregate memory access size and improve
performance by 5.8x
– With CGPC, achieve 2Gbps line rate on three typical
network applications on IXP2400

LCPC 2005

20

Related worky Shangri-la– Michael K. Chen, X. Li, R.Lian, J. Lin, L. Liu, T. Liu, R. Ju. ShangriLa: achieving high performance from compiled network applications
while enabling ease of programming, In PLDI’05, Chicago, IL, June 2005

y Click– Kohler, E., Morris, R., Chen, B., Jannotti, J. and Kaashoek, M.F. The
Click Modular Router. In ACM TCS, 18(3) pp. 263-297, August 2000.

y Memory access combining– Davidson, J. and Jinturkar, S. Memory Access Coalescing: A Techniquefor Eliminating Redundant Memory Accesses. In PLDI’94, Orlando, FL,June 1994.

y Packet buffer caching– S. Iyer, R.R. Kompella, and N. McKeown. Analysis of a memoryarchitecture for fast packet buffers. In Proc. IEEE Workshop High
Performance Switching and Routing(HPSR), 2001.

LCPC 2005

21

Q&A

LCPC 200522