Boosting Capacity Utilization in MPLS
Networks using Load-Sharing
MPLS JAPAN 2007

Sanjay KhannaFoundry Networks
skhanna@foundrynet.com

Agenda

Why we need Load-Sharing



Methods to boost capacity







Trunks/Link Aggregation
Routing Protocols ECMPMPLS Signaling Protocols ECMP
Mapping MPLS payloads to LSPs

Methods for efficient utilization


Packet Forwarding Schemes
Enhancements to Existing Schemes

MPLS JAPAN 2007

2

Copyright 2007 Foundry Networks

Why Load Sharing
 Utilize investment in existing infrastructure Ability to add bandwidth in small increments
 Cost-effective to add bandwidth 1+N link protection
 End to End protection with diverse paths
 Avoid idling of backup paths
 Transport carrier not offering higher bandwidthlinks
 100GbE not available yet
MPLS JAPAN 2007

3

Copyright 2007 Foundry Networks

Factors affecting MPLS Load Sharing

Trunks: Provide multiple links for load-sharing

Link Aggregation/bundling

Protocols: Determine multiple paths for load-sharing
Routing Protocols: IGP, BGP
 MPLS Protocols: LDP, RSVP-TE

MPLS payload mapping: Assign MPLS services to LSPs
BGP/MPLS-VPN & IPv4/v6 mapping to LSPs
 PW mapping to LSPs


Data Forwarding: Decision on how packets are loadshared
Per packet based Flow based


MPLS JAPAN 2007

4

Copyright 2007 Foundry Networks

Link Aggregation/Trunking A very popular method of bundling multiple physicallinks between 2 devices
 Used on both local side and network side

CE

PE

MPLS NetworkPE Typically, higher layer protocols unaware of the linkbundling
MPLS JAPAN 2007

5

Copyright 2007 Foundry Networks

Routing Protocols ECMPIGP (ISIS/OSPF): Determine multiple equal cost paths to a
destinationAffects paths taken by MPLS LSPs
IGP affects ECMP determination for LDP and non-CSPF RSVP-TE LSPs
 IGP-TE affects CSPF RSVP-TE ECMP determination


Import LSPs as IGP Shortcuts into IGPAllows router to use multiple equal cost LSP paths

BGP: Determine multiple equal cost paths to a destination
Multiple equal cost paths to different next-hops reachable by diverseLSPs Multiple LSP paths to a BGP next-hop BGP can calculate multiple equal cost label paths for Inter-AS
MPLS JAPAN 2007

6

Copyright 2007 Foundry Networks

MPLS Signaling Protocols ECMPMPLS signaling allows multiple LSPs to the same destination



RSVP-TE: Selects a path for a LSP from multiple equal cost paths
that satisfy the LSP constraints, as determined through CSPF


Typical criteria used:
Hops: Pick the path with least number of hops
 Less probability of failure Least-fill: Pick the path with highest available bandwidth Even spread of traffic
 Most-fill: Pick the path with lowest available bandwidth Leave room for higher bandwidth LSPs

LDP: Allows a prefix to be reachable through multiple equal costlabel paths

MPLS JAPAN 2007

7

Copyright 2007 Foundry Networks

IP Mapping to LSPs:

For IPv4/v6 Routing and BGP/MPLS-VPNs
BGP advertisement from Router Y

Pick from equalcost paths torouters Y & Z

Router Y
Router X
Router Z

BGP advertisement from Router ZTypical mapping criteria used:

Assign a prefix to single LSP

Map prefixes within a VRF to single LSP

Better operator control

Load-share on per flow basis
MPLS JAPAN 2007

Better predictability

Better traffic distribution
8

Copyright 2007 Foundry Networks

Network N

PW Mapping to LSPs:

For VPWS and VPLS
Pick fromequal cost
LSPs A & B

PE1

Local Circuit


PE2

PW

Local Circuit

Typical mapping criteria used:Bind PW to least used LSP (LSP with lowest number of PWs)


Bind PW to LSP with most available bandwidth or same class of service

Useful for services with dedicated bandwidth requirements

Explicitly bind PW to LSPGood distribution of traffic

Better operator control

PW traffic split across multiple LSPs


MPLS JAPAN 2007

Difficult to monitor and guarantee SLAs, potential packet reordering

9

Copyright 2007 Foundry Networks

Load-Sharing in the Forwarding Plane Routing Protocols ECMP, MPLS Signaling Protocols

ECMP and Link Aggregation provide multiple paths fora MPLS router to forward traffic
 2 common schemes to load-share traffic over theseequal cost paths/links
 Packet based
 Hash/Flow based

MPLS JAPAN 200710

Copyright 2007 Foundry Networks

Packet based Forwarding Each packet in turn is sent on the next link

Pkt 4

Pkt 3

Pkt 2

Pkt 3

Pkt 1

Pkt 4

Pkt 2

Pkt 1

 Perfect load balancing Potential packet reordering issues
 Possible increase in latency and jitter for some flows

MPLS JAPAN 2007

11

Copyright 2007 Foundry Networks

Flow/Hash based Forwarding Identifies packets as flows
 Based on packet content such as IP header Keeps flows on the same path
 Maintains packet ordering

Flow A

Flow B

Flow B

Flow A

Pkt 4

Pkt 3

Pkt 2

Pkt 1

Flow A

Flow A

Pkt 4

Pkt 1

Flow B

Flow B

Pkt 3

Pkt 2

 Hashing is the most popular scheme for flow based

forwarding
MPLS JAPAN 2007

12

Copyright 2007 Foundry Networks

Hash Based Forwarding for L3 Flows (1)Flows based on Source IP & Destination IP addressesAll traffic between 2 hosts relegated to one pathCan lead to over-utilization of one path
All traffic between
Host A and Host B
takes the same path

• Source IP Address• Destination IP AddressIM

Telnet HTTP

FTP

Pkt

Pkt

Pkt

Pkt

Host A

MPLS JAPAN 2007

Host B

13

Copyright 2007 Foundry Networks

Hash Based Forwarding for L3 Flows (2)Flows based on L3 and L4 information

Better traffic distribution for applications between 2 hosts• Source IP Address• Destination IP Address• Source TCP/UDP Port
• Destination TCP/UDP Port

Traffic between Host
A and Host B now
utilizes different paths
HTTP

FTP

Pkt

Pkt

IM

TelnetPkt

Pkt

Host A

Host B Applicable to IPv4 and IPv6 packets
MPLS JAPAN 2007

14

Copyright 2007 Foundry Networks

Hash Based Forwarding for L3 Flows (3)


Note:


Fragmentation Issue:

Example: If payload is fragmented in IP packets, only the first IP
packet carries the L4 information

Solution:
Set TCP segment size lower than the IP MTU to avoid
fragmentation
 Load balance fragmented packets using L3 information only

MPLS JAPAN 2007

15

Copyright 2007 Foundry Networks

Hash based Forwarding for L2 flows (1)Flows based on Layer 2 information in packet
However, IP packets between 2 routers will always take the samepath
IP Traffic between Router X
and Router Y will take thesame path since MAC
addresses in the packets will
be of the routers themselves• Source MAC Address• Destination MAC Address

Router X

IM

Telnet HTTP

FTP

Pkt

Pkt

Pkt

Pkt

Router Y

Host A

MPLS JAPAN 2007

Host B

16

Copyright 2007 Foundry Networks

Hash based Forwarding for L2 flows (2)Determine IPv4/v6 packets in L2 flows:

Flows based on Layer3/4 (and optionally Layer2) information for IPv4/v6
packets
Flows based on Layer 2 information for other packets
IP Traffic between Router X
and Router Y now distributed
over different paths• Source MAC Address• Destination MAC Address

Router X

Host A

MPLS JAPAN 2007

HTTP

FTP

Pkt

Pkt

IM

TelnetPkt

Pkt

Router Y• Source IP Address• Destination IP Address• Source TCP/UDP Port
• Destination TCP/UDP Port
17

Host B

Copyright 2007 Foundry Networks

Forwarding on LSRs
Load-Sharing at
Transit LSR

LSP
Transit LSR can load-share over multiple links between 2 LSRs
 Typical criteria used:Per packet: Potential packet reordering issue
 LSP assigned to one link: High usage LSPs will over-utilize one link Per Inner/Outer Labels: High usage PWs/VPN labels will over-utilize
one link Per flow: Better distribution of traffic Requires Packet Speculation


MPLS JAPAN 2007

18

Copyright 2007 Foundry Networks

Forwarding on LSRs:

Packet Speculation
Transit LSRs have no information on packet payloadHow will Transit LSR
load-share overtrunks using Flows?

Originating LER loadbalances using L2/L3/L4
hashing

Terminating LSR loadbalances using L2/L3/L4
hashing

LSPTransit LSR speculates on the packet type
Checks first nibble after bottommost label
 If 4/6, speculates on packet as IPv4/IPv6
 Else (optionally) speculates on packet as Ethernet
 Can now load-share using “LSP Label/VC label/L2/L3/L4 headers”


MPLS JAPAN 2007

19

Copyright 2007 Foundry Networks

Hash based forwarding issues and solutions:

Polarization Effect

Similar routers pick the same path for flows with identical hash

Leads to over-utilization of some parts of the network
In this example, eachrouter picks the firstlink for flows withsame hashFlows A & B have the same hashFlow A

Flow B

Packets to Network X

MPLS JAPAN 2007

Network X

20

Copyright 2007 Foundry Networks

Hash based forwarding issues and solutions:
Hash Diversification (Neutralizes Polarization Effect)Each router uses a unique-id per router in hash calculations


Alternatively, hashing using Source and Destination MACs may givecomparable results in most scenarios

Similar routers now pick different links
 However, flows are still together on same links


Flow A

Flow B

Packets to Network X

Network X

Flows A & B have the same hash

MPLS JAPAN 2007

In this example, each routerdoesn’t pick the first link forflows with same hash, thusachieving link diversity but
not flow diversity

21

Copyright 2007 Foundry Networks

Hash based forwarding issues and solutions:
Hash Variation (Neutralizes Polarization Effect)Each router uses a different variant of the hash algorithm

Flows can now be distributed

In this example, each router
may pick a different link forflows with same hash, thusachieving both link diversity
and flow diversity

Flow A Flow B Flow C
Packets to Network X

Network X

Router X

Flows A, B & C have the same hash on Router
X but different hash on other routers

MPLS JAPAN 2007

22

Copyright 2007 Foundry Networks

Summary

Load-Sharing is a cost-effective technique to improve
network utilization
Multiple methods to boost capacity at various layersCan effectively increase throughput beyond the current limits ofphysical link capacity

Flow/Hash based forwarding offers many advantages for
efficient utilization of the increased capacityWorks over multiple paths and links

Watch out for polarization effects and neutralize themNot a one size fits all approach

Choose optimal schemes based on traffic types and operator
policyMPLS JAPAN 2007

23

Copyright 2007 Foundry Networks

Questions?

MPLS JAPAN 200724

Copyright 2007 Foundry Networks