Optimizing fabric path forwarding for virtual nodes within an electronic device

The techniques describe directly forwarding a packet from an ingress packet forwarding engine to a particular destination packet forwarding engine (PFE) when internal packet load balancing may otherwise result in an increased number of fabric hops. For example, a source PFE may receive incoming packets destined for a router reachable only by a particular destination PFE (e.g., egress PFE). Rather than load balancing the incoming packets to a destination PFE that is likely to be a non-egress PFE, a source PFE obtains fabric path information associated with the egress PFE from a destination PFE such that source PFE may forward incoming packets directly to the egress PFE.

TECHNICAL FIELD

The invention relates to computer networks and, more specifically, to packet routing and switching within computer networks.

BACKGROUND

A computer network is a collection of interconnected computing devices that can exchange data and share resources. In a packet-based network, such as an Ethernet network, the computing devices communicate data by dividing the data into variable-length blocks called packets, which are individually routed across the network from a source device to a destination device. The destination device extracts the data from the packets and assembles the data into its original form.

Certain devices, referred to as routers, maintain routing information representative of a topology of the network. The routers exchange routing information so as to maintain an accurate representation of available routes through the network. A “route” can generally be defined as a path between two locations on the network. Upon receiving an incoming data packet, a router examines information within the packet, often referred to as a “key,” to select an appropriate next hop to which to forward the packet in accordance with the routing information.

Routers may include one or more forwarding components, e.g., packet forwarding engines, interconnected by an internal switch fabric. Packet forwarding engines receive and send data with other external devices via interface cards. The switch fabric provides an internal interconnect mechanism for forwarding data within the router between the packet forwarding engines for ultimate transmission over a network. In some examples, a router or switching device may employ a distributed, multi-stage switch fabric architecture, in which network packets traverse multiple stages of the switch fabric located in distributed forwarding components of the router to travel from an ingress point of the switch fabric to an egress point of the switch fabric.

A router may be virtualized into multiple virtual network nodes by portioning hardware resources of the router, such as packet forwarding engines. One or more links may be provisioned between two virtual nodes. For example, an aggregated fabric interface (AF) link that is a logical link construct and provides virtual node connectivity. A source packet forwarding engine may load balance incoming data across the internal switch fabric via the AF link towards a destination packet forwarding engine for ultimate transmission over a network.

SUMMARY

In general, this disclosure describes techniques for directly forwarding a packet from an ingress packet forwarding engine to a particular destination packet forwarding engine (PFE) when internal packet load balancing may otherwise result in an increased number of fabric hops. For example, a source PFE may receive incoming packets destined for a router reachable only by a particular destination PFE (e.g., egress PFE). Rather than load balancing the incoming packets to a destination PFE that is likely to be a non-egress PFE, a source PFE obtains fabric path information associated with the egress PFE from a destination PFE such that source PFE may forward incoming packets directly to the egress PFE. The techniques may provide specific technical improvements, such as reduced fabric hops, especially in situations where a router has been partitioned into multiple virtual nodes and traffic from an ingress PFE and destined for a virtual node is typically internally load balanced across the egress PFEs for the virtual node.

As one example, to obtain fabric path information associated with the egress PFE, a source PFE modifies an incoming packet to include a fabric path header that instructs a receiving PFE to send fabric path information associated with the egress PFE to a path module rather than forwarding the packet to a next fabric hop. Source PFE in turn receives from the path module an indicator (e.g., a hash value) of a particular fabric path to egress PFE for which source PFE may use to forward incoming packets directly to the egress PFE rather than load balancing the packet.

In one example, a method includes receiving, by a source virtual routing node of a single-chassis network device having a plurality of packet forwarding engines (PFEs) and a plurality of fabric links coupling respective pairs of the plurality of PFEs at respective fabric interfaces of the plurality of PFEs, a packet. The method may also include sending, by the source virtual routing node, a modified packet to a receiving PFE of the plurality of PFEs, wherein the modified packet includes a fabric path header added to the packet to request fabric path information associated with the egress PFE. The method may further include receiving, by the source virtual routing node and from the path module, the fabric path information associated with the egress PFE. The method may also include storing, by the source virtual routing node, the fabric path information associated with the egress PFE in forwarding information of the source virtual routing node. The method may further include sending, by the source virtual routing node, a next packet directly to the egress PFE instead of load balancing the next packet.

In another example, a method may include receiving, by a receiving virtual routing node of a single-chassis network device having a plurality of packet forwarding engines (PFEs) and a plurality of fabric links coupling respective pairs of the plurality of PFEs at respective fabric interfaces of the plurality of PFEs, a modified packet from a source virtual routing node, wherein the modified packet includes a fabric path header to request fabric path information associated with the egress PFE. The method may also include determining, by the receiving virtual routing node, that the modified packet includes the header. The method may further include retrieving, by the receiving virtual node and from forwarding information of the receiving virtual node, the fabric path information associated with the egress PFE. The method may also include sending, by the receiving virtual node, the fabric path information associated with the egress PFE to the path module instead of forwarding the modified packet to a next fabric hop.

In another example, a single-chassis network device includes a plurality of packet forwarding engines (PFEs); a plurality of fabric links coupling respective pairs of the plurality of PFEs at respective fabric interfaces of the plurality of PFEs; a first virtual routing node and a second virtual routing node of a plurality of virtual routing nodes, wherein the first virtual routing node is configured to: receive a packet; send a modified packet to a receiving PFE of the plurality of PFEs, wherein the modified packet includes a fabric path header added to the packet to request fabric path information associated with the egress PFE; receive the fabric path information associated with the egress PFE from the path module; store the fabric path information associated with the egress PFE in forwarding information of the first virtual routing node; and send a next packet directly to the egress PFE instead of load balancing the next packet.

The details of one or more examples of the techniques described herein are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the techniques described herein will be apparent from the description and drawings, and from the claims.

DETAILED DESCRIPTION

FIG. 1is a block diagram illustrating an example network environment10that includes a logical view of a single-chassis router12configured in accordance with techniques described in this disclosure. For purposes of example, the techniques of this disclosure are described with respect to a simplified network environment10ofFIG. 1in which single-chassis router12communicates with core routers (CR)30A-30B (“core routers30”) to provide client devices22A-22B (“client devices22”) with access to services provided by devices in Internet Protocol (IP)/Multi-Protocol Label Switching (MPLS) core network16.

The configuration of network environment10illustrated inFIG. 1is merely an example. Although not illustrated as such, IP/MPLS core network16may be coupled to one or more networks administered by other providers, and may thus form part of a large-scale public network infrastructure, e.g., the Internet. Ethernet aggregation network14may be viewed as an access network to the Internet. A service provider network may provide computing devices coupled to client devices22with access to the Internet, and may allow the computing devices within customer networks (not shown) to communicate with each other. In another example, IP/MPLS core network16may provide network services within the core of the Internet. In either case, IP/MPLS core network16may include a variety of network devices (not shown) other than single-chassis router12, provider edge (PE) router28, and core routers30, such as additional routers, switches, servers, or other devices.

Single-chassis router12includes a virtual provider edge (vPE) node20(“vPE20”) and virtual core router (vP) nodes18A-18B (“vPs18”), which are cooperative virtual routing components operating as multiple distinct nodes from the perspective of network devices external to single-chassis router12. Single-chassis router12is a router having a single physical chassis, which is virtualized into multiple virtual network nodes (referred to as “vNodes”) by portioning hardware resources of the router, such as packet forwarding engines (PFEs). In the example ofFIG. 1, each of vNodes18,20includes one or more PFEs, e.g., PFEs24A-24F (“PFEs24”). Individual PFEs24are associated with a particular vNode and are not shared among multiple vNodes. In the example ofFIG. 1, vPE20may include PFEs24A and24B; vP18A may include vPEs20C and20D; and vP18B may include PFEs24E and24F.

Client devices22may be devices associated with one or more customer networks (not shown) coupled to customer edge (CE) router23. In some examples, client devices22may include computing devices, such as personal computers, laptop computers, handheld computers, workstations, servers, switches, printers, customer data centers or other devices, for example. In other examples, client devices22may be endpoint devices such as a switch, a router, a gateway, or another terminal that operates as a demarcation point between customer equipment, such as subscriber devices, and service provider equipment. In one example, client devices22may comprise a digital subscriber line access multiplexer (DSLAM) or other switching device. For example, client devices22may be connected to one or more wireless radios or base stations (not shown) to wirelessly exchange packetized data with subscriber devices. Client devices22may comprise a switch, a router, a gateway, or another terminal that aggregates the packetized data received from the wireless radios to CE router23. In some examples, aggregation network14may include an optical access network. For example, CE router23may comprise an optical line terminal (OLT) connected to one or more client devices22or optical network units (ONUs) via optical fiber cables.

Client devices22may be access nodes coupled to customer networks and subscriber devices. Client devices22are clients of services provided by PE router28. In this example, a service provider network includes client devices22and customer edge (CE) router23that provide subscriber devices with access to aggregation network14. In some examples, CE router23may comprise a router that maintains routing information between subscriber devices and aggregation network14. CE router23, for example, may include Broadband Remote Access Server (BRAS) functionality to aggregate output from one or more client devices22into a higher-speed uplink to aggregation network14.

Single-chassis router12includes multiple routing components (e.g., routing processes) and forwarding components (e.g., PFEs24) that are physically coupled and configured to operate as separate logical routers. To core routers30and CE router23of network environment10, single-chassis router12appears as multiple routing devices, specifically, virtual PE (vPE) router20, and virtual provider (vP) routers18. For example, although single-chassis router12includes a single chassis, from the perspective of core routers30, single-chassis router12has multiple externally-advertised network addresses and maintains multiple peer routing sessions for each routing protocol maintaining peer routing sessions with each of the core routers30.

Each vNode's control plane (vCP)25A-25C (“vCPs25”) instantiates with virtual machine (VM) technology. The vCP25either could be within the routing engine (RE) of single-chassis router14or outside the RE. Each vNode could serve the role of different network functions, such as Internet service provider edge (PE), Virtual Private Network (VPN) service PE and Multiprotocol Label Switching (MPLS) Label Switching Router (LSR). Apart from these vNodes, in some examples single-chassis router12may also include an administrative VM instantiated for shared resources management (e.g., a management plane, not shown inFIG. 1).

Between two vNodes in single-chassis router12, one logical layer-3 link is provisioned that is visible to devices external to single-chassis router12. For example, inFIG. 1, aggregated fabric interface (AF) links32A-32B (“AF links32”) provide a logical link between vPE20and vP18A, and vPE20and vP18B, respectively. Each of AF links32are layer-3 logical link constructs and provide vNode to vNode connectivity. AF links32bundle those fabric interconnects that connect the same vNodes. AF provide a single logical link connectivity between the vNodes, and could have many layer-1, layer-2, or layer-3 fabric bundling within, depending on implementation.

AF32A includes fabric interconnects33A-33D, and AF32B includes fabric interconnects33E-33H (collectively, “fabric interconnects33”). Fabric interconnects terminate at fabric interfaces of one of PFEs24. The fabric interfaces have identifiers that are not generally advertised to devices external to single-chassis router12. The fabric interconnects33are modelled as point-to-point Ethernet links between a pair of PFEs24. AF and fabric interfaces (FIs) emulate point-to-point interfaces.

In one example, assume vPE20provides MPLS-based VPN services, and CE23connects to PFE24A. Assume also that vPE20connects to vP18A with fabric equal cost logical paths via PFE24C and PFE24D, and vPE20connects to vP18B with fabric equal cost logical paths via PFE24E and PFE24F. When a packet comes to vPE20from Ethernet aggregation network14and destined for PE28, in the absence of the techniques of this disclosure vPE20would send data traffic to any of PFEs24C-24F based on ECMP load balancing. For example, AF link32A may include fabric interfaces33A-33D to interconnect PFEs of vPE20to PFEs of vP18A, and fabric interfaces33E-33H to interconnect PFEs of vPE20to PFEs of vP18B. vPE20may perform a hashing algorithm to determine which one of fabric interconnects33that is used for an outgoing interface. However, the hash is independent of the egress interface on the egress PFE.

Assume that PE28is reachable from vP18B only via PFE24E. Conventionally, vPE20load balances traffic toward the fabric. If a non-egress destination PFE receives the traffic, the non-egress PFE typically forwards the traffic to egress PFE24E to forward the packet out a WAN link to IP/MPLS core16. This results in an additional fabric hop that consumes fabric bandwidth. By load balancing the traffic, the chances of the traffic forwarded to a non-egress destination PFE is [(n−1)/n]*100 percent. In the example ofFIG. 1, when PFE24A load balances traffic toward the fabric, the chances of forwarding the traffic to a PFE other than PFE24E is 83.3%. As the number of PFEs increase, the chances of forwarding the traffic to a non-egress PFE approaches 100%, thereby increasing the likelihood of performing an extra fabric hop that wastes bandwidth.

In accordance with the techniques of this disclosure, source PFE may directly send traffic to a particular destination PFE (e.g., egress PFE) instead of load balancing the traffic. For example, a virtual node of single-chassis router12may send fabric path information associated with an egress PFE to path module27such that path module27may push the fabric path information to a source virtual node to steer incoming traffic directly to the egress PFE instead of load balancing the traffic. In some examples, path module27may include a telemetry server that receives statistics and performs additional functions such as monitoring traffic flows and aging out traffic flows. Although path module27is illustrated as within the single-chassis router12, path module27may be external to the single-chassis router12.

In the example ofFIG. 1, a source PFE, e.g., PFE24A, of source virtual node vPE20may receive traffic from Ethernet aggregation network14that is destined for PE28. Source PFE24A may perform a lookup to determine whether a fabric path to egress PFE is known. Responsive to a determination that the egress PFE is not known, source PFE24A may e.g., modify a copy of the incoming packet to include, e.g., a fabric path header, that instructs a receiving PFE to send fabric path information associated with the egress PFE to path module27rather than forwarding the packet to the next fabric hop. In some examples, source PFE24A may modify incoming packets identified as having a high packet rate, as further described below inFIG. 2. Source PFE24A then load balances the modified packet and the original packet towards the fabric.

In one instance, a non-egress PFE (also referred to herein as “receiving PFE”), e.g., PFE24D, may receive the modified packet and the original packet. Non-egress PFE24D performs a lookup of its forwarding information to determine the egress PFE, e.g., PFE24E. In some examples, the original packet is forwarded to the next fabric hop to egress PFE24E. Rather than forward the modified packet to the next fabric hop, non-egress PFE24D may send fabric path information36to path module27based on the modified packet. Fabric path information36may include a lookup value (i.e., information associated with egress PFE24E), the Hash value associated with source PFE224A, and information associated with source PFE224A, for example. In some examples, receiving PFE24D may send fabric path information36directly to source PFE24A.

PFE24D may communicate with path module27using a connection34. Connection34may be a communication channel established between PFEs24and path module27to exchange fabric path information. Connection34may be, for example, established using User Datagram Protocol (UDP). In the example ofFIG. 1, connection34is established between vPE20and path module27, between vP18A and path module server27, and between vP18B and path module27. Although not illustrated, other virtual nodes may establish connections with path module27.

Path module27may receive fabric path information36from non-egress PFE24D and may push the fabric path information (“fabric path information37”) to source PFE24A. Source PFE24A may receive the fabric path information37and may store the information associated with egress PFE24E. In this way, upon receiving incoming traffic, source PFE24A may perform a lookup of its forwarding information and determine a fabric path to egress PFE24E known, and directly forwards the incoming traffic to egress PFE24E instead of load balancing the packet.

In some examples, path module27may monitor the fabric path to egress PFE24E. In instances where the fabric path to PFE24E is unavailable or no longer valid, path module27may remove the fabric path or redirect the traffic to another fabric path to another destination PFE configured with the techniques described above.

In this way, incoming packets received from a source PFE are directly forwarded to a particular destination PFE, thereby eliminating the possibility of forwarding the packets to a non-egress PFE and incurring additional fabric hops that waste bandwidth.

FIG. 2is a block diagram illustrating an example single-chassis router212that provides directly forwarding packets to a particular destination PFE instead of load balancing, in accordance with the techniques described in this disclosure. Single-chassis router212may represent single-chassis router12ofFIG. 1. Single-chassis router212may include multiple virtual nodes operating as, for example, virtual provider edge or virtual customer edge routers, virtual autonomous system border routers (ASBRs), virtual area border routers (ABRs), or another type of network device, such as a virtual switch.

In this example, single-chassis router212includes a control unit218that provides control plane functionality for single-chassis router212. Control unit218may be distributed among multiple entities, such as one or more routing units and one or more service cards insertable into single-chassis router212. In such instances, single-chassis router212may therefore have multiple control planes. In some examples, each virtual routing node of single-chassis router212may have its own virtual control plane, e.g., vCPs25ofFIG. 1.

Control unit218may include a routing engine220that provides control plane functions, storing network topology in the form of routing tables, executing routing protocols to communicate with peer routing devices, and maintaining and updating the routing tables. Routing engine220also provides an interface to allow user access and configuration of single-chassis router212.

Single-chassis router212also includes a plurality of forwarding components in the form of example packet forwarding engines224A-224N (“PFEs224”) and a switch fabric226, that together provide a forwarding plane for forwarding and otherwise processing subscriber traffic. PFEs224may be, for example, any of PFEs24ofFIG. 1.

Control unit218is connected to each of PFEs224by internal communication link230. Internal communication link230may comprise a 100 Mbps or 1 Gbps Ethernet connection, for instance. Routing engine220may execute daemons (not shown), e.g., user-level processes that may run network management software, to execute routing protocols to communicate with peer routing devices, execute configuration commands received from an administrator, maintain and update one or more routing tables, manage subscriber flow processing, and/or create one or more forwarding tables for installation to PFEs224, among other functions.

Control unit218may include one or more processors (not shown inFIG. 2) that execute software instructions, such as those used to define a software or computer program, stored to a computer-readable storage medium (again, not shown inFIG. 2), such as non-transitory computer-readable mediums including a storage device (e.g., a disk drive, or an optical drive) and/or a memory such as random-access memory (RAM) (including various forms of dynamic RAM (DRAM), e.g., DDR2 SDRAM, or static RAM (SRAM)), Flash memory, another form of fixed or removable storage medium that can be used to carry or store desired program code and program data in the form of instructions or data structures and that can be accessed by a processor, or any other type of volatile or non-volatile memory that stores instructions to cause the one or more processors to perform techniques described herein. Alternatively, or in addition, control unit218may include dedicated hardware, such as one or more integrated circuits, one or more Application Specific Integrated Circuits (ASICs), one or more Application Specific Special Processors (ASSPs), one or more Field Programmable Gate Arrays (FPGAs), or any combination of one or more of the foregoing examples of dedicated hardware, for performing the techniques described herein.

PFEs224receive and send data packets via interfaces of interface cards222A-222N (“IFCs222”) each associated with a respective one of PFEs224. Each of PFEs224and its associated ones of IFCs222may reside on a separate line card (not shown) for single-chassis router212. Example line cards include flexible programmable integrated circuit (PIC) concentrators (FPCs), dense port concentrators (DPCs), and modular port concentrators (MPCs). Each of IFCs222may include interfaces for various combinations of layer two (L2) technologies, including Ethernet, Gigabit Ethernet (GigE), and Synchronous Optical Networking (SONET) interfaces. In various aspects, each of PFEs224may comprise more or fewer IFCs.

Switch fabric226provides a high-speed interconnect for forwarding incoming data packets to the selected one of PFEs224for output over a network. Switch fabric226may include multiple fabric links (not shown), such as fabric links33ofFIG. 1. In some examples, switch fabric226may be a distributed, multi-stage switch fabric architecture, in which network packets traverse multiple stages of the switch fabric located in distributed forwarding components of the router to travel from an ingress point of the switch fabric to an egress point of the switch fabric. As one example, switch fabric226may be implemented as a single multi-stage Clos switch fabric, which relays communications across the stages of the switch fabric. A typical multi-stage Clos switch fabric has a plurality of switches interconnected to form a plurality of stages. In a typical arrangement, the switch fabric includes an ingress (or “first”) stage, one or more intermediate stages, and an egress (or “final”) stage, with each stage having one or more switches (e.g., crossbar switches—often referred to more simply as “crossbars”). Moreover, the switch fabric may be implemented such that the switches are arranged as multiple parallel fabric planes that each provide independent forwarding from ingress ports to egress ports through the multiple stages, one or more of which may be treated as a spare fabric plane. In other words, each of the parallel fabric planes may viewed as an independent portion of the multi-stage Clos switch fabric, where each plane provides switching redundancy.

PFEs224process packets by performing a series of operations on each packet over respective internal packet processing paths as the packets traverse the internal architecture of single-chassis router212. Operations may be performed, for example, on each packet by any of a corresponding ingress interface, an ingress PFE224, an egress PFE224, an egress interface or other components of single-chassis router212to which the packet is directed prior, such as one or more service cards. The result of packet processing determines the way a packet is forwarded or otherwise processed by PFEs224from its input interface on one of IFCs222to its output interface on one of IFCs222.

The example ofFIG. 2shows each of PFEs224as including forwarding information228(otherwise referred to as forwarding information base (FIB)). Forwarding information228A, provides forwarding information for use in the forwarding plane when looking up next hops.

To illustrate by way of an example, assume PFE224A represents PFE24A ofFIG. 1. PFE224A may receive incoming traffic via an ingress interface on one of IFCs222. PFE224A may determine, from a lookup of forwarding information228A, whether information associated with a particular destination PFE, e.g., egress PFE information275A (“egress PFE info275A”), is known.

If PFE224A determines that egress PFE info275A is not in forwarding information228A, PFE224A may initiate instructions to request fabric path information associated with the egress PFE. For example, PFE224A may modify a copy of the packet to include a fabric path header that instructs a receiving PFE to send fabric path information associated with the egress PFE to path module227instead of forwarding the packet to a next fabric hop. In some examples, the header may include a hash value associated with source PFE224A and information associated with source PFE224A.

PFE224A may load balance the original packet and the modified packet towards the fabric via an egress interface of one of IFCs222of PFE224A. In load balancing, a hash algorithm chooses an egress interface from the aggregated fabric interface (e.g., AF32A ofFIG. 1) when Equal-Cost Multi-Path (ECMP) is available for a prefix. That is, PFE224A may select a next hop from a list of next hops of the AF and sends the original packet and modified packet to the selected next hop towards a receiving PFE via an outbound interface on one of IFCs222.

Assume for example, PFE224N represents non-egress PFE24D ofFIG. 1. PFE224N may receive, from PFE224A, the original packet and modified packet via an ingress interface on one of IFCs222. Based on a determination that the modified packet includes the fabric path header, PFE224N may perform a lookup of forwarding information228N to determine information associated with the egress PFE (e.g., egress PFE information275N) and may send the lookup information236to path module227. In some examples, non-egress PFE224N may send the fabric path information directly to source PFE224A. In some examples, the receiving PFE224N may send information including the lookup information associated with the egress PFE, information associated with the source PFE, and the hash value (collectively referred to herein as, “fabric path information”). Path module227may push the fabric path information237to source PFE224A. The source PFE may install the fabric path information as egress PFE information275A in forwarding information228A. The receiving PFE224N may forward the original packet to egress PFE224E for ultimate transmission over the network and may drop the modified packet.

Upon receiving incoming packets, the source PFE224A may lookup forwarding information228A and determine a particular destination PFE based on egress PFE information275A. In this way, source PFE224A may directly forward incoming packets to a particular destination PFE instead of load balancing the traffic.

FIG. 3is a block diagram illustrating example instances of routing engine318and packet forwarding engines324(“PFEs324”) of routing engine218and PFEs224ofFIG. 2in further detail. In this example, routing engine318provides a control plane302operating environment for execution of various user-level daemons314executing in user space306. Daemons314are user-level processes that may run network management software, execute routing protocols to communicate with peer routing devices, execute configuration commands received from an administrator, maintain and update one or more routing tables, manage subscriber flow processing, and/or create one or more forwarding tables for installation to PFEs324, among other functions. In this example, daemons314include command-line interface daemon332(“CLI332”), routing protocol daemon334(“RPD334”), and Simple Network Management Protocol daemon336(“SNMP336”). In this respect, control plane302may provide routing plane, service plane, and management plane functionality for single-chassis router212. Various instances of routing engine318may include additional daemons314not shown inFIG. 3that perform other control, management, or service plane functionality and/or drive and otherwise manage forwarding plane functionality for single-chassis router212.

Daemons314operate over and interact with kernel343, which provides a run-time operating environment for user-level processes. Kernel343may comprise, for example, a UNIX operating system derivative such as Linux or Berkeley Software Distribution (BSD). Kernel343offers libraries and drivers by which daemons314may interact with the underlying system. PFE interface316of kernel343comprises a kernel-level library by which daemons314and other user-level processes or user-level libraries may interact with programming interface364of PFE324A. PFE interface316may include, for example, a sockets library for communicating with PFE324A over dedicated network links.

Hardware environment350of routing engine318comprises microprocessor352that executes program instructions loaded into a main memory (not shown inFIG. 3) from storage (also not shown inFIG. 3) in order to execute the software stack, including both kernel343and user space306, of routing engine318. Microprocessor352may comprise one or more general- or special-purpose processors such as a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or any other equivalent logic device. Accordingly, the terms “processor” or “controller,” as used herein, may refer to any one or more of the foregoing structures or any other structure operable to perform techniques described herein.

RPD334executes one or more interior and/or exterior routing protocols to exchange routing information with other network devices and store received routing information in routing information base345(“RIB345”). For example, RPD334may execute protocols such as one or more of Border Gateway Protocol (BGP), including interior BGP (iBGP), exterior BGP (eBGP), multiprotocol BGP (MP-BGP), Label Distribution Protocol (LDP), and Resource Reservation Protocol with Traffic-Engineering Extensions (RSVP-TE). RPD334may additionally, or alternatively, execute User Datagram Protocol (UDP) to send and receive data for various system resources, such as physical interfaces. For example, RPD334may use UDP to send and receive data from path module327. Although illustrated with UDP, RPD334may execute any protocol to exchange data for system resources with path module327.

RIB345may include information defining a topology of a network, including one or more routing tables and/or link-state databases. RPD334resolves the topology defined by routing information in RIB345to select or determine one or more active routes through the network and then installs these routes to forwarding information base328A (“FIB328A”). Typically, RPD334generates FIB328A in the form of a radix or other lookup tree to map packet information (e.g., header information having destination information and/or a label stack) to next hops and ultimately to interface ports of interface cards associated with respective PFEs324. Kernel343may synchronize FIB328A of routing engine318with forwarding information348of PFE324A.

Command line interface daemon332(“CLI332”) provides a shell by which an administrator or other management entity may modify the configuration of single-chassis router212using text-based commands. SNMP336comprises an SNMP agent that receives SNMP commands from a management entity to set and retrieve configuration and management information for single-chassis router212. Using CLI332and SNMP336, for example, management entities may enable/disable and configure services, manage classifications and class of service for packet flows, install routes, enable/disable and configure rate limiters, configure traffic bearers for mobile networks, and configure interfaces, for example. RPD334, CLI332, and SNMP336in this example configure forwarding plane304via PFE interface316to implement configured services, and/or add/modify/delete routes. PFE interface316allows daemons314to drive the installation and configuration of packet processing path372of PFE324A. In particular, PFE interface316includes an application programming interface (API) by which daemons314may map packet flows to fabric interfaces for forwarding.

PFE324A, in combination with other PFEs324of single-chassis router212, implements forwarding plane304(also known as a “data plane”) functionality to handle packet processing from ingress interfaces on which packets are received to egress interfaces to which packets are sent. Forwarding plane304determines data packet forwarding through single-chassis router212, applies services, rate limits packet flows, filters packets, and otherwise processes the packets using service objects and lookup data installed by control plane302to forwarding plane304. AlthoughFIG. 3illustrates only PFE324A in detail, each of PFEs324of single-chassis router212comprises similar modules that perform substantially similar functionality.

PFE324A includes application-specific integrated circuit based packet processors (“ASICs368”) that execute processing path372in accordance with techniques described herein. ASICs368include one or more programmable application-specific integrated circuits having a key engine370that executes microcode (or “microinstructions”) to control and apply fixed hardware components of ASICs368to process packet “keys.” A packet key includes packet fields and other parameters that determine a flow of packet processing for the packet along an internal processing path, such as paths373. Key engine370includes key buffer363to store packet field data for corresponding packets that the key engine is currently processing. Key buffer363may also provide limited writable memory to which elements of the internal processing path may write to pass messages accessible by future elements. Some instances of ASICs368may include a plurality of key engines each having an associated key buffer.

Internal processing path372(“processing path372”) of ASICs368comprises programmable, executable microcode and fixed hardware components that determine the packet processing actions and other operations performed by key engine370. PFE324A may store executable instructions of processing path372in computer-readable storage media, such as static random access memory (SRAM). While illustrated within ASICs368, in some examples executable instructions of processing path372may be stored in memory external to ASICs368in PFE324A.

In some aspects, processing path372includes a next hop data structure to initiate processing. At the end of each processing step by key engine370, the result is a next hop that may specify additional processing or the termination of processing, for instance. In addition, next hops may specify one or more functions to be executed by key engine370and/or one or more hardware elements to be applied (e.g., policers). Key engine370may be associated with a result (or “lookup”) buffer (not shown) that stores results for executing next hops. For example, key engine370may execute a lookup specified by a list of next hops and store the result of the lookup to the associated result buffer. The contents of a result buffer may affect the actions of the next hop.

Logical interfaces371(“IFLs371”) is a table or other data structure that includes one or more logical interfaces. Each of IFLs371is an interface to a processing path of paths373. Paths373represents one or more processing paths for execution by key engine370on key buffer363.

PFE microprocessor360manages ASICs368and executes programming interface364to provide an interface for/to routing engine318. Programming interface364may comprise one or more user- or kernel-level libraries, programs, toolkits, application programming interfaces (APIs) and may communicate control and data messages to PFEs324via internal communication link (e.g., communication link230) using sockets, for example. PFE microprocessor360may execute a microkernel362to provide an operating environment for interfaces. Programming interface364receives messages from routing engine318directing packet forwarding engine324A to configure logical interfaces371.

In operation, a source PFE, e.g., source PFE324A, may receive an incoming packet to be communicated through the fabric. Key engine370of source PFE324A performs a lookup of forwarding information348to determine whether fabric path information associated with a particular destination PFE (e.g., egress PFE324E) is known. As one example, key engine370may perform a longest prefix match on the incoming packet to determine whether fabric path information associated with egress PFE324E is known. A successful match of the longest prefix provides source PFE324A next hop information to an egress PFE, whereas a failed match of the longest prefix indicates that fabric path information associated with egress PFE324E is not known and source PFE324A may initiate instructions to obtain fabric path information associated with egress PFE324E.

In response to a failed longest prefix match, source PFE324A initiates instructions to obtain fabric path information associated with egress PFE324E. For example, micro-kernel362of source PFE324A modifies a copy of the incoming packet to include a fabric path header before load balancing the modified packet. The fabric path header may provide an indication for a receiving PFE to send fabric path information associated with egress PFE324E to a path module327instead of forwarding the packet to the next fabric hop. In some examples, the fabric path header may include the hash value and information associated with source PFE324A. In some examples, micro-kernel362may modify an incoming packet determined to have a high packet rate. To determine high packet rate data flows, micro-kernel362may perform a shorter prefix match on the incoming packets and count the number of matches within a defined period of time. For example, for each match, micro-kernel362may store each match count into forwarding information348. If the number of matches exceeds a defined threshold packet rate, micro-kernel362may store the next incoming packet that matches the shorter prefix, e.g., as a hash value, in forwarding information348. Micro-kernel362may modify the next incoming packet to include a fabric path header. This way, PFE324A may add the fabric path header to a packet determined to have a high packet rate.

Source PFE324A then load balances the original packet and the modified packet towards the fabric. For example, micro-kernel362outputs the original packet and modified packet via the corresponding one of IFLs371towards a selected next-hop to a PFE (e.g., PFE324N) among a list of next hops to PFEs in paths373.

Non-egress PFE324N receives the original packet and modified packet and key engine370of non-egress PFE324N determines that the modified packet includes the fabric path header. Instead of forwarding the modified packet to egress PFE324E, non-egress PFE324N performs a lookup of the destination PFE (e.g., egress PFE324E), and may send the lookup information to path module327via micro-kernel362. In some examples, non-egress PFE324N may also send the hash value and information associated with the source PFE that was included in the fabric path header. In some examples, receiving PFE324N may send the fabric path information directly to source PFE324A. Non-egress PFE324N may forward the original packet to egress PFE324E for ultimate transmission over the network and may drop the modified packet.

Path module327pushes the fabric path information to source PFE324A. For example, path module327may send via kernel343a hash entry of a fabric path to egress PFE (“EGRESS INFO375”) that is stored in forwarding information348. The hash entry may be an entry with the longest prefix. Kernel343may also mark the outgoing interface of IFLs371towards egress PFE324E. In this way, when source PFE324A performs a lookup of the next incoming packet, source PFE324A may determine the fabric path to egress PFE324A is known for which key engine370may output the packet via the marked interface of IFLs371towards a next hop to egress PFE324E.

For example, in response to receiving the next incoming packet, key engine370of source PFE327A may perform the longest prefix match as described above. Responsive to a successful longest prefix match, PFE324A, based on the lookup information, sends the packet out of the marked interface of IFLs371directly towards egress PFE324E instead of load balancing the packet.

In some examples, path module327may monitor the availability of fabric path to egress PFE324E. Path module327may determine if a fabric path to egress PFE324E is unavailable, path module327may determine a fabric path to another egress PFE324. For example, forwarding information348may also store in forwarding information348the hash entry of the fabric path to egress PFE324E as a sample to be monitored. Each time an incoming packet is determined to have a high packet rate, micro-kernel362may send a copy of the hash value for the sample to path module327to determine if the fabric path to egress PFE is still available.

FIG. 4is a block diagram illustrating an example modified packet400, in accordance with the techniques described herein. In the example ofFIG. 4, packet400may include a destination MAC address field402, a source MAC address field404, a fabric path header field406, an Ethernet header field408, a type/length field410, and a payload412.

The destination MAC address field402may include the MAC address of the destination node (e.g., PE device28ofFIG. 1). The source MAC address field404may include the MAC address of the source node (e.g., CE device23ofFIG. 1). Ethernet header field408may be based on the Institute of Electrical and Electronics Engineers (IEEE) 802.1Q standard, for example. The type field410may include the type of connection as well as the length of packet400. The payload412may include the data of the packet.

The fabric path header field406may be based on the IEEE 802.1BR standard, for instance. In some examples, fabric header field406may be 16-bytes wide, wherein 2 bytes are reserved for the Ethernet Type, and the remaining 14 bytes may be used to include additional information414, such as a hash value416, and information associated with the source PFE418, as described above.

FIG. 5is a flowchart illustrating an example operation of network devices in accordance with one or more aspects of the disclosure.FIG. 4will be described for purposes of example with respect toFIGS. 1A-1B, 2, and 3. A source PFE24A of a virtual node, e.g., vPE20A ofFIG. 1, receives a packet (502). Source PFE24A determines whether fabric path information associated with an egress PFE is known (504). As described above, a source PFE24A may perform a longest prefix match to determine if fabric path information associated with egress PFE24E is known.

Responsive to a determination that the information associated with the egress PFE is known (“YES” branch of504), source PFE24A sends an incoming packet directly to egress PFE24E (522).

In response to a determination that the information associated with the egress PFE is not known (“NO” branch of504), source PFE24A sends, by load balancing, a modified copy of the packet that includes a fabric path header to the fabric (506). The fabric path header instructs a second virtual routing node to send fabric path information associated with the egress PFE to path module27instead of to the next fabric hop. As described above, the fabric path header may include the hash value and information associated with source PFE24A. Source PFE24A may load balance the original packet and the modified packet towards a next-hop to a PFE selected among a list of next hops to PFEs.

Receiving PFE24D receives the modified packet (508) and determines whether the packet includes the fabric path header (510). In response to a determination that the received packet includes the fabric path header (“YES” branch of510), receiving PFE24D may send lookup information including fabric path information to path module27(512). As described above, the fabric path information may include information associated with egress PFE24E, and the hash value and information associated with source PFE24A that was included in the fabric path header. In some examples, receiving PFE24D may send the fabric path information directly to source PFE24A. Alternatively, responsive to a determination that a received packet does not include the fabric path header (“NO” branch of510), receiving PFE24D forwards the received packet to the next fabric hop. Although not shown, receiving PFE24D may forward the original packet to egress PFE24E for ultimate transmission over the network and may drop the modified packet.

Path module27may receive the fabric path information from receiving PFE24D (516) and may send the fabric path information to source PFE24A (518). For example, path module27may send a hash entry for the fabric path to egress PFE24E. In some example, the hash entry for the fabric path to egress PFE24E may be a longest prefix.

Source PFE24A may receive, from path module27, the fabric path information associated with egress PFE24E (520). Source PFE24A may store the fabric path information associated with egress PFE24E in its forwarding information.

In response to receiving a next incoming packet, source PFE24A, as described above, determines whether fabric path information associated with an egress PFE is known (504) and upon a successful lookup of the fabric path to egress PFE24E, sends the incoming packet directly to egress PFE24E (522).