Patent Publication Number: US-7899930-B1

Title: Integration of an operative standalone router into a multi-chassis router

Description:
This is a divisional of U.S. application Ser. No. 11/217,017, filed Aug. 31, 2005, the entire content of which is incorporated herein by reference. 
    
    
     TECHNICAL FIELD 
     The invention relates to computer networks and, more particularly, to routing packets within computer networks. 
     BACKGROUND 
     A computer network is a collection of interconnected computing devices that can exchange data and share resources. In a packet-based network, such as an Ethernet network, the computing devices communicate data by dividing the data into small blocks called packets, which are individually routed across the network from a source device to a destination device. The destination device extracts the data from the packets and assembles the data into its original form. Dividing the data into packets enables the source device to resend only those individual packets that may be lost during transmission. 
     Certain devices, referred to as routers, maintain tables of routing information that describe routes through the network. A “route” can generally be defined as a path between two locations on the network. Upon receiving an incoming data packet, the router examines destination information within the packet to identify the destination for the packet. Based on the destination, the router forwards the packet in accordance with a routing table. 
     Service providers, for example an Internet service provider, must meet increasing bandwidth demands. This requires service providers to continuously add, replace or upgrade routing equipment within their networks. The relatively short deployable lifetime of routing equipment causes increased capital expenditure and investment. 
     SUMMARY 
     In general, the invention is directed to techniques for transition of an operative standalone router into a multi-chassis router with reduced interruption of packet-forwarding functions. A standalone router typically includes a plurality of interface cards to receive packets and send packets and a standalone switch fabric with a plurality of switch planes that enables packet forwarding between the plurality of interface cards of the standalone router. The techniques described herein allow the switch fabric of the standalone router to be incrementally transitioned to switch fabric capable of integration with a multi-chassis router. 
     In particular, standalone switch fabric of the operative standalone router is transitioned to switch fabric capable of operating as a component within a multi-stage switch fabric of a multi-chassis router. For example, the standalone router includes a set of switch cards that provide the switch planes for routing data packets. The standalone switch cards are incrementally replaced without-chassis switch cards that provide components of the multi-stage switch fabric. Each standalone switch card can be independently replaced and tested. Multi-chassis switch cards may be coupled to other switch cards in the multi-chassis router to form independent switch planes. Each independent switch plane may be tested without loss of forwarding functionality of the router. When the switch cards have been completely transitioned, the standalone router is then configured and rebooted to operate as a line card chassis (LCC) as part of the multi-chassis router. In this way, an operative standalone router can be smoothly integrated into a multi-chassis router in which the router functions as one of a plurality of cooperating routing components operating as a single routing node. 
     The techniques may provide one or more advantages. For example, the techniques allow the standalone router to continue to actively forward network packets during a transition into a component of a larger multi-chassis router, thereby increasing network availability and reducing downtime. Furthermore, the techniques also allow installed multi-stage switch fabric to be tested during the transition. For example, the packet forwarding ability of LCC switch cards to a switch card chassis may be tested using active switch card chassis (SCC) switch cards connected to only a single LCC. In the event that a test is not completed successfully, the invention provides an ability to gracefully back out of the transition into a multi-chassis router without a disruption to network traffic. These and other benefits of the invention will become apparent from the description, drawings and claims. 
     In one embodiment, a method comprises transitioning an operative standalone router into a component of a multi-chassis router by incrementally replacing standalone switch cards within the standalone router with multi-chassis switch cards. 
     In a second embodiment, a method comprises transitioning an operative standalone router into a component of a multi-chassis router while continuing to operate the standalone router in a standalone mode. 
     In another embodiment, a standalone router comprises an operative standalone switch fabric and an operative multi-chassis switch fabric. A multi-chassis switch plane including the operative multi-chassis switch fabric also includes a switch fabric contained in another chassis of a multi-chassis router. 
     In another embodiment, a system comprises a router operating in a standalone mode and coupled to a switch card chassis of a multi-chassis router. The router operates in the standalone mode to forward traffic to the switch card chassis and the switch card chassis forwards the traffic back to the router. 
     In one more embodiment, a router comprises a first switch fabric and a second switch fabric. The first switch fabric forms one or more standalone switch planes. The second switch fabric combines with a third switch fabric to form one or more additional switch planes. The third switch fabric is external to the router. 
     The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the invention will be apparent from the description and drawings, and from the claims. 
    
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         FIG. 1  is an example system comprising a service-provider network, illustrating a standalone router transitioned to a multi-chassis router in accordance with the principles of the invention. 
         FIG. 2  is a block diagram illustrating a standalone router. 
         FIG. 3  is a block diagram illustrating a four-by-four crossbar switch. 
         FIG. 4  is a block diagram illustrating a multi-chassis router. 
         FIG. 5  is a block diagram illustrating a multi-stage network. 
         FIG. 6  is a block diagram illustrating flow-paths of data packets during incremental transition from a standalone-chassis node to a multi-chassis router according to an embodiment of the invention. 
         FIG. 7  is a flowchart illustrating an example method of incremental transition from a standalone-chassis node to a multi-chassis router according to an embodiment of the invention. 
     
    
    
     DETAILED DESCRIPTION 
       FIG. 1  is a block diagram illustrating an example system  2  in which core router  4  of service provider network  6  is transitioned to a multi-chassis router  4 ′ in accordance with the principles of the invention. In this example, core router  4  communicates with edge routers  5 A and  5 B (“edge routers  5 ”) to provide customer networks  8 A- 8 C (“customer networks  8 ”) with access to service provider network  6 . 
     Although not illustrated, service provider network  6  may be coupled to one or more networks administered by other providers, and may thus form part of a large-scale public network infrastructure, e.g., the Internet. Consequently, customer networks  8  may be viewed as edge networks of the Internet. Service provider network  6  may provide computing devices within customer networks  8  with access to the Internet, and may allow the computing devices within the customer networks to communicate with each other. Service provider network  6  may include a variety of network devices other than core router  4  and edge routers  5 , such as additional routers, switches, servers, and the like. 
     In the illustrated embodiment, edge router  5 A is coupled to customer network  8 A via access link  9 A and edge router  5 B is coupled to customer networks  8 B and  8 C via access links  9 B and  9 C, respectively. Customer networks  8  may be networks for geographically separated sites of an enterprise. Each of customer networks  8  may include one or more computing devices (not shown), such as personal computers, laptop computers, handheld computers, workstations, servers, switches, printers, and the like. The configuration of network  2  illustrated in  FIG. 1  is merely exemplary. For example, an enterprise may include any number of customer networks  8 . Nonetheless, for ease of description, only customer networks  8 A- 8 C are illustrated in  FIG. 1 . 
     In this example, core router  4  is a standalone router, which is transitioned to a multi-chassis router  4 ′. As part of this transition, the standalone switch fabric of core router  4  may be incrementally augmented or replaced with switch fabric that may operate in multi-chassis router  4 ′. During the transition, the switch fabric in the standalone router  4  is connected to the switch fabric of an SCC, which provides central control of multi-chassis router  4 ′. At this time, the SCC switch fabric returns data packets forwarded by the replaced switch fabric to standalone router  4 . In this way standalone router  4  continues to forward data packets in service provider network  6  during the transition to multi-chassis router  4 ′. Once the entire switch fabric in routing node  4  has been incrementally transitioned, routing node  4  is rebooted to operate as part of multi-chassis router  4 ′, which includes the SCC. 
     While the techniques of transitioning a standalone router to a multi-chassis router have been described with respect to core router  4 , the techniques may be applied to upgrade any routing node. For example, edge routers  5  and/or customer networks  8  may also comprise standalone routers that may be upgraded to multi-chassis routers. 
       FIG. 2  is a block diagram illustrating an example standalone routing mode  10  that may be transitioned to serve as a routing component within a multi-chassis router. In this example, standalone routing mode  10  includes a control unit  12  that includes a routing engine  16 . Standalone router  10  also includes packet forwarding engines  20 A through  20 N (“PFEs  20 ”) and a switch fabric  18 . PFEs  20  may receive and send data packets via interface cards  21 A through  21 N (“IFCs  21 ”) and IFCs  22 A through  22 N (“IFCs  22 ”). In other embodiments, each of PFEs  20  may comprise more or fewer IFCs. Switch fabric  18  forwards incoming data packets to the correct one of PFEs  20  for transmission over a network, e.g., the Internet. 
     Routing engine  16  maintains routing tables, executes routing protocol and controls user access to standalone routing mode  10 . Routing engine  16  is connected to each of PFEs  20  by a dedicated link  24 . For example, dedicated link  24  may comprise a 100 Mbps Ethernet connection. Routing engine  16  maintains routing information that describes a topology of a network, and derives a forwarding information base (FIB) in accordance with the routing information. Routing engine  16  copies the FIB to each of PFEs  20 . This allows the FIB in each of PFEs  20  to be updated without degrading packet forwarding performance of standalone router  10 . Alternatively, routing engine  16  may drive separate FIBs which are copied to respective PFEs  20 . 
     In a routing node, a “switch plane” is generally capable of providing a communication path between any two PFEs  20 . In this example, switch fabric  18  consists of multiple standalone switch planes  19 A through  19 M (“switch planes  19 ”). In some embodiments, each of switch planes  19  is provided by a switch fabric on a separate, removable switch card. Other routing nodes may comprise additional or fewer switch planes. A majority of the switch planes, e.g., switch planes  19 A- 19 D, may be active at any given time with data packets distributed over the active switch planes. The inactive switch plane(s) of switch fabric  18  serves as back-up(s) such that if one or more of the active switch planes goes offline, the back-up(s) automatically activate, and the bandwidth capacity of standalone router  10  is not diminished. 
     As part of a standalone router, switch planes  19  form a standalone switch fabric. That is, each of switch planes  19  is capable of providing a connection between any of PFEs  20  within standalone routing mode  10 . Standalone routing mode  10  may be transitioned to serve as part of a multi-chassis router. In that case, switch planes  19  would be incrementally replaced with switch planes spanning a multi-chassis router, which would only perform a part of the switching required to connect any two PFEs  20 . 
     An example flow-path of data packets through standalone routing mode  10  is as follows. Initially, an incoming data packet is received by one of IFCs  21  or IFCs  22 , e.g., IFC  22 A. The IFC is coupled to one of PFEs  20 . In this example, PFE  20 A segments the data packet into sixty-four-byte data cells. The data cells are written into ingress memory. PFE  20 A then performs a forwarding lookup, determines a destination for the packet, and transfers the cells of the packet across the active switch planes. Each of the active switch planes forward the cells to the egress PFE. When the data cells arrive at the egress PFE, e.g., PFE  20 N, they are written into egress memory and reassembled into the original packet. The data packet is then transmitted into the network (not shown) via one of IFCs  21  or  22 , e.g., IFC  21 N. 
     By dividing the data packet into cells and evenly transmitting the packet on a cell-by-cell basis across the switch planes, a PFE guarantees that traffic is load-balanced across each of the active switch planes. In standalone routing mode  10 , the back-up switch plane(s) may be identical to the active switch planes and act as hot spare(s) to maintain bandwidth capacity in the event that one or more of the active switch planes fail. Each of switch planes  19  is operationally independent; therefore, standalone routing mode  10  may continue to forward packets as long as at least one of switch planes  19  remain active, but possibly at a reduced bandwidth capacity. 
     In this manner, switch planes  19  form a standalone switch fabric that enables packet forwarding between the plurality of PFEs  20  of standalone routing mode  10 . The techniques described herein allow switch planes  19  of the standalone router to be incrementally transitioned to switch planes capable of integration with a multi-chassis router. In particular, switch fabric  18  of the standalone routing mode  10  is transitioned to a switch fabric capable of operating as a component within a multi-stage switch fabric of a multi-chassis router, e.g., multi-chassis router  4 ′. For example, switch fabric  18  may be provided by a set of removable switch cards, where each removable switch card provides a respective one of switch planes  19 . In accordance with the principles of the invention, the switch cards are incrementally replaced with multi-chassis switch cards that provide components of the multi-stage switch fabric. Each switch card can be independently replaced, coupled to the multi-stage switch fabric of the multi-chassis router and tested without loss of forwarding functionality of the router. When the switch fabric has been completely transitioned, standalone router  10  may then be configured and rebooted to operate as an LCC in the larger multi-chassis router. In this way, an operative standalone router  10  can be integrated into a multi-chassis system in which the router functions as one of a plurality of cooperating routing components operating as a single routing node. 
       FIG. 3  is a block diagram illustrating an exemplary standalone switch plane, e.g., switch planes  19  ( FIG. 2 ), which may be transitioned to a multi-stage switch-plane capable of operating within a multi-chassis system. In particular,  FIG. 3  illustrates a four-by-four crossbar switch  30 . Each of inputs  36 A,  36 B,  36 C and  36 D (collectively “inputs  36 ”) of crossbar switch  30  has a crosspoint with each of outputs  38 A,  38 B,  38 C and  38 D (collectively “outputs  38 ”). Crossbar switch  30  is a four-by-four crossbar switch because it contains exactly four inputs  36  and exactly four outputs  38 . Other crossbar switches may contain more or less inputs and outputs. For example, each of switch planes  19  illustrated in  FIG. 2  may comprise a single sixteen-by-sixteen crossbar switch. 
     Crossbar switch  30  comprises sixteen crosspoints, one at every point where a horizontal input path intersects a vertical output path. In crossbar switch  30 , crosspoints  32  and  34  are activated, i.e., they are providing a connection between a horizontal input path and a vertical output path. Crossbar switch  30  is strictly non-blocking, in that it is always possible to establish a connecting path between any of inputs  36  any of outputs  38 , regardless of existing connections across the switch. For example, active crosspoint  34  provides a direct connection for data transmission between input  36 D and output  38 B. Concurrently, active crosspoint  32  provides a direct connection for data transmission between input  36 A and output  38 A without interfering with the connection between input  36 D and output  38 B. 
       FIG. 4  is a block diagram illustrating an example multi-chassis router  40 . In this example, multi-chassis router  40  comprises four substantially identical LCCs  48 A,  48 B,  48 C and  48 D (collectively “LCCs  48 ”) and an SCC  42 . For examples one of LCCs  48  may have been transitioned from an operative standalone router into an LCC or may have been initially coupled to SCC  42  prior to operation. Each of LCCs  48  may be contained within a physically separate chassis and may include a routing engine  51 A,  51 B,  51 C or  51 D (collectively “routing engines  51 ”), switch fabric  52 A,  52 B,  52 C or  52 D (collectively “switch fabrics  52 ”), a PFE set  50 A,  50 B,  50 C or  50 D (collectively “PFEs  50 ”), and a set of network interfaces  54 A,  54 B,  54 C or  54 D (collectively “network interfaces  54 ”). SCC  42  comprises a centralized routing engine  46  connected to each of routing engine  51  via links  56 A,  56 B,  56 C and  56 D (collectively “links  56 ”). 
     LCCs  48  are substantially similar to standalone routing mode  10  ( FIG. 2 ), except that switch fabric  18  ( FIG. 2 ), which comprises switch planes  19 , has been replaced by one of switch fabrics  52 , each consisting of multiple multi-chassis switch cards. In standalone routing mode  10 , each of switch planes  19  contains a standalone switch plane; while in multi-chassis router  40 , each switch plane may be viewed as a multi-stage switch-plane that is distributed over a multi-chassis switch card from each of LCC switch fabrics  52  and an SCC switch card from switch fabric  44 . 
     In other words, each multi-chassis switch card in switch fabrics  52  performs the beginning and final stages in a multi-stage network (see, e.g.,  FIG. 7 ). E.g., multi-chassis cards in switch fabrics  52  may perform the first and third stages, while switch cards in switch fabric  44  perform the second stage of a three-stage network. Together, one multi-chassis switch card from each of LCCs  48  and one SCC switch card form a switch plane. For example, SCC switch fabric  44  and LCC switch fabrics  52  may form a total of five independent switch planes. In this manner, multi-chassis router  40  may comprise five independent switch planes, each one providing a multi-stage switched interconnect for forwarding packet cells between PFEs  50 . As with switch fabric  18  ( FIG. 2 ), multi-chassis router  40  may consist of multiple active switch planes and additional redundant switch plane(s) to serve as hot spares. For example, multi-chassis router  40  may consist of four active switch planes and one redundant switch plane. 
     In this embodiment, each multi-chassis switch card in LCC switch fabrics  52  is connected to a single switch card in SCC switch fabric  44 . A separate link connects each SCC switch card to each multi-chassis switch card. For example, links  56 A may consist of five fiber-optic array cables. If multi-chassis router  40  has five switch planes, a total of twenty cables may be used to connect switch fabrics  52  with switch fabric  44 . 
     Each PFE set  50  can include multiple PFEs. For example each PFE set  50  may include eight PFEs. Each PFE in PFE sets  50  may connect to one or more network interfaces. For example, a PFE may send and receive information packets for two network interfaces, e.g., IFCs  21 A and  22 A from  FIG. 2 . An LCC  48  containing sixteen PFEs with two network interfaces per PFE would have a total of thirty-two network interfaces. Because multi-chassis router  40  comprises four LCCs, in this example, multi-chassis router  40  would have a total of one-hundred-twenty-eight network interfaces. 
     Routing engine  46  maintains routing information that describes a topology of a network, and derives a FIB in accordance with the routing information. 
     Routing engine  46  copies the FIB to each routing engines  51  via links  58 A,  58 B,  58 C and  58 D (collectively “links  58 ”). For example, links  58  may comprise Unshielded Twisted Pair (UTP) Category 5 Ethernet cables, each less than one-hundred meters in length. Routing engines  51  then copy the FIB to PFEs  20  on their chassis. An ingress PFE (any of PFEs  50 ) uses the FIB to direct packets arriving from the network (not shown) to the proper egress PFE (also any PFE from PFEs  50 ). The egress PFE relays the packets to the network. The FIB in each of PFEs  50  may be updated without degrading packet forwarding performance of multi-chassis router  40  because FIB updates are distributed separately from the data packets, which use links  56 . Alternatively, routing engine  46  may derive separate FIBs which are copied to respective PFEs  20 . 
     Each of PFEs  50  is connected to all five switch planes of switch fabrics  52  and  44 . Each active switch plane is responsible for providing a portion of the required bandwidth of multi-chassis router  40 . By dividing the data packet into cells and evenly transmitting the packet on a cell-by-cell basis over the active switch planes, PFEs  50  load-balance traffic across the active switch planes of switch fabrics  52 . 
     The flow path of a data packet through multi-chassis router  40  is similar to the flow path through standalone routing mode  10  of  FIG. 2 . An incoming data packet is received by a network interface (not shown) by a PFE in PFEs  50 , i.e., the ingress PFE. The ingress PFE segments the data packet into sixty-four-byte data cells. The data cells are written into ingress memory, and the ingress PFE performs a FIB lookup. Using an active route from the FIB provided by routing engine  46 , the ingress one of PFEs  50  transfers the to an egress PFE using switch fabrics  52  and switch fabric  44 . When the data cells arrive at an egress one of PFEs  50 , they are written into egress memory and reassembled into the original packet. The reassembled packet is then transmitted via the corresponding one of network interfaces  54 . 
     The techniques described herein allow switch planes  19  of standalone routing mode  10  ( FIG. 2 ) to be incrementally transitioned to switch planes of switch fabrics  52  capable of integration with multi-chassis router  40 . In accordance with the principles of the invention, switch cards of standalone router  10  are incrementally replaced with multi-chassis switch cards that provide switch fabrics  52 . In this way, an operative standalone router  10  can be integrated into multi-chassis router  40  in which router  10  functions as one of LCCs  48 . The configuration shown in multi-chassis router  40  is merely exemplary. For example, PFEs  50  may communicate directly with SCC routing engine  46 , rather than communicating via LCC routing engines  51 . In this manner, a routing engine in a standalone router may be turned off once the standalone router is transitioned to an LCC as part of multi-chassis router  40 . Other embodiments may not require SCC or any other centralized switching or control. For example, standalone switch cards in a standalone router may be transitioned with switch cards capable of switching directly between one or more routing devices, without having to use a central switch fabric. These and other numerous router configurations are included in embodiments of the invention. 
       FIG. 5  is a block diagram illustrating a three-stage network  70  that operates as a three-stage switch plane. For example, a switch plane including an LCC multi-chassis switch card from each of LCCs  48  and an SCC switch card from SCC  42  in multi-chassis router  40  of  FIG. 4  may be used to form three-stage network  70 . For example, multi-chassis router  40  ( FIG. 4 ) may comprise five switch planes, each comprising a single three-stage network. Network  70  has three stages: stage 1 consisting of crossbar switches  72 A- 72 N (collectively “switches  72 ”), stage 2 consisting of crossbar switches  74 A- 74 N (collectively “switches  74 ”), and stage 3 consisting of crossbar switches  76 A- 76 N (collectively “switches  76 ”). Switches  72  receive data packets via inputs  78 A- 78 N (collectively “inputs  78 ”). Switches  76  relay the data packets via outputs  80 A- 80 N (collectively “outputs  80 ”). 
     Assume there are sixteen inputs and sixteen outputs on each of switches  72 ,  74  and  76 . Also for example, let N equal four. Then each stage consists of exactly four sixteen by sixteen crossbar switches, which are larger versions of crossbar switch  30  in  FIG. 3 . Because each of switches  72  has sixteen inputs, there are a total of sixty-four inputs in network  70 . Similarly, each of switches  76  in stage 3 has sixteen outputs for a total of sixty-four outputs. 
     For example, maintain the assumption that there are sixteen inputs and sixteen outputs on each of switches  72 ,  74  and  76  and that N equals four. The outputs of switches  72  are connected to the inputs of switches  74 . Each of switches  72  has four outputs connected to four inputs on each of switches  74 . Likewise, each of switches  74  has four outputs connected to four inputs of each of switches  76 . These redundant connections between each crossbar switch create additional paths through network  70  which may be required to make network  70  strictly non-blocking. 
     To establish a path through network  70 , from an input to an output, an ingress PFE, e.g., PFE  50 A ( FIG. 4 ) determines an available first-stage switch  72  that allows a connection to the egress PFE. For example, the ingress PFE may select switch  72 A. Switch  72 A then determines an available second stage switch which allows a connection to the egress PFE. For example, switch  72 A may select switch  74 B. Switch  74 B then determines an available third-stage switch which allows a connection to the egress PFE. For example, switch  74 B may select switch  76 A. Switch  76 A then determines an available path to the egress PFE. Each switch determines an available path for the packet on-the-fly. In this manner, a data packet received by a switch  72  in stage 1 may go through any of switches  74  in stage 2 and continue on to the required switch  76  in stage 3 before being forwarded by an egress PFE (not shown) across a network. 
     While the switch planes in multi-chassis router  40  ( FIG. 4 ) are described as containing three-stage networks, the switch planes may contain different switch architecture. For example, the second stage in a three-stage network may be replaced with a three-stage network, thereby forming a five-stage network. In accordance with the described techniques, any of the switch plane components may be independently transitioned and tested without substantially compromising forwarding functionality. 
       FIG. 6  is a block diagram illustrating an example standalone router  102  during incremental transition of standalone switch fabric in a standalone router to multi-stage switch fabric for a multi-chassis router according to an embodiment of the invention. In particular,  FIG. 6  illustrates switch plane path  100  after a first switch plane has been transitioned from a standalone switch fabric I 0  (not shown) to a multi-stage switch fabric M 0  provided by multi-chassis switch card  116 . As described, during the transition process, switch plane path  100  may be viewed as a hybrid of flow through standalone router  10  of  FIG. 2  and the flow path through the multi-chassis router  40  of  FIG. 4 . 
     Standalone router  102  may be the same as standalone routing mode  10  in  FIG. 2  and SCC  104  may be the same as SCC  42  in  FIG. 4 . Standalone router  102  includes a network interface (not shown), ingress PFE  110 , switch planes  114 A,  114 B,  114 C and  114 D (collectively “switch planes  114 ”), a multi-chassis switch card  116 , and egress PFE  112 . SCC  104  includes switch cards  109 A,  109 B,  109 C,  109 D and  109 E (collectively “switch cards  109 ”). While switch plane path  100  only depicts a single ingress PFE  110  and a single egress PFE  112 , standalone router  102  comprises additional PFEs (not shown). Each PFE in standalone router  102  acts both as an ingress PFE and an egress PFE. In switch plane path  100  the distinction is made between ingress PFE  100  and egress PFE  112  is made for illustrative purpose only. 
     Each of standalone switch planes  114  may comprise identical architecture, and may be embodied on a respective switch card. In contrast, multi-chassis switch card  116  contains a first-stage switch  106  and a third-stage switch  108  as part of a three-stage switch fabric. Switch card  109 A contains the second-stage of a three-stage switch fabric. For example, the three-stage switch fabric may comprise a three-stage network  70  ( FIG. 5 ). 
     In this example, multi-chassis switch card  116  has been inserted in router  102  and connected to switch card  109 A via a single cable. While  FIG. 6  shows just one standalone router  102 , which is in the process of being converted to an LCC, the second-stage switches on switch cards  109  may provide a connection between multiple, operative LCCs. For example, SCC  104  may be capable of connecting four LCCs, as in SCC  42  ( FIG. 4 ). 
     In the example of  FIG. 6 , there are five available switch plane paths. During the incremental transition stage shown in  FIG. 6 , the first four switch plane paths are provided by standalone switch planes  114 . The fifth switch plane path is provided by multi-chassis switch card  116  and switch card  109 A. Four of the five available paths are active and the fifth acts as a hot spare to maintain bandwidth capacity in the event that one of the other switch plane paths fails. 
     Assume for example, an incoming data packet is received by a network interface (not shown) of standalone router  102 , and that the network interface is coupled to ingress PFE  110 . PFE  110  segments the data packet into sixty-four-byte data cells. PFE  110  writes the data cells into ingress memory, performs a FIB lookup) and distributes the cells across the four active switch plane paths. 
     Assume, for example, multi-chassis switch card  116  has been activated and that switch plane  114 A has been rendered inactive, i.e., designated as the current spare. Switch planes  114 B,  114 C and  114 D form three of the four active switch planes and direct incoming cells directly to egress PFE  112 . The fourth active switch plane is formed by multi-chassis switch card  116  and SCC switch card  109 A. The data cells traveling to multi-chassis switch card  116  are sent to multi-chassis switch card  116  and first-stage switch  106 , which forward the cells to switch card  109 A in SCC  104 . Switch card  109 A directs the cells to return to multi-chassis switch card  116  according to switch plane path  100 . Then, third-stage switch  108  directs the cells to egress PFE  112 . When the data cells arrive at PFE  112  from switch planes  114 B,  114 C,  114 D and multi-chassis switch card  116  they are written into egress memory and reassembled into the original packet. The data packet is then transmitted via a network interface (not shown) coupled to egress PFE  112 . 
     In this manner, the techniques can be used to incrementally transition an operative standalone router, for example standalone routing mode  10  of  FIG. 2 , into a multi-chassis router, for example multi-chassis router  40  of  FIG. 4 . As described,  FIG. 6  illustrates a stage in the transition an administrator has replaced a single switch plane, i.e., I 0 , with a multi-chassis switch card, i.e., multi-chassis switch card  116 , M 0 . During the future stages of incremental transition, as described in the description of  FIG. 7 , the administrator replaces each of switch planes  114  with a multi-chassis switch card, connects each multi-chassis switch card to SCC switch cards S 1 -S 4  to form multi-stage switch planes and tests the operation of the multi-chassis switch cards and multi-stage switch planes without loss of forwarding capabilities. For example, in the next step of the incremental transition, the administrator may replace switch plane  114 A, I 1 , with a new multi-chassis switch card, M 1  (not shown). The administrator then connects M 1  to SCC switch card  109 B, S 1 , to form a new switch plane. That multi-stage switch plane will replace the packet forwarding capabilities of standalone switch plane  114 A, I 1 . The incremental transition continues until the administrator replaces each of switch planes  114 , i.e., I 0 , I 1 , I 2 , I 3  and I 4  with multi-chassis switch cards M 0 , M 1 , M 2 , M 3  and M 4  and connects them to switch cards  119 , i.e., S 0 , S 1 , S 2 , S 3  and S 4 . A method of incremental transition according to an embodiment of the invention is described in greater detail in the description of  FIG. 7 . 
     Data cells that travel along path  100  may take longer to arrive at egress PFE  112  than the cells that travel along the other switch plane paths. Therefore, egress PFE  112  may buffer data cells from the other paths in memory long enough to receive the cells traveling over switch plane path  100 . Egress PFE  112 , and all PFEs, which each act as egress PFEs, may have additional egress memory to buffer the cells. 
     However, alternative techniques may be used to allow the egress PFEs to use less egress memory. For example, standalone switch cards  114  can be designed or programmed to have a delay approximately equal to the delay caused by the longer path  100 . An added delay would be insignificant relative to the time it takes packets to travel from a source device over a network, e.g., the Internet, to a destination device. If either programmed or part of the physical design of switch cards  114 , the delay results in all data cells of a data packet arriving at egress PFE  112  at approximately the same time. A delay can be added in any place along data cell paths including standalone switch cards  114 . For example, the delay may be an extended physical path, e.g., a coiled wire, between PFEs and switch cards  114  within router  102 . For example, in the case of an extended physical path, implementing the delay could be as simple as flipping a switch to connect an extended path. 
     The delay for cells traveling via path  100  relative to cells traveling via standalone switch cards  114  is dependent on the time it take for data cells to travels path  100 . E.g. if the connection between multi-chassis switch card  116  and switch card  109 A required one or more cables, the length of the cables would affect the delay. A programmable or physical delay added to data cell paths through standalone switch planes  114  can be adjustable to account for an actual or calculated delay over path  100 . The delay could be instituted during a scheduled service outage prior to an incremental transition of switch cards in router  102 . It is also possible for data cell paths including standalone switch cards  114  to always include a delay in anticipation of an incremental transition. However implemented, including a delay to data cell paths through standalone switch planes  114  could reduce PFE resources, in particular egress memory, required during an incremental transition. 
     In the illustrated example of  FIG. 6 , multi-chassis switch card  116 , M 0 , combines with SCC switch card  109 A, S 0 , to form a switch plane. In other embodiments SCC switch card  109 A may comprise the entire switch plane. For example, SCC switch cards  109  may each comprise a sixty-four-by-sixty-four crossbar switch. In other embodiments, SCC switch cards  109  may perform every stage in a network. In such embodiments, the multi-chassis switch cards, e.g., multi-chassis switch card  116 , M 0 , simply provide a constant path between an SCC switch plane, e.g., SCC switch card  109 A, and every PFE in router  102 . 
       FIG. 7  is a flowchart illustrating a method of incremental transition from a standalone router to an LCC as part of a multi-chassis router according to an embodiment of the invention. For exemplary purposes, the method is described with reference to  FIG. 6 . 
     Initially, an administrator initializes and configures SCC  104  as a central control node for a multi-chassis router ( 142 ). For example, the administrator configures SCC  104  with five switch cards  109 . For reference, these switch cards are labeled S 0 , S 1 , S 2 , S 3  and S 4 . Setting up SCC  104  with switch cards  109  may include assembling SCC  104 , powering on SCC  104 , connecting SCC  104  to a management network, installing software on SCC  104 , performing diagnostics on the switch fabric, i.e., switch cards  109 , of SCC  104  and setting control board switches to receive multi-chassis switch card connections. If the administrator performs diagnostics on the switch fabric of SCC  104 , this may include testing for one or more ports by offlining a port, inserting a loopback connector and onlining the port. 
     Next the administrator replaces switch plane I 0  with multi-chassis switch card  116  on standalone router  102  ( 144 ). In this step, the administrator takes offline the standby switch card, e.g., I 0 , in standalone router  102  and physically replaces the switch card with multi-chassis switch card  116 . 
     Following the replacement of I 0  with M 0 , the administrator connects multi-chassis switch card  116 , M 0 , with SCC switch card  109 A, S 0  ( 150 ). For example, multi-chassis switch card  116  may be connected to a port on SCC  104  with a dedicated fiber-optic array cable less than one-hundred meters in length. 
     The administrator then issues software commands, first at standalone router  102 , then at SCC  104 , to start inter-chassis links between M 0  and S 0  ( 152 ). The software command at standalone router  102  may be entered at a command line interface (CLI) to turn on the receive end at standalone router  102 . Likewise, the software command at SCC  104  may be entered at a CLI to turn on the receive end at SCC  104 . 
     Next, the administrator repeats the process of steps  155 ,  156 ,  144 ,  150  and  154  for each switch card still to be replaced with a multi-chassis switch card without interruption of forwarding capabilities ( 154 ). For example, to begin the next iteration, the administrator takes switch plane  114 A, I 1 , on standalone router  102  offline ( 156 ). As a result, the spare switch plane, in this case the combination of M 0  and S 0 , automatically becomes active. 
     At this time, network traffic from standalone router  102  to SCC  104  is returned to the standalone router as illustrated in  FIG. 6 . Moreover, standalone router  102  continues to operate as a standalone router even though traffic flows through SCC  104 . While traffic flows in this manner, the administrator is able to confirm correct operation of both M 0  of router  102  and S 0  of SCC  104 . For example, verification may be performed by seeing illuminated light emitting diodes (LEDs) corresponding to the multi-chassis switch card inserted in step  144  and the SCC switch card connected in step  150 . In the event that verification of a multi-chassis switch plane fails, e.g., the combination of M 0  and S 0 , the method of incremental transition may be aborted without any interruption in traffic across router  102 . To abort incremental transition  140 , the multi-chassis switch card inserted in step  144 , e.g., M 0 , may be replaced with the original switch card removed in step  144 , e.g., I 0 . 
     Steps  144 ,  150 ,  152  and  156  are repeated until each of the five switch planes  114  I 0 , I 1 , I 2 , I 3  and I 4  in standalone router  102  have been replaced with multi-chassis switch cards M 0 , M 1 , M 2 , M 3  and M 4  and connected to SCC switch cards  109 : S 0 , S 1 , S 2 , S 3  and S 4  ( 154 ). At this point in the incremental transition, standalone router  102  is physically ready for incorporation within a multi-chassis router, but is still operating as a standalone router. 
     Once the administrator is finished replacing each of switch planes  114  ( 154 ), the administrator physically connects the routing engine (not shown) of SCC  104  to standalone router  102  ( 158 ). For example, the routing engine of SCC  104  may be connected to standalone router  102  via a category-5 Ethernet cable. 
     The administrator completes the incremental transition by issuing one or more commands to set standalone router  102  to operate in “LCC mode”. This may require rebooting the router ( 160 ). Once set to operate in LCC mode, what was formerly standalone router  102  is now an LCC integrated with SCC  104  as part of a multi-chassis router. As incremental transition is performed, standalone router  102  continues to forward traffic until it is rebooted in step  160 . When rebooting standalone router  102 , routing service will be interrupted at the routing node. Thus, this short period is the only time during the incremental transition that routing service is degraded or interrupted. Once standalone router  102  has been set to operate in LCC mode, operation of the system, including routing engines, switching planes and interfaces can be verified with CLI commands at the SCC. Standalone router  102 , now functioning as an LCC, and SCC  104  are elements in a multi-chassis router and operate as a single routing node 
     A number of embodiments of the present invention have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the invention. As one example, described techniques may be performed remotely. For example, a remote administrate device may provide software for a standalone router to operate an LCC. Furthermore, a remote device may be used to provide the FIB to each device a multi-chassis router. A multi-chassis router may or may not include a central switch fabric, and could using switch fabric in one or more LCCs to provide connections between the multiple devices of a multi-chassis router. Such a router may or may not include a centralized control node; e.g., LCCs may share control functions in a multi-chassis router. A standalone router may be incrementally transitioned to operate in conjunction with one or more other network devices without a centralized switch fabric or SCC. In one instance, two chassis may be coupled to operate as a single router, which each chassis including network interfaces. Accordingly, these and other embodiments are within the scope of the following claims.