Method and apparatus for optimization of redundant link usage in a multi-shelf network element

A method and apparatus for optimizing redundant link usage allows a portion of the wasted bandwidth in a redundant link system to be utilized for additional data traffic without compromising the ability of the system to respond to and correct for a failure of a link. In one embodiment, one of two independent, individually addressable links is selected as a nominal communication path, and the other as a standby communication path. The path independent traffic is sent via the nominal communication path, while the path dependent data is sent via the nominal communication path or the standby path, in accordance with the dependence of the traffic. In another embodiment, critical time sensitive traffic is sent via a nominal and standby time sensitive paths, while normal traffic is sent via nominal normal path, and non-critical traffic is sent via a standby normal path.

FIELD OF THE INVENTION

The present invention relates to the field of network elements, and more particularly to a method and apparatus for optimization of redundant link usage in a multi-shelf network element.

BACKGROUND OF THE INVENTION

An example of a multi-shelf network element with which the present invention may be used is multi-shelf switch system100shown in schematic block form inFIG. 1. Switch system100comprises peripheral shelves110a-n, switching shelves120a-b, and control complex130. Peripheral shelves110a-nprovide input/output (I/O) interfaces for connection to data paths115a-nfor which switching services are provided by switch system100. Switching shelves120a-bprovide the switching services for the system. Control complex130provides central management for the switch system. The various shelves of switch system100are interconnected by intershelf links140a-hand150a-f. An example embodiment of the switch system100ofFIG. 1is the Alcatel7670Router Switch Platform (RSP) Multi-Shelf system (Alcatel7670). The Alcatel7670is a multiprotocol backbone system designed to switch ATM cells and route IP traffic through the same switching fabric. Depending on its configuration, the Alcatel7670provides from 14.4 Gbps up to 450 Gbps of switching capacity.

In one embodiment of switch system100ofFIG. 1, each of switching shelves120a-bcontains a switching fabric core and up to 32 switch access cards (SAC), each providing 14.4 Gbps of cell throughput to and from the core. Switching shelves120a-bare configured as a redundant pair and are referred to as switching shelf X and switching shelf Y.

In one embodiment, two types of peripheral shelves110a-nmay be used in switching system100. The first type, called a high speed peripheral shelf (HSPS), comprises high speed line processing cards (HLPC) each providing 10 Gbps throughput, high speed I/O cards (HIOC), high speed fabric interface cards (HFIC) and two high speed shelf controllers (HSC). The second type, called simply a peripheral shelf (PS), comprises lower speed line processing cards (LPC) each providing 2.5 Gbps throughput, I/O Cards (IOC) and dual or quad port fabric interface cards (DFIC/QFIC). Each of peripheral shelves110a-nalso typically includes a shelf control system which may comprise control complex130(in the case of peripheral shelf110a) or one or preferably two redundant peripheral shelf controllers (PSC).

In one embodiment, switching shelves120a-bconnect to the peripheral shelves110a-nin the system via fabric interface cards (FIC) to provide cell switching to the line processing cards (also referred to as “line cards”) in the peripheral shelves. Examples of these connections are illustrated inFIG. 2.

FIG. 2includes two switching shelves120a(designated “switching shelf X”) and120b(designated “switching shelf Y”) and three peripheral shelves: two normal speed peripheral shelves110a(designated “PS1”) and110b(designated “PS2”) and one high speed peripheral shelf110c(designated “HSPS3”). In one embodiment, each of the peripheral shelves may comprise up to sixteen (16) line processing cards. InFIG. 2, two line cards225a-bare shown for peripheral shelf110a, four line cards235a-dfor peripheral shelf110b, and four high speed line cards245a-dfor high speed peripheral Shelf110c. Each pair of line cards in each peripheral shelf communicate with an associated pair of I/O cards, which connect to the data stream for which switching services are being provided. Thus, in the embodiment ofFIG. 2, line cards225a-bcommunicate with associated I/O cards220a-b, line cards235a-bcommunicate with I/O cards230a-b, line cards235c-dcommunicate with I/O cards230c-d, high speed line cards245a-bcommunicate with I/O cards240a-b, and high speed line cards245c-dcommunicate with I/O cards240c-d. Each line card is also connected to two redundant fabric interface cards (FIC) which provide high speed connections between each line card and each switching shelf120aand120b. One of each pair of FIC's is designated FIC “X” and the other is designated FIC “Y”. Three types of FIC's are shown inFIG. 2. Peripheral shelf110acontains two dual FIC's (DFIC)270aand270b. Peripheral shelf110bcontains two quad FIC's (QFIC)275a-b, and high speed peripheral shelf110ccontains four high speed FIC's (HFIC)280a-d.

In one embodiment, each switching shelf120a-bcomprises a switch core and up to 32 switching access cards (SAC). In the embodiment ofFIG. 2, switching shelf X120acomprises switch core250aand SAC's255a-h, and switching shelf Y120bcomprises switch core250band SAC's265a-h.

In one embodiment, the connections between the various components within a shelf are provided via one or more circuit boards in each shelf, referred to as “midplanes,” to which the respective components are mounted.

In the embodiment ofFIG. 2, each “X”-designated FIC on a peripheral shelf is connected to one or more SAC's on switching shelf X120a, and each “Y”-designated FIC on a peripheral shelf is connected to one or more SAC's on switching shelf Y120b. A DFIC connects to up to two SAC's, a QFIC to up to four SAC's, and a HFIC to a single SAC. In the embodiment ofFIG. 2, the connection between the FIC's on a peripheral shelves110a-cand the SAC's on switching shelves120a-bis via high speed intershelf links (HISL)290a-p, which carry the cell-switched data path traffic. Thus DFIC X270aof peripheral shelf110ais connected to SAC's255a-bon switching shelf X120avia HISL's290a-b, and DFIC Y270bof peripheral shelf110ais connected to SAC's265a-bon switching shelf Y120bvia HISL's290i-j. Similarly, on peripheral shelf110b, QFIC X275ais connected to SAC's255c-fon switching shelf X120avia HISL's290c-f, and QFIC Y275bis connected to SAC's265c-fon switching shelf Y120bvia HISL's290k-n. Finally, on high speed peripheral shelf110c, HFIC's X280aand280care connected via HISL's290g-hto SAC's255g-hon switching shelf X120a, and HFIC's Y280band280dare connected via HISL's290o-pto SAC's265g-hon switching shelf Y120b, respectively.

In the switch system ofFIGS. 1 and 2, peripheral shelf110a, designated “Peripheral Shelf 1,” contains control complex130. In one embodiment, control complex130comprises a logical grouping of cards that provide the central management for all features of switch system100. The local and remote user interfaces to the system are provided by control complex130via user terminals160and170. In addition, control complex130maintains a database for all cards on the system. One embodiment of control complex130is illustrated inFIG. 3.

In the embodiment ofFIG. 3, control complex130comprises two control cards300a-b(designated “Control Card A” and “Control Card B”, respectively), two intershelf connection (ICON) cards310a-b, two control interconnect (CIC) cards315a-b, two intershelf connection I/O (ICON I/O) cards320a-b, and two intershelf connection I/O expansion cards (ICON I/O Exp)330a-b. Control complex130also comprises a facility card (FAC) that provides an RS232 management port and two timing ports that can be attached to a timing source for system synchronization. In addition to control complex130, peripheral shelf110aalso comprises its own set of line processing cards340that communicate with control complex130via CIC cards315a-b.

Control complex130communicates with each shelf in the switch system via shelf controllers on the peripheral and switching shelves. In the embodiment ofFIG. 3, peripheral shelf110bcomprises a redundant pair of shelf controllers340a-b, high speed peripheral shelf110ccomprises a redundant pair of high speed shelf controllers350a-b, and switching shelves120aand120beach comprise a switching shelf controller360aand360b, respectively.

ICON I/O cards320a-band ICON I/O Exp cards330a-bprovide interfaces for connecting control complex130to other shelves (such as peripheral I/O and switching shelves) in the switch system via physical connections. In one embodiment, these connections are referred to as Control Services Links (CSL)370a-h. In the embodiment ofFIG. 3, each ICON I/O and ICON I/O Exp card contains eight CSL ports. ICON I/O A320aand ICON I/O B320bprovide ports for CSL connections to the two switching shelves X and Y120a-band to the first six peripheral I/O shelves (designated peripheral shelves2to7), while ICON I/O Exp A330aand ICON I/O Exp B330bprovide ports for CSL connections to eight additional peripheral I/O shelves (designated peripheral shelves8to15). ICON cards310, ICON I/O cards320, ICON I/O Exp cards330, CSL's370, and shelf controllers340,350and360provide an intershelf connection (ICON) infrastructure that allows the transfer of control traffic between control complex130and the controllers and cards in peripheral shelves110and switching shelves120without using any bandwidth on the main data path. Data path traffic goes through the switching fabric shown inFIG. 2whereas the control traffic uses the out of band ICON infrastructure (also sometimes referred to as the “control infrastructure”) ofFIG. 3to transfer data. Operation of the control infrastructure is not affected by the switching fabric, and the switching fabric is not affected by the control infrastructure.

In one embodiment, a CSL link is physically embodied in a twelve-conductor cable comprising three separate, internal 4-conductor Cat-5-type cables. Each 4-conductor internal cable provides one of three different types of communications channels: a time division multiplexed (TDM) channel (providing E1-type capabilities), a full duplex (Ethernet) messaging channel, and a simplex differential channel.

The TDM channel (also sometimes referred to as the “E1 channel”) is used by the ICON infrastructure to transport time sensitive transport activity, shelf numbering control and system timing information throughout the system. In one embodiment, the TDM channel operates at a frequency of 8000 Hz, providing 32 time slots with 8-bits per slot at a 125 microsecond refresh rate resulting in a data rate of 2.048 Mbps. In this embodiment, the TDM channel provides a guaranteed point-to-point channel between the ICON cards on peripheral shelf1and any shelf connected to one of peripheral shelf1's CSL ports.

The full duplex messaging channel (also sometimes referred to as the “Ethernet channel”) is used for general communications with any shelf in the system. In one embodiment, the Ethernet channel operates at 100 Mbps. Each Ethernet link to a shelf is shared among all components (cards) within the shelf. As a result, every element in the system is capable of communicating with the control complex via the Ethernet channel.

The Ethernet channel may be used for a large variety of communications, including connection information, software downloading to the individual components, debugging, alarm management, and configuration transfers. Communications between the control complex and the shelf controllers that does not travel over the TDM channel in general travels over the Ethernet channel.

The simplex differential channel (also sometimes referred to as the “RTS channel”) is used to distribute a real time stamp (RTS) from the controller to each of the shelves in the system. The simplex differential channel is used instead of the TDM channel or the Ethernet channel because the RTS is sensitive to time delays that result from the applications running on the TDM and Ethernet channels. The RTS is used to align all elements of the system to the same time stamp for purposes such as debugging and billing. The RTS signal is generated by the control complex and is provided to the ICON cards via a direct circuit board (mid-plane) connection. The ICON cards are responsible for extracting the signal and transmitting it via the differential channel to all shelves in the system.

The use of redundant pairs of control cards and icon cards in peripheral shelf1and redundant shelf controllers in other peripheral I/O shelves creates redundant pathways over which control traffic can flow between each control card of the control complex in peripheral shelf1and each line card of each peripheral I/O shelf. Examples of redundant pathways between control card300aand a line card550of high speed peripheral shelf110care shown inFIGS. 5a-b. For simplicity, only the main relevant components are shown inFIGS. 5a-b. For the same reason, the ICON I/O and ICON I/O Exp cards are not shown separately but are included as part of the respective ICON cards.

FIG. 4illustrates the control traffic flow of the pathway ofFIG. 5a. In the embodiment ofFIG. 4, controller card a300acomprises a microprocessor405, a real time stamp source415, and an Ethernet port410.

Microprocessor405exchanges time sensitive control data and normal control data with line card420on peripheral shelf2110c. Time sensitive control data travels over a set of “n” direct links420to a field programmable gate array (FPGA)450on ICON card A310a. FPGA450assembles the data for transmission via TDM, and transmits the time sensitive data via E1 channel485of CSL370cto FPGA470of shelf controller A350aon peripheral shelf2110c. FPGA470extracts the time sensitive data for line card420from the TDM stream, and passes it via direct midplane links495to line card420.

Normal control data, on the other hand, travels from microprocessor405to Ethernet port410of control card A300a, and from there via midplane Ethernet link425to switch445of ICON card A310a. Switch445passes the data via CSL Ethernet channel490to switch465on shelf controller A350aof peripheral shelf2110c. Switch465in turn transmits the data to Ethernet port482on line card420. A microprocessor466of shelf controller A350aon peripheral shelf2110cis coupled to FPGA470via connection467, switch465via connection468, and Ethernet port482via connection469, allowing microprocessor466access to the time sensitive data and the normal control data and/or allowing microprocessor466to exert control over the operations of FPGA470, switch465, and/or Ethernet port482(or, more generally, line card420via Ethernet port482).

Real time stamp (RTS) source415may generate its own time signal, or may receive a time signal from an external source. RTS source415communicates its time signal via midplane link430to RTS transponder455on ICON card A310a. RTS transponder455converts the time stamp into proper form for transmission over the differential channel of CLS370aand sends it to RTS transponder475on shelf controller A350aof peripheral shelf2110c. RTS transponder475extracts the RTS signal from CSL370c's RTS channel and distributes it to line card420via midplane link497.

Even thoughFIG. 4only shows the control data pathways between one control card and one ICON card, it will be understood that the same pathways exist between each control card and both of the ICON cards in peripheral shelf1110a.2. APS Automatic Protection System

A common method used to prevent data communications interruptions is known as Automatic Protection Switching (APS) 1+1. In an APS 1+1 system, components are connected via redundant, mirrored links. Each transmitting component sends the identical data over both links, and each receiving component listens to both links. One of the links is the normal “work” link, while the other is the “protection” link. As long as the work link is operating properly, the receiving component processes the data received from that link and discards the data received via the protection link. However, if at any time the work link becomes inoperable or defective, the receiving component immediately begins processing the data received over the “protection” link. Both links must carry identical data so that no data loss occurs when such a switch is made.

APS is useful because it prevents communications failures resulting from a single link failure. However, it does so at the expense of bandwidth: because the same data is sent simultaneously over two separate links, the total bandwidth needed is twice the what is required by the data stream itself, even though the data transferred over the protection link is used only during the infrequent times when there is a failure of the work link, and is otherwise discarded.

SUMMARY OF THE INVENTION B1

The present invention comprises a method and apparatus for optimizing redundant link usage so as to allow a portion of the wasted bandwidth in a redundant link system to be utilized for additional data traffic without compromising the ability of the system to respond to and correct for a failure of a link. In one embodiment, two independent, individually addressable links are established between two endpoints. One of the links is selected as a nominal communication path, and the other as a standby communication path. Data traffic between the endpoints is divided into path independent traffic and path dependent traffic. The path independent traffic is sent via the nominal communication path, while the path dependent data is sent via the nominal communication path or the standby path, in accordance with the dependence of the traffic. In another embodiment, the nominal communication path and standby communication path each comprise a time sensitive path and a normal path. Critical time sensitive traffic is sent via the nominal and standby time sensitive paths, while normal traffic is sent via the nominal normal path, and non-critical traffic is sent via the standby normal path.

DESCRIPTION OF THE PREFERRED EMBODIMENT

A method and apparatus for optimizing redundant link usage in a multi-shelf network element is disclosed. In the following description, numerous specific details are set forth to provide a thorough description of the invention. However, it will be apparent to one skilled in the art that the invention may be practiced without these specific details. In other instances, well-known features have not been described in detail so as not to obscure the invention.

FIGS. 5A and 5Bshow intershelf control data path connections between two shelves of a multi-shelf element in an embodiment of the invention.FIG. 5Ashows the control data path connections between control card A300ain peripheral shelf1110aand shelf controller A350aof peripheral I/O shelf110c. These control data path connections are the same as those ofFIG. 4, in simplified form. In addition,FIG. 5Bshows a similar control data path between control card A300aand shelf controller B350bin peripheral I/O shelf110c.

As shown inFIG. 6, the control data path between control card A300aand shelf controller A350aincludes an intrashelf control data path670between control card A300aand ICON card A310aand an intershelf CSL link370cbetween ICON card A310aand shelf controller A350a. Similarly, the control data path between control card A300aand shelf controller B350bincludes intrashelf control data path675between control card A300aand ICON card B310band intershelf CSL link370dbetween ICON card B310band shelf controller B350b. Similar intrashelf control data paths (not shown inFIG. 6) exist between control card B and ICON cards A310aand B310b. For clarity, they are not shown here. However, in the discussion below, it will be understood that the description of the operation of control card A applies to control card B as well.

Each of intrashelf paths670and675and CSL links370cand370dcomprise three separate data channels: an RTS channel, an Ethernet channel, and a time sensitive data channel. Intrashelf path670includes an RTS channel425, an Ethernet channel425, and a direct multi-line channel420. Intershelf CSL link370cincludes an RTS channel480, an Ethernet channel490, and a TDM/E1 channel485. RTS channel425is connected through ICON A310ato RTS channel480, Ethernet channel425is connected to Ethernet channel490, and multi-line channel420is connected to E1 channel485.

In other embodiments, time sensitive data channels, for example, indirect multi-line channels, may be provided between a dual or quad fabric interface card (DFIC/QFIC) and each of control card A300a, ICON card A310a, and ICON card B310b. Such channels may be provided in addition to, or in lieu of, direct multi-line channel420and multi-line channel650.

The combination of intrashelf link670and intershelf link370cforms a first threechannel communication path between control card A300aand peripheral1/0shelf110c. This first communication path will be referred to as communication path A. The combination of intrashelf link675and intershelf link370dforms a second, independent three-channel communication path between control card A300aand peripheral I/O shelf110c. This second communication path will be referred to as communication path B. Control card A300ais configured to allow it to individually address each of communication paths A and B.

A concept that is relevant to the embodiment ofFIG. 6is the “health” of a communication path or link. The health of a link refers to the ability of the link to carry traffic without data degradation or loss.

In the embodiment ofFIG. 6, the health of communication paths A and B are continually monitored both by shelf controllers350aand350bon peripheral I/O shelf110cand by control card A300aon peripheral shelf1110a. The health of a link is determined by examining the data coming over each channel of the link to determine the amount of data loss or degradation. The relative health of a link is then determined according to preset criteria. Different weight may be assigned to the contribution by each channel to the overall health of a link. For example, in one embodiment, the state of the channel that carries the most critical and/or time-sensitive data (e.g., in the embodiment ofFIG. 6, the TDM/E1 channel) carries the greatest weight, while the channel that carries the least critical data (e.g. in the embodiment ofFIG. 6, the RTS channel), is given the least weight. In one embodiment that uses a three channel link like CSL links370c-dofFIG. 6, the RTS channel is given no weight at all in calculating the overall health of the link—the health is determined solely based on the conditions of the TCM/E1 and Ethernet channels.

The relative health of two available communication paths or links between two endpoints (such as, for example, control card A300aand peripheral I/O shelf110cinFIG. 6) can be used to choose one of the two communication paths as a “nominal path” and the other as a “standby path.” The nominal path may be used to transfer the normal data traffic between endpoints, while the “standby path” is kept available to carry the normal data traffic should the health of the “nominal path” deteriorate.

Various paradigms can be used for selection of the “nominal” and “standby” paths. In the embodiment ofFIG. 6, the responsibility for selection of the “nominal” path is normally assigned to shelf controllers350a-bof peripheral I/O shelf110c, who communicate with each other over link640, and who, together, can be considered to comprise a shelf control complex for peripheral I/O shelf110c. Each of shelf controllers350a-bmonitors the health of the particular CSL link that it is connected to. That is, shelf controller A350amonitors the health of CSL link370cand shelf controller B350bmonitors the health of CSL link370d. The shelf controllers negotiate among themselves based on the relative health of their respective CSL links to determine which link at any time is selected as the “nominal” link and which as the “standby” link. In addition, if a CSL link to a shelf controller deteriorates below a minimum threshold, the shelf controller may designate the link as being inoperative. Furthermore, in certain circumstances, for example if one of the links is being shut down for service, control card A may send a request to the shelf controllers to designate a particular link as the “nominal” link. However, in the embodiment ofFIG. 6, the final decision rests with the shelf controllers.

The identity at any point in time of the “nominal” link constitutes critical, time sensitive information. For example, a control card such as control card A330auses the identity of the nominal link to determine which path to use for data traffic. In one embodiment, the data identifying the nominal link is simultaneously sent by both shelf controllers via the respective TDM/E1 channels of their respective CSL links to the control cards to ensure that the identity of the nominal link is known, preferably at all times, by the control cards. Sending data regarding the current nominal path via the TDM/E1 channels of both CSL links allows the control card to be instantly informed of a switch in the nominal link, even if the previous link, including its TDM/E1 channel is completely severed. The control card can instantly redirect the normal data traffic to the standby link with little to no loss of data. It should be noted that the normal data traffic is considered to be “path independent” between the endpoints (i.e. control card A300aand peripheral I/O shelf110c). That is, it does not matter whether communication path A or communication path B is used—all that matters is that the data is transferred between endpoints.

In the embodiment ofFIG. 6, because TDM/E1 channels of CSL links370cand370dallow control card A to be instantly informed of a switch in the nominal link, it is not necessary, as in traditional APS schemes, for the normal data traffic over the Ethernet channel of the nominal link to be mirrored over the Ethernet channel of the standby link. As a result, the Ethernet channel of the standby link is available to carry additional data traffic. However, because the Ethernet channel of the standby link may at any time be called into use should a switch in the nominal link be commanded by the shelf controllers, such additional data traffic should normally constitute supplemental, non-critical data. Examples of such non-critical data include diagnostic data concerning the various elements and components in the standby path, and non-time critical software upgrades for elements and components in the standby path. Such data can be considered “path dependent” because it comprises data intended not for the endpoints, but for elements along a particular path.

FIG. 7is a flow chart showing a process used to optimize redundant link usage in an embodiment of the invention. Two independent communication links are established between endpoints at step700(e.g. a control complex and a peripheral or switching shelf or a card on a peripheral or switching shelf), and the health of each link is monitored at step710. At step720, a determination is made as to whether at least one link has sufficient health to be currently usable. If not, an alarm is set at725and the process returns to monitoring the health of the links at step710.

If it is found at step720that at least one of the links is currently usable, then one usable link is designated as the nominal link based on the relative health of the links at step730. At step740, path independent data, and data dependent on the nominal path (i.e. the path formed by the link that has been designated as the nominal link) is transferred between endpoints via the nominal link. At step750, a determination is made as to whether the second link is currently usable. If the second link is found to be not usable, an alarm is set at step755and the process returns to step710. If the second link is found to be usable, then data dependent on the second path (i.e. the path formed by the second link) is transferred between endpoints via the second link. Thereafter the process returns to step710.

FIG. 8is another flow chart showing a process used to optimize redundant link usage in an embodiment of the invention. Two independent links between endpoints are established at step800. At step805, each of the two channels are provided with a time sensitive channel (for example, the TDM/E1 channel of the embodiment ofFIG. 6) and a second channel (for example the Ethernet channel of the embodiment ofFIG. 6). The health of each link is monitored at step810, which may include separately monitoring the health of each channel of each link. At step815a determination is made as to whether the health of at least one link is sufficient to be currently usable. If it is determined that no link is currently usable, an alarm is set at step820and the process resumes monitoring the health of the links at step810.

If it is determined at step815that at least one link is currently usable, then one link is designated as the nominal link based on the relative health of the links at step825. The designation of that link as the nominal link is communicated between endpoints via the time sensitive channel of the nominal link at step830, and normal data traffic is transferred between endpoints via the second channel of the nominal link at step835.

At step840, a determination is made as to whether the health of the second link is sufficient to render it currently usable. If the second link is not found to be usable, an alarm is set at step845and the process returns to monitoring the health of the links at step810. If the second link is found to be usable, the designation of the nominal link (which is sent via the time sensitive channel of the nominal link at step830) is also sent via the time sensitive channel of the second link at step850. In addition, supplemental, non-critical traffic is transferred between endpoints via the second channel of the second link at step860. Thereafter the process returns to step810.

Thus, a method and apparatus of optimizing redundant link usage in a multi-shelf network element has been presented. Although the invention has been described using certain specific examples, it will be apparent to those skilled in the art that the invention is not limited to these few examples. For example, although the invention has been described with respect to use in a multi-shelf network element, the invention may be used for optimization of redundant link usage within and between other communicating devices. Other embodiments utilizing the inventive features of the invention will be apparent to those skilled in the art, and are encompassed herein.