Abstract:
Normal 802.3 Ethernet requires a tree topology. If a ring or a loop exists, then packets will be forwarded around the ring indefinitely. If the ring is broken, then there is no possibility of packets being propagated forever. This invention shows how to quickly impose a virtual break in the ring such that all nodes can communicate with each other, and how to remove the virtual break when a real failure occurs. This is accomplished by placing intelligent nodes on the ring that work together to virtually break and restore the ring. An embodiment is disclosed that handles a unidirectional break in a communication link. This abstract is provided as an aid to those performing prior art searches and not a limitation on the scope of the claims.

Description:
[0001]    This application claims priority to U.S. Provisional Application 60/490,764 filed Jul. 29, 2003 and U.S. Provisional Application 60/468,325 filed May 6, 2003. This application incorporates by reference these two provisionals. 
     
    
     
       FIELD OF THE INVENTION  
         [0002]    This invention relates to communication networks, and more particularly, to an apparatus and method for Ethernet equipment in a ring topology.  
         BACKGROUND OF THE INVENTION  
         [0003]    As Ethernet is deployed in Metro and Access networks, and services are offered on these networks, there is a desire to maintain SONET-like resiliency (i.e. recover from a fault in less than 50 ms). One common means of providing resiliency is with a ring topology. However, Ethernet switches will not work properly if there is a ring or loop in the topology. Protocols such as IEEE 802.1d Spanning Tree Protocol (STP) or IEEE 802.1w Rapid Reconfiguration were invented to detect and remove loops. However, they are slow and cannot achieve path restoral within 50 ms as desired.  
           [0004]    To solve this problem, the IEEE is working on 802.17 Resilient Packet Ring (RPR). Others are looking at Multiprotocol Label Switching (MPLS) with Fast Reroute capabilities. Both of these approaches are quite complex. RPR requires a new Media Access Control (MAC) Layer, and MPLS requires extensive signaling. Because of the complexities, these approaches will drive up the cost of the nodes on the ring.  
           [0005]    This invention introduces a new way (Protected Switching Ring or “PSR”) of providing protection for Ethernet deployed in a ring topology with resiliency that does not require a new MAC layer, and that can be built using low cost Ethernet chips and methods.  
           [0006]    This invention differs from some previous inventions. One of interest is described in U.S. Pat. No. 6,430,151, granted on Aug. 6, 2002. PSR is similar to &#39;151 in that:  
           [0007]    Both are based on nodes arranged in a ring topology.  
           [0008]    Both aim to overcome the limitations of STP.  
           [0009]    Both describe making or breaking a ring based on the passage or blockage of test messages.  
           [0010]    Both have two classes of nodes on the ring, one of which is a controller or master.  
           [0011]    Some of the differences between PSR and the teachings of the &#39;151 include:  
           [0012]    &#39;151 is composed of bridging nodes that do dynamic layer 2 learning, while PSR is composed of nodes that are configured to switch (add and/or drop) packets based on a VLAN tag.  
           [0013]    &#39;151 has a single redundancy manager (RM), while PSR can support dual redundancy Ring Arbiters (RA). PSR can operate in the presence of a failed RA, thus providing a higher level of availability.  
           [0014]    The nodes in &#39;151 learn an association between ports and MAC addresses for ring traffic. When the topology changes, the RM of the &#39;151 must send a “flush” message to tell the nodes to clear their databases. In contrast, the Ring Relay (“RR”) nodes in PSR always send messages (both data and control) around the ring in both directions, thus removing half of the propagation delay from the recovery time. In this way, a flush command is not needed to redirect traffic on the ring, thus reducing the recovery time.  
           [0015]    &#39;151 can cause packets to be duplicated during a restoral as there will be a ring upon restoral. Duplication of packets violates the IEEE 802.3 specifications. The state machines in PSR prevent this.  
           [0016]    Since nodes in PSR are not performing learning for ring traffic, there is less overhead and a higher packet rate can be sustained for a given amount of processing power.  
           [0017]    Another approach to the problem is described in U.S. Pat. No. 4,354,267. The &#39;267 patent describes a set of homogeneous layer 2 devices arranged in a ring. Each node in &#39;267 forwards packets around the ring, and the originator removes the packet.  
           [0018]    Some of the differences between PSR and the teachings of the &#39;267 include:  
           [0019]    &#39;267 assumes that data sent that is sent one way around the ring makes it all the way around. In layer 2 systems, each node may pick off packets addressed to it, so this assumption is not valid.  
           [0020]    &#39;267 assumes that each node can repair a fault. See claim 1 in column 10, starting at line 34, and claim 5, in column 12, starting at line 38. In contrast, PSR concentrates the recovery mechanism in just two nodes.  
         SUMMARY OF THE DISCLOSURE  
         [0021]    Normal 802.3 Ethernet requires a tree topology. If a ring or a loop exists, then packets will be forwarded around the ring indefinitely. STP was created to solve this problem by detecting and breaking any rings. If the ring is broken, then there is no possibility of packets being propagated forever.  
           [0022]    This invention shows how to virtually break the ring such that all nodes can communicate with each other, and how to remove the virtual break when a real failure occurs. This is accomplished by placing intelligent nodes on the ring that work together to virtually break and restore the ring.  
           [0023]    In PSR, the nodes communicate between and among themselves to determine when and where a break occurs. The relevant state machines for a preferred embodiment of the present invention are contained within this disclosure. 
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0024]    [0024]FIG. 1 shows an example of prior art.  
         [0025]    [0025]FIG. 2 shows an example Protected Switching Ring in the Full Ring mode in normal operation.  
         [0026]    [0026]FIG. 3 shows an example Protected Switching Ring in the Full Ring mode during a failure.  
         [0027]    [0027]FIG. 4 shows an example Protected Switching Ring in the High Availability mode in normal operation.  
         [0028]    [0028]FIG. 5 shows an example Protected Switching Ring in the High Availability mode during a failure.  
         [0029]    [0029]FIG. 6 shows the state machine for a Ring Arbiter node in the Full Ring mode.  
         [0030]    [0030]FIG. 7 shows the state machine for a Ring Relay node in the Full Ring mode.  
         [0031]    [0031]FIG. 8 shows the state machine for the Ring Side of a Ring Relay node in the High Availability mode.  
         [0032]    [0032]FIG. 9 shows the state machine for the Extension Side of a Ring Relay node in the High Availability mode.  
         [0033]    [0033]FIG. 10 illustrates a unidirectional ring break.  
         [0034]    [0034]FIG. 11 shows the “Dual Homing” embodiment providing User Ports  1140  with redundant links to the existing network. 
     
    
     DESCRIPTION  
       [0035]    The present invention will now be described more fully hereinafter with reference to the accompanying drawings, in order to disclose selected embodiments. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Like numbers refer to like elements throughout.  
         [0036]    Overview  
         [0037]    The Protected Switching Ring (PSR) enables building of partial or full ring topologies from low-cost Ethernet equipment, while providing for sub-50 millisecond recovery from equipment or link faults. PSR nodes support the transport of point-to-point port-switched connections across the ring topology. During normal, non-fault operation, one port in the ring will be blocked to user traffic, thus preventing a loop. In the event of a fault in the ring, the blocked port will be unblocked, allowing access to all nodes on the ring.  
         [0038]    Two topologies using the present invention are described below. The first topology is the PSR Full Ring (“FR”) configuration that consists of a full ring of PSR nodes. Port-switch connections can be configured between any two subscriber ports on the ring. The second topology is the PSR High-Availability (HA) configuration. This configuration provides a partial-ring extension of a SONET or RPR ring, or a partial-ring addition to existing layer-2 switching equipment. In either case, a path is engineered through the existing equipment to complete the path for the PSR protocol traffic and user data.  
         [0039]    PSR nodes are designated as Ring Arbiters or Relays. Each ring contains two Ring Arbiters. The Ring Arbiters communicate with a “hello” protocol to coordinate the blocking or forwarding of user traffic. In a preferred embodiment, the PSR Ring Arbiter ports take on the role of master or slave on the ring according to their relative node priority. In a preferred embodiment, the priority could be a unique identifier, such as a MAC address. In a highly preferred embodiment, the priority can be the concatenation of an operator-configurable priority with the MAC address (or other unique identifier) such that the priorities of two nodes would never be equal. In either of these preferred embodiments, the reception of a HELLO with the same priority would indicate a ring with only one arbiter, where that arbiter was receiving its own HELLOs.  
         [0040]    In general, during normal fault-free operation of two Ring Arbiters, the slave Ring Arbiter will block one of its ring ports in order to terminate the ring loop. A ring may contain one or more Relay nodes. The Relay nodes may be distributed in any fashion around the ring, although some benefit is provided by distributing approximately equal numbers of Relays on each “side” of a full-ring configuration.  
         [0041]    In addition to the “hello” protocol, each node performs a “discovery” protocol that allows each node to know about all the other nodes on the ring. The discovery protocol is also used to detect persistent ring faults and to generate the associated alarms. Both protocols operate at layer 2, employing reserved multi-cast MAC addresses.  
         [0042]    IP connectivity between all ring nodes is accomplished over a control VLAN used only for that purpose. This allows Telnet and a UDP-based signaling protocol to operate between any nodes on the ring. (An explanation of Telnet is not critical to the understanding of the present invention but Telnet is a terminal emulation program used with TCP/IP networks that allows remote entry of commands that are treated as if input at the network device.) Bridging techniques are used to provide the connectivity for these IP-based applications; all user traffic is transported across the ring using port-switching. As such, all user traffic is point-to-point across the ring; traffic from a subscriber-port/VLAN on one node is connected to a subscriber-port/VLAN on another node.  
         [0043]    An additional embodiment of the present invention addresses a partial failure of a network link so that the communication link is lost in only one direction. Yet, another embodiment uses a single arbiter to provide a high reliability connection of user ports to an existing network ring by creating a switching ring with the arbiter and two network ring access points.  
         [0044]    Ring Nomenclature  
         [0045]    When the PSR is configured, two ports are designated as the ring ports and may be called East and West ports. Also the node type is given to distinguish Ring Arbiter types and Relays (also called Ring Relays or Relay Nodes). The Ring Arbiter type may be High-Availability (HA) or Full-Ring (FR). The two Ring Arbiters on the ring must be of the same type. When speaking of a specific ring port, the partner port refers to the other port of the pair of ring ports on that Ring Arbiter or Ring Relay.  
         [0046]    An additional distinction is made in the case of a HA Ring Arbiter. The port of the HA Ring Arbiter connected to the existing SONET or RPR ring is designated the “extension side” (ES) port. This port interfaces with the existing equipment for which we wish to extend a ring segment. The other Ring Arbiter port is referred to as the “ring side” port. It is connected to a string of one or more Ring Relays or directly to the other Ring Arbiter.  
         [0047]    HELLO Protocol  
         [0048]    Each PSR Ring Arbiter periodically issues a “HELLO” protocol packet out each ring port. In a preferred embodiment each PSR Ring Arbiter issues a “HELLO” protocol packet out each ring port every 10 milliseconds. The packet uses a special multicast MAC address as the destination address. The Relay nodes are configured to have the data plane pass the packet from one ring port to the other, so a Relay node adds only a small amount of delay as the packet moves from one Ring Arbiter to the other. The remote Ring Arbiter node will terminate the packet and send the packet to the control plane. The control plane uses the presence of the new packet and some control information to drive its state machine for the Ring Arbiter ports. The absence of a new HELLO message for 30 milliseconds constitutes a ring timeout. If the timeout persists for 1.5 seconds, a ring failure is declared and the appropriate alarm is issued.  
         [0049]    The significantly longer period used as a trigger for a ring failure keeps a short intermittent problem from being deemed ring failures though the problems may be handled by the declaration of ring timeouts. In one embodiment, the ring failure is detected by loss of Discovery messages, described below. One of skill in the art could implement the ring failure to be based on the absence of HELLO messages rather than Discovery messages. One of skill in the art would appreciate that the HELLOs are not processed at the RR nodes, whereas the Discovery messages are. HELLOs therefore propagate around the ring faster than Discovery messages. A timeout threshold for loss of HELLOs can be set lower than an equivalent threshold for Discovery messages.  
         [0050]    A ring timeout causes the state machines to transition a slave Ring Arbiter port to a FORWARDING state. This response ensures that any loss in connectivity due to a single failure across the ring will only persist for 50 milliseconds or less.  
         [0051]    In a preferred embodiment the sequence number in the HELLO PDU is used at the receiving Ring Arbiter to distinguish the arrival of a new HELLO PDU. Those of skill in the art will recognize that other methods could be employed to detect the arrival of a new HELLO PDU. The Relay nodes do not process the HELLO PDUs; they only forward them between ring ports.  
         [0052]    Discovery Protocol  
         [0053]    The discovery protocol is an optional protocol that can be implemented in order to add functionality. Note since the discovery protocol is not a necessary requirement of the state machines for any of the Ring Arbiters, Protected Switching Rings in accordance with the teachings of the present invention could be implemented without implementing the discovery protocol.  
         [0054]    The discovery protocol also uses a special multicast destination MAC, but runs every 500 milliseconds. The discovery PDU is originated by the Ring Arbiters, appended to by intervening Relay nodes, and terminated at the remote Ring Arbiter. As the discovery PDU traverses the path between Ring Arbiters, each node in the path appends its management IP address, egress port for the PDU, and node type to the PDU. Since the discovery messages are flowing in both directions on the ring, each node on the ring can see the path of nodes to each Ring Arbiter on each of its ring ports. For example in FIG. 2, the Ring port  210  will receive a discovery message on one port directly from the RA  200  and will receive the other discovery message from the RA  225  after that discovery message passes through the ring port  220 . Thus after receiving the two discovery PDUs, each ring port knows the identity of all devices between the ring port and each RA.  
         [0055]    Additionally, as each Ring Arbiter constructs the discovery message to send out a ring port, the Ring Arbiter adds the completed node list received at its partner port. This allows every node in the PSR to know all the IP addresses of the nodes in the ring.  
         [0056]    In the event of a ring or node failure, the Relay nodes closest to the point of failure will originate the discovery message. In other words, if a relay fails to receive a discovery PDU from its upstream neighbor (due to a link or node failure), then the relay will create and send a discovery PDU in the downstream direction. All downstream nodes will detect that the Ring Arbiter is no longer the originator of the discovery message and declare a fault alarm. If a node either does not receive a Discovery message or receives a Discovery message without a ring Arbiter as the originator, a ring failure is declared after 1.5 seconds. The fault is cleared when the node receives a Discovery message with a ring Arbiter as the originator.  
         [0057]    PSR Data Plane for User Traffic  
         [0058]    User traffic may enter and leave the PSR at any Ring Arbiter or relay node. A PSR connection defines the entry and exit points for a full-duplex flow of user traffic across the ring. This flow is defined by a pair of port/VLAN ID/PSR Node Address tuples. The connection defines a path through the ring between 2 user ports, each residing on a PSR node, configured to carry the user traffic for specific or all VLAN IDs on that port.  
         [0059]    As the user traffic enters the ring, a ring tag is added to the packet. The ring tag is a VLAN tag and is unique on the ring. The ring tag defines a given connection between two ring nodes. At the egress node of the PSR connection, the ring tag is removed from the frame before forwarding to the user port. In this way, the VLAN tags present in the user data are transparently transported across the ring. VLAN IDs used on one user port do not interfere with IDs used on another user port.  
         [0060]    A PSR node is either an endpoint of a given connection or a transit node for that connection. A PSR node is an endpoint for a connection if one of its user ports is specified in the definition of the given connection. The node is a transit node if neither endpoint of the connection resides on the node. In either case, a switch table used by the data plane is configured on each PSR node to either terminate one end of a given connection or to act as a transit node for that connection. When a node is a transit node for a given connection, the node simply transfers frames from one ring port to the other, based on the ring tag, without modification. When a node is an endpoint node for a given connection, the data plane directs the data arriving on a ring port to the correct user port and removes the ring tag. Conversely, the node&#39;s data plane directs user packets from the given user port with the given VLAN ID to the ring ports, adding the correct ring tag in the process.  
         [0061]    PSR Control Plane for Control Traffic  
         [0062]    A PSR requires a mechanism to transport HELLO PDUs, discovery PDUs, and IP traffic for ring control applications between PSR nodes. While user traffic transport is transported using switching techniques, in a preferred embodiment the control functions are transported using bridging techniques. By using bridging techniques, full PSR node control connectivity is attained with all nodes appearing on the same IP subnet. This makes configuration much simpler.  
         [0063]    One ring tag is reserved for PSR control traffic. The data plane uses learning procedures and forwarding table lookups to direct control traffic to the correct PSR node. Note that the use of learning procedures and forwarding table lookups for the direct control traffic imposes an overhead burden that is orders of magnitude smaller than the overhead needed to use learning techniques for user data traffic. In the preferred embodiment the HELLO and discovery messages use known multicast MAC addresses and thus do not add additional learned database entries to be forwarded. Flushing is not needed for the control traffic upon failure, recovery, or reconfiguration of the ring, as the new port entries are learned from bidirectional traffic after a short period of time.  
         [0064]    While the use of bridging for control traffic is preferred, it is not required in order to implement the present invention. The present invention could be implemented to use switching techniques for data packets and some or all types of control traffic. Care must be taken in creating this variation that the control traffic described in this application as passing when data packets are blocked, must be allowed to pass.  
       EXAMPLE RECOVERY FOR FULL RING  
       [0065]    Fault Detection  
         [0066]    [0066]FIG. 3 shows a full ring where the link  1325  fails between nodes  310  and  320 . This means that RA nodes  300  and  325  are unable to communicate with each other via the left hand side of the ring. Prior to the failure, assume that RA node  325 , the slave Ring Arbiter, is blocking traffic on link  1330  (thus no counterclockwise communication on  1330 ) and forwarding traffic on link  1335 . Also, any user traffic arriving on link  1330  is discarded. So clockwise traffic on  1330  is discarded at the  1330  side of RA  325 . Communications to subscriber ports connected to RA  325  reach those ports through counterclockwise communication over link  1335  to RA  325 .  
         [0067]    Assuming RA node  300  was the master Ring Arbiter, when RA node  325  detects the loss of communication; RA node  325  will start forwarding traffic to the right hand side of the ring onto link  1330  and accepting user traffic arriving on link  1330  and relaying the traffic to link  1335  and to the subscriber ports of RA  325 . This will restore communications between all of the nodes on the ring. At this point, RA  325  is forwarding traffic on both ring ports. The ring port that is facing link  1335  is in MASTER FORWARDING state, and the ring port that is facing link  1330  is in SLAVE FORWARDING state.  
         [0068]    Link Restoral  
         [0069]    When link  1325  is restored, RA node  325  needs to block one of its ring ports to prevent a loop in the ring. When RA node  325  receives the first HELLO on link  1335  (due to the restoration of link  1325 ), RA node  325  will see that the partner port to the port that is facing link  1335  is in SLAVE FORWARDING state. RA node  325  will move the port that is facing link  1335  to the BLOCKING state. Assuming that the Ring ports of nodes  310  and  320  connected to link  1325  went to an OPER DOWN state during the failure, the TIMING state in the relay nodes  310  and  320  will prevent forwarding of traffic until the slave Ring Arbiter has time to switch from MASTER FORWARDING to BLOCKING on the  325  side of the Ring Arbiter. OPER DOWN is an indication from the physical or transport layer that a link is not operational. It is normally based on the detection of loss or corruption of the incoming electrical or optical signal.  
         [0070]    The advance to the TIMING state is triggered by the reception of a HELLO message. This TIMING state delay in the resumption of operation of relay nodes  310  and  320  prevents duplicate packets from reaching a given destination when the failed link is restored. To illustrate the value of this delay in the Ring Relay ports, consider a message coming to Ring Relay  305  to a subscriber port connected with Ring Relay  310  just before link  1325  is restored. Ring Relay  305  operating normally will send the same message onto link  1300  and link  1320 . The message traveling counterclockwise reaches Ring Relay  310  in a conventional way. The message traveling clockwise to Ring Relay  310  will pass through Ring Arbiter  325  onto link  1335  as the West Port is operating in MASTER FORWARDING. When link  1325  is restored, there is a path for a duplicate message to cross link  1325  to Ring Relay  310 . This potential is eliminated if the Ring Relay ports undergo a suitable delay between receipt of the first HELLO and the ultimate state of FORWARDING. Note that the HELLO messages from Ring Arbiter  300  to Ring Arbiter  325  and from Ring Arbiter  325  to Ring Arbiter  300  will pass over link  1325  as soon as it is restored as the HELLO messages are not blocked at any port in any state.  
         [0071]    The preferred embodiments disclose using a timing delay to ensure that a port progressing from OPER DOWN to operational delays sending data packets long enough for the slave arbiter to impose a virtual break. One of skill in the art will recognize that the use of the timer could be replaced by a control signal sent by the slave arbiter after it has successfully imposed the virtual break. In either case, the port does not go to fully operational until after the virtual break has been imposed to preclude the creation of a ring for data packets.  
       EXAMPLE RECOVERY FOR HA RING  
       [0072]    Fault Detection  
         [0073]    [0073]FIG. 4 shows a HA ring under normal fault-free operation. The ES slave port  1440  is in the BLOCKING state to prevent a ring loop. FIG. 5 shows a HA ring where the link  1520 , between nodes  510  and  520 , fails. As for the full ring case, the bidirectional failure of link  1520  means that the Ring Arbiter nodes  500  and  530  are unable to communicate over the left side (Ring Side) portion of the HA ring. Assuming Ring Arbiter node  530  is the slave, its ES port (the facing link  1540 ) would be un-blocked when the failure is detected. Fault detection and subsequent un-blocking of the slave Ring Arbiter port is fundamentally the same as for the full ring case described above.  
         [0074]    Link Restoral  
         [0075]    In a preferred embodiment, the HA ring favors the Ring Side once the fault is removed. Instead of leaving the slave Ring Arbiter ES port (the port facing link  1540 ) in the forwarding state and blocking the Ring Side port (the port facing link  1530 ), the HA slave Ring Arbiter  530  always returns to a FORWARDING state on the Ring Side segment and blocks the ES port.  
         [0076]    The Ring Side segment of the HA ring is favored in order to minimize HA ring traffic on the existing SONET or RPR ring as this will cut some of the user traffic on the SONET ring segment between the Ring Access Equipment as one direction will be blocked (thus counterclockwise traffic from port  1440  will be blocked while clockwise traffic from  1400  will continue to travel on the SONET Ring.  
         [0077]    Nomenclature for State Diagrams  
         [0078]    In the following descriptions, “isMaster” is based on the last received HELLO. If no HELLO was ever received on the port, then isMaster is based on the partner&#39;s last HELLO. If no HELLOs have ever been received by either port, then isMaster is “true”. The Boolean variable “isSlave” is the logical negation of “isMaster”.  
         [0079]    The term “PartnerHelloTimeout” indicates that the partner port&#39;s age timer has timed out. The designation “RxHello&lt;Node” means a HELLO message has been received whose priority is less than that of the receiving node. This event would cause the receiving node to consider itself a master.  
         [0080]    Full Ring Mode—Ring Arbiter  
         [0081]    [0081]FIG. 6 shows the state diagram for a RA node. Each of the two ports on an RA node runs a copy of this state machine.  
         [0082]    Description of States  
         [0083]    The state machine of FIG. 6 has the following states.  
                       TABLE A                       Number   State   Description                   600   PORT DOWN   The port is operationally down or has just               been initialized Entered from any state.       610   BLOCKING   The node is sending HELLOs, but not               forwarding data traffic.       620   SLAVE TIMING   Node knows that it is a slave, but port is               waiting for a timer to expire before               moving to a forwarding state.       630   MASTER TIMING   Node knows that it is a master, but port is               waiting for a timer to expire before               moving to a forwarding state.       640   SLAVE   The port on a Slave Node is forwarding           FORWARDING   packets       650   MASTER   The port on a Master Node is forwarding           FORWARDING   packets                  
 
         [0084]    Description of State Transitions  
         [0085]    The table below describes the transitions of the state machine shown in FIG. 6.  
         [0086]    Note the fd timer reference below runs using a time that is a small fraction of the time used for the age timers in the RA and Relay nodes. This ensures that the relays are timed for a period long enough after the expiration of the fd timer for the loop to be broken. For example, one embodiment uses a 10 millisecond timer for the RA and Relay nodes and the fd timer at just one “tick” (a single 10 millisecond delay). This 10 millisecond interval is a small fraction of the 30 millisecond interval used to detect a ring timeout when a new HELLO message does not arrive within that interval.  
         [0087]    Note that the state machine for ring arbiters in the preferred embodiment does not wait indefinitely to see a HELLO as long as the ports of the ring arbiter are operationally UP. The goal is to let the parts of the network ring operate even if other portions of the network ring cannot operate.  
                       TABLE B                       Num   Event   Action                   1610   port operationally down OR init   block user traffic, cancel               all timers       1615   port operationally up   start age timer       1620   age timer expires OR RxHello &lt; Node   start fd timer       1625   fd timer expires   restart age timer, forward               user traffic       1630   RxHello &gt; Node   start fd timer       1635   fd timer expires AND partner not   restart age timer, forward           SLAVE FORWARD   user traffic       1640   age timer expires OR RxHello &lt; Node   restart fd, age timer       1645   RxHello &gt; Node   restart fd timer       1650   Age timer expires OR RxHello &lt; Node   restart age timer       1655   RxHello &gt; Node AND partner not   restart age timer           SLAVE FORWARD       1660   RxHello &gt; Node AND partner SLAVE   restart age timer           FORWARD                  
 
         [0088]    The following table shows the complete state transitions for the Full-Ring Arbiter as shown FIG. 6.  
                                       TABLE C                           PORT       SLAVE   MASTER   SLAVE   MASTER       Current State   DOWN   BLOCKED   TIMING   TIMING   FORWARDING   FORWARDING       Event   600   610   620   630   640   650                   Oper Down   N/A   PORT DOWN   PORT DOWN   PORT DOWN   PORT DOWN   PORT DOWN       Oper Up   BLOCKED   N/A   N/A   N/A   N/A   N/A       Current State       Age Timer   N/A   MASTER   MASTER   MASTER   MASTER   MASTER       Expires       TIMING   TIMING   TIMING   FORWARDING   FORWARDING       fd Timer Expires   N/A   N/A   N/A   MASTER   N/A   N/A                       FORWARDING       fd timer Expires   N/A   N/A   SLAVE   N/A   N/A   N/A       AND Partner           TIMING       SLAVE       FORWARDING       fd Timer Expires   N/A   N/A   SLAVE   N/A   N/A   N/A       AND Partner not           FORWARDING       SLAVE       FORWARDING       RxHello &lt; Node   N/A   MASTER   MASTER   MASTER   MASTER   MASTER               TIMING   TIMING   TIMING   FORWARDING   FORWARDING       RxHello ≧ Node   N/A   SLAVE   SLAVE   SLAVE   N/A   N/A               TIMING   TIMING   TIMING       RxHello ≧ Node   N/A   N/A   N/A   N/A   N/A   BLOCKED       AND Partner       SLAVE       FORWARDING       RxHello ≧ Node   N/A   N/A   N/A   N/A   SLAVE   SLAVE       AND Partner not                   FORWARDING   FORWARDING       SLAVE       FORWARDING                  
 
         [0089]    In a preferred embodiment, every 10 milliseconds, the two ports are checked in the same order. The combination of variations in when the HELLOs were generated plus transit delays may cause one HELLO on one port to arrive before the other HELLO on the other port. In any case, since one port is checked before the other then the other, it always appears as though one HELLO arrives before the other. The order that the ports are checked does affect which slave port is set to BLOCKING on the full ring.  
         [0090]    One of skill in the art will recognize that any embodiment that does not check one port before the other would need to address the case of two HELLOs arriving essentially simultaneously at the two ports.  
                                                                                                               TABLE D                       Time   Port A   Input   State Change   Port B   Input   State Change                                1   Port Down   Port up   1615 to   Port Down   Port up   1615 to                   Blocking           Blocking            HELLOs generated by other RA and sent towards ports A and B of this RA. One HELLO       arrives slightly before the other.            2   Blocking   RxHello &gt; node   1630 to Slave   Blocking                           Timing       3   Slave Timing           Blocking   RxHello &gt; node   1630 to Slave                               Timing       4   Slave Timing   fd timer   1635 to   Slave Timing               expires and   SLAVE               partner not   FORWARD               SLAVE               FORWARD       5   Slave forward           Slave Timing   [cannot                           advance to                           Slave Forward                           as partner is in                           Slave                           Forward]       6   Link breaks       7   Slave Forward   Link breaks,   1650 to Master   Slave Timing               age timer   Forwarding               expires       8   Master           Slave Timing   fd timer   1635 to           Forward               expires and   SLAVE                           partner not   FORWARD                           SLAVE                           FORWARD       9   Master           Slave Forward           Forward       10   Link Restored       11   Master   RxHello &gt; Node   1660 to   Slave Forward           Forward   and   Blocking               partner Slave               Forward       12   Blocking   RxHello &gt; Node   1630 to Slave   Slave Forward                   Timing            This continues until a port goes down, a link goes down, or the node number of the other RA       changes to become less than Node (normally this would take a reconfiguration from the       operator or the substitution of another RA unit).                  
 
         [0091]    Full Ring Mode—Ring Relay  
         [0092]    [0092]FIG. 7 shows the state machine for a Ring Relay node.  
         [0093]    Description of States  
         [0094]    The state machine of FIG. 7 has the following states.  
                       TABLE E                       Number   State   Description                   700   PORT DOWN   The port is operationally down or has just               been initialized. Entered from any state               on an indication of the port going down               due to a loss of signal or other similar alarm.       710   AWAITING   Port is operationally up, but no HELLO           HELLO   has been received       720   TIMING   The port is waiting for the fd timer to expire       730   FORWARDING   Normal forwarding.                  
 
         [0095]    Description of State Transitions  
         [0096]    The table below describes the transitions of the state machine shown in FIG. 7.  
                       TABLE F                       Number   Event   Action                   1710   port operationally down OR init   block user traffic, cancel all               timers       1715   port operationally up   start age timer       1720   age timer expires OR RxHello   start fd timer       1725   fd timer expires   forward user traffic                  
 
         [0097]    High Availability Mode—Ring Arbiter—Ring Side  
         [0098]    [0098]FIG. 8 shows the state machine for the Ring Side (RS) of a Ring Arbiter in HA mode.  
         [0099]    Description of States  
         [0100]    The state machine of FIG. 8 has the following states.  
                       TABLE G                       Number   State   Description                   800   PORT DOWN   The port is operationally down or has just               been initialized. Entered from any state.       810   BLOCKING   The port is sending HELLOs, but is not               forwarding data traffic.       820   SLAVE   The port on a Slave Node is forwarding           FORWARDING   packets       830   MASTER   The port on a Master Node is forwarding           FORWARDING   packets                  
 
         [0101]    Description of State Transitions  
         [0102]    The table below describes the transitions of the state machine shown in FIG. 8.  
                       TABLE H                       Number   Event   Action                   1810   port operationally down OR init   block user traffic, cancel               age timer       1815   port operationally up   start age timer       1820   (age timer expires AND isMaster)   forward user traffic           OR RxHello &lt; Node       1825   (age timer expires AND isSlave)   forward user traffic           OR RxHello &gt; Node       1830   RxHello &lt; Node   No action       1835   RxHello &gt; Node   No action                  
 
         [0103]    High Availability Mode—Ring Arbiter—Extension Side  
         [0104]    [0104]FIG. 9 shows the state machine for the Extension Side (ES) of a Ring Arbiter in HA mode.  
         [0105]    Description of States  
         [0106]    The state machine of FIG. 9 has the following states.  
                       TABLE I                       Number   State   Description                   900   PORT DOWN   The port is operationally down or has just               been initialized. Entered from any state.       910   BLOCKING   The node is sending HELLOs, but not               forwarding data traffic.       920   SLAVE   The port on a Slave Node is forwarding           FORWARDING   packets       930   MASTER   The port on a Master Node is forwarding           FORWARDING   packets                  
 
         [0107]    Description of State Transitions  
         [0108]    The table below describes the transitions of the state machine shown in FIG. 9.  
                       TABLE J                       Number   Event   Action                   1910   port operationally down OR init   block user traffic, cancel               age timer       1915   port operationally up   start age timer       1920   (age timer expires AND isMaster)   forward user traffic           OR RxHello &lt; Node       1925   (age timer expires AND isSlave)   forward user traffic       1930   RxHello &lt; Node   continue forwarding user               traffic       1935   RxHello &gt; Node AND PartnerHello   continue forwarding user           Timeout   traffic       1940   RxHello &gt; Node AND NOT   block user traffic, start           PartnerHello Timeout   age timer       1945   RxHello &gt; Node AND NOT   block user traffic, start           PartnerHello Timeout   age timer                  
 
         [0109]    As shown in the sequence of events reported in the table below, the RS ports of the Arbiters are always forwarding, unless the ports are OPER DOWN. The point of interest is the state of the ES port of the slave Arbiter. In essence, the ES slave port is FORWARDING if there is a HELLO timeout on either the RS or ES.  
                                                                                       TABLE K                                       Port Status               (before trigger)                500   500   530   530               TIME   RS   ES   RS   ES   Trigger   Reaction                    1   800   900   800   900   500 initialized   500 RS goes Blocking,                               500 ES Goes to Blocking       2   810   910   800   900   530 initialized   530 RS goes Blocking,                               530 ES Goes to Blocking       3   810   910   810   910   500 receives HELLO from 530   500 RS state change 1820 to                           and RxHello &lt; node   Master Forwarding                               500 ES state change 1920 to                               Master Forwarding       4   810   910   830   930   530 received HELLO from 500   530 RS state change 1825 to Slave                           and RxHello &gt; node   Forwarding                               530 ES does not leave Blocking                               unless RS or ES has HELLO                               timeout       5   820   910   830   930       Continues operation with the                               virtual break in the HA ring at the                               ES of the slave (RA 500).       6   820   910   830   930   Break in link 1520 (ring side)       7   820   910   830   930   RxHellos stop coming on RS   500 RS no change                               530 ES state change 1925 to Slave                               Forwarding       8   820   920   830   930       All four ports forward traffic while                               there is a physical break       9   820   920   830   930   Break fixed       10   820   920   830   930   RxHello received at 530 RS and &gt; node   530 ES state change 1945 to                               blocking       11   820   920   830   930       Continues operation with virtual                               break.       12   820   910   830   930   Link break ES       13   820   910   830   930   HELLOs stop on ES side of both   530 ES notes that its age timer                           RA units   expires and it isSlave and has state                               change 1925 to slave forwarding       14   820   920   830   930       All four ports forward traffic while                               there is a physical break       15   820   920   830   930   Break fixed       16   820   920   830   930   530 ES receives RxHellos &gt; node   530 ES moves along state                           and not   transition 1945 to Blocking                           PartnerHelloTimeout       17   820   910   830   930       Until next break, port down, or                               switch in node numbers sufficient                               to change master/slave                               relationship.                  
 
       ALTERNATIVE EMBODIMENTS  
       [0110]    Unidirectional Break  
         [0111]    The control system described above assumes that a break in a network ring will be a bidirectional break as it connects both the clockwise and counterclockwise virtual breaks upon failure to receive a HELLO. This bidirectional response could cause a loop in the event of a unidirectional failure.  
         [0112]    [0112]FIG. 10, adds additional detail to the drawing shown in FIG. 2. More specifically, the links are shown in their unidirectional components rather than as bidirectional links.  
         [0113]    For example, when the network ring is fully operational, Master Arbiter  1000  can receive HELLOs from Slave Arbiter  1025  via link  11037 , relay  1020 , link  11027 , relay  1010 , and link  11012 . Likewise, Slave Arbiter  1025  can receive HELLOs from Master Arbiter  1000  via link  11010 , relay  1010 , link  11025 , relay  1020 , and link  11035 .  
         [0114]    If link  11027  was cut but link  11025  was left in service, then the West port on Master Arbiter  1000  would soon stop receiving HELLOs from Slave Arbiter  1025 , while Slave Arbiter  1025  continued to receive HELLOS from Master Arbiter  1000 . In the previously described embodiment, this unidirectional cut at link  11027  would not trigger the Slave Arbiter  1025  to unblock as it continues to receive HELLOs from Master Arbiter  1000  across intact link  11025 . Thus ring relay  1020  as well as connected subscriber ports  1045 would be cut off from the east side of the ring as Slave Arbiter  1025  is still blocking data, including data that would otherwise travel from Slave Arbiter  1025  to ring relay  1020 .  
         [0115]    One alternative embodiment is to react to a port going to an OPER DOWN state by stopping the transmission of HELLOs and all data from that port in the opposite direction, effectively creating a virtual unidirectional break in the other direction. Hence when ring relay  1010  observes an OPER DOWN associated with link  11027 , ring relay  1010  would stop sending HELLOs and all data on link  11025 . After Slave Arbiter  1025  fails to receive HELLOs in an allotted time, the Slave Arbiter  1025  would remove the virtual break on its west side to allow data traffic from link  11035  to proceed towards link  11030  or the user ports  1060  and to allow traffic from link  11032  or user ports  1060  to proceed onto link  11037 .  
         [0116]    Dual Homing Using a Single Node Ring  
         [0117]    [0117]FIG. 11 shows an application of a particular embodiment of the present invention that is referred to as “dual homing”. Dual homing allows a Slave Arbiter Node  1130  to provide protected access for User Ports  1140  to network via Ring Access Equipment nodes  1110  and  1120  using redundant links  1160  and  1170 .  
         [0118]    In this alternative embodiment, the Slave Arbiter node  1130  would see its own HELLOs. As described in Table C, one side of  1130  (for example the West side of the Slave Arbiter connected to link  1160 ) would go to the SLAVE FORWARDING state and one side (for example, the East side of the Slave Arbiter connected to link  1170 ) would go to the BLOCKED state.  
         [0119]    Now, in response to a fault on the Ring Access Equipment  1110  or the link  1160 , the East side of the Slave Arbiter  1130  would unblock, and the User Ports  1140  would continue to have access to the network. The network access for User Ports  1140  is therefore protected against faults in either the access links ( 1160  and  1170 ) as well as in the Ring Access Equipment nodes ( 1110  and  1120 ).  
         [0120]    The preferred embodiments disclose using a timing delay to ensure that a port progressing from OPER DOWN to operational delays sending data packets long enough for the slave arbiter to impose a virtual break. One of skill in the art will recognize that the use of the timer could be replaced by a control signal sent by the slave arbiter after it has successfully imposed the virtual break.  
         [0121]    One of skill in the art will recognize that alternative embodiments set forth above are not universally mutually exclusive and that in some cases alternative embodiments can be created that implement two or more of the variations described above.  
         [0122]    Those skilled in the art will recognize that the methods and apparatus of the present invention have many applications and that the present invention is not limited to the specific examples given to promote understanding of the present invention. Moreover, the scope of the present invention covers the range of variations, modifications, and substitutes for the system components described herein, as would be known to those of skill in the art.  
         [0123]    The legal limitations of the scope of the claimed invention are set forth in the claims that follow and extend to cover their legal equivalents. Those unfamiliar with the legal tests for equivalency should consult a person registered to practice before the patent authority which granted this patent such as the United States Patent and Trademark Office or its counterpart.  
                                             ACRONYMS                                    ES   Extension Side           FR   Full Ring           HA   High Availability           IP   Internet Protocol           MAC   Media Access Control           MPLS   Multiprotocol Label Switching           PDU   Packet Data Unit           PSR   Protected Switching Ring           RA   Ring Arbiter           RPR   Resilient Packet Ring           RR   Ring Relay           RS   Ring Side           TCP   Transmission Control Protocol           UDP   User Datagram Protocol