Detecting intermittent network link failures

An apparatus, system, and method are disclosed for detecting intermittent network link failures. A tracking module tracks link failures of a network link of a network over a specified time interval. A failure module determines the network link is failing in response to a number of link failures exceeding a specified failure threshold. A mitigation module mitigates communications over the network link.

BACKGROUND

The subject matter disclosed herein relates to detecting network failures and more particularly relates to detecting intermittent link failures.

2. Description of the Related Art

Networks are used to communicate data between multiple devices. A network will typically include a plurality of nodes, with network links between the nodes. Data may move from node to network link to node until the data arrives at a destination node.

If a network link between nodes fails, the data may be unable to reach a destination node. This is particularly true in a loop-free topology. As a result, networks often employ failure detection algorithms to detect the failure of network links.

In addition to a complete failure, a network link may fail intermittently. Such intermittent failures may not be detected by failure detection algorithms as the failure ends and the network link resumes transmitting data. As a result, no corrective action may be taken, although network link performance may be degraded or unreliable.

BRIEF SUMMARY

From the foregoing discussion, it should be apparent that a need exists for an apparatus, system, and method that detect intermittent network link failures. Beneficially, such an apparatus, system, and method would detect unreliable and degraded network links.

The present invention has been developed in response to the present state of the art, and in particular, in response to the problems and needs in the art that have not yet been fully solved by currently available network link failure detection methods. Accordingly, the present invention has been developed to provide an apparatus, system, and method for detecting intermittent network link failures that overcome many or all of the above-discussed shortcomings in the art.

The apparatus for detecting intermittent network link failures is provided with a plurality of modules configured to functionally execute the necessary steps of tracking link failures, determining a network link is failing, and mitigating communications. These modules in the described embodiments include a tracking module, a failure module, and a mitigation module.

The tracking module tracks link failures of a network link of a network over a specified time interval. The failure module determines the network link is failing in response to a number of link failures exceeding a specified failure threshold. The mitigation module mitigates communications over the network link.

A system of the present invention is also presented to detect intermittent network link failures. The system may be embodied in an Ethernet network. In particular, the system, in one embodiment, includes a network and a data processing device.

The data processing device is in communication with the network. The data processing device includes a tracking module, a failure module, and a mitigation module. The tracking module tracks link failures of a network link of a network over a specified time interval. The failure module determines the network link is failing in response to a number of link failures exceeding a specified failure threshold. The mitigation module mitigates communications over the network link.

A method of the present invention is also presented for detecting intermittent network link failures. The method in the disclosed embodiments substantially includes the steps necessary to carry out the functions presented above with respect to the operation of the described apparatus and system.

A tracking module tracks link failures of a network link of a network over a specified time interval. A failure module determines the network link is failing in response to a number of link failures exceeding a specified failure threshold. A mitigation module mitigates communications over the network link.

References throughout this specification to features, advantages, or similar language do not imply that all of the features and advantages may be realized in any single embodiment. Rather, language referring to the features and advantages is understood to mean that a specific feature, advantage, or characteristic is included in at least one embodiment. Thus, discussion of the features and advantages, and similar language, throughout this specification may, but do not necessarily, refer to the same embodiment.

These features and advantages of the embodiments will become more fully apparent from the following description and appended claims, or may be learned by the practice of embodiments as set forth hereinafter.

DETAILED DESCRIPTION

The computer readable medium may also be a computer readable signal medium. A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electrical, electro-magnetic, magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport computer readable program code for use by or in connection with an instruction execution system, apparatus, or device. Computer readable program code embodied on a computer readable signal medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, Radio Frequency (RF), or the like, or any suitable combination of the foregoing.

FIG. 1is a schematic block diagram illustrating one embodiment of a network100. The network100includes a plurality of nodes105. The nodes105may be routers, bridges, and the like. The nodes105are in communication via network links120. One or more data processing devices110may be in communication with the network100through the nodes105. The network100may also be in communication with an external network115through a first node105a.

In one embodiment, the network100is an Ethernet network. The Ethernet network may be selected from the group consisting of a one Gigabit Fiber Channel over Ethernet (FCoE) network, a ten Gigabit FCoE network, an Internet Small Computer System Interface (iSCSI) network, a Hyper Small Computer System Interface (HyperSCSI) network, and an Advanced Technology Attachment (ATA) over Ethernet (AoE) network, and the like.

In one embodiment, the network links120employs transport medium selected from the group consisting of a fiber optic transport medium, a copper transport medium, and a wireless transport medium. In an alternate embodiment, the network links120employ a combination of two of more of the fiber optic transport medium, the copper transport medium, and the wireless transport medium.

In one embodiment, the network100may be organized as a loop-free topology. For example, the network100may be organized as mesh of connected nodes105with a single active path between any two nodes105. Network links120that result in loops in the network100are disabled. In one embodiment, the network100employs a spanning tree protocol to remove loops from the network100. The spanning tree protocol may be a Digital Equipment Corporation Spanning Tree Protocol (DEC STP), Institute of Electrical and Electronic Engineers (IEEE) Rapid Spanning Tree Protocol (RSTP), per-VLAN spanning tree (PVST), Multiple Spanning Tree Protocol (MSPT), Rapid per-VLAN spanning tree (R-PVST), and the like.

Tests for a failed network link typically send test messages, referred to hereafter as heartbeat messages, to determine if a network link120is still operating or is failed. If the heartbeat messages are not received, a test may determine that the network link120is failed. However, a network link120may intermittently fail. If the network link120intermittently fails then occasionally a heartbeat message will be successfully communicated across the network link120. As a result, a test may not discover that the network link120is operating with intermittent failures.

FIG. 2is a schematic diagram illustrating one embodiment of a timeline200of network communications. The timeline200shows the transmission of exemplary messages210,225between a start time205and an end time220. The messages210,225may be communicated over the network100ofFIG. 1. The description of the timeline200refers to elements ofFIG. 1, like numbers referring to like elements.

The messages210,225include heartbeat messages210and data messages225. The messages210,225may be sent as packets as is well known to those skilled in the art.

In one embodiment, a heartbeat message210is sent after each heartbeat time interval215. In a certain embodiment, a specified number of heartbeat messages210are sent during each specified time interval. The specified time interval may be in the range of 6 to 18 seconds.

FIG. 3is a schematic block diagram illustrating one embodiment of a node105. The node105may be the node105FIG. 1. The description of the node105refers to elements ofFIGS. 1 and 2, like numbers referring to like elements. The node105includes a processor305, a memory310, and communication hardware315.

The memory310stores a computer readable program. The computer readable program may comprise instruction code and data. The memory310may be configured as a semiconductor memory, a hard disk drive, and optical storage medium, a micromechanical storage medium, and the like. In one embodiment, the memory310is a computer readable storage medium.

The processor305executes the computer readable program. The computer readable program may manage the communication of messages210,225over the network links120. The communication hardware315may comprise connectors, semiconductor logic, and the like. The communication hardware315may send and receive messages210,225over the network links120.

In one embodiment, the communication hardware315embodies a network layer1. In addition, the computer readable programs may embody a network layer2. The network layer1may be a physical layer. In addition, the network layer2may be a data layer.

In one embodiment, the network layer1is an Ethernet physical layer and the network layer2is an Ethernet data link layer. The Ethernet physical layer and Ethernet data link layer may conform to the Open System Interconnection (OSI) reference model and IEEE specification802.

Alternatively, the network layer1may be a Transmission Control Protocol/Internet Protocol (TCP/IP) transmission medium. In addition, the network layer2may be a TCP/IP Physical Protocol.

FIG. 4is a schematic block diagram illustrating one embodiment of a detecting apparatus400. The apparatus400may be embodied in one or more nodes105and/or one or more data processing devices110of the network100FIG. 1. The description of the apparatus400refers toFIGS. 1-3, like numbers referring to like elements. The apparatus includes a tracking module405, a failure module410, a mitigation module415, and a detection module420.

The tracking module405, failure module410, mitigation module415, and detection module420may be embodied in hardware circuits, computer readable programs stored on a computer readable storage medium, and in a combination of hardware circuits and computer readable programs. In one embodiment, the hardware circuits are embodied in the communication hardware315ofFIG. 3. In addition, the computer readable programs may be stored on the memory310and executed by the processor305ofFIG. 3.

The detection module420detects a link failure of a network link120. The link failure may be an intermittent network link failure. In one embodiment, the detection module420comprises an Ethernet physical layer. The Ethernet physical layer may be embodied in the communication hardware315FIG. 3. The Ethernet physical layer may look for a heartbeat message210each heartbeat time interval215. In one embodiment, the heartbeat message210is a fast link pulse. If the heartbeat message210is not received within the heartbeat time interval215, the detection module420may detect a link failure. In one embodiment, the link failure may be an Ethernet bouncing link condition.

The tracking module405may track the link failures of each network link120of the network100. In one embodiment, the tracking module405tracks each link failure of each network link120over the specified time interval. The tracking module405may be embodied in an Ethernet data link layer of a node105. In a certain embodiment, the tracking module405is embodied in a computer readable program stored on the memory310of a node105.

In one embodiment, the tracking module405records a log entry for each link failure. The log entry may be stored in the memory310of the node105. Alternatively, the log entry may be communicated to a network agent425. The network agent425may reside on a data processing device110in communication with the network100. In a certain embodiment, the network agent425may communicate with the network100through the external network115. The tracking module405may time stamp each log entry. Alternatively, the tracking module405may record each log entry in a log entry bin for each specified time interval. In one embodiment, the tracking module405overwrites previous log entries after each specified time interval.

In an alternate embodiment, the tracking module405increments a counter for each link failure such as when a heartbeat message210is missed. The tracking module405may reset the counter after each specified time interval.

In one embodiment, the tracking module405and communicates link failure information to the failure module410. The failure module410may be embodied in the network agent425.

The failure module410determines the network link120is failing in response to a number of link failures exceeding a specified failure threshold. In one embodiment, the failure module410determines the network link120is failing if the counter exceeds the specified failure threshold. In an alternate embodiment, the failure module410analyzes the log entries of link failures for each network link120. In one embodiment, the specified failure threshold is zero (0) link failures per the specified time interval. For example, if three heartbeat messages are typically sent each specified time interval, the specified failure threshold may be zero link failures per specified time interval, so that the failure module410determines the network link120is failing if one heartbeat message is not received.

The mitigation module415mitigates communications over the network link120. In one embodiment, the mitigation module415mitigates communications over the network link120by removing the network link120from the network100. In addition, the mitigation module415may route communications over an alternate network link120. In one embodiment, the mitigation module415disables each port in communication with the network link120to remove the network link120from the network100.

In one embodiment, the mitigation module415flags the network link120as a reduced performance network link. The network100may route communications to avoid each network link120flagged as a reduced performance network link unless no other option is available for communicating with a specified node105.

FIG. 5is a schematic flow chart diagram illustrating one embodiment of a network link failure detection method500. In one embodiment, the detection module420employs the network link failure detection method500to detect a link failure. The description of the method500refers to elements ofFIGS. 1-4, like numbers referring to like elements. The method500may be performed by hardware logic, computer readable programs stored on a computer readable storage device such as the memory310and executing on a processor such as the processor305, or any combination thereof.

The method500starts, and in one embodiment the network layer1detects505the missed heartbeat message210. The network layer1may reset a heartbeat message flag and a heartbeat message timer at the beginning of a heartbeat time interval215. The heartbeat time interval215may be less a specified time interval. The network layer1may set the heartbeat message flag when a heartbeat message210is received. If the heartbeat message flag is set before the heartbeat message timer times out the heartbeat time interval215, the heartbeat message210is not missed. However, if the heartbeat message flag is not set before the heartbeat message timer times out the heartbeat time interval215, the heartbeat message210is missed.

In an alternate embodiment, the network layer2detects505the missed heartbeat message210. The network layer2may reset the heartbeat message flag and the heartbeat message timer at the beginning of a heartbeat time interval215.

The network layer1may notify510the network layer2of the missed heartbeat message210. In one embodiment, the network layer1writes a digital value to register to notify510the network layer2of the missed heartbeat message210. Alternatively, the network layer1may assert an interrupt to notify510the network layer2of the missed heartbeat message210.

The network layer2notifies515the network agent425of the missed heartbeat message210and method500ends. In one embodiment, the network agent425resides on the same node105as the network layer2. Alternatively, the network agent425may reside on a node105distinct from the network layer2. The network layer2may notify515the network agent425using an internet control message protocol message. In an alternate embodiment, the network layer2notifies515the network agent425using a set-link-info command.

FIG. 6is a schematic block diagram illustrating one embodiment of protocol layers600of the network100. The protocol layers600include a physical layer605, a data link layer610, a network layer615, a transport layer620, a session layer625, a presentation layer630, and an application layer635. The description of the protocol layers600refers to elements ofFIGS. 1-5, like numbers referring to like elements

The protocol layers600may be for an Ethernet protocol. The physical layer605may be the Ethernet physical layer and the data link layer610may be the Ethernet data link layer. The application layer635may initiate communication of the message over the network100. The presentation layer630may encrypt the message. The session layer625may coordinate communication of the message. The transport layer620may transmit complete messages. The network layer615may route the message across the network100. The data link layer610may encode and decode the message. The physical layer605may send and receive the encoded message.

In an alternate embodiment, the protocol layers600are for the TCP/IP protocol. The physical layer605may be the TCP/IP transmission medium, the data link layer610may be a TCP/IP physical protocol, the network layer615may be a TCP/IP network protocol, the transport layer620may be TCP/IP TCP and UDP protocols, the session layer625may be a TCP/IP transport protocol, the presentation layer630may be a TCP/IP session protocol, and the application layer635may be a TCP/IP Application protocol.

In one embodiment, the network agent425resides in a layer selected from the group consisting of the data link layer610, the network layer615, the transport layer620, the session layer625, the presentation layer630, and the application layer635. In a certain embodiment, the network agent425resides in a layer selected from the group consisting of the session layer625, the presentation layer630, and the application layer635.

FIG. 7is a schematic flow chart diagram illustrating one embodiment of an intermittent network link failure detecting method700. The method700may embody the functions of the tracking module405, the failure module410, the mitigation module415, and the detection module420ofFIG. 4. The description of the method700refers to elements ofFIGS. 1-6, like numbers referring to like elements. The method700may be performed by hardware logic, computer readable programs stored on a computer readable storage device such as the memory310and executing on a processor such as the processor305, or any combination thereof.

The method700starts, and in one embodiment the detection module420resets705one or more counters and timers. In one embodiment, the detection module420resets705a heartbeat message counter. The heartbeat message counter may count heartbeat messages210for a specified network link120. In addition, the detection module420may reset a specified time interval timer. The specified time interval timer may time the specified time interval.

The tracking module405tracks710the link failures of a network link120. In one embodiment, the tracking module405tracks710each link failure of each network link120of the network100. In an alternate embodiment, the tracking module405tracks710link failures for one or more selected network links120. For example, the tracking module405may track710link failures for one or more randomly selected first network links120for a first testing time interval and then track710link failures for one or more second network links120for a second testing time interval. Alternatively, the tracking module405may systematically track710link failures for each network link120in turn for a specified tracking time interval.

In one embodiment, the failure module415determines715if the specified time interval is elapsed. If the specified time interval is not elapsed, the tracking module405tracks710the link failures of a network link120. If the specified time interval is elapsed, the failure module415determines720if the network link120is failing in response to a number of link failures exceeding a specified failure threshold.

In one embodiment, the specified failure threshold is zero link failures per the specified time interval. For example, the specified failure threshold may be 0 and/or 1 failed heartbeat messages210per specified time interval. Thus, for example, if there is one link failure at a network link120followed by one or more successfully received heartbeat messages210, the failure module415may still determine that the network link120is failing.

In one embodiment, the specified failure threshold is set in response to a service level. For example, if the service level is a high service level, the specified failure threshold may be set to tolerate no intermittent failures. Alternatively, if the service level is a low service level the specified failure threshold may be set to allow on occasional intermittent failure of the network link120.

In one embodiment, the specified failure threshold t is calculated using Equation 1, where k is a non-zero constant, s is a service level between 0 and 100%, and n is a number of heartbeat messages210per specified time interval.
t=ksn  Equation 1

If the failure module415determines720of the number of link failures during the specified time interval does not exceed the specified failure threshold, the detection module420resets705the counters and timers. If the failure module415determines720that the number of link failures during the specified time interval exceeds the specified failure threshold, the mitigation module415mitigates725communications over the network link120and the method700ends.

The mitigation module415may mitigate725communications over the network link120by removing the network link120from the network100. In one embodiment, the mitigation modules415employs spanning tree protocol to remove the network link120from the network100. Alternatively, the mitigation module415may mitigate725communications over the network link120by communicating a removal notice to remove the network link120from the network100.

FIG. 8is a schematic diagram illustrating one embodiment of a timeline800of heartbeat messages210. The timeline800may be the timeline200ofFIG. 2, but only showing the heartbeat messages210. The description of the timeline800refers to elements ofFIGS. 1-7, like numbers referring to like elements. The timeline800shows the specified time interval805.

In the depicted embodiment, the three (3) heartbeat messages210are sent during the specified time interval805. In one embodiment, if three (3) heartbeat messages210are received each specified time interval805, the failure module415does not determine720the network link120is failing.

FIG. 9is a schematic diagram illustrating one alternate embodiment of a timeline800of heartbeat messages210. The timeline800is the timeline ofFIG. 8. The description of the timeline800refers to elementsFIGS. 1-8, like numbers referring to like elements.

In the depicted embodiment, only two (2) heartbeat messages210are received during the specified time interval805. The detection module420detects a link failure as one heartbeat message210was not received. The tracking module405tracks710one (1) link failure. If, for example, the link failure threshold is 0, the failure module410may determine720that the network link120is failing in response to the number of link failures exceeding the specified failure threshold. The mitigation module415may mitigate communications over the network link120by removing the network link120from the network100.

Thus if the network link120is intermittently failing, the intermittent failure is still detected. As a result, the network100may be reconfigured to use an alternate network link120. By removing intermittently failing network links120from the network100, the performance and reliability of the network100may be improved.