Method and apparatus for detecting failures in network components

A monitoring system includes a monitoring component that compares a number of application-specific request packets sent with a number of response packets received pursuant to the sent request packets during a designated monitoring period to detect failure in a monitored component without using any pinging packets. The application-specific request and response packets contain data to perform tasks specific to an application that is not associated with operational status monitoring information and the request and response packets are not specifically ping packets. A method for monitoring includes counting a number of application-specific request packets sent with a number of response packets received during a monitoring period without any pinging. If one or more request packets are sent but no response packets are received during a designated monitoring period, the monitored component is identified as failed. If some, but not all, response packets are received, the monitored component is identified as operational. Alternatively, if no response packets are received after a first monitoring period, the monitored component is identified as potentially failed. If no response packets are received after the end of a second monitoring period, the monitored component is identified as failed.

BACKGROUND

The operational status of different network processing devices needs to be monitored to ensure reliable network operation. If a network device is identified as being down, the network administrator or automated mechanisms can reconfigure the network around the failed device to a standby device. The standby device is maintained in the same state as a primary device so that network operations can be maintained if the primary device fails.

It is difficult to accurately identify a device failure. Techniques such as pinging have been developed to monitor network components. Pinging operations monitor devices by sending test packets. If the ping test packet is not returned, a failure condition is identified for the network processing device. However pinging uses substantial network bandwidth to send test packets back and forth between the monitoring device and the device under test.

A Resource Policy Management System (RPMS) consists of multiple stateful components and a state-less component called a Remote Access Service Router (RASER). All of these components are implemented as individual processes and can be deployed on separate hosts. These components communicate with each other via a User Datagram Protocol (UDP). A response always follows a sent request. The RASER acts as a front-end for the system and receives all traffic from Universal Gateways (UGs). The RASER routes this traffic to the appropriate RPMS component. The RASER also routes inter-component traffic.

The stateful RPMS components can be deployed as hot-standby pairs. For fault-tolerance and in case the component fails, all traffic can be re-directed to the RPMS standby component. Since the RASER routes traffic, it should be able to detect component failures and redirect traffic. This includes process, host or network failures.

However, using UDP in RPMS communications can indicate component failures even when the host machine is available. To solve this failure detection problem, pinging is used to periodically send test packets to the RPMS components. If the test packets are not returned (ping failure), communication is switched to a standby component. However, as described above, pinging each RPMS component uses substantial network bandwidth.

SUMMARY OF THE INVENTION

Application-specific packets sent to a component and received back from the component are counted for a monitoring period. If one or more request packets are sent during the monitoring period, but no response packets are received back during the monitoring period, the component is identified as down. Alternatively, the component can be identified as potentially down after the first monitoring period. If no response packets are received back after the end of a second monitoring period, the component is identified as down.

DETAILED DESCRIPTION

FIG. 1shows a system that includes a component A and a component B. In one example, components A and B are software applications that run on network processing devices that communicate over an Internet Protocol (IP) network. In this example, the network processing devices operating components A and B can be routers, switches, network access servers, or any other type of device that communicates over a packet switched Local Area Network (LAN) or Wide Area Network (WAN).

The components A and B may both be on the same machine or separate machines. For example, component A may be located on a network access server and component B may be located on a separate authentication server. Alternatively, components A and B may be associated with software applications or individual processors on the same device.

A software application associated with component A sends application-specific requests12to component B. The application associated with component A expects a response14pursuant to the request12. The requests12comprise request packets13and the responses14comprise response packets15.

There may be a requirement that component A monitor component B and that component A take some appropriate action when it detects that component B has failed. For example, component A may send requests12to a standby component (not shown) that can provide similar services when component B is down.

The request12and response14are not normally used for identifying operational status of component B. However, since each request12requires a response14, a request and response monitor17can use the application communication protocol to also determine the operational status of component B. Request packets13may be sent to component B, but no response packets15may be sent back from component B. From this information the request and response monitor17can determine when component B is down. This prevents having to send special ping test packets between the components A and B that require additional network bandwidth.

Any application that sends requests and expects responses to those requests can also be used to monitor operational status of a component. For example, the application may be requesting access to an Internet Service Provider (ISP) network, requesting establishment of a Voice Over IP (VoIP) connection with another endpoint, requesting access a webserver, etc.

The requests12and responses14are application-specific. This means that either the request12, response14, or both, contain information that is used by the receiving component to perform an application that is not associated with component operational status monitoring. For example, the request12may send login information that component B uses to authorize component A to access a network system.

FIG. 2describes the monitoring scheme performed by the request and response monitor17ofFIG. 1. The monitor17in component A initializes a PacketSent counter and a PacketReceived counter both to zero in block16. A monitoring interval is also reset in block16. The monitoring interval is used to determine how often to evaluate the PackSent and PacketReceived counters. The monitoring interval can be user configurable and typically is set to a time longer than the typical time required for component A inFIG. 1to receive a response packet15back from component6pursuant to a request packet13.

In block18the PacketSent counter is incremented by one each time a request packet13is sent by component A. The PacketSent counter is incremented only for request packets13that require component B to send back a response packet15. As explained above, the request packets13contain application-specific payloads. For example, component A may be associated with a webbrowser application that requests data from a webserver associated with component B. The webbrowser expects a response back from the webserver for each data request12.

The PacketRecieved counter is incremented by one in block20each time a response packet15is received by component A. The number of request packets13sent by component A and the number of response packets15received by component A continue to be counted until the monitoring interval has been reached in decision block22.

In decision block24, the PacketSent count is compared with the PacketReceived count. If the PacketSent count is greater than zero and the PacketReceived count is not zero, then component B is determined to be operational and the monitoring process jumps back to block16. However, if component A has sent request packets to component B during the monitoring period (PacketSent>0) but no response packets have been received back (PacketReceived=0), component B is identified as dead in block26.

Component A can take an appropriate action if component B is identified as dead in block26. For example, component A may send a message to a network system administrator indicating component B as down. Component A can also record the error in memory. Component A could also automatically switch over to another standby component if a failure condition is detected.

Since the monitoring system described above does not rely on periodic pinging or similar mechanisms, traffic overhead is not introduced into the network for component monitoring. The failure detection latency (the time before a failure is detected) is no worse than a ping-based monitoring approach. The monitoring system is also very simple to implement and uses very little programming overhead.

Identifying Failures Over Multiple Monitoring Intervals

A monitoring component could send out a request packet13near the end of a monitoring period. In this situation, the monitored component may not be able to send a response packet15back before the end of the monitoring period. Because a response packet15is not received prior to the end of the monitoring period, the monitoring component could falsely identify the monitored component as dead.

For example, assume that the monitoring period is five seconds. Also assume that the minimum time that component B can send back a response packet is two seconds. Consider a situation where no request packets are sent from component A to component B for the first 4 seconds of the monitoring period. During the fifth second of the monitoring period component A sends a request packet13to component B.

When the count check is performed at the end of the monitoring period after the fifth second, a failure is identified for component B since it takes at least two seconds for a response packet to be received back from component B. In other words, component B has not been given enough time to respond to the request packet.

This problem can be solved by identifying component B as potentially dead when it fails during a first monitoring period. Component B is then monitored for an additional monitoring period. If no response packets are received during the first or second monitoring period, component B is identified as dead. This provides enough time for component B to respond to request packets sent at the end of the first monitoring period.

FIG. 3shows in more detail how a component uses multiple monitoring periods to more accurately detect component failures. In block30the monitoring component initializes an additional FailureCount counter in addition to the PacketSent and PacketReceived counters. The monitoring period is reset in block32. The number of request packets13and response packets15are counted in blocks34–38in a manner similar to that described above inFIG. 2.

If the PacketSent count is greater than zero and the PacketReceived count is zero, then the monitoring component increments the FailureCount in block42. If the FailureCount is less than two in decision block44, the monitoring component considers component B potentially dead. In the potentially dead state, the monitoring component jumps back to block32and resets the monitoring interval. The PacketSent count and the PacketReceived count continue to be tracked for a second monitoring interval in blocks34–38.

If the PacketReceived count is no longer zero after the end of the second monitoring period, the monitored component is no longer considered potentially dead. The PacketSent and PacketReceived counters are reinitialized in block30and another monitoring interval is started.

However, if the PacketReceived counter still remains at zero at the end of the second monitoring period in decision block40, the FailureCount is incremented again in block42. The monitored component changes state from potentially dead to dead.

It should be noted that in the examples above the monitored component is not identified as dead even if it responds to some, but not all, of the request packets. This means that the monitored component is still available, even though it may not be operating at maximum efficiency. Knowledge of the monitored component only responding to some of the request packets can be used by the monitoring component to take other actions. For example, an alert email may be sent to the system administrator.

The monitoring component can also vary the criteria for identifying a failure. For example, a failure may be identified if the PacketReceived count does not reach some minimal percentage of the PacketSent count. The monitoring component can also report the PacketReceived and PacketSent counts to the system administrator or switch to communicating with a standby component, when the PacketReceived count falls below a predetermined minimum percentage of the PacketSent count. The number of monitoring periods can also be increased to more than two, before identifying a dead component.

FIG. 4shows a specific example where the monitoring system is used in a network access sever50to monitor the status of an Authentication Authorization and Accounting (AAA) server54. Of course this is just one example, and the Network Access Server (NAS)50can be any network device that needs to monitor status of another network processing device.

In this example, the NAS50receives a dial up request72from an endpoint70. The endpoint70may be any Personal Computer (PC), server, Voice Over IP (VoIP) phone, etc. that needs to access another network processing device, endpoint or system in an IP network. For example, the endpoint70may want to access an Internet Service Provider (ISP) network. But the endpoint70is not allowed to connect into the ISP network until it is authenticated by authentication server54.

In order to maintain operation even during a server failure, multiple standby servers56and58may be maintained by the ISP. Each standby server56and58is maintained in the same state as server54. One method for maintaining a standby server in the same state as a primary server54is described in co-pending U.S. patent application entitled: High Availability Network Processing System, Ser. No. 10/143,066, filed on May 10, 2002 which is herein incorporated by reference.

A proxy52can be used for relaying requests60and responses62between the NAS50and the primary server54. The proxy52also maintains a configuration file (not shown) that contains the locations of standby servers56and58. The proxy52can hide these connection details from the NAS50thus simplifying transmit operations. The proxy52may be located on a separate network device or may be located on one of servers54,56or58.

The proxy52typically sends requests to primary server54and will switch over to one of the standby servers56or58if the primary server54does not respond to request64. Any switch-over by proxy52can be initiated using the monitoring scheme described above. In this example, the proxy52is analogous to component A inFIG. 1and the server54is analogous to component B inFIG. 1.

Responses66should be received back for proxied requests64. The proxy52tracks the number of proxied request packets sent to the server54. The number of packets in request64is compared with the number of response packets in response66received back from server54. If during the designated monitoring period(s) one or more request packets are sent, and no response packets are received, the proxy52identifies the server54as down. The proxy52then may switch over to another one of the servers56or58. Operational status of server54is identified by piggy backing the operational status monitoring on top of the requests64and responses66that are already conducted between NAS50and server54. Thus, there is no need to send additional ping packets.

In one specific implementation, the NAS50uses a Remote Access Dial-In User Service (RADIUS) request and respond protocol to communicate with proxy52. In another example, a Resource Policy Management System (RPMS) uses the monitoring scheme described above.

The RADIUS protocol may be used by NAS50for transferring authentication and accounting information. The RADIUS protocol is based on the User Datagram Protocol (UDP). Typically, a login for user70consists of a query (Access-Request) from the NAS50to the RADIUS server54and a corresponding response (Access-Accept or Access-Reject) from the RADIUS server54. In this example proxy52and server54can be considered the RADIUS server54.

An access-request packet in request60contains the username, encrypted password, NAS IP address, and port information. The format of the request60also provides information on the type of session that the user70wants to initiate. When the RADIUS server54receives the access-request60from the NAS50, it searches a database for the username listed. If the username does not exist in the database, either a default profile is loaded or the RADIUS server54immediately sends an access-reject message in response62. This access-reject message can be accompanied by an optional text message, which can indicate the reason for the refusal.

If the username is found and the password is correct, the RADIUS server54returns an access-accept response62, including a list of attribute-value pairs that describe the parameters to be used for this session. Typical parameters include service type, protocol type, IP address to assign the user, access list to apply, or a static route to install in the NAS routing table. The configuration information in the RADIUS server54defines what will be installed on the NAS50. The monitoring system described above is incorporated in the RADIUS application to automatically identify failures in servers54–58.

In yet another implementation, block50inFIG. 4may represent a web-browser component, the proxy52may represent a web content cache and the primary server54may be a web server. The web cache52may send certain requests to the web-server54that require a response66. If no response is received for web-browser requests64within a designated monitoring period, the web cache switches over to a standby web-server56or58.

The system described above can use dedicated processor systems, micro controllers, programmable logic devices, or microprocessors that perform some or all of the operations. Some of the operations described above may be implemented in software and other operations may be implemented in hardware.

Having described and illustrated the principles of the invention in a preferred embodiment thereof, it should be apparent that the invention may be modified in arrangement and detail without departing from such principles. Claims are made to all modifications and variation coming within the spirit and scope of the following claims.