System and method for reliability-based load balancing and dispatching using software rejuvenation

A method of operating a node of a computer network which uses a plurality of servers, by determining that one of the servers has degraded health due to software aging, assigning tasks to the other servers while reducing workload at the first server, rejuvenating the first server once its workload has terminated and, after rejuvenation, assigning tasks to the first server. The servers are clustered to provide service based on a single server address (TCP/IP). The node may include a gateway interface which receives the server requests and passes them on to a dispatcher at the node. Tasks are assigned in response to health-related messages sent by the servers and received by a workload monitor agent of the dispatcher.

DESCRIPTION OF AN ILLUSTRATIVE EMBODIMENT The present invention is directed to a method of enhancing the performance and reliability of a distributed processing system, particularly a system that is part of a computer network such as a local area network (LAN) or the Internet, similar to that depicted in FIG. 1 . The invention may, however, be implemented in other networks so, while the present invention may be understood with reference to FIG. 1 , this reference should not be construed in a limiting sense. With further reference to FIG. 2 , there is depicted one embodiment 12 of a multi-server network node constructed in accordance with the present invention. Node 12 is adapted to act as a single network location, e.g., a single TCP address. In an exemplary implementation, node 12 is an internet server, and may provide web pages in hypertext transfer protocol, or provide other electronic information using other conventional protocols. Node 12 is generally comprised of a gateway interface 14 , a plurality of servers 16 a , 16 b and 16 c , and a task dispatcher 18 . While three servers are shown, those skilled in the art will appreciate that a smaller or larger number of servers may be utilized in variations of the present invention. Gateway 14 uses a conventional interface to communicate with the remainder of the network 20 , i.e., other gateways, routers or bridges which provide connectivity with end users at client workstations. While gateway 14 and dispatcher 16 are shown as separate logical entities, they may be implemented on a single data processing system. This data processing system may be a conventional, general-purpose computer programmed according to the teachings herein, and provided with one or more network interface devices such as an ethernet card. This same data processing system may also act as one of the servers. Dispatcher 18 acts to spread out the workload among the servers 14 a , 14 b and 14 c . Dispatcher 18 includes a workload monitor 22 which receives performance and health-related messages from each of the servers. As with the prior art, dispatcher 18 uses this information to balance the overall workload across all of the servers. Dispatcher 18 receives client requests via gateway 14 , and task assignment logic 24 assigns the next task to the server with the lightest current workload, to avoid any given server from becoming overloaded. Each server has an application performance and health monitor 26 a , 26 b , and 26 c . The application performance and health monitors are processes running on each server which use conventional techniques to evaluate server performance and health based on the current usage of various system resources. Application performance and health monitors 26 a , 26 b , and 26 c construct a performance and health-related message to inform dispatcher 18 how busy and healthy the particular server is. Application performance and health monitors 26 a , 26 b , and 26 c additionally provide the novel function of informing dispatcher 18 whenever a server requires software rejuvenation. Rejuvenation services may be indicated by observing various signs of software aging including, but not limited to, excess memory usage or overflows, software exceptions, livelocks, deadlocks, etc. This invention improves the overall system availability of a web by applying the software failure prediction technology to the existing framework in which a Network Dispatching (ND) component is used. Currently, the TCP servers used in this configuration send performance related information (via messages) to the ND so that Load Balancing can be accomplished. This invention extends this concept, so that the TCP servers will also send health-related information to the ND. In one implementation, instead of providing an indication of how busy the server is, a health-related message indicates that the server needs to go offline completely. This message is recognized by service indicator logic 28 , and dispatcher 18 then begins transitioning workload off of this server and onto other active and operational servers. In an alternative implementation, the service (health-related) message can be appended to the performance-related message, to inform the ND of the current workload as well. In the depicted embodiment, service indicator logic 28 is integrated into workload monitor 22 . The workload will dwindle to zero as new workload is steered to other servers and old requests on the aging server are completed. When all the workload has been removed, the selective rejuvenation process can begin; the server can be taken offline with little or no disruption in the overall service of node 12 . The server may be rejuvenated in a conventional manner by, e.g., re-initializing the server application, middleware, or operating system. Once rejuvenation has been completed, the rejuvenated server can rejoin the server group by notifying dispatcher 18 (via workload monitor 22 ) that it is available, and begin accepting workload again. The present invention thus helps to eliminate unplanned partial system outages by predicting an imminent failure, taking the appropriate steps to move user sessions to an alternative operational and healthy server, proactively servicing the unhealthy server via software rejuvenation, and returning it to active service. This procedure improves the overall system availability to the end user, eliminates disruptive unplanned outages and transparently transitions them to a more reliable operating environment. This implementation of the present invention may further be understood with reference to the flow chart of FIG. 3 . The process begins with each server evaluating its current performance ( 30 ). The servers then transmit performance-related and/or health related messages to the dispatcher ( 32 ). The messages are received by the dispatcher and processed by the workload monitor/service indicator ( 34 ), and a determination is made as to whether any of the servers requires rejuvenation ( 36 ). If not, the task assignment logic at the dispatcher uses its normal workload distribution routine ( 38 ), and assigns various tasks to the specified servers ( 40 ). The servers process those tasks ( 42 ), and the process repeats in an iterative fashion. If the determination step 36 indicates that rejuvenation is required, then the task assignment logic instead begins to transition the workload away from the aged server ( 44 ). Tasks are again assigned ( 40 ), although now in a manner which will eliminate new tasks being assigned to the aged server. When activity has ceased, the aged server can be taken offline. After rejuvenation has been completed, the aged server can rejoin the group by notifying the dispatcher. Although the invention has been described with reference to specific embodiments, this description is not meant to be construed in a limiting sense. Various modifications of the disclosed embodiments, as well as alternative embodiments of the invention, will become apparent to persons skilled in the art upon reference to the description of the invention. For example, while the illustrative embodiment has been described in the context of a client-server network, those skilled in the art will appreciate that it can be practiced in a peer-to-peer network as well. In addition, this technique is applicable to other computing environments where load-based dispatching to an aggregate of servers is used; examples include transaction processing, file serving, application serving, messaging, mail serving, and many others. It is therefore contemplated that such modifications can be made without departing from the spirit or scope of the present invention as defined in the appended claims.