Abstract:
A technique for maintaining server cluster consistency is disclosed. When a front-end server of a plurality of servers detects that the front-end server has reconnected to a backend server or in the situation of a backend server failover, the front-end server sends resource ownership information to a backend server. The resource ownership information relates to ownership information for each respective resource owned by the front-end server. The back-end server verifies the ownership information for contention with static resources and dynamic resources owned by other servers and sends back verification information to the server. The backend maintains resource ownership information for all the servers in the cluster than sent resources for verification. Based on the received verification information, the server maintains ownership information at the server for each respective resource indicated in the verification information to be owned by the server. The server discards ownership of resources that were verified by the backend server to be not owned by the server. According to another aspect, resource ownership information is received at a backend server from a server of a plurality of servers. The received resource ownership information contains resource ownership information for the server. The received resource ownership information is compared to resource ownership information stored at the backend server. The backend server resolves any contention for a resource and sends verification information to the server. The verification information contains information verifying ownership for each resource indicated by the backend server that is owned by the server.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     The present application is related to U.S. patent application Ser. No. 09/611,318, entitled “Identification of Active Server Cluster Controller,” invented by F. Mohammed et al., and to U.S. patent application Ser. No. 09/611,314, entitled “Load Balancing of Chat Servers Based on Gradients,” invented by N. Dani et al., both of which are incorporated by reference herein and filed concurrently herewith. 
    
    
     TECHNICAL FIELD 
     The present invention relates to the field of computer network servers. More particularly, the present invention relates to a method for recovering cluster consistency following failover of a backend server. 
     BACKGROUND OF THE INVENTION 
     In a server cluster consisting of a backend server and a plurality of front-end servers, front-end servers can become temporarily disconnected from the backend server. Later when a disconnected server reconnects to the backend server, resource ownership information maintained by the reconnecting server might conflict with resource ownership information maintained on other servers of the server cluster. Additionally, in situations of a backend server failover, resource ownership information maintained in a newly active backend server may conflict with resource ownership information maintained by the respective servers of the server cluster. 
     What is needed is a technique maintaining server cluster consistency. 
     SUMMARY OF THE INVENTION 
     The present invention provides a technique for maintaining server cluster consistency. One aspect of the present invention provides a method for recovery of server cluster consistency. According to this aspect of the invention, when a front-end server detects that it has reconnected to a plurality of servers or in the situation of a backend server failover, each front-end server from the plurality of servers sends resource ownership information to the backend server. The resource ownership information relates to ownership information for each respective resource owned by the server. The backend server verifies the ownership information for contention with static resources and dynamic resources owned by other servers and sends back verification information to the server. The backend server maintains resource ownership information for all the servers in the cluster that sent resources for verification. Based on the received verification information, the server maintains ownership information at the server for each respective resource indicated in the verification information to be owned by the server. The server discards ownership of resources that were verified by the backend server to be not owned by the server. 
     According to another aspect of the invention, resource ownership information is received at a backend server from a server of a plurality of servers. The received resource ownership information contains resource ownership information for the server. The received resource ownership information is compared to resource ownership information stored at the backend server. The backend server resolves any contention for a resource and sends verification information to the server. The verification information contains information verifying ownership for each resource indicated by the backend server that is owned by the server. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     The present invention is illustrated by way of example and not limitation in the accompanying figures in which like reference numerals indicate similar elements and in which: 
     FIG. 1 is a schematic diagram of a conventional general-purpose digital computing environment that can be used to implement various aspects of the invention; 
     FIG. 2 is a schematic diagram of a server cluster including three servers each coupled to both a primary controller and a secondary controller. 
     FIG. 3 shows a flow diagram for choosing either the primary controller or the secondary controller as the active server cluster controller. 
     FIG. 4 shows a schematic block diagram of a server cluster that is suitable for the load prediction and balancing technique of the present invention according to the present invention; 
     FIG. 5 shows a flow diagram for determining the present and future load distributions for chat servers of a chat server cluster; 
     FIG. 6 shows a flow diagram setting forth the details of evaluating the loads carried by the respective chat servers of the chat server cluster and selecting a chat server for a new channel resource; 
     FIG. 7 shows an exemplary sliding window having a plurality of timeslots that are used for determining a load gradient of a chat server according to the present invention; 
     FIG. 8 shows a flow diagram for determining a consistent state of a server cluster when one or more chat servers in the cluster temporarily lose and regain connection to the rest of the server cluster according to the present invention; and 
     FIG. 9 shows a flow diagram determining a consistent state for a server cluster for the situation of a failover according to the present invention. 
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     Although not required, the invention will be described in the general context of computer-executable instructions, such as program modules, that are executed by a personal computer or a server. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the invention may be practiced with other computer system configurations, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices. 
     FIG. 1 is a schematic diagram of a conventional general-purpose digital computing environment that can be used to implement various aspects of the invention. Computer  100  includes a processing unit  110 , a system memory  120  and a system bus  130  that couples various system components including the system memory to the processing unit  110 . System bus  130  may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. System memory  120  includes a read only memory (ROM)  140  and a random access memory (RAM)  150 . 
     A basic input/output system (BIOS)  160  containing the basic routines that help to transfer information between elements within the computer  100 , such as during start-up, is stored in ROM  140 . Computer  100  also includes a hard disk drive  170  for reading from and writing to a hard disk (not shown), a magnetic disk drive  180  for reading from or writing to a removable magnetic disk  190 , and an optical disk drive  191  for reading from or writing to a removable optical disk  192 , such as a CD ROM or other optical media. Hard disk drive  170 , magnetic disk drive  180 , and optical disk drive  191  are respectively connected to the system bus  130  by a hard disk drive interface  192 , a magnetic disk drive interface  193 , and an optical disk drive interface  194 . The drives and their associated computer-readable media provide nonvolatile storage of computer readable instructions, data structures, program modules and other data for personal computer  100 . It will be appreciated by those skilled in the art that other types of computer readable media which can store data that is accessible by a computer, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memories (RAMs), read only memories (ROMs), and the like, may also be used in the exemplary operating environment. 
     A number of program modules can be stored on the hard disk, magnetic disk  190 , optical disk  192 , ROM  140  or RAM  150 , including an operating system  195 , one or more application programs  196 , other program modules  197 , and program data  198 . A user can enter commands and information into computer  100  through input devices, such as a keyboard  101  and a pointing device  102 . Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit  110  through a serial port interface  106  that is coupled to the system bus, but may be connected by other interfaces, such as a parallel port, a game port, a universal serial bus (USB) or through a PCI board. A monitor  107  or other type of display device is also connected to system bus  130  via an interface, such as a video adapter  108 . In addition to the monitor, personal computers typically include other peripheral output devices (not shown), such as speakers and printers. 
     Computer  100  can operate in a networked environment using logical connections to one or more remote computers, such as a remote computer  109 . Remote computer  109  can be a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to computer  100 , although only a memory storage device  111  has been illustrated in FIG.  1 . The logical connections depicted in FIG. 1 include a local area network (LAN)  112  and a wide area network (WAN)  113 . Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet. 
     When used in a LAN networking environment, computer  100  is connected to local network  112  through a network interface or adapter  114 . When used in a WAN networking environment, personal computer  100  typically includes a modem  115  or other means for establishing a communications over wide area network  113 , such as the Internet. Modem  115 , which may be internal or external, is connected to system bus  130  via serial port interface  106 . In a networked environment, program modules depicted relative to personal computer  100 , or portions thereof, may be stored in the remote memory storage device. 
     It will be appreciated that the network connections shown are exemplary and other ways of establishing a communications link between the computers can be used. The existence of any of various well-known protocols, such as TCP/IP, Ethernet, FTP, HTTP and the like, is presumed, and the system can be operated in a client-server configuration to permit a user to retrieve web pages from a web-based server. Any of various web browsers can be used to display and manipulate data on web pages. 
     It will be appreciated that the network connections shown are exemplary and other ways of establishing a communications link between the computers can be used. The existence of any of various well-known protocols, such as TCP/IP, Ethernet, FTP, HTTP and the like, is presumed, and the system can be operated in a client-server configuration to permit a user to retrieve web pages from a web-based server. Any of various conventional web browsers can be used to display and manipulate data on web pages. A primary aspect of the invention provides a failover algorithm that synchronously and automatically allows servers in a server cluster to determine the identity of the active server cluster controller. 
     The failover algorithm ensures that each server will properly identify which controller is active at any particular time, including, but not limited to, upon starting up or bootstrapping the server cluster, upon adding one or more servers to a cluster that is already operating, and upon failure of an active controller, one or more servers, or a link between an active controller and a server. 
     Referring to FIG. 2, a schematic view of a simple server cluster is shown. The server cluster includes servers  24 ,  26 , and  28 , each of which is coupled in a well-known manner to two servers designated as primary controller  20  and secondary controller  22 . 
     FIG. 3 shows a flow chart of steps that may be performed by servers, such as server  24 ,  26 , and  28  to determine whether primary controller  20  or secondary controller  22  is active at any particular time. For instance, when any of servers  24 ,  26 , and  28  are brought into service, by performing the steps set forth in FIG. 3, any such server may determine which controller is active. Although any number of servers may simultaneously be attempting to determine which controller is active, FIG. 3 will be discussed in the context of server  24  determining which controller is active. 
     At steps  302  and  304 , server  24  attempts to connect to primary controller  20  and secondary controller  22 , respectively. At step  306 , server  24  makes a determination of its own assessment of which controller is active based upon success or failure in establishing a connection with either or both controllers. If a connection is successfully established with both primary controller  20  and secondary controller  22 , then server  24  may assess one of the controllers as the active controller in accordance with a predetermined strategy, such as, for instance, assessing primary controller  20  as the active controller. If a connection is successfully established with one controller but not the other controller, then server  24  makes an own assessment of the active controller as the controller to which a successful connection was established. If the attempts to establish a connection between server  24  and both controllers fails, then steps  302  through  310  are repeated in a loop that will terminate upon successfully establishing a connection to either or both controllers. 
     Upon successfully connecting to primary controller  20 , the yes branch from step  308  is taken. Similarly, upon successfully connecting to secondary controller  22 , the yes branch from step  310  is taken. At steps  312  and  314 , server  24  notifies the controllers of the assessment that server  24  has made as to which controller is active. This assessment is referred to as an own assessment of the active controller because it is not based on the assessments made by any of the other servers in the cluster as to which controller is active. 
     At steps  316  and  318 , server  24  queries primary controller  20  and secondary controller  22  for the assessments of other servers in the cluster. Primary controller  20  may respond to such a query by sending the assessments of the other servers in the cluster, which have notified controller  20  of each of those servers&#39; own assessments of the active controller. Similarly, secondary controller  22  may respond to such a query by sending the assessments of the other servers in the cluster, which have notified controller  20  of each of those servers&#39; own assessments of the active controller. 
     At step  320 , server  24  makes a determination regarding whether either primary controller  20  or secondary controller  22  has a quorum of servers assessing that particular controller as the active controller. A quorum is preferably defined as at least a majority of the number of servers in the cluster, not including the primary and secondary controllers. For instance, in the server cluster shown in FIG. 2, a quorum would preferably be at least 2. If there is not at least a quorum of servers assessing either of the controllers as the active controller, then the no branch is taken from step  320  and steps  302  through  320  are repeated. Because each server preferably assesses only one controller as the active controller, and because a quorum is preferably defined as at least a majority of the servers, excluding the controllers, the situation in which a quorum of servers assesses both of the controllers as the active controller is preferably avoided. If one of the controllers has a quorum of servers assessing that controller as the active controller, then the yes branch is taken from step  320  to step  322 . At step  322 , server  24  treats as the active controller the controller with the quorum count of servers assessing it as the active controller, which is also referred to herein as the consensus active controller. 
     Referring to FIG. 4, the failover algorithm described above may be used to detect a failure of the active controller, which in the cluster server shown in FIG. 4 may be either primary backend server  43  or secondary backend server  44 . The failover algorithm may have directory service servers  41   a ,  41   b  and  41   c  and chat servers  42   a ,  42   b ,  42   c , and  42   d  each perform the steps depicted in FIG.  3 . While groups of chat servers, such as the directory service servers and the chat servers may have their own quorum counts, a consensus active controller is preferably determined based upon a quorum of substantially all servers in the server cluster. 
     Another primary aspect of the invention provides a predictive load balancer for distributing and balancing channels (chatrooms) and users (clients) among a plurality of chat servers. This aspect of the invention will be described with reference to FIGS. 4-6. 
     FIG. 4 shows a schematic block diagram of a server cluster  40  that is suitable for the load prediction and balancing technique of the present invention. Server cluster  40  includes a plurality of directory service servers  41   a ,  41   b  and  41   c , a plurality of chat servers  42   a ,  42   b ,  42   c  and  42   d , a primary backend server  43  and a secondary backend server  44 . Directory service servers  41   a - 41   c  and chat servers  42   a - 42   d  are each connected to a computer network  45 , such as the Internet, in a well-known manner. Each of directory service servers  41   a - 41   c  and chat servers  42   a - 42   d  are connected to each of primary backend server  43  and secondary backend server  44  in a well-known manner. Primary backend server  43  is designated as the active backend server during system initialization and provides well-known backend services, such as administrative functions and resource ownership monitoring. Secondary backend server  44  is designated as the inactive backend server during system initialization and operates as a backup backend server. In the event of a failover, secondary backend server  44  becomes the active backend server, as described above/elsewhere in this disclosure. While three directory service servers  41   a - 41   c  and four chat servers  42   a - 42   d  are shown in FIG. 4, any number of directory service servers  41  and any number of chat servers  42  can be used in server cluster  40 . 
     Directory service servers  41   a - 41   c  provide well-known directory service functions for a chatroom environment. For example, when a user accesses any of directory service servers  41   a - 41   c  and requests to enter an already existing chatroom (channel resource), the particular chatrooms existing on server computers  42   a - 42   d  are known to each of the directory service servers. The user is then redirected and connected to the requested chatroom. 
     In the situation when a user accesses a directory service server and requests a chatroom that does not exist on any of the chat servers, the directory service server assigns the new chatroom to a selected chat server  42   a - 42   d  based on present and future load distributions for the various chat servers  42   a - 42   d  the cluster  40 . The process used for determining the present and future load distributions among chat servers  42   a - 42   d  is described in greater detail below. The directory service server sends a message to the active backend server containing information relating to the channel resource assignment to the selected chat server. The active backend server in turn sends a message to the selected chatroom server notifying the selected chat server of the channel resource assignment. The active backend server also sends a message to each of the other directory service servers so that new users requesting the newly created chatroom can be redirected accordingly. The active backend server sends a message to each directory service servers  41   a - 41   c  and the particular chat server  42  to which the chatroom has been assigned. The directory service server requesting assignment of the channel resource then redirects the user to the appropriate chat server in a well-known manner. 
     A user at client computer  46  that is connected to computer network  45 , and who desires to enter a chatroom hosted by server cluster  40 , accesses one of the directory service servers  41   a - 41   c  in a well-known manner. That is, a browser application program operating on client computer  46  responds to a computer network address, such as a domain name, entered by the user by accessing a domain name service (DNS)  47  in a well-known manner for obtaining the computer network address, or universal resource locator (URL), for the domain name. DNS  47  sends a return message to the browser in client computer  46  containing the URL for the desired domain name. The browser then sends a page request message to one of the directory service servers  41   a - 41   c . The page request message is routed through computer network  45  in a well-known manner and is directed to the particular directory service server  41   a - 41   c  corresponding to the URL. 
     FIG. 5 shows a flow diagram  50  for determining the present and future load distributions for chat servers of a chat server cluster according to the present invention. At step  51 , a user (client) accesses a directory service server  42  and requests access to a chatroom be created. At step  52 , the directory service server determines whether the requested chatroom exists. If so, flow continues to step  53  where the user is connected to the existing chatroom. If, at step  52 , the requested chatroom is not yet existing, flow continues to step  54  where the directory service server evaluates the loads carried by the respective chat servers of the chat server cluster for determining which chat server to assign the new chatroom (channel resource). A chat server is selected based on the present loads and the future load estimations for each respective chat server so that the loads carried by each chat server will be balanced. 
     FIG. 6 shows a flow diagram  60  setting forth the details of evaluating the loads carried by the respective chat servers  42  of the chat server cluster and selecting a chat server for a new channel resource. The flow chart of FIG. 6 corresponds to step  54  in FIG.  5 . At step  61 , each chat server sends a message to the directory service server indicating the total number of users (clients) connected to the chat server during a predetermined time interval for all channel resources assigned to the chat server. At step  62 , the directory service server assigns an initial estimated weight to the newly created channel. The estimated weight is a measure of the number of users expected to join the channel, and is used to initially influence the future load prediction for the server until the load on the newly created channel ramps up and a load gradient on the channel can be explicitly determined. At step  63 , the directory service server determines a load gradient for each respective chat server over all channels resources assigned to the chat server. 
     Each respective chat server  42   a - 42   d  updates the directory service servers  41   a - 41   c  with its total user count (weight) at every timeslot. Directory service servers  41  use the gradient and weight for each chat server  42   a - 42   d  for predicting the future load for each respective chat server. The load distribution among the chat servers is then balanced based on the gradients and weights provided by the respective chat servers. 
     FIG. 7 shows an exemplary sliding window of time  70  having a plurality of timeslots  71  that are used for determining a load gradient of a chat server according to the present invention. 
     Preferably, sliding window  70  spans about 60 seconds and each time slot  71  preferably spans about five (5) seconds (twelve slots per sliding window). The load gradient for a chat server is preferably calculated by determining the difference between the total number of users (all channels) connected to the server in a current timeslot and the total number of users (all channels) connected to the server in a timeslot occurring 60 seconds earlier. For example, an exemplary load gradient for a chat server is calculated by subtracting the total number of users on the chat server at timeslot  71   a  from the total number of users on the chat server at time slot  71   b . Of course, a sliding window having a different time span than 60 seconds can be used. Similarly, a timeslot having a time interval different from 5 seconds can also be used. 
     As time progresses, the window slides to include a new current timeslot, while excluding an earlier timeslot that now is outside of the sliding window. That is, the sliding window at time to includes timeslots between timeslots  71   a  and  71   b , inclusive. At subsequent time t 1 , the sliding window includes timeslots between timeslots  71   c  and  71   d.    
     Returning to FIG. 5, at step  55 , the directory service server selects a chat server for assigning the new channel resource based on the current and future loads and the current load gradients. The selection can be based on, for example, an equitable load balancing policy so that each chat server carries about the same amount of traffic, or any other suitable load balancing policy. At step  56 , the directory service server sends a message to the selected chat server informing the chat server of the newly assigned channel resource, in addition to the estimated weight for the new channel. The selected chat server adds the initial estimated weight for the new channel to the total number of users in the current timeslot of the sliding window. The assigned chat server and the directory server use the estimated weight until the load on the newly created channel ramps up and a load gradient on the channel can be explicitly determined. 
     At step  57 , a message is sent to each of the other directory service servers indicating the assignment of the new channel resource. At step  58 , the directory service server connects the user to the new chatroom. 
     The present invention is not limited to a chat server cluster environment, but is applicable to any server cluster environment in which resources and clients are assigned among a plurality of servers. 
     Yet another primary aspect of the invention provides a technique for determining a consistent state for a server cluster, such as resource ownership for a chat server cluster, in situations in which one or more chat servers in the cluster temporarily lose and subsequently regain connection to the server cluster. This aspect of the present invention also determines a consistent state of a server cluster after a failover, that is, when the server cluster switches from a primary backend server to a secondary backend server as a result of a failure of the primary backend server. This aspect of the invention will be described with reference to FIGS. 4,  8  and  9 . 
     According to this aspect of the invention, when a server disconnects and subsequently reconnects, or when a failover occurs, the current ownership of resources assigned among the servers of a server cluster is redetermined and any contention for a resource among servers of the cluster is resolved. 
     FIG. 8 shows a flow diagram  80  for determining a consistent state of a server cluster when one or more chat servers in the cluster temporarily lose and subsequently regain connection to the rest of the server cluster. At step  81 , a chat server  42  determines that it has temporarily lost and regained connection to the server cluster  40  and begins the process of determining a consistent state of the cluster. At step  82 , the reconnecting chat server forwards resource ownership information to the active backend server for both static and dynamic resources maintained by the server internally. A static resource is, for example, a channel (chatroom) that is perpetually open, such as a “cars” chatroom. A dynamic resource is, for example, a chatroom that is created on an as needed basis, such as a “bicycle tire repair” chatroom. 
     At step  83 , the active backend server compares the resource ownership information received from the reconnecting chat server with resource ownership information maintained within the active backend server. At step  84 , the active backend server resolves ownership contention issues for both static and dynamic resources and a resource ownership response message is sent to the reconnecting server. At step  85 , the reconnecting chat server then updates its internal resource ownership information by discarding any resources indicated to no longer be owned by the reconnecting server, adding any resources that (from the point of view of the reconnecting server) are newly assigned, and maintaining any resources that have been verified to be owned by the reconnecting server. 
     FIG. 9 shows a flow diagram  90  for determining a consistent state for a server cluster for the situation of a failover. In such a situation, the secondary backend server cannot rely on ownership information received from the failed primary backend server. The process begins at step  91  where, just after the failover, each chat server  42  of the cluster sends resource ownership information for each resource it internally owns to the now-active secondary backend server  44  for verification. At step  92 , secondary backend server  44  resolves any contention for any static resources that the second backend server already has recorded internally. At step  93 , secondary backend server  44  then verifies each resource indicated to be owned by each chat server  42  and resolves any contention for the same resource when the resource ownership information received from chat servers  42  indicates that multiple servers own the resource. At step  94 , the backend server sends a message to each chat server  42  responding to the ownership information from each chat server. At step  95 , each chat server then updates its internal resource ownership information by discarding any resources indicated to no longer be owned by the chat server, adding any resources that (from the point of view of the server) are newly assigned, and maintaining any resources that have been verified by active backend server  44 . The active backend server maintains resource ownership information for all of the resources that were sent by the servers for verification and were successfully verified. The backend server also updates all directory services with the new resource ownership information. 
     Of course, this aspect of the present invention is not limited to a chat server cluster environment, but is applicable to any server cluster environment in which resources and clients are assigned among a plurality of servers. 
     While the present invention has been described in connection with the illustrated embodiments, it will be appreciated and understood that modifications may be made without departing from the true spirit and scope of the invention.