Abstract:
A method and systems for a redundant server automatic fail-over system is provided. The system includes a plurality of client devices communicatively coupled to a network wherein the plurality of client devices each includes an active server identification location. The system also includes a first server system communicatively coupled to the network that is configured to operate as the active server on the network wherein messages sent to the first server system are addressed to the first server system using the active server identification location on each client device. The system further includes a second server system communicatively coupled to the network that is configured to operate as a standby server on the network and is configured to switch to being the active server on the network when it is determined that the first server system is unable to operate as the active server.

Description:
BACKGROUND OF THE INVENTION 
       [0001]    This invention relates generally to process control networks and, more particularly, to systems and a method for automatic failover of redundant servers in a process control network. 
         [0002]    At least some known process control networks include a plurality of HMI clients connected to a pair of redundant SCADA servers via Local Area Networks (LAN). One SCADA server is in control as an active server while the other SCADA server is in standby mode. The data between the SCADA servers are synchronized. When the active server fails or is disconnected from the network for various reasons, the standby SCADA switches to the active role. The plurality of HMI clients need to switch to the newly active SCADA server to query and process the process data with minimal interruption. One of the problems with redundant schemes is that each client needs to have a connection to the active SCADA server of the logical pair. In such known networks, to maintain continuous connection to the active SCADA server, a custom script or application running on each HMI client polls the status of the SCADA server pair and switches between them when the active connection failed. However, such polling increases the computational overhead of each of the HMI clients and causes increased traffic on the network. Additionally, managing custom scripts or applications at the HMI client introduces a probability of configuration errors and compatibility issues. 
       BRIEF DESCRIPTION OF THE INVENTION 
       [0003]    In one embodiment, system for a redundant server automatic fail-over system includes a plurality of client devices communicatively coupled to a network wherein the plurality of client devices each includes an active server identification location. The system also includes a first server system communicatively coupled to the network that is configured to operate as the active server on the network wherein messages sent to the first server system are addressed to the first server system using the active server identification location on each client device. The system further includes a second server system communicatively coupled to the network that is configured to operate as a standby server on the network and is configured to switch to being the active server on the network when it is determined that the first server system is unable to operate as the active server. The active server identification location is configured to receive an active server identification when the first server system is unable to operate as the active server. 
         [0004]    In another embodiment, a method for automatic failover includes operating a first server system as an active server on a network wherein the first server system is configured to communicate with a plurality of clients. Messages sent to the first server system are addressed to the first server system using an active server identification location on the sending client. The method also includes operating a second server system as a standby server on the network, switching the second server to being the active server on the network when it is determined that the first server system is unable to operate as the active server, and changing the active server identification location on the plurality of clients to the identification of the second server system. 
         [0005]    In yet another embodiment, a redundant server system includes a network, a first server system communicatively coupled to said network operable as an active server on said network, a second server system communicatively coupled to said network operable as a standby server on said network, and a plurality of clients communicatively coupled to said network, at least some of the plurality of clients comprising an active server identification location containing an identification of the active server on the network. The second server system is configured to switch to being the active server and at least one of the plurality of clients is programmed to receive a message including an identification of the active server and to change the active server identification location associated with that client using the message. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0006]      FIGS. 1 and 2  show exemplary embodiments of the method and systems described herein. 
           [0007]      FIG. 1  is a schematic block diagram of a redundant server system  100  in accordance with an exemplary embodiment of the present invention; and 
           [0008]      FIG. 2  is a table that illustrates a response of the server status manager shown in  FIG. 1  in various operational activities. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0009]    The following detailed description illustrates embodiments of the invention by way of example and not by way of limitation. It is contemplated that the invention has general application to redundant control systems in industrial, commercial, and residential applications. 
         [0010]    As used herein, an element or step recited in the singular and proceeded with the word “a” or “an” should be understood as not excluding plural elements or steps, unless such exclusion is explicitly recited. Furthermore, references to “one embodiment” of the present invention are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features. 
         [0011]      FIG. 1  is a schematic block diagram of a redundant server system  100  in accordance with an exemplary embodiment of the present invention. In the exemplary embodiment, a first server  102  of a redundant pair of servers operates in an active mode. First server  102  includes a processor  103 . A second server  104  in the redundant pair of servers operates in a standby mode. Second server  104  includes a processor  105 . A plurality of view nodes  106  such as Human-Machine Interfaces (HMI) includes a first view node  108 , a second view node  110 , and an Nth view node  112 . Each of the plurality of view nodes  106  directs communications to active server  102  based on an identification of active server  102  held in a respective active server identification location  114 ,  116 , and  118  stored within each of the plurality of view nodes  106 . Standby server  104  also directs communications to active server  102  based on an identification of active server  102  held an active server identification location  120  stored within standby server  104 . In addition active server  102  includes an active server identification location  122  that also stores an identification of active server  102  as the active server. 
         [0012]    In the exemplary embodiment, each of the plurality of view nodes  106  and servers  102  and  104  includes active server  102  in their respective active server identification locations. Accordingly, active server  102  is referred to as the master and standby server  104  is referred to as the slave. All nodes such as plurality of view nodes  106  and servers  102  and  104  should point to the master, in the exemplary embodiment, server  102 . Nodes that do not point to the master or include an identification of active server  102  in their respective active server identification location may not receive all the first failover logic module  124  services available from active server  102  or the services may be delayed. For example, standby server  104  operating as the slave does not run an I/O driver but is getting database synchronization from active server  102 , which operates as the master, consequently data read from standby server  104  may not be the most current. The nodes pointing to the salve cannot write data to or configure the slave. Additionally, because the slave is not running a module referred to as scan, alarm, and control (SAC) the node will not receive any new alarms. The SAC program is responsible for looking through the process database and deciding what locations need to be updated and when. 
         [0013]    Further, nodes that direct communications to and subsequently receive communications from the slave may not receive up to date information regarding changes to for example, setpoint changes when a user makes a setpoint change to active server  102 . Moreover, information entered by the user may not update an alarm file of active server  102  if the view node from which it was entered is pointing to standby server  104 . 
         [0014]    To ensure all nodes point to the master or active server  102 , active server  102  pulls view nodes to it. On failover and at a predetermined time period and/or event the master pulls all view nodes to itself To pull one of the plurality of view nodes  106  to active server  102 , active server  102  tells the view node failover to active server  102 . Active server periodically checks the active connections of view nodes  106  and writes the identification of the active server for each view node  106  that is not connected to active server  102 . 
         [0015]    Such a failover connection between view nodes  106  and active server  102  of the logical pair is fast and automatic for the user in that view nodes  106  are pulled by active server  102  to communicate with it rather than each view node  106  having to poll servers  102  and  104  to determine which is the current active server and then having to switch itself to the new active server  102  in the event of a failover. The active server pulling each of the plurality of view nodes  106  facilitates minimizing configuration errors, eliminating the need for custom scripts or applications, and providing maximum availability. 
         [0016]    Having the scada server “pull clients” to it when the server status becomes active permits retrofitting the automatic failover using software executing on the servers rather than software and hardware on the plurality of view nodes  106 . The active failover also facilitates minimizing the amount of time that view nodes  106  attempt to retry communications to the newly disabled or disconnected server. 
         [0017]    Servers  102  and  104  each maintain a list of HMI clients or view nodes  106  having active connections to server  102 . When standby server  104  is assigned an active status, the now active server  104  cycles through the client list and switches the logical connection on each to the newly active server. In the exemplary embodiment, the logical connections are switched sequentially. In an alternative embodiment, the logical connections are switched simultaneously using for example, but not limited to a broadcast message. The newly active server creates a bi-directional connection back to each view node  106 , verifies which server the view node is connected to, and calls a remote procedure to set the logical connection to the newly active server. 
         [0018]    A first failover logic module  124  executes on server  102 . A second failover logic module  126  executes on server  104 . Likewise, when more servers are present each may execute a respective failover logic module. Each of the plurality of view nodes  106  executes logic that causes each to failover on connection loss to the master or active server  102 . If one of the plurality of view nodes  106  loses a connection to the master, view node  106  fails to the slave or standby server  104 . The view node logic may be disabled by modifying a configuration field. Additionally, a view node  106  may be manually or programmatically failed to point at slave. View node  106  fails back to master within a predetermined time period as first failover logic module  124  pulls all view nodes  106  to it on a periodic or event driven cycle. A server status manager  128  monitors redundant server system  100  to determine a status of at least one of the operating and connected servers on a network  130 . Typically, all servers that are operating and connected to network  130  are either in a standby or an active mode. However, the status of a server that is not connected but operating, a server that is shutdown, or server with a loss of power may be determined to be in an unknown state. 
         [0019]    Additionally, view nodes  106  execute logic that periodically reads network status display (NSD) fields on servers  102  and  104  to determine which server is the master. The NSD fields are a collection of numeric and ASCII values that are used to view various information on network status. View node  106  ensures it is pointing to the master by writing the identification of the determined active server into the respective active server identification location. 
         [0020]    During startup, each server determines whether it is operating as active server  102  or standby server  104 . Each server then builds an easy data access (EDA) group for further processing. EDA is an application programming interface layer used to access real-time process data. An EDA group is a reference to one or more data locations that are read as a group. If the server determines it is operating as the master or active server  102 , at a predetermined time period or predetermined event, server  102  sets its own active server identification location to itself, for each connection that is at least incoming (could be both incoming and outgoing) server  102  either transmits its own server identification to each of the connected view nodes&#39; active server identification location or requests the view nodes to transmit the server location in each view nodes&#39; active server identification location. In the exemplary embodiment, reading of the EDA is sequential. In an alternative embodiment, reading of the EDA is parallel. 
         [0021]    If the server determines it is operating as the slave or standby server  104 , at a predetermined time or event, server  104  determines the identification of the master if necessary and then writes the identification of the master into its own active server identification location thereby saving the master from having to write its identification into the slave&#39;s own active server identification location. 
         [0022]    During a manual failover, first failover logic module  124  writes the identification of the slave into the slave&#39;s active server identification location  120  and into the master&#39;s active server identification location  122 . The master then drops offline or is switched to being the slave and the slave assumes the role of master. The master (formerly slave) transmits its identification to the plurality of view nodes  106  to pull them into communication with the new master. The identification is written into each respective active server identification location  114 ,  116 , and  118  for each connected view node  106 . Any new view nodes  106  can connect to either server  102  or  104 , but will be pulled to the master at the first predetermined time period when the master pulls all view nodes to itself. 
         [0023]    During a loss of power to the master, second failover logic module  126  determines that active server  102  can no longer serve as the master and switches standby server  104  to an active mode. View nodes  106  connected to active server  102  automatically failover to the new master either by network timeout logic executing on view nodes  106  or by second failover logic module  126  pulling each view node  106  to the new active server. All connected view nodes  106  then request a boot queue update. The boot queue is a list of current alarms that occurred prior to view node  106  connected to server  102 . When a view node  106  re-connection occurs, view nodes  106  request active server  102  to re-send the current active alarms. The boot queue is sent to view nodes  106  so any current alarms can be displayed. 
         [0024]    If a connection between one or more of the plurality of view nodes  106  and the master are lost, the affected view nodes failover to the slave if it is present. If the affected view nodes can only connect to the slave, the affected view nodes will maintain the connection to the slave because there is no logic on the affected view node or the slave to cause the affected view node to connect to another server. The logic in first failover logic module  124  that executes the pullover of the view nodes  106  executes only on the master. If the master doesn&#39;t pull the affected view nodes to itself the slave will not affect the connection of the affected view nodes to itself. If the affected view nodes reconnect to master, first failover logic module  124  on master will pull the affected view nodes back to itself. By connecting to the slave when a connection to the master is lost the affected view nodes have access to data that is relatively old depending upon the synchronism rate between the slave and the master. If any of the plurality of view nodes  106  loses a connection to the slave, there will be no effect on the operation of the plurality of view nodes  106  until a failover occurs. In such a case, the effect on the plurality of view nodes  106  is similar to the loss of connection to the master described above after the failover. 
         [0025]      FIG. 2  is a table that illustrates a response of server status manager  128  (shown in  FIG. 1 ) in various operational activities. In the exemplary embodiment, server status manager  128  monitors the servers connected to network  130  and determines whether each server is in an active mode, a standby mode, or an unknown state. In the active mode, view nodes  106  connect the data session to the active server  102 . The active server SAC processes the database blocks. In the standby mode, the standby server  104  SAC is in standby mode (does not process the database blocks) and active server  102  updates the database (in memory) on the standby server  104 . Server status manager  128  also provides for switching the status of a server to facilitate reducing conflicts between servers. For example, if more than one server is active, each will continually try to pull view nodes  106  to itself increasing the computational overhead experienced by each view node  106  as they comply with first one server pulling it and then the next server pulling it. When the server partners are both running in the same mode, an arbitration procedure is used to determine which one should be the Active node. When server status manager  128  requests the status from the servers, part of the status response includes an indication of whether the server is configured as a primary node. If both servers agree on which one is the primary node, the primary node becomes active. If both servers do not agree on which is the primary node, an alternate method, for example, a server name string comparison is performed, and the server having a lower ASCII value in its server name string becomes active. 
         [0026]    The term processor, as used herein, refers to central processing units, microprocessors, microcontrollers, reduced instruction set circuits (RISC), application specific integrated circuits (ASIC), logic circuits, and any other circuit or processor capable of executing the functions described herein. 
         [0027]    As used herein, the terms “software” and “firmware” are interchangeable, and include any computer program stored in memory for execution by processors  103  and  105 , or processors executing on view nodes  106  including RAM memory, ROM memory, EPROM memory, EEPROM memory, and non-volatile RAM (NVRAM) memory. The above memory types are exemplary only, and are thus not limiting as to the types of memory usable for storage of a computer program. 
         [0028]    As will be appreciated based on the foregoing specification, the above-described embodiments of the disclosure may be implemented using computer programming or engineering techniques including computer software, firmware, hardware or any combination or subset thereof, wherein the technical effect is having a newly active server of a redundant server pair use an existing network connection to create a dynamic (bi-directional) connection to one or more clients and sending a command to each client to switch the logical connection to the newly active server. Any such resulting program, having computer-readable code means, may be embodied or provided within one or more computer-readable media, thereby making a computer program product, i.e., an article of manufacture, according to the discussed embodiments of the disclosure. The computer readable media may be, for example, but is not limited to, a fixed (hard) drive, diskette, optical disk, magnetic tape, semiconductor memory such as read-only memory (ROM), and/or any transmitting/receiving medium such as the Internet or other communication network or link. The article of manufacture containing the computer code may be made and/or used by executing the code directly from one medium, by copying the code from one medium to another medium, or by transmitting the code over a network. 
         [0029]    The above-described embodiments of a method and systems for automatic failover of redundant servers in a process control network provides a cost-effective and reliable means for having a newly active server of a redundant server pair use an existing network connection to create a dynamic (bi-directional) connection to one or more clients and sending a command to each client to switch the logical connection to the newly active server. More specifically, the method and systems described herein facilitate ensuring minimal disruption in the operation of the process controlled by the active server. In addition, the above-described method and systems facilitate upgrading existing system because there is no code modification to older versions of the client required to implement the automatic failover as the software resides on the servers or may reside on an external system. Furthermore, the method and systems described herein facilitate reducing client computational overhead because the clients do not have to periodically discover which server is currently the active server. As a result, the method and systems described herein facilitate automatic failover of redundant servers in a process control network in a cost-effective and reliable manner. 
         [0030]    While the disclosure has been described in terms of various specific embodiments, it will be recognized that the disclosure can be practiced with modification within the spirit and scope of the claims.