Patent Publication Number: US-7711980-B1

Title: Computer system failure management with topology-based failure impact determinations

Description:
BACKGROUND 
     As an alternative to purchasing computer systems, a user can lease portions of a massive computer system, much as a traveler might lease a hotel room or an event planner might lease a hotel-dining hall. This business model, introduced by Hewlett-Packard Company as a “Utility Data Center” (“UDC”), allows flexible access to computer resources without the burden of maintaining a computer system. 
     Of course, the owner of the computer system must maintain it. Not only must the owner of the computer system provide maintenance, but do so in a way that ensures that contractual obligations are met. Since failures are inevitable in a large system, provisions must be made to move a user&#39;s workload to working hardware that meets user specification. 
     Computer system maintenance can be automated to a large extent. An automated workload manager can test for or otherwise detect failures. For example, the workload manager can send out requests and check for responses. If a device does not respond, it may have failed or it may be inaccessible due to failure of another device (e.g., a server may be inaccessible because a switch port to which it is connected has failed.) In either case, a failure can be noted, e.g., in a device database. 
     If possible, workload on a failed device can be migrated to an available device. In any event, a failed device will not be targeted for installation of a new workload or the target of a software migration. In due course, hardware or replacement of devices marked “failed” can obviate the failure. 
     Herein, related art is described to facilitate understanding of the invention. Related art labeled “prior art” is admitted prior art; related art not labeled “prior art” is not admitted prior art. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWING 
       The figure depicts implementations/embodiments of the invention and not the invention itself. 
         FIG. 1  is a schematic diagram of a managed computer system in accordance with an embodiment of the invention. 
         FIG. 2  is a portion of a topology corresponding to the computer system of  FIG. 1 . 
         FIG. 3  is a combination schematic diagram of a computer system and a flow chart of a method implemented in that computer system in accordance with embodiments of the invention. 
         FIG. 4  is another portion of the topology of  FIG. 2 . 
     
    
    
     DETAILED DESCRIPTION 
     In the course of the present invention, it was recognized that, while distinguishing between failed and available devices suffices to maintain the functionality of a system, further distinctions are required for manageability purposes. For example, if a terminal server used for managing other servers, fails, the managed servers may continue functioning. However, the ability to reconfigure the managed servers will have been lost or impaired. This loss or impairment should be taken into account when a need arises to reconfigure a system; for example, an administrator should not try to migrate a workload to a device with no or impaired manageability. 
       FIG. 1  depicts a customer-leased portion of a vendor-operated data center AP 1 , which includes managed server groups MG 1 - 3 , storage-array network (SAN) disks SD 1 , SAN storage fabric SF 1 , first tier switches T 11 , and two groups of second tier switches T 21  and T 22 . First tier switches T 11  includes switch  51  and switch S 2 ; second tier switches T 21  includes switches S 3  and S 4 , and second tier switches T 22  includes switches S 5  and S 6 . Each of these switches S 1 -S 6  is shown with eight ports, P 11 -P 18 , P 21 -P 28 , P 31 -P 38 , P 41 -P 48 , P 51 -P 58 , and P 61 -P 68 . Typically, there would be more, e.g., twenty-four, ports per switch. 
     Managed group MG 1  includes a terminal server TS 1  and customer-leased servers S 11 -S 14 . Managed group MG 2  includes a terminal server TS 2  and customer-leased servers S 21 -S 24 . Managed group MG 1  includes a terminal server TS 3  and customer-leased servers S 31 -S 34 . For each managed group, MG 1 , MG 2 , and MG 3 , the terminal server TS 1 , TS 2 , and TS 3 , is used to manage the customer-leased servers S 11 -S 14 , S 21 -S 24 , and S 31 -S 34 . The grouping of servers into managed groups is invisible to the customer. The customer can group servers into task groups independent of the membership in managed groups. For example, serves S 11 , S 21 , and S 31  are arranged into a customer task group TG 1 . 
     If a customer-leased server fails, it will be marked “failed” in a change database. Generally, a new server will be added to the customer&#39;s pool and the workload on the failed server will be migrated to the new server or another server of the customer&#39;s if possible. Thus, if server S 11  fails, its workload can be migrated to server S 22 . Server S 11  will be repaired or replaced as a maintenance schedule permits. It may or may not be returned to its role for its former lessee. 
     If a terminal server fails, the ability to manage the customer-leased servers in the group is lost or compromised. For example, if terminal server TS 1  fails, servers S 11 -S 14  can become unmanageable. They may, however, continue functioning. In this case, terminal server TS 1  is marked “failed”, while servers S 11 -S 14  are marked “impacted” or “partially failed” in view of their functional dependencies from failed terminal server TS 1 . The “impacted” marking can be supplemented with a comment or other indication of why they are so marked. 
     The “impacted” indication does not require an immediate workload migration in all situations. However, contractual terms might require migration from an impacted machine. On the other hand, in addition, physical repair or replacement of an impacted device is usually not required. On the other hand, due to loss of manageability, an impacted server would not be a good migration target. Marking an impacted server as such could serve as a warning to an allocation planner (whether human or automated) that the migration is not possible due to lack of manageability or, if possible, not desirable, due to lack of manageability. 
     Thus, expanding a change database to provide for “impacted” devices to be distinguished from “available” and “failed” devices, facilitates data center management by 1) preventing un-implementable reallocation plans from being generated; 2) avoiding migrations to less desirable locations; and 3) avoiding attempts to repair devices that are not really damaged. 
     For another scenario, consider the case where port P 37  provides the only connection to terminal server TS 1 . If port P 37  fails, it is marked “failed”. However, terminal server TS 1  becomes disconnected from the rest of data center AP 1 , and thus is non-functional. A manager program can detect failures of both port P 37  and terminal server TS 1 . In view of a connection dependency of terminal server TS 1  on port P 37 , it can be determined that the inaccessibility of terminal server TS 1  may be due to the failure of port P 37 , rather than any actual problem with terminal server TS 1 . Thus, the problem with port P 37  might be repaired or an alternative connection might be provided before an attempt to repair or replace terminal server TS 1  is made. 
     In this scenario, terminal server TS 1  is marked “impacted” with a comment indicating that it is inaccessible. Also, devices dependent on a failed or impacted device are marked. Thus, servers S 11 -S 14  are marked impacted in view of their functional dependency on terminal server TS 1 . 
     Also in this scenario in which port P 37  fails, incorporating switch S 3  is marked “impacted” because of its inclusion relationship with failed port P 37 . This indication is used to discourage allocation to devices directly serviced by switch S 3 , because it is partially defective and because a failure of one port may presage other ports failing on the same switch. Future allocation should marginalize the role of switch S 3  so that it can be replaced with minimal interference of devices in data center AP 1 . 
     In practice, a data center is designed with many redundant connections. For example, terminal server TS 1  would likely have two network connections, e.g., one connected to port P 37  and another connected to port P 57 . A failure of either one of the ports would not render terminal server TS 1  inaccessible, but would increase the risk of its being rendered inaccessible. For example, if port P 37  fails, redundancy is lost and only one more port needs to fail for terminal server TS 1  to become inaccessible. 
     If one of two redundant connections to terminal server TS 1  fails, the terminal server remains accessible. Nonetheless, it is marked “impacted” in view of its connection dependency on a failed device. Also, servers S 11 -S 14  are marked “impacted” in view of their functional dependencies on an impacted terminal server. This marking is appropriate even though servers S 11 -S 14  remain functional and manageable, as they are at increased risk of becoming unmanageable. A workload manager should be discouraged from migrating workloads to those servers. Of course, if better choices are not available, impacted servers can still be migration targets. For this reason, intelligent commenting can permit the person or program doing the allocation to weigh allocation choices. 
       FIG. 2  shows a portion of a topology TP 1  generated to facilitate identification of indirect (intergenerational) dependency relationships. In the illustrated topology portion, the top item is switch port P 37 . Switch S 3  has an inclusion dependency relationship on switch port P 37 . No device depends directly on switch S 3 . In addition, a server port (network interface card) of terminal server TS 1  has a connection dependency on port P 37 . Terminal server TS 1  has an inclusion dependency on this server port. Servers S 11 -S 14  are managed by terminal server TS 1 , and thus have (non-inclusion, non-connection) functional dependencies on terminal serve TS 1 . Note that it is immediately apparent that servers S 11 -S 14  will be impacted by a failure of switch port P 37 . 
     The invention applies to devices other than switches and servers. For example, failures in SAN storage fabric SF 1  can impact storage disks SD 1 . Other examples are described with reference to  FIG. 3 , which depicts another portion of data center AP 1 . 
     The portion of data center AP 1  shown in  FIG. 3  includes first tier switches T 11 , second tier switches SW 1  and SW 2 , severs SV 3 -SV 6 , other service level devices SV 7  including servers, firewalls, load balancers, and other infrastructure devices, and SV 8 , including switches, terminal servers, fiber channel, and power servers. Switch SW 1  is shown with eight ports Q 11 -Q 18 , and switch SW 2  is shown with eight ports Q 21 -Q 28 . In practice, switches can have many more ports. 
     Manager server SV 3  has two network ports Q 31  and Q 32  respectively connected to switch ports Q 11  and Q 21  for redundancy. Customer server SV 4  has two ports Q 41  and Q 42  respectively connected to switch ports Q 12  and Q 22 . Firewall server SV 5  has two network ports Q 51  and Q 52  respectively connected to switch ports Q 13  and Q 23 . Customer server SV 6  has one network port coupled to switch port Q 24 . Service level devices SV 7  and infrastructure devices SV 8  all have connection ports as appropriate for their functions. 
     Manager server SV 3  includes software for managing data center AP 1 . Failure monitor  13 , also known as an infrastructure and resource finger listener, periodically sends requests to resource and infrastructure devices in data center AP 1 . Some devices can report their own failures or failures of their components to failure monitor  13 . In other cases, failures are determined by lack of responses. A failure marker  15  marks devices as “available”, “impacted” or “failed”, along with additional indications and comments as appropriate for planning purposes. A dependency rule parser  17  analyzes dependency data to determine which devices are impacted and how. Topology generator  19  uses the dependency analyses provided by dependency rule parser  17  to generate and update a data center topology  21 . An allocation planner  23  plans allocations of resources to workloads based in part of topology and failure data. 
     Change database  25  provides characterizations for all resource and infrastructure devices of data center AP 1 . The characterizations include “device ownership”  27 , “device types”  29 , “device roles”  31 , “device dependencies”  33 , “device dependency relations”  41 , “device failure status”  43 , and “redundancy”  45 . “Device ownership”  27  refers to the customer or other entity to which a resource device is assigned; ownership is not listed for infrastructure devices. “Device types”  29  specifies servers, switches, power server, fiber channel, disk array, etc. “Device roles”  31  is a field that depends on the device type. For a switch, tier is listed. For a server, it can be general (no dependencies issue from it), firewall, load-balancer, terminal server, etc. 
     “Device dependencies”  33  lists all devices that have a direct dependency relationship for a device. For each dependency, a type is listed as either an inclusion type  35 , a connection type  37 , or a functional type  39 . “Inclusion” means that a failed component is part of an incorporating device that may still be partially functional. For example, if a network port fails, an incorporating switch or server can be impacted. If a switch fails, its ports are impacted. Connection type dependencies involve network (e.g., Ethernet, SAN) connections. If a port fails, the port on the other side of the connection is impacted. In this case, the device incorporating the impacted port is also impacted via an inclusion dependency. A functional dependency is a dependency that is based on the role of a device. For example, a resource server can be functionally dependent on a terminal server, a firewall, or a load balancer, despite the lack of a direct connection. 
     For each device dependency there is a relationship, either parent, child, or sibling. In an inclusion relationship, the component is the parent and the incorporating device is the child. In a connection relationship, the parent is the device higher in the hierarchy, e.g., a port on a first tier switch is the parent in a connection with a port on a 2 nd  tier switch. In a functional relationship, the device providing the service is the parent, while the device receiving the service is the child. For example, a firewall is a parent to a resource server protected by the firewall. A sibling relationship applies mainly to connection type relationships, e.g., between switch ports at the same level a hierarchy. 
     Database  25  lists device failure status, e.g., available, impacted, or failed. The present embodiment uses comments to distinguish different failure and impact types. In other embodiments, other labels and categories are used. 
     “Redundancy”  45  indicates the required (e.g., by lease agreement or standard practices) and actual redundancy available to an impacted device. The redundancy fields are primarily intended for “impacted” yet fully functioning devices. If sufficient redundancy remains for an impacted device, it may be retained in service and even used as an allocation target. 
     For example, a customer may require a resource server with redundant network connections. Server SV 4  meets this requirement, and so is assigned to the customer. If switch port Q 12  fails, server SV 4  still has a network connection, but has lost redundancy of that connection. In this circumstance, port Q 41  is marked impacted as a result of a connection dependency from switch port Q 12 ; no redundancy is marked for switch port Q 41  since it has only one network connection. Server SV 4  is marked “impacted” as a result of an inclusion dependency on port Q 41 . Server SV 4  is marked as having a required redundancy of “2” (will remain functional with one failure), and an actual redundancy rating of “1” (functional but no redundancy) for this network connection. 
     In this impacted condition, two-port server SV 4  may be as capable of non-impacted one-port server SV 6 . However, since server SV 6  is not impacted, it remains fully available for allocation planning. Of course, allocation planner  23  can determine from database  25  that server SV 6  has only one network connection and will not assign it to a customer or workload requiring a redundant network connection. The impacted status of server SV 4  on the other hand, is an impediment to its use as an allocation target. If better alternatives are available, it will be withheld from the pool of allocation targets. If better alternatives are not available, it can be used as a one-port server for planning purposes. 
     In another scenario, if switch port Q 13  fails, port Q 51  of firewall server SV 5  is impacted via a connection dependency, and firewall sever SV 5  is impacted in turn by an inclusion dependency. All servers relying on firewall SV 5  are also impacted via a functional dependency. Impacted firewall server SV 5 , in this scenario, is marked with a “2” required redundancy and a “1” actual redundancy. If, then, switch port Q 23  fails, the required redundancy for firewall server SV 5  is still two, but the actual redundancy is marked “0”, meaning “no redundancy and not functional”. Devices depending on firewall server SV 5  for network protection are impacted more severely in the event of a loss of functionality due to two failures than they are when only one network connection is lost. Comments can be used to distinguish the severity of consequences for impacted devices, e.g., resource servers relying on firewall server SV 5 . 
     Method ME 1 , flow-charted in  FIG. 2 , is implemented in data center AP 1 . At method segment MS 1 , topology generator  19  generates or updates topology  21 . To this end, topology generator  19  accesses database  25  to determine and characterize direct dependencies. 
     Indirect dependencies are traced by following child relationships. The dependency topology  21  can involve inclusion, connection, and functional relationships. In alternative embodiment, separate topologies are generated for different dependency types. 
     A portion of topology  21  is shown in  FIG. 4 . This portion focuses on firewall server SV 5 . It has an inclusion relationship with its two network interface cards or ports Q 51  and Q 52 . Each of these has a respective connection relationship with a respective switch port Q 13  and Q 23 . Each switch port has a parental inclusion relationship with the respective including switch SW 1  and SW 2 . Firewall SV 5  is set up to protect servers SV 4  and SV 6 , which thus have child functional dependencies on firewall SV 5 . From topology  21 , it is apparent that servers SV 4  and SV 6  will be impacted if either switch port Q 13  or Q 23  fails. In this case, for example, a failure of switch port Q 13  will increase the risk that severs SV 4  and SV 6  will lose firewall protection. 
     When a device failure occurs, it is detected at method segment MS 2 . In some cases, a device can report its own failure, or an incorporating device can report a failed component. In other cases, repeated failures to receive responses to queries by failure monitor  13  can lead to a determination of a failure. If only one device fails, it is marked “failed” at method segment MS 3 . If more than one device has recently failed, method ME 1  can complete method segment MS 4  before determining which device is subject to a physical failure, and which device appears to have failed because of a dependency. 
     Method segment MS 4  involves determining dependencies of a device detected as having failed at method segment MS 2 . If there is only one failed device, this involves tracing all child and sibling dependencies from the failed device. This means tracing down topology  21  inclusion, connection, and functional dependencies to devices such as resource servers that have no dependents. All of these devices are marked “impacted”. 
     If two or more devices are detected as having failed at method segment MS 2 , there is the possibility that one or more may still be functional and appear to have failed only because of a dependency on another device that has actually failed. For example, a server may be inaccessible due to a failure of the only switch port connected to the terminal server. If topology  21  indicates a device that is detected as having failed could appear failed because of its dependency on another device that has failed, it is marked “impacted” rather than failed. Other devices depending from the failed device are marked as impacted as described above. 
     Once devices are marked as failed or impacted in database  25 , subsequent planning can take the new device statuses into account. Failed devices are targeted for repair or decommissioning; of course, they are avoided as targets for future workload allocations until repaired. To the extent possible, impacted devices are avoided in favor of “available” devices as allocation targets at method segment MS 6 . Method segment MS 6  involves planning by allocation planner  23 , and implementation, e.g., via terminal servers TS 1 - 3 , in  FIG. 1 . 
     A “computer” is a machine that manipulates data in accordance with instructions. A “server” is a computer that provides service, e.g., serves web pages to or responds to database queries to other computers. A “terminal server” is a computer with human interface devices; in the present context, the interface is used to allow a human administrator to manage other servers. “Network infrastructure” refers to devices, such as switches, other than servers that handle inter-device communications; in the present context device is a device. 
     If a first device provides a service to a second device, the two have a dependency relationship. If the dependency is reciprocal, the relationship is a “sibling” relationship. Typically, sibling relationships are between devices of the same type, e.g., a pair of switches. If the relationship is hierarchical, the device providing the service is the “parent” and the device receiving the service is the “child”. 
     All dependency relationships are “functional” in a general sense. However, herein, unless otherwise indicated by context, “functional dependencies” are those that are not “inclusion” relationships or “connection relationships”. In an inclusion relationship, one device incorporates another device. For example, a switch device includes switch port devices. This relationship is termed an “inclusion” relationship herein. In an inclusion relationship, the including device is the “child” and the included device is the “parent”. 
     In a connection relationship, one device communicates with the other over a network or peer-to-peer connection. For example, a switch port can connect to a network interface card of a server. Typically, connections are arranged hierarchically. In a connection relationship, the device higher in the hierarchy is the parent. If two connected devices are at the same level of the hierarchy, there is a sibling relationship. 
     “Failure” indicates the functionality of a device is lost due to some problem with the device itself. “Impacted” indicates that functionality of a device is lost, impaired, or put at increased risk due to the failure of another device. 
     A “firewall” is an information technology (IT) security device that is configured to permit, deny, or proxy data connections set and configured by the organization&#39;s security policy. A firewall&#39;s basic task is to control traffic between computer networks with different zones of trust. A “load balancer” is an IT device that evenly distributes work between servers. A “power server” is a device that distributes power to other devices to match their demands. A “disk array” is a storage media consisting of a (typically redundant) array of magnetic disks. A many-to-many connection between servers and disk arrays can be managed using a storage array network, which can include fiber channel network technology. 
     A “program” is an ordered series of computer-executable instructions. As used herein, a “program” is inherently tangibly embodied in computer-readable media. A “server” is a computer that provides services. 
     “Computer-readable media” refers to media that can be accessed by a computer and includes volatile and nonvolatile, removable and non-removable media. By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media. “Computer storage media” includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. 
     “Computer storage media” encompasses, but is not limited to, random access memory (RAM), read-only memory (ROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CDROM), digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computer. 
     “Communication media” encompasses computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media. Combinations of any of the above should also be included within the scope of “computer-readable media”. 
     Herein, “allocation” involves matching workloads to resources, e.g., software programs to servers. Herein, an allocation planner gives a higher priority to a first resource than to another if, all else being equal, it allocates a workload to the first resource. Thus, in the present context, an allocation planner will assign a workload to a non-impacted server instead of an impacted server, even though the capabilities of the two servers are the same. In some cases, some overriding consideration might result in an allocation of a workload to an impacted server instead of a non-impacted server. E.g., where the workload is already running on the impacted server or where an impacted server is more capable than any non-impacted server that is not otherwise occupied. 
     Alternative embodiments use different labels and to distinguish devices that have actually failed completely, actually failed partially, appear to have failed completely due to a dependency, and appear to have failed partially due to a dependency, and devices that appear to be functioning properly but are impacted by a failure of a device on which is depends. In addition, some embodiments limit impacted designations when sufficient redundancy remains despite a failure. For example, if one of three connections fails, and redundant connections remain, some embodiments do not mark the redundantly connected device as “impacted”. These and other variations upon and modification to the illustrated embodiments are provided for by the present invention, the scope of which is defined by the following claims.