Multiple node/virtual input/output (I/O) server (VIOS) failure recovery in clustered partition mobility

A system and computer program product utilizes cluster-awareness to effectively support a live partition mobility (LPM) event and provide recovery from node failure within a Virtual Input/Output (I/O) Server (VIOS) cluster. An LPM utility creates a monitoring thread on a first VIOS on initiation of a corresponding LPM event. The monitoring thread tracks a status of an LPM and records status information in the mobility table of a database. The LPM utility creates other monitoring threads on other VIOSes running on the (same) source server. If the first VIOS sustains one of multiple failures, the LPM utility provides notification to other functioning nodes/VIOSes. The LPM utility enables a functioning monitoring thread to update the LPM status. In particular, a last monitoring thread may perform cleanup/update operations within the database based on an indication that there are nodes on the first server that are in failed state.

BACKGROUND

1. Technical Field

The present invention relates in general to clustered data processing systems and in particular to management and utilization of shared storage within a clustered data processing system. Still more particularly, the present invention relates to an improved method and system for node failure recovery based on utilization of shared, distributed storage within a clustered data processing system.

2. Description of the Related Art

Virtualized data processing system configuration, which provides the virtualization of processor, memory and Operating System (OS) resources are becoming more and more common in the computer (and particularly the computer server) industry. To a lesser extent, storage virtualization is also known and provided in limited environments. However, within the virtualization computing environment, storage virtualization and management is implemented as a separate virtualization model from server virtualization and management. Thus, different client logical partitions (LPARs) associated with different virtualized server systems may access the same storage access network (SAN) storage. However, the client LPARs on one server do not have any “knowledge” of whether the storage access network (SAN) disk that the client LPAR is trying to access is being used by some other client LPAR belonging to another server. The conventional implementation of distributed server systems providing storage virtualization within shared SAN storage can cause data integrity issues and may potentially cause data corruption and client partition crashes.

Live partition mobility (LPM) is the practice of moving a virtualized client partition from one server to another without appearing to interrupt operations on the client. However, failures occasionally occur during these LPM events. Unfortunately, conventional approaches have not been effective in handling hardware failures during LPM operations. This is a very complex problem because the state permutations are considerable, and there is no single node that can be relied upon to survive the failure. The traditional approach does not work in the clustered environment because in addition to cleaning up storage resources that were orphaned by the failure, there is not synchronization of relational information for the various nodes. With current approaches, recovery from the failure(s) would require a slow node by node accounting and cleanup process which would limit the function of the cluster (potentially a loss of service for the client partition) for some amount of time.

BRIEF SUMMARY

Disclosed are a system and computer program product for utilizing cluster-awareness to effectively support a Live Partition Mobility (LPM) event and provide recovery from node/Virtual Input/Output (I/O) Server (VIOS) failure within a VIOS cluster. A Live Partition Mobility (LPM) utility creates a monitoring thread on a first VIOS upon initiation of a corresponding LPM event. The monitoring thread tracks the status of an LPM event and records status information in the mobility table of a database. The LPM utility creates other monitoring threads on other VIOSes running on the (same) source computing electronic complex (CEC). If the first VIOS sustains one of multiple failure conditions (e.g., the VIOS can no longer perform I/O operations, goes offline or is removed from the cluster), the LPM utility provides notification to other functioning nodes/VIOSes. The LPM utility enables a functioning monitoring thread to update the LPM event status. In particular, a last monitoring thread may perform update operations within the database based on indication that there are nodes on the first server that are in a failed state.

DETAILED DESCRIPTION

The illustrative embodiments provide a method, data processing system, and computer program product for utilizing cluster-awareness to effectively support a live partition mobility (LPM) event and provide recovery from node failure within a Virtual Input/Output (I/O) Server (VIOS) cluster. A Live Partition Mobility (LPM) utility creates a monitoring thread on a first VIOS upon initiation of a corresponding LPM event. The monitoring thread tracks the status of an LPM event and records status information in the mobility table of a database. The LPM utility creates other monitoring threads on other VIOSes running on the (same) source computing electronic complex (CEC). If the first VIOS sustains one of multiple failure conditions (e.g., the VIOS can no longer perform I/O operations, goes offline or is removed from the cluster), the LPM utility provides notification to other functioning nodes/VIOSes. The LPM utility enables a functioning monitoring thread to update the LPM event status. In particular, a last monitoring thread may perform update operations within the database based on indication that there are nodes on the first server that are in a failed state.

Within the descriptions of the different views of the figures, similar elements are provided similar names and reference numerals as those of the previous figure(s). The specific numerals assigned to the elements are provided solely to aid in the description and are not meant to imply any limitations (structural or functional or otherwise) on the described embodiment.

It is understood that the use of specific component, device and/or parameter names (such as those of the executing utility/logic/firmware described herein) are for example only and not meant to imply any limitations on the invention. The invention may thus be implemented with different nomenclature/terminology utilized to describe the components/devices/parameters herein, without limitation. References to any specific protocol or proprietary name in describing one or more elements, features or concepts of the embodiments are provided solely as examples of one implementation, and such references do not limit the extension of the invention to embodiments in which different element, feature or concept names are utilized. Thus, each term utilized herein is to be given its broadest interpretation given the context in which that terms is utilized.

As further described below, implementation of the functional features of the invention is provided within processing devices/structures and involves use of a combination of hardware, firmware, as well as several software-level constructs (e.g., program code). The presented figures illustrate both hardware components and software components within example data processing architecture having a specific number of processing nodes (e.g., computing electronic complexes). The illustrative and described embodiments assume that the system architecture may be scaled to a much larger number of processing nodes.

In the following descriptions, headings or section labels are provided to separate functional descriptions of portions of the invention provided in specific sections. These headings are provided to enable better flow in the presentation of the illustrative embodiments, and are not meant to imply any limitation on the invention or with respect to any of the general functions described within a particular section. Material presented in any one section may be applicable to a next section and vice versa. The following sequence of headings and subheadings are presented within the specification:A. General ArchitectureB. Cluster-Aware VIOSC. CA VIOS Communication ProtocolD. VIOS Shared DB for Cluster ManagementE. VIOS Cluster MobilityF. Failure Recovery During Clustered Partition Mobility
A. General Architecture

With specific reference now toFIG. 1A, there is depicted a block diagram of an example cluster-aware (CA), distributed data processing system (DPS) architecture100, within which the functional aspects of the described embodiments may advantageously be implemented. For simplicity, cluster-aware, distributed DPS architecture100shall be referred to herein simply as DPS100. DPS100comprises a plurality of computing nodes, each referred to herein as a computing electronic complex (CEC), of which CECs110A and110B are illustrated. The number of CECs within DPS100may vary, ranging from a single CEC in a smaller system extending up to hundreds or thousands of CECs, in larger scaled systems. For simplicity, the embodiments shall be described from the perspective of a single CEC (CEC110A) or two CECs (CECs110A,110B). Each CEC110A-110B comprises at least one (and in most instances a plurality of) Virtual Input/Output Server112(also referred to herein as a VIO Server or VIOS), with functionality as described below. The actual number of VIOSes112within each CEC110of DPS100is a design feature and may vary. Also supported within each CEC110A-110B are client logical partitions (interchangeably referred to as client LPARs or “clients”), of which a first two clients, clientA114aand clientB114b, are illustrated. As described below, with reference toFIG. 2, client LPARs114are logical partitions of a virtualized (or operating system partitioned) computing system. The actual number of clients within each CEC110may vary and could range from a single client to hundreds or thousands of clients, without limitation. For efficiency in presenting the inventive concepts herein, only two clients are presented within each CEC110of the various illustrative and described embodiments.

DPS100also comprises a distributed storage facility, accessible to each of the CECs110and the components within the CECs110. Within the described embodiments, the distributed storage facility will be referred to as distributed data store150, and the distributed data store150enables several of the client level functional features provided by the embodiments described herein. Distributed data store150is a distributed storage facility providing a single view of storage that is utilized by each CEC110and for each client114of each CEC110within a cluster-aware, distributed system. Distributed data store150comprises local physical storage160and network storage161, both of which comprise multiple physical storage units162(e.g., disks. solid state drives, etc.). The physical disks making up distributed data store150may be distributed across a storage network (e.g., a SAN). Additionally, distributed data store150provides a depository within which is stored and maintained the software utility, instruction code, OS images, client images, data (system, node, and client level), and/or other functional information utilized in maintaining the client-level, system management, and storage-level operations/features of DPS100. In addition to distributed data store150, DPS100also comprises a VIOS database (DB)140, which may also be a distributed storage facility comprising physical disks across a storage network. VIOS DB (or DB)140is a repository that stores and provides access to various cluster configuration data and other functional components/modules and data structures that enable the various cluster-aware functionality described herein. In one embodiment, portions of distributed data store150may be allocated to provide storage pools for a cluster. Each VIOS112of the cluster maintains a local view of the DB140and updates the cluster level information/data/data structures within DB140as such information/data is created or updated.

Communication between each VIOS112of each CEC110as well as with the VIOSes of at least one other CEC110is generally supported via a plurality of inter-CEC interconnects, illustrated as bi-directional, dashed lines connecting pairs of VIOSes112. The arrows indicated two way data exchange or communication between components. In addition to the inter-CEC interconnects, each VIOS112is also connected to Distributed data store150via CEC-to-Store interconnects, which are also illustrated as full lined bi-directional arrows. Also, each VIOS112is connected to DB140via VIOS-to-DB interconnects, presented as dashed and dotted lines. With the exception of the inter-CEC connectors running from a first VIOS (e.g., VIOS112a) of a first CEC to a second VIOS (e.g., VIOS112b) on the same CEC, the various interconnects represent a network level connectivity between the VIOS nodes of the cluster and the DB140and the distributed data store150. As utilized herein, references to one or more “nodes”, are assumed to refer specifically to a VIOS within the cluster. DPS100also comprises a management console175on which a management tool (not shown) executes.

Turning now toFIG. 1B, there is illustrated another view of DPS100illustrating the network-based connection of the CECs110to the distributed storage repository150and DB140.FIG. 1Billustrates in greater detail the network connectivity of VIOSes and CECs to each other and to Distributed storage repository150. With this view, CEC_A (Node_A)110A and CEC_B (Node_B)110B comprise similar constructs as presented inFIG. 1A. Each CEC110within DPS100connects to distributed storage repository150via one or more networks and/or I/O interconnect/switch fabric (generally illustrated as interconnect/network fabric170). The descriptions and illustrations assume that at least some of the CECs110of DPS100and distributed storage repository150are located remotely from each other, including being located in different countries, for example, such that no direct physical connectivity exists between the respective devices. For simplicity, the embodiments are described as having primary interconnect/network170comprising a private wide area network (WAN) or a public WAN (such as the Internet), although other network types (e.g., a local area network) are possible and supported.

As depicted, in one or more embodiments, each CEC110is also connected to one or more neighbor CECs110, in order to provide efficient fail-over and/or mobility support and other functions, as described hereinafter. As utilized herein, the term neighbor refers to a connected second CEC with which a first CEC is able to communicate, and references to a neighbor CEC is not limited to a second CEC in geographic proximity to the first CEC. CEC_A110A and CEC_B110B are illustrated connected to each other via some connecting medium, which may include a different network (such as a local area network)172or some type of direct interconnect (e.g., a fiber channel connection) when physically close to each other. The connection between neighbor CECs110A and110B is illustrated as a direct line connection or a secondary network connection (172) between CECs110A and110B. However, it is appreciated that the connections are not necessarily direct, and may actually be routed through the same general interconnect/network170as with the other CEC connections to distributed storage repository150. In one or more alternate embodiments, the connections between CECs may be via a different network (e.g., network172,FIG. 1B), such as a local area network (LAN).

As depicted, each CEC110comprises one or more network interfaces134and one or more I/O adapters132to enable the CEC110and thus the other components (i.e., client partitions) of the CEC110to engage in network level communication. Each VIOS112emulates virtual client I/O adapters226a-22cto enable communication by specially-assigned client LPARs114a-114cwith distributed storage repository150and/or other clients, within the same CEC or on a different CEC. The VIOSes112emulate these virtual I/O adapters226a-226cand communicates with distributed storage repository150by connecting with corresponding virtual sever I/O adapters (SVA)152a-152cat distributed storage repository150. Internal CEC communication between VIOS112and client LPARs114a-114care illustrated with solid connecting lines, which are routed through the virtualization management component, while VIOS to server communication is provided by dashed lines, which connect via the network/interconnect fabric172. Management console175is utilized to perform the setup and/or initialization of the backup and restore operations described herein for the individual VIOSes112and/or of the VIOS cluster as a whole, in various embodiments. The VIOSes112within each CEC110are thus able to support client level access to distributed storage150and enable the exchange of system level and client level information with distributed storage repository150.

In addition, each VIOS112also comprises the functional components/modules and data to enable the VIOSes112within DPS100to be aware of the other VIOSes anywhere within the cluster (DPS100). From this perspective, the VIOSes112are referred to herein as cluster-aware, and their interconnected structure within DPS100thus enables DPS100to also be interchangeably referred to as cluster-aware DPS100. As a part of being cluster-aware, each VIOS112also connects to DB140via network170and communicates cluster-level data with DB140to support the cluster management functions described herein.

Also illustrated byFIG. 1Bis an initial view of the component make-up of an example distributed storage repository150and an initial listing of some components of DB140. To support the virtual I/O operations with the VIOSes112and the associated virtual client I/O adapters, distributed storage repository150comprises communication infrastructure151. Communication infrastructure151comprises network interface(s)153and a plurality of server I/O adapters152utilized for cluster-level communication and enabling access to data/code/software utility stored on distributed storage repository150to complete I/O operations thereto. Specifically, these server I/O adapters are also presented as virtual sever I/O adapters, which are paired with virtual I/O adapters (132) that are assigned to clients114of CECs110.

As shown, distributed data store150generally comprises general storage space160(the available local and network storage capacity that may be divided into storage pools) providing assigned client storage165(which may be divided into respective storage pools for a group of clients), unassigned, spare storage167, and backup/redundant CEC/VIOS/client configuration data storage169. In one embodiment, the assigned client storage is allocated as storage pools, and several of the features related to the sharing of a storage resource, providing secure access to the shared storage, and enabling cluster-level control of the storage among the VIOSes within a cluster are supported with the use of storage pools. When implemented within a VIOS cluster, storage pools provide a method of logically organizing one or more physical volumes for use by the clients supported by the VIOSes making up the VIOS cluster.FIG. 4Aillustrates an example configuration of a storage pool utilized within a cluster aware DPS100. Specifically,FIG. 4Aprovides details on how these physical volumes are used within the storage pool. As shown, storage pool460within the cluster contains one or more Disk Groups462. Disks Groups462provide administrators the ability to provide access policies to a given subset of physical volumes162within the storage pool460. Once a disk group462has been defined, administrators can further categorize the subset into Storage Tiers464based on disk characteristics. Once a Disk Group462and Storage Tier464have been defined, administrators carve Logical Units (LU)466to be exported to client partitions (114).

With the capability of virtual pooling provided herein, an administrator allocates storage for a pool and deploys multiple VIOSes from that single storage pool. With this implementation, the SAN administration functions is decoupled from the system administration functions, and the system administrator can service customers (specifically clients114of customers) or add an additional VIOS if a VIOS is needed to provide data storage service for customers. The storage pool may also be accessible across the cluster, allowing the administrator to manage VIOS work loads by moving the workload to different hardware when necessary. With the cluster aware VIOS implementation of storage pools, additional functionality is provided to enable the VIOSes to control access to various storage pools, such that each client/customer data/information is secure from access by other clients/customers.

As illustrated, DSR150further comprises a plurality of software, firmware and/or software utility components, including DSR configuration utility154, DSR configuration data155(e.g., inodes for basic file system access, metadata, authentication and other processes), and DSR management utility156.

To support the cluster awareness features of the DPS100, and in accordance with the illustrative embodiment, DPS100also comprises VIOS database (DB)140, in which is stored various data structures generated during set up and/or subsequent processing of the VIOS cluster-connected processing components (e.g., VIOSes and management tool). DB140comprises a plurality of software or firmware components and/or and data, data modules or data structures, several of which are presented inFIG. 1B, for illustration. Among these components are cluster management (CM) utility182, VIO AdapterID data structure183, cluster configuration data184, Client identifying (ID) data185, active nodes list186, and I/O redundancy data187, among others. These various components support the various clustering functionality and cluster-aware I/O operations of the one or more VIOSes112, as described herein. In the present embodiment, VIOS DB140also comprises a mobility table510by which the nodes of the VIOS cluster are able to track and support movement and/or re-location of VIOS partitions and/or client partitions within the VIOS cluster. Additional features of DB140and distributed storage repository150as well as the specific components or sub-components that enable the various clustering functionality are presented within the description of the remaining figures and throughout the description of the various presented embodiments.

The various data structures illustrated by the figures and/or described herein are created, maintained and/or updated, and/or deleted by one or more operations of one or more of the processing components/modules described herein. In one embodiment, the initial set up of the storage pools, VIOS DB140and corresponding data structures is activated by execution of a cluster aware operating system by management tool180and/or one or more VIOSes112. Once the infrastructure has been established, however, maintenance of the infrastructure, including expanding the number of nodes, where required, is performed by the VIOSes112in communication with DB140and the management tool180.

Also associated with DPS100and communicatively coupled to distributed storage repository150and DB140and VIOSes112is management console175, which may be utilized by an administrator of DPS100(or of distributed storage repository150or DB140) to access DB140or distributed storage repository150and configure resources and functionality of DB140and of distributed storage repository150for access/usage by the VIOSes112and clients114of the connected CECs110within the cluster. As shown inFIG. 1Band described throughout the specification, management tool180is implemented within management console175. However, it is appreciated that (resources of) any node within DPS100may be selected/elected to perform the functions of management tool180, and the selected node would then perform one or more of the below described cluster creation and the other cluster monitoring and management functions, utilizing the availability of the resources provided by DB140and distributed storage repository150.

In an alternate embodiment, management tool180is an executable module that is executed within a client partition at one of the CECs within DPS100. In one embodiment, the management tool180controls the operations of the cluster and enables each node within the cluster to maintain current/updated information regarding the cluster, including providing notification of any changes made to one or more of the nodes within the cluster. In one embodiment, management tool180registers with a single VIOS112band is thus able to retrieve/receive cluster-level data from VIOS, including FFDC data (191) of the entire cluster.

With reference now toFIG. 2A, there is presented a third view of an example DPS100, emphasizing a processing system architecture200(i.e., architecture of the individual CECs, and specifically CEC_A110A). CEC_A110A (CEC110A) serves as the example CEC that is described in greater detail inFIG. 2Aand throughout the specification. CEC110A is presented as a server that comprises hardware components and software/firmware/OS components that are logically partition to create a plurality of virtualized machine partitions, which are assigned as client logical partitions (LPARs) and virtual I/O servers (VIOSes). Hardware components230of example CEC110A comprises one or more processors231A-231P, one or more memories233A-233M, and local storage234. The processors230A-230P are interconnected with one or a plurality of memories233A-233M and with local storage234via a bus, interconnect/switch or an interconnect fabric (not specifically shown). The specific internal connectivity of components, which may be distributed across a large scale interconnect fabric, is not germane to the described embodiments, and no further detail is presented regarding the particular type of interconnectivity between the system hardware components.

Also included within hardware components230are one or more physical network interfaces134by which CEC_A110A connects to an external network, such as network170, among others. Additionally, hardware components230comprise a plurality of I/O adapters232A-232E, which provides the I/O interface for CEC_A110A. I/O adapters232A-232E are physical adapters that enable CEC_A110to support I/O operations via an I/O interface with both locally connected and remotely (networked) connected I/O devices, including SF storage150. Examples of I/O adapters include Peripheral Component Interface (PCI), PCI-X, or PCI Express Adapter, and Small Computer System Interconnect (SCSI) adapters, among others. CEC110is logically partitioned such that different I/O adapters232are virtualized and the virtual I/O adapters may then be uniquely assigned to different logical partitions. In one or more embodiments, configuration data related to the virtualized adapters and other components that are assigned to the VIOSes (or the clients supported by the specific VIOS) are maintained within each VIOS and may be maintained and updated by the VIOS OS, as changes are made to such configurations and as adapters are added and/or removed and/or assigned.

Logically located above the hardware level (230) is a virtualization management component, provided as a Power Hypervisor (PHYP)225(trademark of IBM Corporation), as one embodiment. While illustrated and described throughout the various embodiments as PHYP225, it is fully appreciated that other types of virtualization management components may be utilized and are equally applicable to the implementation of the various embodiments. PHYP225has an associated service processor227coupled thereto within CEC110. Service processor227may be used to provide various services for one or more logical partitions. PHYP225is also coupled to hardware management controller (HMC)229, which exists outside of the physical CEC110. HMC229is one possible implementation of the management console175illustrated byFIGS. 1A-1B, and the use of HMC229specifically within this illustration is solely for illustration of one actual embodiment among several available options. Operations of the different logical partitions may be controlled through HMC229, which is a separate data processing system from which a system administrator may perform various functions, such as reallocation of resources to different logical partitions. Importantly, features related to backup and restoration of OS partitions and in particular of the VIOSes and the VIOS cluster are controlled through the HMC, in the present embodiment, but those features are described more generally with reference to the management console175in the various other embodiments presented herein.

CEC_A110A further comprises a plurality of user-level logical partitions (LPARs), of which a first two are shown, represented as individual client LPARs114A-114B within CEC110A. According to the various illustrative embodiments, CEC110A supports multiple clients and other functional operating OS partitions that are “created” within a virtualized environment. Each LPAR, e.g., client LPAR114A, receives an allocation of specific virtualized hardware and OS resources, including virtualized CPU205A, Memory210A, OS214A, local firmware216and local storage (LStore)218. Each client LPAR114includes a respective host operating system214that controls low-level access to hardware layer (230) of CEC110A and/or to virtualized I/O functions and/or services provided through VIOSes112. In one embodiment, the operating system(s) may be implemented using OS/400, which is designed to interface with a partition management firmware, such as PHYP225, and is available from International Business Machines Corporation. It is appreciated that other types of operating systems (such as Advanced Interactive Executive (AIX) operating system, a trademark of IBM Corporation, Microsoft Windows®, a trademark of Microsoft Corp, or GNU®/Linux®, registered trademarks of the Free Software Foundation and The Linux Mark Institute) for example, may be utilized, depending on a particular implementation, and OS/400 is used only as an example.

Additionally, according to the illustrative embodiment, CEC110A also comprises one or more VIOSes, of which two, VIOS112A and112B, are illustrated. In one embodiment, each VIOS112is configured within one of the memories233A-233M and comprises virtualized versions of hardware components, including CPU206, memory207, local storage208and I/O adapters226, among others. According to one embodiment, each VIOS112is implemented as a logical partition (LPAR) that owns specific network and disk (I/O) adapters. Each VIOS112also represents a single purpose, dedicated LPAR. The VIOS112facilitates the sharing of physical I/O resources between client logical partitions. Each VIOS112allows other OS LPARs (which may be referred to as VIO Clients, or as Clients114) to utilize the physical resources of the VIOS112via a pair of virtual adapters. Thus, VIOS112provides virtual small computer system interface (SCSI) target and shared network adapter capability to client LPARs114within CEC110. As provided herein, VIOS112supports virtual real memory and virtual shared storage functionality (with access to distributed storage repository150) as well as clustering functionality. Relevant VIOS data and cluster level data are stored within local storage (L_ST)208of each VIOS112. For example, in one embodiment VIOS configuration data of the local VIOS hardware, virtual and logical components. Additionally, local storage (L_ST)208comprises cluster configuration data184, cluster state data185, active nodes list186.

Within CEC110A, VIOSes112and client LPARs114utilize an internal virtual network to communicate. This communication is implemented by API calls to the memory of the PHYP225. The VIOS112then bridges the virtual network to the physical (I/O) adapter to allow the client LPARs114to communicate externally. The client LPARs114are thus able to be connected and inter-operate fully in a VLAN environment.

Those of ordinary skill in the art will appreciate that the hardware, firmware/software utility, and software components and basic configuration thereof depicted inFIGS. 1A,1B,2A and2B may vary. The illustrative components of DPS100and specifically those within CEC110A are not intended to be exhaustive, but rather are representative to highlight some of the components that are utilized to implement certain of the described embodiments. For example, different configurations of data processing systems/CECs devices may be provided, containing other devices/components, which may be used in addition to or in place of the hardware depicted, and may be differently configured. The depicted example is not meant to imply architectural or other limitations with respect to the presently described embodiments and/or the general invention. The CEC110depicted in the various figures may be, for example, an IBM eServer pSeries system, a product of International Business Machines Corporation in Armonk, N.Y., running the Advanced Interactive Executive (AIX) operating system or LINUX operating system.

Certain of the features associated with the implementation of a cluster aware VIOS (e.g., VIOS112ofFIGS. 1A,1B and2) are introduced above with reference to the description of the previous figures, and particularlyFIG. 2. Descriptions of the specific functionality of the VIOS will continue to be provided with reference to the illustrations ofFIGS. 1A,1B and2. As presented byFIG. 2, each VIOS112is a virtual machine instance that emulates hardware in a virtualized environment. The VIOS112is tasked with emulating SCSI storage devices and grants access to Distributed data store150in cooperation with the PHYP225. Configuration of the VIOS is performed through the hardware management tools of HMC229. SCSI storage devices support a set of commands that allow SCSI initiators the ability to control access to storage. Data base programs, for example, may manage access to distributed data store150through a set of SCSI commands commonly referred to as persistent reserve. Other types of reserves are also supported by VIOS112, and the collective group of such commands is referred to herein as reserve commands.

As provided herein, each VIOS112allows sharing of physical I/O resources between client LPARs, including sharing of virtual Small Computer Systems Interface (SCSI) and virtual networking These I/O resources may be presented as internal or external SCSI or SCSI with RAID adapters or via Fibre-Channel adapters to Distributed data store150. The client LPAR114, however, uses the virtual SCSI device drivers. In one embodiment, the VIOS112also provides disk virtualization for the client LPAR by creating a corresponding file on distributed data store150for each virtual disk. The VIOS112allows more efficient utilization of physical resources through sharing between client LPARs, and supports a single machine (e.g., CEC110) to run multiple operating system (OS) images concurrently and isolated from each other.

As provided within VIOS112of CEC110A, VIOS112comprises cluster aware (CA) OS kernel220(or simply CA_OS220), as well as LPAR function code224for performing OS kernel related functions for the VIOS LPARs114. In one or more embodiments, the VIOS operating system(s) is an enhanced OS that includes cluster-aware functionality and is thus referred to as a cluster aware OS (CA_OS). One embodiment, for example, utilizes cluster aware AIX (CAA) as the operating system. CA_OS220manages the VIOS LPARs112and enables the VIOSes within a cluster to be cluster aware.FIG. 2Billustrates an example CA_OS kernel220with several functional modules, according to one embodiment. In the illustrative one embodiment, CA_OS kernel220comprises cluster management (CM) utility222which supports the VIOS configuration for cluster-level functionality. Also illustrated within CA_OS220are primary node election/operation utility182, node monitoring and reporting utility183, and Events module181, among others. Each of these additional software components may be a functional module within CM utility, in one embodiment, and are described as such throughout the remainder of this specification. In one embodiment, CM utility222may be a separate utility that is locally installed or downloaded (from DB140, for example) as an enhancement to an existing OS within a CEC110. CM utility222is then executed when configuring VIOS to join a cluster and become a cluster-aware node within the cluster, and CM utility enables the OS to support the various cluster-awareness and other cluster-level features and functionality. In an alternate embodiment, CA_OS220includes all the clustering features and functionality and rolls out the various features when the CEC110/VIOS112joins the cluster and/or during configuration of VIOS112to become cluster-aware.

According to one embodiment, cluster-awareness enables multiple independent physical systems to be operated and managed as a single system. When executed within one or more nodes, CA_OS220enables various clustering functions, such as forming a cluster, adding members to a cluster, and removing members from a cluster, as described in greater detail below. In one embodiment, CM utility222may also enable retrieval and presentation of a comprehensive view of the resources of the entire cluster. It is appreciated that while various functional aspects of the clustering operations are described as separate components, modules, and/or utility and associated data constructs, the entire grouping of different components/utility/data may be provided by a single executable utility/application, such as CA OS220. Thus, in one embodiment, CA_OS executes within VIOS112and generates/spawns a plurality of functional components within VIOS112and within DB140. Several of these functional components are introduced withinFIG. 1B, and others are described throughout the various embodiments provided herein. For simplicity in the descriptions which follow, references to cluster management utility and CA_OS220will be assumed to be referring to the same general component (i.e., CM utility222being a subcomponent of CA_OS220), and the terms may be utilized interchangeably throughout the specification.

As further presented by the illustrative embodiments (e.g.,FIG. 2A), VIOS112includes one or more additional functional modules/components, such as VIO adapter(s) (interface)226, and virtual I/O drivers/utility228, which provides I/O functionality to VIOS112and enables VIOS112to route data traffic to and from data structures and storage within distributed data store150and/or DB140. Virtual I/O adapter(s)226and CM utility222also enable the VIOS112to provide each client LPAR114with access to the full range of storage accessible within distributed data store150and other cluster-supported functionalities, as described herein.

In the illustrative embodiment, each client LPAR114communicates with VIOS112via PHYP225. VIOS112and client LPAR114A-114B are logically coupled to PHYP225, which enables/supports communication between both virtualized structures. Each component forwards information to PHYP225, and PHYP225then routes data between the different components in physical memory (233A-233M). In one embodiment, a virtualized interface of I/O adapters is also linked to PHYP225, such that I/O operations can be communicated between the different logical partitions and one or more local and/or remote I/O devices. As with local I/O routing, data traffic coming in and/or out of I/O adapter interface or network interface from a remote I/O device is passed to the specific VIOS112via PHYP225.

With the above introduced system configuration ofFIGS. 1A,1B and2A, a first VIOS112a(through a communication channel established via PHYP225), grants access to another VIOS112bthrough one or more virtual adapters. VIOS112includes the functionality to query PHYP225for the identity of the Client LPAR114on the CEC110where the VIOS112is currently running

C. CA VIOS Communication Protocol

One embodiment provides a communication protocol that enables efficient communication between the Clients114and distributed data store150via the respective VIOS112and virtual I/O adapters assigned within the VIOSes112to the specific client114. The embodiment further provides storage virtualization and management via the specific communication mechanisms/protocols implemented with respect to the use of cluster awareness and the Distributed data store150such that the virtualization is presented within the context of the server (CEC110) virtualization and management. With the presented protocol, different VIOSes112associated with different CECs110access the same single distributed DB140and cluster-level information is shared/communicated with each Client I/O process such that a first client on a first CEC is aware of which SAN disk resources are being accessed by a second client on a second CEC (or on the same CEC). With this awareness factored into the I/O exchange with the distributed data store150, the first client can avoid accessing the same storage resource that is concurrently being utilized by the second client, thus preventing data integrity issues, which would potentially cause data corruption and client partition crashes.

The communication protocol provides a highly integrated server-based storage virtualization, as well as distributed storage across clustered VIOS partitions. This protocol comprises one or more query features, which enables dynamic tracking of storage resource usage across the entire cluster. Throughout the following description, the communication and management protocol shall be described as a VIOS protocol. VIOS protocol provides distributed storage across clustered VIOS partitions. With the VIOS protocol, the storage is considered as a one large storage pool which chunks of storage (i.e., logical units or LUs) allocated to each client114. The VIOSes within the overall system (DPS100) are now structured as part of the cluster, with each VIOS being a node in the cluster. Each VIOS node communicates with other VIOS nodes utilizing the VIOS protocol. With this configuration of VIOSes, when two or more client LPARs114belonging to different CECs110share storage on the SAN (e.g., two clients assigned overlapping LUs), the VIOS protocol enables each node to query (each client within the cluster) to determine the current usage of the storage device. When this information is received, the VIOS may then disseminate this information to other VIOSes. Each client is thus made aware of whether the SAN storage device that the client is trying to access is currently being used by some other client.

According to the described implementation, the different clientID-vioAdapterID pairings are unique throughout the cluster, so that no two clients throughout the entire cluster can share a same virtual adapter and no two vioAdapterIDs are the same within a single client.FIG. 3is a flow chart illustrating the method by which a VIOS112on a CEC110with DPS100enables cluster level communication between a client LPAR114and distributed data store150, according to one embodiment. The process begins at block302at which the VIOS112queries PHYP225for the identity of the client LPAR114. At block304, the VIOS112creates a unique identifier (ID) for the client (i.e., a ClientID). The VIOS112then stores the unique ClientID in ClientID data structure159(FIG. 1B) within DB140(block306). The DB140and by extension the ClientID data structure159are accessible to each VIOS partition in the cooperating cluster (DPS100). At block308, the VIOS112also generates an identifier for each virtual IT nexus (virtual I/O AdapterID) that is utilized for each virtual adapter assigned to the client LPAR114. In one embodiment, a client LPAR114can have multiple virtual adapters assigned thereto. These vio AdapterIDs are stored in the AdapaterID data structure158(block310) and are associated with their corresponding clientIDs (block312). The method illustrated byFIG. 3ends at termination block314, with each clientID having been associated with the corresponding one or more vio AdapterIDs with DB140.FIG. 4Bdescribed below illustrates these data structures as well as several of the other components stored within DB140.

VIOS SCSI emulation code (an executable module provided by VIO software utility228) utilizes the vioAdapterID to emulate reserve commands. Secure access to storage pools are managed by the unique ClientID, which is provided on an access list associated with each storage pool. In one embodiment, the VIOS112supports commands that are invoked as part of moving a client LPAR114from a first (source) CEC (110A) to a second (destination) CEC (110B) in a cluster environment. The commands generate data streams describing the virtual devices, which include the vio Adapter information. That information is used to modify the ClientID database159so that the identity of the Client on the destination CEC (110B) is associated with the unique ClientID of that client, and the unique identifiers of the VIO adapters (VIO AdapterIDs) on the source CEC (110A) are inherited by the I/O adapters on the destination CEC (110B).

D. VIOS Shared DB for Cluster Management

As described herein, implementation of the cluster awareness with the VIOSes of the cluster enables the VIOSes to provide cluster storage services to virtual clients (114). The VIOS software stack provides the following advanced capabilities, among others: Storage Aggregation and Provisioning; Thin Provisioning; Virtual Client Cloning; Virtual Client Snapshot; Virtual Client Migration; Distributed Storage Repository; Virtual Client Mirroring; and Server Management Infrastructure integration. More generally, the VIOS protocol allows distributed storage to be viewed as centralized structured storage with a namespace, location transparency, serialization, and fine grain security. The VIOS protocol provides storage pooling, distributed storage, and consistent storage virtualization interfaces and capabilities across heterogeneous SAN and network accessible storage (NAS). In order to provide block storage services utilizing the distributed repository, each VIOS configures virtual devices to be exported to virtual clients. Once each virtual device is successfully configured and mapped to a virtual host (VHOST) adapter, the clients may begin utilizing the devices as needed. In one embodiment, the virtualization is performed utilizing POWER™ virtual machine (VM) virtualization technology, which allows the device configuration process to occur seamlessly because the physical block storage is always accessible from the OS partition. When a virtual target device is removed, the corresponding ODM entries are deleted. Within the clustered environment, removal of any of the LUs is noticed to the other VIOSes. According to the described method, a distributed device repository and local repository cache are utilized to ensure the nodes within the cluster become device level synchronized from each node (VIOS) in the cluster.

According to one embodiment, information needed to configure a virtual target device (VTD) is stored in DB140. This database (DB140) can be accessed by all the nodes in the VIOS cluster, utilizing services provided by Cluster-Aware OS, such as but not limited to Cluster-Aware AIX (CAA). Additionally, certain small levels of cluster data are stored in a local database (ODM) (e.g., virtualized portions of storage234,FIG. 2) on each node for the devices which exist on that node. This local storage is necessary in order for the processes running on the local node to be able to match the VIOS device with the correct information in the distributed database.

With information about each device being stored in the DB140, operations on those devices can be performed from any VIOS node in the cluster, not just the node on which the device resides. When an operation on a device is performed on a “remote” (non-local) node (i.e. one other than the node where the device physically resides), the operation is able to make any changes to the device's information in the DB140, as necessary. When corresponding changes are needed in the device's local database, the corresponding CM utility222enables the remote node to send a message (using cluster services) to the local node to notify the local node to make the required changes. Additionally, when a node in the cluster is booted up, or when the node rejoins the cluster after having been lost for any period of time, the node will autonomously reference the DB140in order to synchronize the data there with the local data of the node.

As an example, if an operation to delete a VIOS device from the local mode is executed on a remote node, the operation will remove the information associated with that device from the DB140, and send a message to the local node to tell the local node to remove the device from the local database. If the local node is down or not currently a part of the cluster, when the local node first boots up or rejoins the cluster, the local node will automatically access the DB140, retrieve current data/information that indicates that the information for one of the local devices has been removed, and delete that device from the local database records.

FIG. 4Bis a block diagram representation of functional components of a source node, a target node and shared storage (DB140) to enable cluster level information/data storage, management and exchange between the nodes and VIOS shared storage (DB140) during cluster level operations, including a live partition mobility operation. In one embodiment, a local copy of DB140is shared by each VIOS within the cluster. Each VIOS is then responsible for storing, maintaining and updating the data structures at DB140in one embodiment. As illustrated byFIG. 4B, DB140is accessible to the various VIOS nodes112and to management tool405. Database140comprises several different modules of data, which may be arranged in a plurality of formats (e.g., tables, raw data, sequenced data, etc.) According to the figure, DB140includes a virtual adapter data structure425, which maintains a listing of and configuration information about the virtual adapters. DB140also includes a second data structure430that holds the unique adapter identifiers (AdapterIDs), and is therefore referred to herein as AdapterID data structure430. DB140maintains a listing of and information about the VIOSes within a VIOS data structure435. In one or more embodiments, each of the described data structures425-435can be or can include a table within DB140. VIOS DB140also includes a mobility table510. In one embodiment a copy of the mobility table can be maintained at distributed storage repository150.

When a virtual adapter is first discovered, the cluster management (CM) utility122(FIG. 1B) creates a row within the virtual adapter data structure425and a row within the unique AdapterID data structure430. These two rows in the different data structures are associated with each other, and the identifier (ID) is guaranteed to be unique. In one or more embodiments, adapter names are unique per CEC110, and where VIOS partitions are not “mobile” (i.e., do not move from a first CEC to a second CEC), the adapter names can be identified using a CEC, name tupple. The kernel extension is passed the AdapterID and utilizes the AdapterID to identify the IT Nexus, thus allowing the VIOS cluster to limit access to storage based on the reserve commands. This scheme allows software designed to access physical devices (e.g., SCSI devices) to operate with security, without requiring any modification. This scheme further allows the customer to have access to a full suite of existing software solutions that are familiar to system administrators. The kernel extension is also tasked with sending a message through a socket once the kernel discovers the identity of a VIOS on VIOS login. The VIOS uses a SCSI standard login command, which is part of a protocol known as SRP. The SRP protocol is emulated over the PHYP transport layer by the VIOS. The SRP login can be use as a trigger to send a command over a socket with the CEC relative identity of the VIOS. A message is sent back to the kernel extension once the message is processed. The message triggers access to DB140, which access checks if the VIOS is known. If the VIOS is not known within the cluster, a unique identifier is assigned to the VIOS and a row is inserted in the VIOS data structure435within DB140. The created adapter row of the adapter data structure425in DB140is associated with this row of the VIOS data structure435. The management tool validates that the VIOS has access rights to the storage pools in use by the adapter to emulate logical units. In the described embodiments, the kernel extension does not allow I/O access to logical units making use of restricted storage pools until the VIOS identity is verified.

Among the principal functional features of the illustrative embodiments is the ability to cluster the VIOSes112of the various CECs110within the DPS100(FIG. 1A-1B). Additionally, VIOS provides clustering services that can be exposed to have operations be invoked not only on the local VIOS, but on remote nodes without the consumer being aware of such actions. The introduction of these technologies requires the consumers, namely a management tool, to be able to understand what capabilities a VIOS currently is running with when the VIOS is part of a cluster, and what the VIOS is potentially capable of runningFIG. 4Billustrates the communication between a management tool180, such as Systems Director of IBM Corporation, according to one embodiment, and the VIOS nodes within the DPS100.

According to one or more embodiments, the algorithms/functional software modules provided by CM utility222also account for the VIOS moving from a first CEC, referred to herein as the source CEC, to a second CEC, referred to herein as the destination or target CEC. One of the roles played by the VIOS in enabling performance of a mobility operation within the cluster aware DPS100is to describe the storage that is in use on the source CEC to the VIOS on the destination CEC. The description provide by the first VIOS112aincludes a key into an adapter table for the source adapter. The key is utilized to find the client (114) and unique AdapterID information, based on the data base relationship (e.g., the association of data structures (e.g., tables) within the database). The unique AdapterID is passed to the kernel extension, which verifies storage access. The PHYP signals the termination of the mobility operation, and as part of that completion, the row within the VIOS table is updated with the new CEC relative identifier. Thus, while the move of the particular LPAR is completed, the unique AdapterID assigned to that OS partition is not changed within the database (distributed storage repository150). The CEC relative identifier allows the VIOS to be discovered, while the unique AdapterID allows secure implementation of storage pool access rights. This scheme allows flexibility in the management tools implementation for pool security allowing for convenience of use by the system administrator.

F. Failure Recovery During Clustered Partition Mobility

In one implementation, certain functional components of CM utility222are encoded on local device storage accessible to corresponding VIOS112, such that the VIOS112is able to immediately register with the cluster and retrieve/download or have forwarded from DB140(on successful registration with the cluster) the necessary CM software, information and/or data to become cluster aware when the VIOS is initially activated within the CEC110. In addition to the locally stored software utility components of CM utility222, other functional components of CM utility222may be downloaded from DB140when CEC is powered on or when one or more VIOSes112and/or one or more new client LPARs114are enabled on CEC110. Additionally, according to the presently described embodiments, and additional utility is provided on the CEC to enable failure recovery during live partition mobility. The live partition mobility (LPM) utility executes within a CEC from which a client partition is undergoing a live mobility operation from a source VIOS on the first CEC to a target VIOS on a second CEC. The LPM utility activates a LPM module (of CM utility of CA_OS) within the source VIOS and one or more other VIOSes of the cluster, including the target VIOS.

According to one or more embodiments, and as illustrated byFIGS. 5A and 5B, the LPM utility550is implemented as a part of the management tool180and/or from the management console175. Other embodiments can provide for the LPM utility to be located within or associated with the PHYP225. Referring now toFIG. 5A, there is illustrated a data processing system with hardware and software components that can be utilized to initiate and support live partition mobility within A VIOS cluster, according to one or more embodiments. The illustrated processing system provides/supports the functionality of an example management console and is therefore referred to herein as management console175, for consistency. It is appreciated that the physical configuration of management console175may be different from that illustrated inFIG. 5A, and the specific configuration presented herein is provided for illustrative purposed only.

As illustrated, management console175comprises a processor502, which is communicatively coupled to local memory506and I/O controller/bridge510via system bus/interconnect504. I/O controller/bridge510has an associated I/O bus to which is connected one or more I/O devices, of which keyboard514and pointing device516(e.g., mouse), and display520are illustrated. Display520connects to I/O bus512via a graphics/display adapter518. Also connected to I/O bus512are network interface522and I/O adapter524. Network interface enables connection to an external network, such as is illustrated by network fabric170(FIGS. 1A-1C). I/O adapter524can be any I/O adapter that enables I/O interfacing with an I/O device and/or another data processing system, such as CEC110(FIGS. 1A-1Cand2). Management console175further includes a storage device530within which instructions/code/data related to processes on the management console may be stored.

In addition to these hardware components, located within local memory506are a plurality of software components that enable management console175to function as a management device within a VIOS cluster environment. Among these software components are local OS508and management tool180. Management tool180as previously described supports/provides certain of the functions related to management of a VIOS cluster, including initiating the set up the individual client LPARs, assigned to specific clients, and overall management functions associated with the client LPARs and the VIOSes on a CEC or within the VIOS cluster. Specific to the presently described embodiments, management tool180provides/comprises LPM utility550, which executes on processor502to provide a plurality of functions associated with the live partition mobility operations within a VIOS cluster. Communication of the management tool180(and/or LPM utility550) functions to the VIOSes can be accomplished via the virtualization management component225, in one embodiment. In the provided embodiments, some of the features of LPM utility180can be provided within the VIOSes as well, and the embodiments are described without specific limitation on whether the features are implemented on the management console175or on a VIOS112to which the management tool is communicatively connected.

In one embodiment, LPM utility550provides code/program instructions that are executed on one or more virtual processor resources of one or more VIOSes112within CEC110and/or on processor502of management console175to provide specific functions. Among the functionality provided when LPM utility550is executed and which are described in greater detail herein are the following non exclusive list: (a) creating a first monitoring thread on a first VIOS to track the status of an LPM event; recording information about the LPM event within a database by using said first monitoring thread; (b) identifying a first set of functioning monitoring threads that continue to function on a first, source server if the first VIOS and the first monitoring thread is crashed; (c) determining whether the first set of functioning monitoring threads is a single, last monitoring thread; and (d) if there are nodes on the first server exhibiting one or more of multiple pre-identified failure conditions (from among the non-exclusive list of (a) the VIOS can no longer perform I/O operations, (b) the VIOS goes offline or (c) the VIOS is removed from the cluster, performing, via the last monitoring thread, update operations within the database based on indication that there are nodes on the first server that are in a crashed state.

Turning now toFIG. 5B, there is illustrated an example VIOS cluster (i.e., a cluster aware collection of VIOSes) that is configured to effectively support a Live Partition Mobility (LPM) event, according to one embodiment. According to one or more embodiments, the algorithms/functional software modules provided by LPM utility550also account for the migration of one LPAR (e.g., LPAR114A) from source CEC110A to target/destination CEC110B. Each VIOS within the VIOS cluster (DPS100) can be considered a node in the cluster.

In response to detecting the start of an LPM event, LPM utility550initiates the creation of a first monitoring thread (e.g., by using monitor thread module504) on first VIOS112A corresponding to the LPM event. The LPM event specifically refers to the transfer of LPAR114A on CEC110A (i.e., a first, source server) to CEC110B (i.e., a second, target server) within the same VIOS cluster (DPS100). First VIOS112A runs on the source server (e.g., CEC110A) from which the client/LPAR114A currently accesses storage. In addition, LPM utility550initiates the creation of other monitoring threads (e.g., via monitoring thread module506) on every other VIOS (e.g., including VIOS112B) running on the (same) source server. Furthermore, LPM utility550initiates the creation of monitoring threads on second VIOS on a target server (e.g., CEC110B) from which the client subsequently accesses (i.e., is expected to subsequently access) storage once the (migrated) client is running on the target server (e.g., CEC110B). Similar to the creation of other monitoring threads on the source server, LPM utility550creates a collection of other monitoring threads on every other VIOS (e.g., including VIOS112D) on the target server. LPM utility550provides current/up-to-date information about a particular LPM event by enabling an appropriate monitoring thread(s) to create or update a respective row in the “mobility table” (e.g., table510) in database520of VIOS DB140. In one embodiment, mobility table510may be associated with a database stored within distributed storage repository150. Monitoring threads continually check the state of the LPM by communicating with hypervisor125. If a particular VIOS (e.g., VIOS512A) is in a failed condition (or is experiencing on of several pre-defined failure conditions;), LPM utility550provides notification of the partition failure to other functioning nodes/VIOSes. LPM utility550enables a functioning monitoring thread to update the LPM status within the table510. In particular, a last monitoring thread may perform update operations within the database based on an indication that there are nodes on the first server that are in a failed state. The utilization of mobility table510to efficiently provide multi-node failure recovery in clustered partition mobility is further described inFIG. 6.

FIG. 6illustrates an example mobility table that is updated by a monitoring thread during an LPM event, according to one embodiment. Table510provides information about three example LPM events within three rows of Table510, respectively. In table510, a first monitoring thread associated with monitor thread module504creates row602when an LPM event associated with LPAR114A is initiated. In one embodiment, LPM utility550enables a (relational) database to point to a client partition table to keep track of the client partition and provide information about the client (e.g., “client1”). Furthermore, LPM utility550enables the database to be mapped to several client partition tables, one for each VIOS that is involved in the LPM event and/or is running on the source server. In the database system for cluster VIOSes, these client partition tables may collectively be referred to as the client mobility table. In one embodiment, LPM utility550detects the start of an LPM event and enables a particular monitoring thread to create the row in mobility table510. In one embodiment, LPM utility550determines whether a particular LPM event already has a row reserved in a mobility table. If the particular LPM event has not yet been entered into the mobility table, LPM utility550enables the first monitoring thread of the VIOS on which the LPM is initiated to create the appropriate row in mobility table510. If the particular LPM event has been previously entered into the mobility table, LPM utility550enables the first monitoring thread or other appropriate monitoring thread (e.g., a last monitoring thread) to update the respective row of mobility table510.

In updated/created row502, LPM utility550enables the storage of information about (a) the server (e.g., CEC110A) upon which the monitoring thread is based, (b) the client (e.g., “client1” controlling/utilizing LPAR114A) for which the associated LPM is monitored by the monitoring thread, and (c) the current state of node/VIOS (i.e. whether the node/VIOS is running or has crashed) corresponding to the monitored LPM. Row502, for example, indicates that LPAR1 is currently “running”.

Since a first monitoring thread ceases to exist in the event that the partition (e.g., VIOS512A and associated LPAR) goes down, LPM utility550keeps the state field current at the database (e.g., VIOS DB140) by the functions provided by cluster aware DPS100. That is, if the cluster recognizes a node failure, the cluster updates the state field within the appropriate row of the VIOS table to indicate “failed.” Monitoring threads continually check the state of the LPM by communicating with hypervisor125. If a migration is terminated, the first monitoring thread performs certain actions that include possibly cleaning up storage resources and removing a respective row from mobility table510. The first monitoring thread performs these expected and required functions until a node (VIOS) fails (and, as a consequence, the first monitoring thread also fails) during LPM.

When the first monitoring thread fails (e.g., because of a VIOS failure condition), LPM utility550employs other monitoring threads which remain functional within the same CEC to do the work that the failed first monitoring thread cannot perform, as follows: when a monitoring thread is notified that the LPM operation is over (complete or failed), the monitoring thread is also supplied with a count of how many other threads on the server are currently monitoring the migration. If the notified monitoring thread is the last, LPM utility550enables the “last” monitoring thread to query mobility database table510to determine if there are any nodes on the same server in the failed/crashed state. For any entries that the (last) monitoring thread finds in the crashed state (e.g., LPAR2 of (second) row504), the last monitoring thread performs operations needed to keep the database current/consistent and then removes the row from table510.

Accordingly, multiple node failures are tolerated provided at least a single node involved in the LPM operation survives (and provides a “last” monitoring thread). In one embodiment, LPM utility550is able to roll back migration to a particular stage of the migration process. Thus, for example, in response to the LPM operation having ended as a result of a failure condition, the utility returns to a particular stage of a corresponding migration process in order to resume and complete the mobility operation. If the LPM process fails, LPM utility550may trigger a termination of the use of resources at the target server. On the other hand, if the LPM process succeeds, LPM utility550may terminate the use of resources at the source server. As a result of the cluster awareness features and characteristics of DPS100, the entry of third row506in table510of the same, shared VIOS DB140indicates that a particular partition (i.e., LPAR3) that is involved in an LPM event and is based on a different server (e.g., CEC110B) is also being monitored by a monitoring thread within the cluster (i.e., DPS100).

FIGS. 7-8are flow charts illustrating various methods by which the above processes of the illustrative embodiments are completed. Although the methods illustrated inFIGS. 7-8may be described with reference to components and functionality illustrated by and described in reference toFIGS. 1-6, it should be understood that this is merely for convenience and alternative components and/or configurations thereof can be employed when implementing the various methods. Certain portions of the methods may be completed by LPM utility550executing on one or more (virtual) processors (CPU206A) within VIOS112(FIG. 1or2) or on processing resources of management tool180(within management console175) or DB140. The executed processes then control specific operations of or on CECs110, client LPARs114, VIOSes112, DB140and/or distributed data store150. For simplicity is describing the methods, all method processes are described from the perspective of either/both LPM utility550and VIOS/node112.

FIG. 7illustrates the method by which a VIOS cluster monitors an LPM event, updates a mobility table and provides recovery from node failure, according to one embodiment. The method begins at initiator block702and proceeds to block704at which LPM utility550detects the initiation of an LPM event. At block706, LPM utility550initiates creation of a first monitoring thread on the VIOS corresponding to the LPM event and other monitoring threads on VIOSes on the source server from which the client currently accesses storage. At block708, LPM utility550enables the first monitoring thread to update/create a row of mobility table510. At block710, LPM utility550initiates continual checks of the LPM state by communicating with PHYP via one or more monitoring threads. At block712, LPM utility550detects failure of the node/VIOS corresponding to the LPM event. At block714, LPM utility550notifies the cluster of the node failure. LPM utility550enables the cluster and, in particular, the other monitoring threads on VIOSes on the source server to update the LPM status to crashed, as shown at block716. At decision block718, LPM utility550determines whether the LPAR migration is successfully completed or terminated. If LPM utility550determines that the LPAR migration is terminated or successfully completed, LPM utility550removes the corresponding row from mobility table510, as shown at block720. If LPM utility550determines that the LPAR migration is not terminated or successfully completed, the process moves to block722at which block the process ends.

FIG. 8illustrates the method by which the VIOS cluster utilizes a monitoring thread which remains functional/active after a first monitoring thread has failed to keep a database current and provide database consistency, according to one embodiment. The method begins at block802and proceeds to block804at which LPM utility550monitors an LPM event via a (first) monitoring thread. At decision block806, LPM utility550determines whether the LPM event is ended (i.e., either terminated or completed). If LPM utility550determines that the LPM event is ended, LPM utility550receives a count indicating the quantity of other threads currently monitoring the LPM, as shown at block808. If LPM utility550determines that the LPM event has not ended, the process returns to block804. At block810, LPM utility550initiates a check to determine whether a particular monitoring thread that receives count information is the last monitoring thread. At decision block812, LPM utility550determines whether the count information was received by a single/last remaining (and functioning) monitoring thread. If at decision block812LPM utility550determines that the count information was received by the last monitoring thread, LPM utility550enables the last monitoring thread to query the database/table to determine whether there are nodes on server in failed/crashed state, as shown at block814. In one embodiment, if at decision block812LPM utility550determines that the count information was not received by the last monitoring thread (i.e., more than one functional monitoring thread remain) LPM utility550selects a particular monitoring thread to query the database/table to determine whether there are nodes on server in crashed state, as shown at block815. At decision block816, LPM utility550determines whether there are VIOSes/nodes on the server in the failed/crashed state. If LPM utility550determines that there are VIOSes/nodes on the server in the failed/crashed state, LPM utility550removes the row (via the last or selected monitoring thread) corresponding to the one or more failed/crashed VIOSes from the mobility table510, as shown at block818. If LPM utility550determines that there are no VIOSes/nodes on the server in the crashed state, the process moves to block820at which block the process ends.

In the flow charts above, one or more of the methods are embodied in a computer readable medium containing computer readable code such that a series of steps are performed when the computer readable code is executed (by a processing unit) on a computing device. In some implementations, certain processes of the methods are combined, performed simultaneously or in a different order, or perhaps omitted, without deviating from the spirit and scope of the invention. Thus, while the method processes are described and illustrated in a particular sequence, use of a specific sequence of processes is not meant to imply any limitations on the invention. Changes may be made with regards to the sequence of processes without departing from the spirit or scope of the present invention. Use of a particular sequence is therefore, not to be taken in a limiting sense, and the scope of the present invention extends to the appended claims and equivalents thereof.

As will be further appreciated, the processes in embodiments of the present invention may be implemented using any combination of software, firmware or hardware. As a preparatory step to practicing the invention in software, the programming code (whether software or firmware) will typically be stored in one or more machine readable storage mediums such as fixed (hard) drives, diskettes, optical disks, magnetic tape, semiconductor memories such as ROMs, PROMs, etc., thereby making an article of manufacture in accordance with the invention. The article of manufacture containing the programming code is used by either executing the code directly from the storage device, by copying the code from the storage device into another storage device such as a hard disk, RAM, etc., or by transmitting the code for remote execution using transmission type media such as digital and analog communication links. The methods of the invention may be practiced by combining one or more machine-readable storage devices containing the code according to the present invention with appropriate processing hardware to execute the code contained therein. An apparatus for practicing the invention could be one or more processing devices and storage systems containing or having network access to program(s) coded in accordance with the invention.

Thus, it is important that while an illustrative embodiment of the present invention is described in the context of a fully functional computer (server) system with installed (or executed) software, those skilled in the art will appreciate that the software aspects of an illustrative embodiment of the present invention are capable of being distributed as a program product in a variety of forms, and that an illustrative embodiment of the present invention applies equally regardless of the particular type of media used to actually carry out the distribution.