Method and system for computing a quorum for two node non-shared storage converged architecture

Described is an approach for computing a quorum for a two-node non-shared storage converged architecture via a distributed state machine without the need for a quorum device.

FIELD

This disclosure concerns a method, a computer program product, and a computer system for computing a quorum for a two-node non-shared storage converged architecture without the need for a quorum device via a distributed state machine.

BACKGROUND

Legacy multi-node computing clusters in a converged architecture generally utilize shared storage and/or quorum consensus devices to determine which node is a master node so that multi-node computing clusters may continue operating after a problem occurs, or during a complete cluster failure to determine which node to restart the cluster. A converged architecture operates by grouping multiple information technology components into a single, optimized computing package. Components of a converged architecture may include servers/nodes, data storage devices, networking equipment and software for IT infrastructure management, automation, and orchestration.

A quorum device is a shared storage device and/or quorum server that (1) is shared by two or more nodes and (2) contributes votes that are used to establish a quorum to manage the general state of the cluster. The quorum device(s) often serve as tiebreakers when determining whether the cluster is healthy or not or whether a particular node is a master or not. A cluster may operate only when a quorum of votes is available. Both cluster nodes and quorum devices vote to form a quorum. By default, cluster nodes acquire a quorum vote count of one when they boot and become cluster members. Quorum devices acquire quorum vote counts that are based on the number of node connections to the quorum device. When a quorum device is set up, it acquires a maximum vote count of N−1 where N is the number of connected votes to the quorum device. For example, a quorum device that is connected to two nodes with nonzero vote counts has a quorum count of one (two minus one). When a two-node cluster goes down (e.g., fails) without a quorum device, there is no tiebreaker (e.g., the quorum device) to determine which node of the two-node cluster should be the master node to restart the cluster to ensure data integrity is maintained in the cluster. However, implementing a quorum device with each cluster introduces additional financial and technical costs and complexity to managing the cluster, especially when the cluster is only a two-node non-shared storage cluster.

Recently, with the advent of different types of local storage that may be directly attached to or operatively coupled to nodes, users are able to easily deploy nodes using new types of local storages as non-shared storage devices in a two-node cluster arrangement. For example, different types of local storages may include non-volatile memory (NVM) which is often NAND flash memory that comes in several physical form factors, including solid-state drives (SSDs), PCI Express (PCIe) add in cards and other forms such as M.2 cards. NAND flash memory is a type of nonvolatile storage technology that does not require power to retain data. An important goal of NAND flash development has been to reduce the cost per bit and to increase maximum chip capacity so that flash memory can compete with magnetic storage devices, such as traditional hard disks. NVM Express (NVMe), as a logical device interface, has been designed to capitalize on the low latency and internal parallelism of solid-state storage devices. Thus, many legacy users deploy a large number of two node clusters (e.g., hundreds and thousands of two node clusters) where the users simply want redundancy in their nodes for failover purposes without incurring additional costs associated with installing and managing quorum devices and/or shared disk management functionalities for managing each of the two node clusters.

When implementing a two-node cluster, it is often undesirable to include a tiebreaker node since there are disadvantages when including such a tiebreaker (e.g., a quorum device). The disadvantages are associated with resource availability. For example, when a user is provisioning a two-node cluster in a cloud implementation, the user may only want to deploy a two-node cluster without having to include a shared disk/shared database and/or a quorum device for managing the two-node cluster and/or the shared disk and/or shared database. The user may not want to pay for or manage a complex cluster infrastructure to just manage a two-node cluster. If the user plans to deploy, as an example, one thousand two-node clusters having non-shared storage, including one thousand shared databases and/or one thousand quorum devices may be a huge financial and infrastructure management burden for the user's organization.

However, failure to include a tiebreaker may also cause its own sets of problems. In a two-node cluster with non-shared storage, the absence of a quorum server or a quorum device may result in a non-deterministic, split-brain resolution during failure situations of either or both nodes. For example, the two nodes may both end up surviving independent crashes, which may result in corruption of data.

Therefore, there is a need for a method and/or system for computing a quorum for a two-node non-shared storage converged architecture without the need for a quorum device.

SUMMARY

According to some embodiments, described are improved systems, computer program products, and methods for determining a quorum in a two node non-shared storage cluster via a distributed state machine without the need for a quorum device and/or quorum server.

DETAILED DESCRIPTION

Various embodiments will now be described in detail, which are provided as illustrative examples of the invention so as to enable those skilled in the art to practice the invention. Notably, the figures and the examples below are not meant to limit the scope of the present invention. Where certain elements of the present invention may be partially or fully implemented using known components (or methods or processes), only those portions of such known components (or methods or processes) that are necessary for an understanding of the present invention will be described, and the detailed descriptions of other portions of such known components (or methods or processes) will be omitted so as not to obscure the invention. Further, various embodiments encompass present and future known equivalents to the components referred to herein by way of illustration.

The present disclosure provides an improved approach for computing a quorum for a two-node non-shared storage converged architecture, hereinafter referred to as a “two-node cluster”, without the use of a quorum device and/or tiebreaker. The non-shared storage may be non-shared databases, but logically shared through each respective node. In particular, the approach leverages a state management process for determining which node of the two-node cluster is a master node so that if the cluster fails and needs to be restarted, the master node may be determined based on this state management process. The approach provides a protocol for a planned transfer of mastership such that the participating nodes can confirm whether the re-mastership has been completed or whether the re-mastering was in progress when the cluster failed. In either case, based on the disclosed approach, the participating nodes, upon restart after a cluster failure, may be able to deterministically determine which node is the master node to restart the cluster, regardless of the whether the transfer in mastership completed successfully or failed during the re-mastering process.

The state management process writes values identifying which node is the master node in different persistent storage locations so that when the system fails and the two nodes are coming back online, each node of the two nodes can inspect the various different persistent storage locations that they have access to and based on the values that are stored in the different persistent storage locations, each node can figure out who is the master node deterministically. Furthermore, during a re-mastering process, where mastership of the master node is transferred to the non-master node, new values are written to the different persistent storage locations in a particular order so that even in the event of a cluster failure, e.g., where both nodes fail, in the middle of the re-mastering process, each node, upon startup, may inspect the various different persistent storage locations that they have access to and based on the values that are stored in the different persistent storage locations, each node can figure out which node is the master node deterministically so that the master node may restart the cluster.

For example, during normal operations, a current sequence number (discussed further below) can be found in three of the four persistent storage locations (e.g., local registry file140aand voting file150aof the master node and voting file150bof the non-master node, as discussed further below). During a re-mastering process, a new monotonically increasing sequence number is generated (e.g., a proposed sequence number, discussed further below) and stored in the plurality of persistent storage locations to manage the state of mastership in a particular and predefined sequence of steps. If the re-mastering process completes before a total failure of the cluster (e.g., before both nodes fail), then each node, upon restart, may deterministically determine who is the master node. However, if failure of the cluster occurs in the middle of the re-mastering process, as an example, each node upon their restart, may still be able to deterministically determine which node is the master node; simply by inspecting the plurality of persistent storage locations that each node has access to inspect, and in certain situations, in combination with a predefined timeout variable (discussed further below).

Although the state management process may be able to help determine which node is the master node during most failure scenarios, there are certain scenarios which may require manual intervention to properly restart the cluster. For example, in some embodiments, the master node may fail before a re-mastering process may be initiated (e.g., if someone accidentally unplugs a power cable of the master node and immediately shuts down the master node before a re-mastering process may be initiated by either the master node or the non-master node), a manual restart of the cluster may be required.

The state management process is maintained by first identifying a node from the two nodes in the two-node cluster as the master node the first time the two-node cluster is initialized (e.g., started, formed). When mastership needs to be transferred from the master node to the non-master node, a monotonically increasing sequence number is written into a plurality of persistent storage locations in a pre-defined sequence across the two-node cluster during the transfer in order to keep track of the transfer process in case the cluster fails in the middle of the change of mastership (e.g., re-mastering). By managing the state of the cluster in this manner, each node of the two-node cluster may determine whether it is the master node or non-master node when each node comes back online based upon at least each node inspecting the plurality of storage locations that it has access to in order to determine whether it is the master node or not. This is important because during a cluster restart, only the master node is allowed to restart the cluster. The non-master node cannot restart the cluster. Instead, the non-master node may only join/rejoin the cluster once the master node has restarted the cluster. Determining the master node to restart the cluster is an important process to ensure storage data among the cluster is maintained and not corrupted as a result of a cluster failure.

With the availability of cloud computing, converged architectures are very attractive to users because users may simply (a) provision servers having local storages attached to the servers and (b) connect the servers together without any additional infrastructures such as, for example, a shared database and/or quorum device(s) for operating and managing the cluster of nodes and storage devices. As discussed above, a converged architecture operates by grouping multiple information technology components into a single, optimized computing package. Components of a converged architecture may include servers/nodes, data storage devices, networking equipment and software for IT infrastructure management, automation, and orchestration. Many users are looking to form two-node converged clusters with non-shared local storages to provide fail-safe/fault tolerant across the two-node cluster without having to add on additional infrastructures typically required for managing multi-node clusters.

The approach disclosed herein improves the technological area of cluster management for two-node non-shared storage converged architectures because the present disclosure provides the ability to compute a quorum for a two node non-shared storage converged architecture by using a distributed state machine without the need for a quorum device and/or quorum server. Not having a quorum device included in the system to determine a quorum greatly reduces the amount of cluster management complexities and costs associated with a two-node cluster. In some embodiments, a user's computing infrastructure may include hundreds and/or thousands of two node non-shared storage converged architectures (e.g., a two-node cluster). In these embodiments, not having a third system/component (e.g., quorum device, quorum server, or a tiebreaker) to manage a quorum greatly reduces the complexities associated with systems management as well financial costs overall associated to technical equipment, software, and human resources for managing the two-node clusters.

FIG. 1illustrates a two-node non-shared storage converged architecture for computing a quorum for the cluster without a quorum device, according to some embodiments of the present disclosure. System100includes a two-node cluster103wherein the two-node cluster includes at least a first node110a, a second node110b, a first local storage120a, and a second local storage120b, wherein the first local storage120aand the second local storage120btogether represents a non-shared storage130.

Nodes110are servers. The first node110aincludes a mastership manager105aand a local registry file140athat stores state information for managing the state of the cluster. The mastership managers105manage the state of mastership of the two-node cluster103by ensuring that at any point in time, a master node may be determined by either the first node110aor the second node110b. The mastership managers105may constantly keep track of each node110's health via, as an example, heartbeat monitoring and node resource utilization statistics. If it is determined that the master node may be taken offline (e.g., a planned maintenance outage or an unplanned system failure), the mastership managers105will work together to transfer mastership from the master node to the non-master node by writing to a plurality of persistent storage locations (e.g., local registry files140and voting files150discussed below) within the two-node cluster103.

The local registry file140ais a persistent storage location on the first node110aused to at least hold sequence number(s) as discussed in more detail below. The local registry file140ais managed by the local file system of the first node110a. The local registry file140ais not network accessible and thus the mastership manager105bon the second node110bcannot access the local registry file140aover the network. The local registry file140bis managed by the local file system of the second node110b. The local registry file140bis not network accessible and thus the master manager105aon the first node110acannot access the local registry file140bover the network.

State information stored in the plurality of persistent storage locations may include a current committed sequence number and/or a proposed sequence number. The sequence numbers (e.g., the current committed sequence number and the proposed sequence number) are monotonically increasing numbers that are generated. The sequence number may be associated to a specific state of the system (e.g., currently committed or proposed). A change in sequence number takes place whenever a change in mastership of the nodes is initiated and completed. A local registry file140on a node having only one sequence number may indicate that the node may be a master node. However, a local registry file140of a particular node having no sequence numbers may indicate that the particular node is the non-master node. More disclosures about the relationship of the one or more sequence numbers stored in a local registry file140of a particular node at any moment in time to determine whether the node is a master node is discussed below.

Local storages120are storage media/devices connected to nodes110via, as an example, iSCSI devices and accessible over the network to the other nodes. For example, first node110amay be configured with one or more local storage directly attached to and/or operatively coupled with, the first node110a. The one or more local storage, attached to the first node110a, correspond to first local storage120a. First local storage120amay include a voting file150wherein the voting file150is a block in the local storage120that comprise heartbeats (e.g., status information of the nodes) and information indicating the current state of the system. The information indicating the current state of the system may either be a current sequence number or a proposal that includes both the current sequence number and a proposed sequence number, wherein the proposed sequence number is a greater number than the current sequence number.

A proposal may be a combination of two sequence numbers (e.g., a current sequence number “Sc” and a proposed sequence number “Sp”). A proposal indicates a start of a state transition, a re-mastering of the master node from a first node to a second node, or a transferring of mastership from a master node to a non-master node, all of which may be used interchangeably hereinafter. In some embodiments, a proposal may be two sequence numbers (e.g., Sc and Sp) stored in consecutive memory locations of a node registry file140and/or in a voting file150. For example, assuming node110ais currently the master node of the two-node cluster and no state transition has been initiated yet. The first voting file150ain this example may have a current sequence number Sc without any proposed sequence number Sp. The second voting file150bmay also have the current sequence number Sc without any proposed sequence number Sp and the first local registry file140amay also have the current sequence number Sc without any proposed sequence number Sp. The second local registry file140bon the second node110bwill not have any current sequence number Sc and/or proposed sequence number Sp, which indicates that second node110bis a non-master. In some embodiments, the current and proposed sequence numbers may be designated via a field name designation associated with the value of the respective sequence number.

The plurality of persistent storage locations comprises the local registry file140aon the first node110a, the local registry file140bon the second node110b, and the first voting file150awithin the first local storage120aand the second voting file150bwithin the second local storage120b. During normal operations, only a current sequence number Sc is stored in three of the four persistent storage locations, as discussed above. The three out of the four persistent storage locations correspond to a quorum indicating that the first node110ais currently the master node. During a re-mastering process (e.g., when mastership is being transferred from one node to the other node), a proposed sequence number is introduced into the plurality of persistent storage locations at various steps of the re-mastering process as a proposal. More detailed discussions of how a master node is determined based on the sequence number(s) stored in the plurality of persistent storage locations will be discussed below.

Local storage120may be accessible by both nodes of the two-node clusters when both nodes of the two-node clusters are operational and active. However, the local storage120associated to a particular node may not be accessible by the other node if the particular node fails and/or is offline. For example, when both nodes110are up and running, both local storages120are accessible to both nodes110. However, if the first node110ahas failed, the first local storage120ais no longer accessible by second node110b, hence the non-shared aspect of the local storage on the two nodes. The non-shared storage130comprises local storage devices120directly attached to and/or operatively coupled to respective nodes110. Although both local storages120are non-shared, local storage120are accessible by both nodes110when both nodes110are active and running. Thus, although the local storage120aand120bare non-shared, they are logically shared through each respective node110.

Mastership or assigning a node to be a master node of the two-node cluster occurs in two circumstances. The first circumstance is upon mastering the first time the cluster is formed and the second circumstance is upon re-mastering to change mastership during operation of the cluster. The first circumstance of mastering is performed by the first node to ever come up to form the cluster for the first time. The first node is considered the master node and the second node to come up to join the first node in the cluster the first time the cluster is formed is the non-master node. A user may choose the master node during the initial formation of the cluster. Once the master node forms the cluster, a new sequence number Sc is generated. The Sc is then written or ‘keyed’ to at least two of the persistent storages (e.g., local registry file140aand voting file150a). Once the non-master node joins the cluster, the Sc is written (e.g., keyed) to the voting file150bto complete the initial mastership setup. The remaining disclosure will focus on the re-mastering process to ensure a master node is always deterministic in the two-node cluster, even after a complete cluster failure/crash during the process of re-mastering.

Re-mastering is the mechanism for transferring mastership from one node to another node. There are many reasons why re-mastering is necessary even in a two-node cluster. For example, a master node may need to be taken offline for maintenance purposes, at which point, the master node may request the non-master node to perform the re-mastering and monitoring of the re-mastering process so that the master node may be taken offline for maintenance purposes. As another example, it may be determined that the master node is beginning to fail, at which point, re-mastering is initiated and managed by either the master node or the non-master node just in case the master node does fail, the two-node cluster would not have to fail as long as the mastership was successfully transferred to the non-master node before the master node fails. In some embodiments, when re-mastering is master driven, the master node may ask the non-master node to perform the re-mastering and monitoring of the re-mastering process.

Additionally, re-mastering may be non-master driven such that a non-master node may initiate and monitor the re-mastering process. For example, if the non-master node determines that the master node may be failing, the non-master node may initiate the re-mastering so that mastership may be transferred to the non-master node in order to prevent the failing of the master node to take down the two-node cluster. The managing of the state of the mastership of the two nodes is based at least in part on the writing of the new sequence number as a proposal to a majority of the plurality of persistent storage locations to initiate and monitor the process of re-mastering to ensure that if there are any issues in the middle of the transfer of the re-mastering process, there will be a process of checking the plurality of persistent storage locations to determine which node of the two-node cluster is the master node that will restart the cluster. This way, a master node may be determined at all times, even during failure of the master node in the middle of the re-mastering process.

The determination of which node is the master node is based at least in part on determining at which point during the re-mastering process the failure took place. For example, if the failure took place early during the re-mastering process, the original master node maintains mastership. And, if failure took place late during the re-mastering process, the original non-master node may be determined to be the master if enough of the re-mastering process has already taken place such that the new proposed sequence number is written to the local registry file (e.g., local registry file140b) of the original non-master node. However, in the event that a failure took place towards a middle of the re-mastering process, a determination must be further undertaken to determine mastership. More details regarding how a node is determined to be the master node during a failure in the middle of a re-mastering process is discussed below.

In some embodiments, the re-mastering process is managed and monitored by the non-master node. Determining when a re-mastering needs to be started may be determined by either the master node or the non-master node or by a system administrator, or some other third entity. The non-master node may handle the actual managing and monitoring of the re-mastering process. In some embodiments, database transactional activities are paused while the re-mastering process is in progress to ensure that no new transactional data is written to any storage devices that may result in split brain issues or data corruption if the cluster fails in the middle of the re-mastering process.

FIG. 2shows a high-level flowchart of an approach to implement some embodiments of the present disclosure. Process flow200illustrates the steps involved for re-mastering (e.g., transferring mastership from one node to the other node). At210, a master node may be identified as a first node in a two-node cluster environment. The first node may be the node that originally was designated as the master node with the two-node cluster was formed or created the first time (as discussed above). However, the first node may be the master node by way of a successful re-mastering process that transferred mastership from a second node to the first node. In any case, at any point in time during normal operations of the two-node cluster, one node of the two nodes of the two-node cluster is designated as the master node such that the master node may be easily identifiable.

At220, a transfer of mastership from the first node to a second node of the two-node cluster may be initiated. The initiation of the mastership transfer may be due to a planned maintenance outage of the first node for, as an example, the purpose of upgrading hardware and/or software. Additionally, the initiation of the mastership transfer may be due to a determination that the first node, currently the master node, may be experiencing some potential issues that may indicate that the first node may fail in the near future, in which case, a transfer of mastership from the first node to the second node may be required to ensure in the event the first node fails, the second node is the master node and may continue to keep the cluster alive so that when the first node is restarted, it may rejoin the cluster.

At230, a sequence number is generated as a proposed sequence number Sp to initiate the transfer of mastership from the first node to the second node in a two-node non-shared storage cluster. The proposed sequence number Sp is a monotonically increasing number such that the proposed sequence number Sp is a larger number than the current committed sequence number Sc stored in a majority portion of the plurality of persistent storage locations. For example, assuming the master node is node110a, the majority portion of the plurality of persistent storage locations that may store the current committed sequence number Sc include the local registry file140a, the first voting file150aand the second voting file150b.

At240, the proposed sequence number Sp, recently generated, is written to a plurality of persistent storage locations in a particular order as a proposal (e.g., Sc and Sp) to track the progress of the transferring of the mastership from a first node to a second node. The details of the particular order of writing the proposed sequence number Sp to the plurality of persistent storage locations is further disclosed inFIG. 3below.

FIG. 3illustrates a flowchart to write a proposed sequence number to a plurality of persistent storage locations in a particular order, according to some embodiments of the present disclosure.

At310, the first local registry file140ais updated to include the proposed sequence number Sp as well as the current sequence number Sc (e.g., the proposal including both the Sc and Sp). Because this is the first time the proposed sequence number Sp is written, this update of the local registry file140aindicates that the re-mastering process has been initiated. For example, if the local registry file140ais inspected at this point in time, the presence of both sequence numbers (e.g., Sc/Sp as a proposal) may indicate that a re-mastering process is underway.

At this point in time, if the master node fails, the re-mastering process fails, or the entire cluster fails, the first node may be determined to be the master node of the cluster because an inspection of the plurality of storage locations can quickly show that the re-mastering process failed very early in its processing since nowhere else in the plurality of storage locations can the proposed sequence number Sp or the proposal (e.g., Sc & Sp) be found. Which means if the cluster fails and needs to be restarted, since the first local registry file includes both the Sc and Sp, and no other persistent storage locations of the plurality of persistent storage locations have the Sp, mastership is still maintained by first node (e.g., the current master node). In some embodiments, the request to update the first local registry file is made by the second node (e.g., second node110b), but the actual processing of updating the first local registry file is handled by the first node since the first local registry file is not network accessible by the second node110b.

At320, the first voting file150aon the first local storage120ais updated to include the proposed sequence number Sp as well as the current sequence number Sc (e.g., the proposal). In some embodiments, the second node performs the update to include the proposed sequence number Sp into the first local storage. At this point in the re-mastering process, if the master node fails, the re-mastering process fails, or the entire cluster fails, the first node may still be determined to be the master node.

At330, the second voting file150bon the second local storage120bis updated to include the proposed sequence number Sp as well as the current sequence number Sc (e.g., the proposal). In some embodiments, the second node performs the update of including the proposed sequence number Sp into the first local storage. At this point in the re-mastering process, if the master node fails, the re-mastering process fails, or the entire cluster fails, the first node may still be determined to be the master node because the second node does not yet have a committed sequence number in the local registry file140blocated on the second node. However, there are some situations where the first node may think the second node may be the new master, but the first node may wait for a disk timeout period before concluding that the first node is still the master. A disk timeout period is the amount of time which a particular node would perceive another node to be dead based on heartbeat information written by the other node in the voting file. The disk timeout period may be a preconfigured amount of time determined to provide adequate time for a miscommunication between two nodes to indicate a node failure. Further disclosures as to situations of when the first node may suspect it is still the master node, but must perform a few more checks/steps before concluding that the first node is still the master node, are discussed below.

At340, the proposed sequence number Sp is added to the local registry file140bon the second node110bas a committed sequence number. As discussed above, before adding the proposed sequence number Sp to the local registry file140b, the local registry file does not have the currently committed sequence number Sc. In fact, this is the only persistent storage location of the plurality of storage locations used to monitor the re-mastering process that does not have the currently committed sequence number Sc during normal operation. This is because during a normal failure where no re-mastering is taking place, at restart of the nodes, the first thing the nodes110check is its respective local registry file140. If there are no current sequence number Sc or proposed sequence number Sp in its local registry file, then the node may conclude that it is the non-master node and will not attempt to restart the cluster.

Instead, the non-master node will rejoin the cluster once the master node restarts the cluster. However, if the respective local registry file140has only a current sequence number Sc (i.e., no proposed sequence number Sp) then the respective node may conclude that the respective node is a master node, at which time, the respective node may restart the cluster so that the non-master node, when it is ready to join the cluster, may join the cluster.

Returning back toFIG. 2, at250, a portion of the plurality of persistent storage locations are updated upon the completion of the transfer of mastership from the first node to the second node. The local registry file140ais cleared of the proposal (e.g., the currently committed sequence number Sc and proposed sequence number Sp). Additionally, the first and second voting files (e.g., voting files150aand150b) are updated such that the currently committed sequence number is replaced with the proposed sequence number and the proposed sequence number is replaced with a value of “0” or null to indicate that the currently committed sequence number has a higher value than the proposed sequence number (which has a value of “0” or null) to indicate that no re-mastering process in currently in progress. In some embodiments, the clearing of the local registry file140aand the update of both of the voting files150aand150bare performed in a single transaction to complete the last step of the re-mastering process.

At this point in the re-mastering process, if the master node fails, the re-mastering process fails, or the entire cluster fails, the re-mastering process is deemed to have been effectively completed such that mastership is successfully transferred from the first node110ato the second node110b. For example, once the local registry file140ais cleared and the voting files are updated, there is no longer any question as to which of the two nodes is the master. If the first node fails and is restarted, the mastership manager105aon the first node will check its local registry file140aand determine that there are no sequence numbers in the local registry file. Next, the first node will check its voting file150ato find that the currently committed sequence number has a value while the proposed sequence number has a value of “0” or null, which indicates no re-mastering process is in progress. Thus, the first node may conclude without doubt that the first node is not the master and that the other node (e.g., the second node) is the master node, and the first node is the non-master node. In some embodiments, the mastership manger105amay simply conclude that node110ais the non-master node simply from the fact that the local registry file140aon the first node110adoes not have a committed sequence number and that the proposed sequence number has either a value of “0” or null.

FIGS. 4A-4Gillustrate an example of transferring mastership from a first node to a second node in a two-node non-shared storage cluster, according to some embodiments of the present disclosure.FIGS. 4A-4Gillustrates a system similar to the system ofFIG. 1disclosed above with the exception that local registry files140and voting files150show two storage location within their respective files for storing either a currently committed sequence number Sc or a proposed sequence number Sp.FIGS. 4A-4Gillustrates, according to some embodiments of the present disclosure, a sequential set of steps (seeFIG. 3) for tracking the state of the cluster during a transfer of mastership from a first node (node110a) to a second node (node110b).FIGS. 4A-4Gwill now be discussed with references made to steps fromFIG. 3.

FIG. 4Aillustrates a two-node cluster system400under normal operating conditions where no re-mastering is currently in progress. This is evident by the currently committed sequence number410, shown as Sc, being currently stored in the local registry file140a, the first voting file150a, and the second voting file150b. Additionally, the second node registry file140bdoes not contain any sequence numbers. Currently the master node is the first node110a. This is determined by inspecting the four persistent storage locations (e.g., local registry files140and voting files150). For example, a proposed sequence number is not currently stored in any of the four persistent storage locations. This is clear that a re-mastering has not yet been initiated, and thus node110ais the master node. Additionally, node110bis clearly a non-master node since the local registry file140bdoes not contain any sequence number(s), indicating that node110ashould have at least a current sequence number410.

FIG. 4Billustrates the two-node cluster system400where a proposed sequence number420, shown as Sp in the figures, is first entered into the local registry file140aof the first node110a(e.g., Step310fromFIG. 3) to update the registry file140ato include a proposal depicted as having both the current sequence number410and the proposed sequence number420. As discussed above, the proposed sequence number420is a monotonically increasing number that is generated. Proposed sequence number420is a number larger than the current sequence number Sc currently stored in the registry file140a.

The entry of the proposed sequence number420signals the initiation of a re-mastering process. Since node110ais the current master, writing the proposed sequence number420into the local registry file140aindicates that a re-mastering process is in progress. If the master node was to fail and had to restart at this point in time, the master node, upon restarting, would inspect the registry file140aand determine that a re-mastering is in progress and that additional information may need to be checked to determine whether or not node110ais still the current master or if mastership has already transferred to the second node110b.

The proposed sequence number420is next written into the first voting file150ato update the first voting file150a(e.g., Step320fromFIG. 3) to include the proposed sequence number420, as depicted inFIG. 4C. The first voting file150anow having a proposal comprising the current sequence number Sc and the proposed sequence number420, shown as Sp in figures. Next, the proposed sequence number420is written into the second voting file150bto update the second voting file150b(e.g., Step330fromFIG. 3) to include the proposed sequence number420, as depicted inFIG. 4D. The second voting file150bnow having the proposal comprising the current sequence number Sc and the proposed sequence number420, shown as Sp in the figures. At this point in time, the state of system400has the proposal sequence number420in the local registry file140aand on both voting files150on both nodes.

The proposed sequence number420from the previous steps is next written to the local registry file140b(e.g., Step340fromFIG. 3) as a committed sequence number430, also shown as Sp, as depicted inFIG. 4E. The local registry file140bnow has only the committed sequence number430. Node110bwas originally the non-master node of the two-node cluster since node110awas originally the master node. The non-master node of the two-node cluster does not have any sequence number in its local registry file140, as depicted inFIG. 4A-4D. Now, non-master node110bhas the proposed sequence number420written in its local registry file140bas committed sequence number430. Committed sequence number430is significant because when the second node110brestarts and inspects its local registry file140b, it will notice that there is a committed sequence number430stored within the local registry file140b, indicating that the second node110bis the master node of the two-node cluster.

Next, the currently committed sequence number Sc and the proposed sequence number Sp are both removed/deleted/cleared from the local registry file140a(e.g., Step250fromFIG. 2), as depicted inFIG. 4F. The local registry file140anow has no sequence numbers, thus indicating that the node110ais now the non-master node in the two-node cluster. Next, the first and second voting files are updated such that the currently committed sequence number is replaced with the proposed sequence number (e.g., Step250fromFIG. 2), as depicted as440inFIG. 4Gand the proposed sequence number is replaced with a value of “0”, as depicted as450inFIG. 4Gto indicate that the currently committed sequence number has a higher value than the proposed sequence number (which has a value of “0”) to indicate that there is no re-mastering process in progress. Updating the currently committed sequence number with the proposed number and updating the proposed sequence number with a value of “0” is to avoid monotonically increasing pending sequence numbers.

The merit of having a proposal (e.g., Sc and Sp) is to give the old master a chance to form the cluster if re-mastering fails. Once, the local registry file140aof the old master (e.g., node110a) clears the proposal (Sc/Sp), the old master (e.g., node110a) can no longer become a master. Beyond this point only one node (e.g., the new master node110b) can form the cluster. The old master would renounce the mastership by setting the current sequence number to the proposed sequence number and setting the propose sequence number to a value of “0” on both the voting files150aand150b. In some embodiments, the clearing of the first local registry file and the update of both of the voting files are performed in a single transaction step to commit the proposed sequence number to be the new current sequence number.

Failure Scenarios During Re-Mastering

The following scenarios assume that the re-mastering failed causing both of the nodes (e.g., node110aand node110b) to go down and the two nodes start up at some point in time thereafter. The following scenarios also revolve around a strict assumption that a non-master node cannot have a proposal (Sc/Sp) in its local registry file140and would only have the proposed sequence number. All steps are causal and if one fails all future steps fail. The causality is vital and it leads to deterministic recovery. For simplicity of explanation, node110awill be referred to hereinafter as “node1”, node110bwill be referred to hereinafter as “node2”, the current sequence number410will be referred to hereinafter as “Sc”, the proposed sequence number420will be referred to hereinafter as “Sp”, and the value stored in one of the plurality of persistent storage location for managing the re-mastering process as a combination of the Sc and Sp will be referred to hereinafter as a “proposal”.

All of the Nodes Fail after Completing Step330but Before Completing Steps340and250:

The state of the two-node cluster is: a proposal is in the local registry file140aof node1and both voting files150on node1and node2. Node1is still the master node, but node1does not know that it is still the master node.

When node1is restarted, node1will wait for a disk timeout to make sure the other node (e.g., node2) has not yet formed the cluster. After the disk timeout period has expired, indicating that node2is not the master because if node2was the master, node2would have restarted the cluster within the disk timeout period and sent out heartbeats to node1. At this point, since node2did not form the cluster within the disk timeout period, node1will assume that node1is the still the master and node1will re-key and form the cluster. Re-keying is further disclosed below. If only one of the voting files150is accessible (e.g., voting file150aon node1is accessible and voting file150bis not available because node2is down but may be rebooting), node1should not form the cluster. Node1should instead wait to join the other node, or discover the other voting file150bbecause the other node (e.g., node2would reboot on failure) has not yet rebooted.

For example, there are four persistent storage locations tracking the mastership state management of the cluster (e.g., voting files150a, voting files150b, local registry file140a, and local registry file140b). Node1, upon restart, will need to have access to three of the four persistent storage locations to be able to make a quorum like determination to know for sure whether it is the master node or the other node is the master node. Since node1only has access to only one voting file150aand not the other voting file150bbecause node2may be down, node1cannot establish a quorum because node1has, at most, access to only two of the four persistent storage locations—thus node1does not have access to at least three of the four persistent storage location information to determine whether it is the master node or not the master node. This is to ensure that node2did not already form the cluster and continue processing, which is why node1needs access to both of the voting files (e.g., voting files150aand150b) to make the determination. Therefore, if node1only has access to its own voting file150aand not to voting file150bfrom node2, then node1will have to wait until node2is restarted such that node1may access voting file150bon node2to complete its determination of whether or not node1is the master node.

Node2reboots upon failure and determines that it does not have a proposal in its registry file140b. Node2would then wait for node1to form the cluster and then join the cluster because node2would know that it is the non-master node. Additionally, node2must not send heartbeats to node1so that node1may go ahead and form the cluster after the disk timeout period has expired. A heartbeat sent from node2to node1would result in both the nodes waiting for each other because node1, as discussed above in Situation A, upon receiving a heartbeat from node2, node1would assume that node2is the master node and thus node1would wait for node2to restart the cluster so node1may join it. After joining the cluster formed by node1, the local storages120would sync up and the cluster reconfiguration may be complete. If node2comes up and node1never comes up, node2should not be allowed to form the cluster since node2is the non-master node. A manual human intervention may be required to restart the two-node cluster.

All of the Nodes Fail after Completing Step340but Before Step250:

The state of the two-node cluster is: a proposal is in the local registry file140aof node1and both voting files150on node1and node2. A new sequence number (e.g., the proposed sequence number Sp) is stored in the local registry file140b. Node2is the new master.

With respect to node1restarting, the same Situation A above applies.

There are two possible scenarios: (I) node2starts up and heartbeats within the disk timeout of node1(e.g., from when node1starts counting its disk timeout period). In this case, node2will form the cluster and node1will join the cluster formed by node2. (II) If node2never starts up or node2starts up after the disk timeout period expired for node1, two outcomes may apply: (a) if node2starts after node1has re-keyed, which means node1will continue to be the master and restarts the cluster, this is okay since node2does not have a majority match of its new sequence number of Sp in its local registry file140bsince the other three persistent storage location does not have the new sequence number of Sp (e.g., the local registry file140aof node1and both voting files150on node1and node2have the proposal stored (e.g., Sc and Sp, not just Sp). (b) if node1has not yet re-keyed, node2would be given the go ahead to form the cluster because node1would have detected the heartbeats from node2, clear its local registry file140aand restart as a non-master.

For example, this failure scenario may be explained with reference toFIG. 4E, where the proposed sequence Sp is already written into the local registry file140bof node2. During this period where the cluster is down and the two nodes are restarting, node1may think that it is the master because inspecting its local registry file140aand voting file150a, node1will see both a proposed sequence number Sp and a current sequence number Sc stored at both persistent storage locations. However, node1will wait until a timeout period expires to allow node2the opportunity to restart and perhaps form the cluster, if node2is the master node. Assuming node2starts up and determines that node2is the master node (e.g., by inspecting its local registry file and seeing only a proposed sequence number Sp), node2will then go ahead and restart the cluster and heartbeat to node1, at which point node1will simply join the cluster. However, if after the timeout period expires and node2has not yet restarted, then node1will attempt to restart the cluster and start the re-key process to make sure that when node2eventually restarts, node2will not try to restart the cluster since the data stored in its local registry file140bwill be blank and the information stored in its local voting file150bwill only contain a current sequence number Sc (e.g., seeFIG. 4Awhich shows the state of the system immediately after a rekey by node1(e.g., node110a).

All of the Nodes Fail after Completing Step250:

The state of the two-node cluster is: the local registry file140aof node1is cleared, the Sp is stored as the new current sequence number Sc in the local registry file140bof node2and Sp is stored as 0/null in both voting files150on node1and node2.

Node1is the non-master node and node2is the new master node at this point. Only node2can form the cluster.

Recovery and Re-Keying:

Re-keying is resetting the state management process once a master node has been definitively defined so the state management process may be re-initialized for the next re-mastering request. All failure scenarios disclosed above will leave behind an unfinished proposal to be addressed upon cluster restart. This is termed as recovery. Recovery includes choosing a master as disclosed in this disclosure and re-keying the voting files150aand150band the master node's local registry file140to confirm the proposed sequence number as the new current sequence number Sc. After the master node is determined, the master node will process the unfinished proposal as follows: (a) if the node with the proposal (Sc/Sp) in its local registry file140becomes the master (e.g., if node1is originally the master and during the re-mastering the cluster failed, then the local registry file140aof node1is the only node with Sc/Sp in its local registry), it would rekey its local registry file140and both voting files150with a newly generated sequence number (i.e., not Sc nor Sp); (b) if the node with only Sp in its local registry file140becomes the new master (e.g., the non-master node would be the only node with only Sp in its local registry file140—see Step340andFIG. 4E), then that node would commit Sp in at least its local voting file150and then attempt to commit Sp on both voting files150; (c) the node would clear its respective local registry file140if it does not match the committed sequence number Sp (e.g., the node is the non-master node, wherein the non-master node has an empty local registry file140when the two-node cluster is in a normal operational state, not during a re-mastering process).

What has been disclosed is an approach for computing a quorum for a two-node non-shared converged storage architecture. This approach therefore enhances the efficiencies of computing a quorum in a two-node non-shared converged storage architecture without the need of an additional quorum device, structure, and/or tiebreaker to compute a quorum for the two-node cluster. This approach greatly improves the technical area of two-node cluster management via a distributed state machine of tracking the state of the cluster by writing to a plurality of persistent storage location in a particular way and/or a particular order.

System Architecture Overview

FIG. 5is a block diagram of an illustrative computing system1000suitable for implementing an embodiment of the present disclosure. Computer system1000includes a bus1006or other communication mechanism for communicating information, which interconnects subsystems and devices, such as processor1007, system memory1008(e.g., RAM), static storage device1009(e.g., ROM), disk drive1010(e.g., magnetic or optical), communication interface1014(e.g., modem or Ethernet card), display1011(e.g., CRT or LCD), input device1012(e.g., keyboard), data interface1033, and cursor control.

According to some embodiments of the disclosure, computer system1000performs specific operations by processor1007executing one or more sequences of one or more instructions contained in system memory1008. Such instructions may be read into system memory1008from another computer readable/usable medium, such as static storage device1009or disk drive1010. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the disclosure. Thus, embodiments of the disclosure are not limited to any specific combination of hardware circuitry and/or software. In some embodiments, the term “logic” shall mean any combination of software or hardware that is used to implement all or part of the disclosure.

The term “computer readable medium” or “computer usable medium” as used herein refers to any medium that participates in providing instructions to processor1007for execution. Such a medium may take many forms, including but not limited to, non-volatile media and volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as disk drive1010. Volatile media includes dynamic memory, such as system memory1008.

In an embodiment of the disclosure, execution of the sequences of instructions to practice the disclosure is performed by a single computer system1000. According to other embodiments of the disclosure, two or more computer systems1000coupled by communication link1010(e.g., LAN, PTSN, or wireless network) may perform the sequence of instructions required to practice the disclosure in coordination with one another.

Computer system1000may transmit and receive messages, data, and instructions, including program, e.g., application code, through communication link1015and communication interface1014. Received program code may be executed by processor1007as it is received, and/or stored in disk drive1010, or other non-volatile storage for later execution. A database1032in a storage medium1031may be used to store data accessible by the system1000via data interface1033.

FIG. 6is a simplified block diagram of one or more components of a system environment800by which services provided by one or more components of an embodiment system may be offered as cloud services, in accordance with an embodiment of the present disclosure. In the illustrated embodiment, system environment800includes one or more client computing devices804,806, and808that may be used by users to interact with a cloud infrastructure system802that provides cloud services. The client computing devices may be configured to operate a client application such as a web browser, a proprietary client application, or some other application, which may be used by a user of the client computing device to interact with cloud infrastructure system802to use services provided by cloud infrastructure system802.

It should be appreciated that cloud infrastructure system802depicted in the figure may have other components than those depicted. Further, the embodiment shown in the figure is only one example of a cloud infrastructure system that may incorporate an embodiment of the disclosure. In some other embodiments, cloud infrastructure system802may have more or fewer components than shown in the figure, may combine two or more components, or may have a different configuration or arrangement of components. Client computing devices804,806, and808may be devices similar to those described above forFIG. 5. Although system environment800is shown with three client computing devices, any number of client computing devices may be supported. Other devices such as devices with sensors, etc. may interact with cloud infrastructure system802.

Network(s)810may facilitate communications and exchange of data between clients804,806, and808and cloud infrastructure system802. Each network may be any type of network familiar to those skilled in the art that can support data communications using any of a variety of commercially-available protocols. Cloud infrastructure system802may comprise one or more computers and/or servers.

In certain embodiments, cloud infrastructure system802may include a suite of applications, middleware, and database service offerings that are delivered to a customer in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner.

In various embodiments, cloud infrastructure system802may be adapted to automatically provision, manage and track a customer's subscription to services offered by cloud infrastructure system802. Cloud infrastructure system802may provide the cloud services via different deployment models. For example, services may be provided under a public cloud model in which cloud infrastructure system802is owned by an organization selling cloud services and the services are made available to the general public or different industry enterprises. As another example, services may be provided under a private cloud model in which cloud infrastructure system802is operated solely for a single organization and may provide services for one or more entities within the organization. The cloud services may also be provided under a community cloud model in which cloud infrastructure system802and the services provided by cloud infrastructure system802are shared by several organizations in a related community. The cloud services may also be provided under a hybrid cloud model, which is a combination of two or more different models.

In some embodiments, platform services may be provided by the cloud infrastructure system via a PaaS platform. The PaaS platform may be configured to provide cloud services that fall under the PaaS category. Examples of platform services may include without limitation services that allow organizations to consolidate existing applications on a shared, common architecture, as well as the ability to build new applications that leverage the shared services provided by the platform. The PaaS platform may manage and control the underlying software and infrastructure for providing the PaaS services. Customers can acquire the PaaS services provided by the cloud infrastructure system without the need for customers to purchase separate licenses and support.

By utilizing the services provided by the PaaS platform, customers can employ programming languages and tools supported by the cloud infrastructure system and also control the deployed services. In some embodiments, platform services provided by the cloud infrastructure system may include database cloud services, middleware cloud services, and Java cloud services. In one embodiment, database cloud services may support shared service deployment models that allow organizations to pool database resources and offer customers a Database as a Service in the form of a database cloud. Middleware cloud services may provide a platform for customers to develop and deploy various business applications, and Java cloud services may provide a platform for customers to deploy Java applications, in the cloud infrastructure system.

In certain embodiments, cloud infrastructure system802may also include infrastructure resources830for providing the resources used to provide various services to customers of the cloud infrastructure system. In one embodiment, infrastructure resources830may include pre-integrated and optimized combinations of hardware, such as servers, storage, and networking resources to execute the services provided by the PaaS platform and the SaaS platform.

In some embodiments, resources in cloud infrastructure system802may be shared by multiple users and dynamically re-allocated per demand. Additionally, resources may be allocated to users in different time zones. For example, cloud infrastructure system830may allow a first set of users in a first time zone to utilize resources of the cloud infrastructure system for a specified number of hours and then allow the re-allocation of the same resources to another set of users located in a different time zone, thereby maximizing the utilization of resources.

In certain embodiments, a number of internal shared services832may be provided that are shared by different components or modules of cloud infrastructure system802and by the services provided by cloud infrastructure system802. These internal shared services may include, without limitation, a security and identity service, an integration service, an enterprise repository service, an enterprise manager service, a virus scanning and white list service, a high availability, backup and recovery service, service for enabling cloud support, an email service, a notification service, a file transfer service, and the like.

In certain embodiments, cloud infrastructure system802may provide comprehensive management of cloud services (e.g., SaaS, PaaS, and IaaS services) in the cloud infrastructure system. In one embodiment, cloud management functionality may include capabilities for provisioning, managing and tracking a customer's subscription received by cloud infrastructure system802, and the like.

In one embodiment, as depicted in the figure, cloud management functionality may be provided by one or more modules, such as an order management module820, an order orchestration module822, an order provisioning module824, an order management and monitoring module826, and an identity management module828. These modules may include or be provided using one or more computers and/or servers, which may be general purpose computers, specialized server computers, server farms, server clusters, or any other appropriate arrangement and/or combination.

In operation834, a customer using a client device, such as client device804,806or808, may interact with cloud infrastructure system802by requesting one or more services provided by cloud infrastructure system802and placing an order for a subscription for one or more services offered by cloud infrastructure system802. In certain embodiments, the customer may access a cloud User Interface (UI), cloud UI812, cloud UI814and/or cloud UI816and place a subscription order via these UIs. The order information received by cloud infrastructure system802in response to the customer placing an order may include information identifying the customer and one or more services offered by the cloud infrastructure system802that the customer intends to subscribe to.

After an order has been placed by the customer, the order information is received via the cloud UIs,812,814and/or816. At operation836, the order is stored in order database818. Order database818can be one of several databases operated by cloud infrastructure system818and operated in conjunction with other system elements. At operation838, the order information is forwarded to an order management module820. In some instances, order management module820may be configured to perform billing and accounting functions related to the order, such as verifying the order, and upon verification, booking the order. At operation840, information regarding the order is communicated to an order orchestration module822. Order orchestration module822may utilize the order information to orchestrate the provisioning of services and resources for the order placed by the customer. In some instances, order orchestration module822may orchestrate the provisioning of resources to support the subscribed services using the services of order provisioning module824.

In certain embodiments, order orchestration module822allows the management of business processes associated with each order and applies business logic to determine whether an order should proceed to provisioning. At operation842, upon receiving an order for a new subscription, order orchestration module822sends a request to order provisioning module824to allocate resources and configure those resources needed to fulfill the subscription order. Order provisioning module824allows the allocation of resources for the services ordered by the customer. Order provisioning module824provides a level of abstraction between the cloud services provided by cloud infrastructure system802and the physical implementation layer that is used to provision the resources for providing the requested services. Order orchestration module822may thus be isolated from implementation details, such as whether or not services and resources are actually provisioned on the fly or pre-provisioned and only allocated/assigned upon request.

At operation844, once the services and resources are provisioned, a notification of the provided service may be sent to customers on client devices804,806and/or808by order provisioning module824of cloud infrastructure system802.

At operation846, the customer's subscription order may be managed and tracked by an order management and monitoring module826. In some instances, order management and monitoring module826may be configured to collect usage statistics for the services in the subscription order, such as the amount of storage used, the amount data transferred, the number of users, and the amount of system up time and system down time.

In certain embodiments, cloud infrastructure system802may include an identity management module828. Identity management module828may be configured to provide identity services, such as access management and authorization services in cloud infrastructure system802. In some embodiments, identity management module828may control information about customers who wish to utilize the services provided by cloud infrastructure system802. Such information can include information that authenticates the identities of such customers and information that describes which actions those customers are authorized to perform relative to various system resources (e.g., files, directories, applications, communication ports, memory segments, etc.) Identity management module828may also include the management of descriptive information about each customer and about how and by whom that descriptive information can be accessed and modified.

In the foregoing specification, the invention has been described with reference to specific embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention. For example, the above-described process flows are described with reference to a particular ordering of process actions. However, the ordering of many of the described process actions may be changed without affecting the scope or operation of the invention. The specification and drawings are, accordingly, to be regarded in an illustrative rather than restrictive sense. In addition, an illustrated embodiment need not have all the aspects or advantages shown. An aspect or an advantage described in conjunction with a particular embodiment is not necessarily limited to that embodiment and can be practiced in any other embodiments even if not so illustrated. Also, reference throughout this specification to “some embodiments” or “other embodiments” means that a particular feature, structure, material, or characteristic described in connection with the embodiments is included in at least one embodiment. Thus, the appearances of the phrase “in some embodiment” or “in other embodiments” in various places throughout this specification are not necessarily referring to the same embodiment or embodiments.