Scalable storage cluster mirroring

Scalable storage cluster mirroring is disclosed. A compute instance comprising a processor device determines that storage segments have been modified on a first storage node of a plurality of storage nodes in a first cluster of storage nodes at a first data modification rate. In response to determining that the storage segments have been modified on the first storage node at the first data modification rate, a first mirror process that is configured to copy storage segments from an identified storage node to a mirrored cluster of storage nodes is initiated, and a storage node identifier that identifies the first storage node is communicated to the first mirror process.

BACKGROUND

A storage cluster is made up of a number of storage nodes on which data is stored. A storage node is an electronic device that may unexpectedly fail, or lose power. In the event of a power failure, to ensure that services that rely on the data are not disrupted for a long period of time, it may be desirable to mirror (i.e., copy) the data in one storage cluster to another storage cluster, often substantially in real-time, so that each storage cluster contains an identical copy of the data. Mirroring data from one cluster to another may also be utilized for purposes of data migration.

SUMMARY

The examples disclosed herein implement scalable storage cluster mirroring. The examples utilize tunable, on-demand and scalable mirror processes that are initiated, or terminated, as needed, based on a real-time data modification rate that is occurring on the storage nodes of a storage cluster and a real-time data mirror rate of the mirror processes.

In one example a method is provided. The method includes determining, by a compute instance comprising a processor device, that storage segments have been modified on a first storage node of a plurality of storage nodes in a first cluster of storage nodes at a first data modification rate. The method further includes, in response to determining that the storage segments have been modified on the first storage node at the first data modification rate, initiating a first mirror process that is configured to copy storage segments from an identified storage node to a mirrored cluster of storage nodes. The method further includes communicating, to the first mirror process, a storage node identifier that identifies the first storage node.

In another example a compute instance is provided. The compute instance includes a memory and a processor device coupled to the memory. The processor device is to determine that storage segments have been modified on a first storage node of a plurality of storage nodes in a first cluster of storage nodes at a first data modification rate. The processor device is further to, in response to determining that the storage segments have been modified on the first storage node at the first data modification rate, initiate a first mirror process that is configured to copy storage segments from an identified storage node to a mirrored cluster of storage nodes. The processor device is further to communicate, to the first mirror process, a storage node identifier that identifies the first storage node.

In another example a computer program product is provided. The computer program product is stored on a non-transitory computer-readable storage medium and includes instructions to cause a processor device to determine that storage segments have been modified on a first storage node of a plurality of storage nodes in a first cluster of storage nodes at a first data modification rate. The instructions further cause the processor device to, in response to determining that the storage segments have been modified on the first storage node at the first data modification rate, initiate a first mirror process that is configured to copy storage segments from an identified storage node to a mirrored cluster of storage nodes. The instructions further cause the processor device to communicate, to the first mirror process, a storage node identifier that identifies the first storage node.

DETAILED DESCRIPTION

A storage cluster is made up of a number of storage nodes on which data is stored. A storage node is an electronic device which may unexpectedly fail. To ensure that the data on a failed storage node is not lost, and to ensure that services that rely on the data are not disrupted for a long period of time, it may be desirable to mirror (i.e., copy) the data in one storage cluster to another storage cluster, often substantially in real-time, so that each storage cluster contains an identical copy of the data.

Typically, the mirroring process, sometimes referred to as a form of replication, involves two storage clusters, each of which is made up of a number of storage nodes. In a one-way mirroring environment, as data is modified on an original storage cluster, the data is copied to a mirrored storage cluster. As used herein, the term “modified” in the context of storage refers to any modification of data, including newly stored data, changed data, or deleted data. Typically a finite number of mirror tasks are initiated that await notification of modifications to the original storage cluster, and when notified, copy any modified storage segments from the original storage cluster to the mirrored storage cluster. The term “storage segment” as used herein refers to any uniquely identified unit of storage. Depending on the particular storage system at issue, a storage segment may be a fixed size, such as, by way of non-limiting example, 64 megabytes (MB), or may be a variable size. A storage segment may be, for example, an object in an object-based storage system, or a predetermined fixed-size portion of an object. A storage segment may be, for example, a file in a file-based storage system, or a predetermined fixed-size portion of a file.

There may be long periods of time, such as outside of normal work hours, during which there are few or no data modifications made to the storage cluster. Yet, the mirror tasks continue to utilize scarce resources, such as memory and CPU resources, even when not actively copying storage segments. Moreover, during high-activity periods of time when storage segments may be modified at an unexpectedly high rate, the mirror tasks may not be able to copy the data to the mirror site at the desired rate, which may lead to data vulnerabilities should a storage node unexpectedly fail, or may fail to meet a service level agreement requiring that data be mirrored at some guaranteed rate.

Enterprises, such as businesses, are increasingly utilizing cloud computing environments to implement desired data processing services. A cloud computing environment provider often charges fees based on the amount of resources being utilized. Mirror tasks that are utilizing memory and CPU resources during periods of time when little or no data modifications are being made to a storage cluster incur costs for no benefit.

The examples disclosed herein implement scalable storage cluster mirroring. The examples utilize tunable, on-demand and scalable mirror processes that are initiated, or terminated, as needed, based on a real-time data modification rate that is occurring on the storage nodes of a storage cluster and a real-time data mirror rate of the mirror processes. For example, if a data modification rate is exceeding the data mirror rate, additional mirror processes may be initiated. If the data mirror rate is exceeding the data modification rate, some mirror processes may be terminated. If no data modifications are occurring, all mirror processes may be terminated. In some examples, the number of mirror processes may also be influenced by a guaranteed data mirror rate associated with an enterprise and set forth, for example, in a service level agreement between a cloud computing environment provider and the enterprise. Among other advantages, the examples disclosed herein facilitate finer grained resource usage, such as, by way of non-limiting example, updating the throttling limit of data transmission.

FIG.1is a block diagram of an environment10in which examples can be practiced. The environment10includes a cloud computing environment12. The phrase “cloud computing environment” as used herein refers to a computing environment, often provided by a cloud computing environment provider, which facilitates dynamic and scalable computing resources upon request. The examples have applicability in any public, private, and/or hybrid cloud computing environment, including, by way of non-limiting example, cloud computing environments implemented by Red Hat® OpenStack, Oracle® Cloud, Amazon® Web Services, Google® Cloud, Microsoft® Azure, IBM® cloud services, Salesforce.com®, and the like. It should be noted that while for purposes of illustration the examples are disclosed in the context of a cloud computing environment, the examples disclosed herein are not limited to a cloud computing environment and have applicability to any environment in which a storage cluster is mirrored.

The cloud computing environment12implements processing nodes referred to herein as compute instances (CI). A compute instance, as discussed herein, refers to a discrete runtime environment, and may comprise a physical (sometimes referred to as bare metal) computing device configured to run an operating system, or may comprise a virtual machine that emulates a physical machine. A virtual machine typically runs a guest operating system in conjunction with a virtual machine monitor, such as a hypervisor, that is configured to coordinate access to physical resources of a physical computing device, such as a memory and a processor device, by the virtual machines running on the physical computing device. A compute instance thus, whether a physical computing device or a virtual machine, includes a memory and a processor device.

A compute instance14of the cloud computing environment12includes a serverless function interface16that can be called or otherwise invoked to initiate a serverless function. As used herein, the term “serverless function” refers to computing processes that can be initiated by the cloud computing environment12upon request or upon the occurrence of certain criteria, that have no state, and that can be scaled up or down upon demand. The implementation of a serverless function may differ among different cloud computing environments; however, as an example, in an Amazon Web Services (AWS) cloud computing environment, serverless functions are implemented via the AWS Lambda serverless computing platform. In this example, the compute instance14includes, or is communicatively coupled to, a storage device18on which a mirror process code20is stored. The mirror process code20comprises executable instructions that implement the functionality of a mirror process, as discussed in greater detail below.

The cloud computing environment12includes a storage cluster22(hereinafter “cluster22”) that in turn includes a plurality of storage nodes24-1-24-N. The storage nodes24-1-24-N include processor devices26-1-26-N and memories28-1-28-N, respectively. The storage nodes24-1-24-N also include storage devices30-1-30-N, respectively. The storage device30-1comprises a plurality of storage segments31-1-31-N. The storage devices30-2-30-N similarly comprise a plurality of storage segments (not illustrated). The cluster22includes a compute instance32, which in turn includes a processor device34and a memory36. A cluster manager38executes in the memory36. Among other tasks, the cluster manager38communicates with agents40-1-40-N which execute on the storage nodes24-1-24-N, respectively. The environment10includes a storage cluster42, which includes a plurality of storage nodes44-1-44-N, and to which the storage segments on the storage devices30-1-30-N are mirrored, as discussed in greater detail below. The storage cluster42also includes a compute instance46on which a cluster manager48executes.

A plurality of applications50-1-50-N (generally, applications50) executing in the cloud computing environment12store data on the cluster22. It is noted that the compute instances on which the applications50-1-50-N execute have been omitted due to space considerations. As the applications50modify storage segments of the storage nodes24-1-24-N, each agent40-1-40-N maintains a list of modified storage segments (LOMSS)52-1-52-N, respectively, that identify the particular storage segments that have been modified, and thus that need to be mirrored to the storage cluster42. As the applications50modify storage segments of the storage nodes24-1-24-N, each agent40-1-40-N also determines a data modification rate (DMR)54-1-54-N that identifies a rate at which the storage segments on the respective storage nodes24-1-24-N are being modified. The data modification rate54may be in any desirable units over any desirable increment of time. As an example, in a system wherein storage segments are a fixed size, the data modification rate54may comprise the number of storage segments modified over some interval, such as 100 milliseconds, 1 second, 5 seconds, or the like. In a system wherein storage segments are variable size, the data modification rate54may comprise the number of bytes of data modified over some interval, such as 100 milliseconds, 1 second, 5 seconds, or the like.

For purposes of illustration, assume that the applications50-1-50-N make modifications to the storage nodes24over a period of time. The agent40-1determines the data modification rate54-1that identifies the rate at which storage segments31are being modified on the storage node24-1, and sends the data modification rate54-1to the cluster manager38. Based on the data modification rate54-1, the cluster manager38invokes the serverless function interface16and thereby causes the initiation of a mirror process56on a compute instance58. The cluster manager38communicates information to the mirror process56, including, by way of non-limiting example, a storage node identifier60that identifies the storage node24-1, authentication credentials62that are required by the storage node24-1to gain access to the storage segments31, and an address64of the storage cluster42to which the modified storage segments31are to be copied. In some implementations, the cluster manager38may request, from the storage node24-1, the list of modified storage segments52-1, and communicate the list of modified storage segments52-1to the mirror process56. In other implementations, after authenticating with the storage node24-1, the mirror process56may obtain the list of modified storage segments52-1from the storage node24-1. The communications between the cluster manager38and the mirror process56may take any desired form or forms, such as via parameters, the use of an application programming interface, the use of global environment variables, or any other mechanism by which the serverless function interface16facilitates communications between a serverless function and another process.

The mirror process56uses the authentication credentials62to authenticate with the storage node24-1. The mirror process56may also obtain, from the storage node24-1, the list of modified storage segments52-1. The mirror process56begins obtaining copies of the modified storage segments31and sending the modified storage segments31to the storage cluster42utilizing the address64. The mirror process56maintains a data mirror rate (DMIR)66that identifies a real-time rate at which the mirror process56sends the modified storage segments31to the storage cluster42. The mirror process56sends the data mirror rate66to the cluster manager38.

The cluster manager38receives the data modification rate54-1iteratively from the agent40-1, and receives the data mirror rate66iteratively from the mirror process56. Based on the data modification rate54-1and the data mirror rate66, the cluster manager38determines whether the mirror process56is copying the modified storage segments31to the storage cluster42at a faster rate than the storage segments31are being modified on the storage node24-1, is copying the modified storage segments31to the cluster42at a slower rate than the storage segments31are being modified on the storage node24-1, or is copying the modified storage segments31to the cluster42at a rate substantially equivalent to the rate at which the storage segments31are being modified on the storage node24-1.

In some implementations, the cluster manager38may access service level agreement (SLA) information68, which may identify a guaranteed data mirror rate for the enterprise (i.e., tenant), associated with the storage segments31. For example, the SLA information68may require that every modified storage segment31associated with this tenant be copied to the storage cluster42within 2 minutes of such modification. Based on the data modification rate54-1and the data mirror rate66, the cluster manager38can determine whether the mirror process56is meeting the guaranteed data mirror rate.

The mirror process56may continuously be provided additional storage segment identifiers that identify storage segments31that are modified. In some implementations, the cluster manager38may periodically request the list of modified storage segments52-1, identify newly modified storage segments31that were not previously identified to the mirror process56, and communicate information to the mirror process56that identifies the newly modified storage segments. In another implementation, the mirror process56may iteratively communicate with the agent40-1to obtain information that identifies newly modified storage segments31.

The agent40-2determines a data modification rate54-2that identifies the rate at which storage segments are being modified on the storage node24-2, and sends the data modification rate54-2to the cluster manager38. For purposes of illustration, assume that the data modification rate54-2is a greater rate than the data modification rate54-1. Based on the data modification rate54-2and the SLA information68, the cluster manager38determines that two mirror processes should be initiated, and invokes the serverless function interface16to cause the initiation of a mirror process70on a compute instance72, and the initiation of a mirror process74on a compute instance76. The cluster manager38communicates information to the mirror processes70,74including, by way of non-limiting example, a storage node identifier78that identifies the storage node24-2, authentication credentials62that are required by the storage node24-2to gain access to the storage segments of the storage node24-2, and an address64of the storage cluster42to which the modified storage segments are to be copied.

The cluster manager38requests, from the storage node24-2, the list of modified storage segments52-2. The cluster manager38selects a first subset of storage segments identified on the list of modified storage segments52-2and populates a list80with the first subset of storage segments. The cluster manager38selects a second subset of storage segments identified on the list of modified storage segments52-2and populates a list82with the second subset of storage segments. Each of the storage segments identified on the list of modified storage segments52-2is identified in either the list80or the list82. The cluster manager38communicates the list80to the mirror process70and communicates the list82to the mirror process74.

The mirror process70authenticates with the storage node24-2using the authentication credentials62, and begins copying the storage segments identified on the list80to the storage cluster42. Similarly, the mirror process74authenticates with the storage node24-2using the authentication credentials62, and begins copying the storage segments identified on the list82to the storage cluster42.

The mirror process70periodically generates a data mirror rate84that identifies a real-time rate at which the mirror process70sends the modified storage segments of the storage node24-2to the storage cluster42. The mirror process74also periodically generates a data mirror rate86that identifies a real-time rate at which the mirror process74sends the modified storage segments of the storage node24-2to the storage cluster42of storage nodes. The agent40-2periodically generates the data modification rate54-2that identifies the rate at which the storage segments on the storage node24-2are being modified, and sends the data modification rates54-2to the cluster manager38.

The cluster manager38receives the data mirror rates84, the data mirror rates86, and the data modification rates54-2. The cluster manager38determines that the rate at which the modified storage segments are being copied to the storage cluster42given the rate at which storage segments are being modified meets the guaranteed data mirror rate identified in the SLA information68and thus, that an additional mirror process need not be initiated.

At some point in time, the data modification rate54-2indicates that storage segments are being modified on the storage node24-2at a very slow rate. This could occur, for example, after normal work hours. The cluster manager38determines that, given the data modification rate54-2, a single mirror process can copy modified storage segments on the storage node24-2at a data mirror rate sufficient to meet the guaranteed data mirror rate identified in the SLA information68. The cluster manager38terminates the mirror process74, thus freeing up scarce memory and CPU resources that are otherwise used by the mirror process74, and eliminating any costs that would otherwise be incurred by the mirror process74.

It is noted that, because each of the cluster manager38, the agent40-1, and the mirror process56are components of the respective computing device on which they are executing, functionality implemented by any of the cluster manager38, the agent40-1, and the mirror process56may be attributed to the respective computing device on which they are executing. Moreover, in examples where the cluster manager38, the agent40-1, and the mirror process56comprise software instructions that program a processor device to carry out functionality discussed herein, functionality implemented by the cluster manager38, the agent40-1, and the mirror process56may be attributed herein to the respective processor device of the computing device on which they are executing.

It is also noted that, for simplicity and clarity, networks and communication paths have been omitted fromFIG.1; however, it will be understood that each of the components are communicatively coupled to one another via one or more networks.

FIG.2is a flowchart of a method for scalable storage cluster mirroring from the perspective of the cluster manager38according to one implementation.FIG.2will be discussed in conjunction withFIG.1. The compute instance32determines that the storage segments31have been modified on the storage node24-1of the plurality of storage nodes24-1-24-N in the cluster22of the storage nodes24-1-24-N at the data modification rate54-1(FIG.2, block1000). In response to determining that the storage segments31have been modified on the storage node24-1at the data modification rate54-1, the compute instance32initiates the mirror process56that is configured to copy storage segments from an identified storage node to a mirrored cluster of storage nodes (FIG.2, block1002). The compute instance32communicates, to the mirror process56, the storage node identifier60that identifies the storage node24-1(FIG.2, block1004).

FIG.3is a flowchart of a method for scalable storage cluster mirroring from the perspective of the agent40-1according to one implementation.FIG.3will be discussed in conjunction withFIG.1. The agent40-1determines the data modification rate54-1at which the storage segments31are being modified on the storage node24-1(FIG.3, block2000). The agent40-1sends the data modification rate54-1to the cluster manager38(FIG.3, block2002).

FIG.4is a flowchart of a method for scalable storage cluster mirroring from the perspective of the mirror process56according to one implementation.FIG.4will be discussed in conjunction withFIG.1. The mirror process56receives, from the cluster manager38, the storage node identifier60that identifies the storage node24-1(FIG.3, block3000). The mirror process56obtains, from the storage node24-1, the plurality of storage segments31that have been modified (FIG.3, block3002). The mirror process56sends, to the mirrored cluster of storage nodes44, the plurality of storage segments31(FIG.3, block3004).

FIG.5is a block diagram of the environment10illustrated inFIG.1according to another implementation. In this implementation, the environment10includes an additional storage cluster88that includes a plurality of storage nodes90-1-90-N, and to which the storage segments on the storage devices30-1-30-N are mirrored, as discussed in greater detail below. The storage cluster88also includes a compute instance92on which a cluster manager94executes. When the cluster manager38initiates the mirror processes56,70, and74, as discussed above with regard toFIG.1, the cluster manager94provides, to each of the mirror processes56,70, and74, the address64of the storage cluster42and also an address96of the storage cluster88. Each of the mirror processes56,70, and74then copy modified storage segments from the storage nodes24-1,24-2to both the storage cluster42and the storage cluster88.

FIG.6is a block diagram of the environment10illustrated inFIG.1according to another implementation. Note that only two storage nodes24-1-24-2are illustrated due to space limitations inFIG.2. In this implementation, the applications50-1-50-N are associated with two different divisions of the same company. For example, some of the applications50-1-50-N are associated with a mergers and acquisitions division (i.e., sometimes referred to herein as division1), and some of the applications50-1-50-N are associated with a human resources division (i.e., sometimes referred to herein as division2). Neither division is to be given access to data of the other division, and the enterprise has prioritized the mergers and acquisitions division over the human resources division for purposes of mirroring data segments to the storage cluster42.

As the applications50modify storage segments of the storage node24-1, the agent40-1maintains a list of modified storage segments98that identifies the modified storage segments31that are associated with division1, and a list of modified storage segments100that identifies the modified storage segments31that are associated with division2. The agent40-2maintains similar lists of modified storage segments for the two different divisions.

The agent40-1also determines a data modification rate102that identifies a rate at which the storage segments31associated with division1are being modified, and a data modification rate104that identifies a rate at which the storage segments31associated with division2are being modified. The agent40-1periodically sends the data modification rates102,104to the cluster manager38along with information that identifies the divisions to which the respective data modification rates102,104correspond.

The cluster manager38receives the data modification rates102,104and accesses the SLA information68. The SLA information68indicates that division1has a higher priority than division2. The prioritization may be expressed in any desired manner, including simply a ranking, or, in other implementations, by a higher guaranteed data mirror rate. Based on the data modification rate102and the SLA information68that indicates that division1has a higher priority than division2, the cluster manager38invokes the serverless function interface16to cause the initiation of a mirror process106on a compute instance108, and the initiation of a mirror process110on a compute instance112. The cluster manager38communicates information to the mirror processes106,110including, by way of non-limiting example, the storage node identifier60that identifies the storage node24-1, authentication credentials114that are required by the storage node24-1to gain access to the storage segments31of the storage node24-1that are associated with division1, and an address64of the storage cluster42to which the modified storage segments are to be copied.

The cluster manager38requests, from the storage node24-2, the list of modified storage segments98. The cluster manager38selects a first subset of storage segments identified on the list of modified storage segments98and populates a list116with the first subset of storage segments. The cluster manager38selects a second subset of storage segments identified on the list of modified storage segments98and populates a list118with the second subset of storage segments. Each of the storage segments identified on the list of modified storage segments98is identified in either the list116or the list118. The cluster manager38communicates the list116to the mirror process106and communicates the list118to the mirror process110.

Each of the mirror processes authenticates with the storage node24-1using the authentication credentials114, and begins copying the storage segments31identified on the lists116,118, respectively to the storage cluster42. The mirror processes106,110periodically generate data mirror rates120,122, respectively, that identify the real-time rates at which the mirror processes106,110send the modified storage segments31of the storage node24-1to the storage cluster42. The agent40-1also periodically determines the data modification rate102that identifies a rate at which the storage segments31on the respective storage node24-1that belong to division1are being modified. The cluster manager38receives the data mirror rates120,122, and the data modification rate102to determine whether the rate at which the modified storage segments31are being copied to the storage cluster42, given the rate at which storage segments31are being modified, meets the guaranteed data mirror rate associated with division1identified in the SLA information68.

Based on the data modification rate104and the SLA information68that indicates that division2has a lower priority than division1, the cluster manager38invokes the serverless function interface16to cause the initiation of a mirror process124on a compute instance126. The cluster manager38communicates information to the mirror process124, including, by way of non-limiting example, the storage node identifier60that identifies the storage node24-1, authentication credentials128that are required by the storage node24-1to gain access to the storage segments31of the storage node24-1that are associated with division2, and an address64of the storage cluster42to which the modified storage segments are to be copied.

The mirror process124authenticates with the storage node24-1using the authentication credentials128, and begins copying the storage segments31identified on the list of modified storage segments100to the storage cluster42. The mirror process124periodically generates a data mirror rate130that identifies a real-time rate at which the mirror process124sends the modified storage segments31of the storage node24-1to the storage cluster42. The agent40-1also periodically determines the data modification rate104that identifies a rate at which the storage segments31on the respective storage node24-1that belong to division2are being modified. The cluster manager38receives the data mirror rate130and the data modification rate104to determine whether the rate at which the modified storage segments31are being copied to the storage cluster42given the rate at which storage segments31are being modified meets the guaranteed data mirror rate associated with division2identified in the SLA information68.

FIG.7is a block diagram of the environment10illustrated inFIG.1according to another implementation. Note that only two storage nodes24-1-24-2are illustrated due to space limitations inFIG.2. In this implementation, the applications50-1-50-N are associated with two different enterprises (i.e., different businesses) that both utilize the same cloud computing provider that implements the cloud computing environment12. Different businesses are sometimes referred to as tenants. Thus, some of the applications50-1-50-N are associated with tenant1and some of the applications50-1-50-N are associated with tenant2. Applications50-1-50-N that are associated with tenant1cannot be permitted to access storage segments31that are associated with tenant2, and applications50-1-50-N that are associated with tenant2cannot be permitted to access storage segments31that are associated with tenant1.

As the applications50modify storage segments of the storage node24-1, the agent40-1maintains a list of modified storage segments132that identifies the modified storage segments31that are associated with tenant1, and a list of modified storage segments134that identifies the modified storage segments31that are associated with tenant2. The agent40-2maintains similar lists of modified storage segments for the two different tenants.

The agent40-1also determines a data modification rate136that identifies a rate at which the storage segments31associated with tenant1are being modified, and a data modification rate138that identifies a rate at which the storage segments31associated with tenant2are being modified. The agent40-1periodically sends the data modification rates136,138to the cluster manager38along with information that identifies the tenants to which the respective data modification rates136,138correspond.

The cluster manager38receives the data modification rate136and accesses SLA information140. The SLA information140indicates that tenant1has a particular guaranteed data mirror rate. Based on the data modification rate136and the SLA information140, the cluster manager38invokes the serverless function interface16to cause the initiation of a mirror process142on a compute instance144, and the initiation of a mirror process146on a compute instance148. The cluster manager38communicates information to the mirror processes142,146including, by way of non-limiting example, the storage node identifier60that identifies the storage node24-1, authentication credentials150that are required by the storage node24-1to gain access to the storage segments31of the storage node24-1that are associated with tenant1, and an address152of the storage cluster42to which the modified storage segments31associated with the tenant1are to be copied.

The cluster manager38requests, from the storage node24-1, the list of modified storage segments132. The cluster manager38selects a first subset of storage segments identified on the list of modified storage segments132and populates a list154with the first subset of storage segments. The cluster manager38selects a second subset of storage segments identified on the list of modified storage segments132and populates a list156with the second subset of storage segments. Each of the storage segments31identified on the list of modified storage segments132is identified in either the list154or the list156. The cluster manager38communicates the list154to the mirror process142and communicates the list156to the mirror process146.

Each of the mirror processes142,146authenticate with the storage node24-1using the authentication credentials150, and begin copying the storage segments31identified on the lists154,156, respectively to the storage cluster42. The mirror processes142,146periodically generate data mirror rates158,160, respectively, that identify the real-time rates at which the mirror processes142,146send the modified storage segments31of the storage node24-1to the storage cluster42. The agent40-1also periodically determines the data modification rate136that identifies a rate at which the storage segments31on the respective storage node24-1that belong to tenant1are being modified. The cluster manager38receives the data mirror rates158,160and the data modification rate136to determine whether the rate at which the modified storage segments31are being copied to the storage cluster42given the rate at which storage segments31are being modified meets the guaranteed data mirror rate associated with tenant1and identified in the SLA information140.

The cluster manager38receives the data modification rate138and accesses SLA information162. The SLA information162indicates that tenant2has a particular guaranteed data mirror rate that is lower than that of tenant1. Based on the data modification rate138and the SLA information162, the cluster manager38invokes the serverless function interface16to cause the initiation of a mirror process164on a compute instance166. The cluster manager38communicates information to the mirror process164including, by way of non-limiting example, the storage node identifier60that identifies the storage node24-1, authentication credentials168that are required by the storage node24-1to gain access to the storage segments31of the storage node24-1that are associated with tenant2, and an address170of a storage cluster172to which the modified storage segments31associated with the tenant2are to be copied.

The cluster manager38requests, from the storage node24-1, the list of modified storage segments134. The cluster manager38communicates the list of modified storage segments134to the mirror process164. The mirror process164authenticates with the storage node24-1using the authentication credentials168, and begins copying the storage segments31identified on the list of modified storage segments134to the storage cluster172. The mirror process164periodically generates a data mirror rate174that identifies a real-time rate at which the mirror process164sends the modified storage segments31of the storage node24-1to the storage cluster172. The agent40-1also periodically determines the data modification rate138that identifies a rate at which the storage segments31on the respective storage node24-1that belong to tenant2are being modified. The cluster manager38receives the data mirror rate174and the data modification rate138to determine whether the rate at which the modified storage segments31are being copied to the storage cluster172given the rate at which storage segments31are being modified meets the guaranteed data mirror rate associated with tenant2identified in the SLA information162.

FIGS.8A-8Care message sequence diagrams illustrating example messages communicated between components and actions taken by components during scalable storage cluster mirroring according to one implementation. Referring first toFIG.8A, the cluster manager38receives a data modification rate from the agent40-1that identifies a rate at which storage segments are being modified on the storage node24-1(step4000). The cluster manager38accesses the SLA information68and, based on the guaranteed data mirror rate and the data modification rate, determines that a single mirror process is initially needed to mirror modified data segments from the storage node24-1to the storage cluster42(steps4002-4004).

The cluster manager38requests the list of modified storage segments52-1from the agent40-1(step4006). The agent40-1sends the list of modified storage segments52-1to the cluster manager38(step4008). The cluster manager38initiates the mirror process56(steps4010-4012). The cluster manager38communicates the list of modified storage segments52-1, the storage node identifier60, the authentication credentials62and the mirror site address64to the mirror process56(step4014).

The mirror process56sends the authentication credentials62to the agent40-1(step4016). The agent40-1verifies that the credentials are correct, and sends an acknowledgement to the mirror process56(step4018). An iteration of a number of steps begins. The mirror process56sends a request for one or more storage segments identified on the list of modified storage segments52-1to the agent40-1(step4020). The agent40-1sends the requested storage segments to the mirror process56(step4022). Periodically, the agent40-1sends a data modification rate to the cluster manager38(step4024). The mirror process56sends the storage segments received from the agent40-1to the storage cluster42(step4026). Periodically, the mirror process56sends a data mirror rate to the cluster manager38(step4028). The cluster manager38analyzes the data modification rate, the data mirror rate, and the guaranteed mirror rate from the SLA information68to determine if the rate at which the mirror process56is sending the storage segments to the storage cluster42in view of the rate at which storage segments are being modified comply with the guaranteed mirror rate (step4030). The steps4018-4030may repeat any number of times.

In this example, at some point in time, the cluster manager38determines that the rate at which the mirror process56is sending the storage segments to the storage cluster42in view of the rate at which storage segments are being modified do not (or soon will not) comply with the guaranteed mirror rate (step4032).

Referring now toFIG.8B, the cluster manager38requests the list of modified storage segments52-1from the agent40-1(step4034). The agent40-1sends the list of modified storage segments52-1to the cluster manager38(step4036). The cluster manager38initiates the mirror process70(steps4038-4040). The cluster manager38may enter into one or more communications with the mirror process56to synchronize the storage segments that the mirror process56has already sent to the storage cluster42and the storage segments identified on the list of modified storage segments52-1(step4042). This process may involve an instruction to the mirror process56to temporarily halt sending storage segments to the storage cluster42. Once the cluster manager38determines what storage segments on the list of modified storage segments52-1remain to be sent to the storage cluster42, the cluster manager38selects a first subset of storage segments identified on the list of modified storage segments52-1and populates a first list with the first subset of storage segments. The cluster manager38selects a second subset of storage segments identified on the list of modified storage segments52-1and populates a second list with the second subset of storage segments (step4044).

The cluster manager38sends the first list to the mirror process56and instructs the mirror process56to resume processing (step4046). The cluster manager38communicates the second list of modified storage segments, the storage node identifier60, the authentication credentials62and the mirror site address64to the mirror process70(step4048). The mirror process70sends the authentication credentials62to the agent40-1(step4050). The agent40-1verifies that the credentials are correct, and sends an acknowledgement to the mirror process70(step4052). An iteration of a number of steps begins. The mirror process70sends a request for one or more storage segments identified on the second list of modified storage segments to the agent40-1(step4054). The agent40-1sends the requested storage segments to the mirror process70(step4056). The mirror process70sends the storage segments received from the agent40-1to the storage cluster42(step4058). Concurrently with these steps, the mirror process56sends a request for one or more storage segments identified on the first list of modified storage segments to the agent40-1(step4060). The agent40-1sends the requested storage segments to the mirror process56(step4062). The mirror process56sends the storage segments received from the agent40-1to the storage cluster42(step4064). Periodically, the agent40-1sends a data modification rate to the cluster manager38(step4066). Periodically, the mirror process70sends a data mirror rate to the cluster manager38(step4068). Periodically, the mirror process56sends a data mirror rate to the cluster manager38(step4070).

Referring now toFIG.8C, the cluster manager38analyzes the data modification rates received from the mirror processes56,70, the data mirror rate, and the guaranteed mirror rate from the SLA information68to determine if the rate at which the mirror processes56,70are sending the storage segments to the storage cluster42in view of the rate at which storage segments are being modified comply with the guaranteed mirror rate (step4072). The steps4054-4072may repeat any number of times.

In this example, at some point in time, the cluster manager38determines that the rate at which storage segments are being modified has decreased to a level that the data mirror rate of the mirror process56is sufficent to comply with the guaranteed mirror rate (step4074). The cluster manager38sends an instruction to the mirror process70to terminate (step4076). The mirror process70terminates, reducing memory and CPU utilization (step4078). The mirror process56continues to iteratively request additional modified storage segments from the agent40-1and to send such modified storage segments to the storage cluster42(steps4080-4084). Periodically, the agent40-1sends a data modification rate to the cluster manager38(step4086). Periodically, the mirror process56sends a data mirror rate to the cluster manager38(step4088). This process can continue indefinitely until the cluster manager38again determines either to terminate the mirror process56, or to initiate another mirror process.

FIG.9is a block diagram of a compute instance32-1suitable for implementing scalable storage cluster mirroring according to one implementation. The compute instance32includes a storage segment modification determiner176that is configured to determine that a storage segment has been modified. In some implementations, this may involve the receipt of information, such as a data modification rate, or any other suitable information, from an agent executing on a storage node. The storage segment modification determiner176may comprise executable software instructions configured to program a processor device to implement the functionality of determining that a storage segment has been modified, may comprise circuitry including, by way of non-limiting example, an application-specific integrated circuit (ASIC) or field-programmable gate array (FPGA), or may comprise a combination of executable software instructions and circuitry.

The compute instance32-1also includes a mirror process initiator178that is configured to initiate a mirror process upon request. In some implementations, this may involve interfacing with a serverless function interface, such as the serverless function interface16, of a cloud computing environment. The mirror process initiator178may comprise executable software instructions configured to cause a serverless function interface to initiate a mirror process, may comprise circuitry including, by way of non-limiting example, an ASIC or FPGA, or may comprise a combination of executable software instructions and circuitry.

The compute instance32-1also includes a storage node identifier communicator180that is configured to communicate a storage node identifier to a mirror process. This may be accomplished via any suitable inter-process communications mechanism. The storage node identifier communicator180may comprise executable software instructions configured to communicate a storage node identifier to a mirror process, may comprise circuitry including, by way of non-limiting example, an ASIC or FPGA, or may comprise a combination of executable software instructions and circuitry.

FIG.10is a block diagram of a compute instance32-2suitable for implementing scalable storage cluster mirroring according to another implementation. The compute instance32-2includes a means182for determining that storage segments have been modified on a first storage node of a plurality of storage nodes in a first cluster of storage nodes at a first storage segment modification rate. The means182may be implemented in any number of manners, including, for example, via the storage segment modification determiner176illustrated inFIG.9.

The compute instance32-2includes a means184for initiating, in response to determining that the storage segments have been modified on the first storage node at the first storage segment modification rate, a first mirror process that is configured to copy storage segments from an identified storage node to a mirrored cluster of storage nodes. The means184may be implemented in any number of manners, including, for example, via the mirror process initiator178illustrated inFIG.9. The compute instance32-2includes a means for186communicating, to the first mirror process, a storage node identifier that identifies the first storage node. The means186may be implemented in any number of manners, including, for example, via the storage node identifier communicator180illustrated inFIG.9.

FIG.11is a simplified block diagram of the environment10illustrated inFIG.1according to one implementation. The compute instance32includes the processor device34coupled to the memory36. The processor device34is to determine that the storage segments31have been modified on the storage node24-1of the plurality of storage nodes24-1-24-N in the cluster22of storage nodes24-1-24-N at the data modification rate54-1. In response to determining that the storage segments31have been modified on the storage node24-1at the data modification rate54-1, the processor device34is further to initiate the mirror process56that is configured to copy storage segments from an identified storage node to a mirrored cluster of storage nodes. The processor device34is further to communicate, to the mirror process56, the storage node identifier60that identifies the storage node24-1.

FIG.12is a block diagram of a computing device188that is suitable for implementing various components illustrated in the environment10, including, for example, the compute instance32, the compute instance58, and the storage node24-1. The computing device188may comprise any computing or electronic device capable of including firmware, hardware, and/or executing software instructions to implement the functionality described herein, such as a computer server, a desktop computing device, a laptop computing device, a smartphone, a computing tablet, or the like. The computing device188includes the processor device190, a system memory192, and a system bus194. The system bus194provides an interface for system components including, but not limited to, the system memory192and the processor device190. The processor device190can be any commercially available or proprietary processor.

The system bus194may be any of several types of bus structures that may further interconnect to a memory bus (with or without a memory controller), a peripheral bus, and/or a local bus using any of a variety of commercially available bus architectures. The system memory192may include non-volatile memory196(e.g., read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.), and volatile memory198(e.g., random-access memory (RAM)). A basic input/output system (BIOS)200may be stored in the non-volatile memory196and can include the basic routines that help to transfer information between elements within the computing device188. The volatile memory198may also include a high-speed RAM, such as static RAM, for caching data.

The computing device188may further include or be coupled to a non-transitory computer-readable storage medium such as a storage device202, which may comprise, for example, an internal or external hard disk drive (HDD) (e.g., enhanced integrated drive electronics (EIDE) or serial advanced technology attachment (SATA)), HDD (e.g., EIDE or SATA) for storage, flash memory, or the like. The storage device202and other drives associated with computer-readable media and computer-usable media may provide non-volatile storage of data, data structures, computer-executable instructions, and the like.

A number of modules can be stored in the storage device202and in the volatile memory198, including an operating system and one or more program modules, such as the cluster manager38, the mirror process56, or the agent40-1, which may implement the functionality described herein in whole or in part.

All or a portion of the examples may be implemented as a computer program product204stored on a transitory or non-transitory computer-usable or computer-readable storage medium, such as the storage device202, which includes complex programming instructions, such as complex computer-readable program code, to cause the processor device190to carry out the steps described herein. Thus, the computer-readable program code can comprise software instructions for implementing the functionality of the examples described herein when executed on the processor device190.

An operator may also be able to enter one or more configuration commands through a keyboard (not illustrated), a pointing device such as a mouse (not illustrated), or a touch-sensitive surface such as a display device. Such input devices may be connected to the processor device190through an input device interface206that is coupled to the system bus194but can be connected by other interfaces such as a parallel port, an Institute of Electrical and Electronic Engineers (IEEE) 1394 serial port, a Universal Serial Bus (USB) port, an IR interface, and the like. The computing device188may also include a communications interface208suitable for communicating with a network as appropriate or desired.

Other computer system designs and configurations may also be suitable to implement the systems and methods described herein. The following examples illustrate various additional implementations in accordance with one or more aspects of the disclosure.

Example 1 is a method that includes determining, by an agent executing on a storage node, a first data modification rate at which first storage segments are being modified on the storage node; and sending the first data modification rate to a cluster manager.

Example 2 is the method of example 1 wherein the first storage segments are associated with a first division of an enterprise; and wherein the method further includes determining, by the agent, a second data modification rate at which second storage segments associated with a second division of the enterprise are being modified on the storage node; and sending the second data modification rate to the cluster manager.

Example 3 is the method of example 1 wherein the method further includes receiving, by the storage node, a request from the cluster manager for a list of storage segment identifiers that identify storage segments that have been modified on the storage node; and sending, by the storage node to the cluster manager, the list of storage segment identifiers that identify the storage segments that have been modified on the storage node.

Example 4 is the method of example 1 wherein the method further includes receiving, by the storage node, over a period of time, modifications to the first storage segments from a plurality of applications.

Example 5 is the method of example 1 wherein the first storage segments are associated with a first tenant; and wherein the method further includes determining, by the agent, a second data modification rate at which second storage segments associated with a second tenant are being modified on the storage node; and sending the second data modification rate to the cluster manager.

Example 6 is the method of example 1 wherein the method further includes receiving, by the storage node from a mirror process, a request for a list of modified storage segments; and providing, to the mirror process, the list of modified storage segments.

Example 7 is the method of example 6 wherein the method further includes receiving, from the mirror process, authentication credentials associated with the modified storage segments; and determining that the authentication credentials are valid.

Example 8 is a storage node that includes a memory and a processor device coupled to the memory. The processor device is to determine, by an agent executing on the storage node, a first data modification rate at which first storage segments are being modified on the storage node; and send the first data modification rate to a cluster manager.

Example 9 is a method including receiving, by a mirror process from a cluster manager, a storage node identifier that identifies a storage node; obtaining, from the storage node, a plurality of storage segments that have been modified; and sending, to a mirrored cluster of storage nodes, the plurality of storage segments.

Example 10 is the method of example 9 wherein the method further includes receiving, from the cluster manager, a list of the storage segments that have been modified.

Example 11 is the method of example 9 wherein the method further includes sending, by the mirror process to the storage node, a request for a list of storage segments that have been modified; and receiving the list of storage segments that have been modified from the storage node.

Example 12 is the method of example 9 wherein the method further includes receiving, from the cluster manager, authentication credentials; and sending, to the storage node, the authentication credentials.

Example 13 is the method of example 9 wherein the method further includes receiving, from the cluster manager, an address of the mirrored cluster of storage nodes.

Example 14 is the method of example 13 wherein the method further includes receiving, from the cluster manager, an address of a second mirrored cluster of storage nodes; and sending, to the second mirrored cluster of storage nodes, the plurality of storage segments.

Example 15 is the method of example 9 wherein the method further includes determining, by the mirror process, a data mirror rate that identifies a real-time rate at which the mirror process is sending the plurality of storage segments to the mirrored cluster of storage nodes; and sending the data mirror rate to the cluster manager.

Example 16 is the method of example 9 wherein the method further includes periodically: determining, by the mirror process, a data mirror rate that identifies a real-time rate at which the mirror process is sending the plurality of storage segments to the mirrored cluster of storage nodes; and sending the data mirror rate to the cluster manager.

Example 17 is an apparatus that includes a memory and a processor device coupled to the memory. The processor device is to receive, by a mirror process from a cluster manager, a storage node identifier that identifies a storage node; obtain, from the storage node, a plurality of storage segments that have been modified; and send, to a mirrored cluster of storage nodes, the plurality of storage segments.

Example 18 is an apparatus that includes a means for determining that storage segments have been modified on a first storage node of a plurality of storage nodes in a first cluster of storage nodes at a first storage segment modification rate; a means for initiating, in response to determining that the storage segments have been modified on the first storage node at the first storage segment modification rate, a first mirror process that is configured to copy storage segments from an identified storage node to a mirrored cluster of storage nodes; and a means for communicating, to the first mirror process, a storage node identifier that identifies the first storage node.

Example 19 is an apparatus that includes a storage segment modification determiner that is configured to determine that a storage segment has been modified; a mirror process initiator that is configured to initiate a mirror process upon request; and a storage node identifier communicator that is configured to communicate a storage node identifier to the mirror process.