Method and apparatus for load balancing virtual data movers between nodes of a storage cluster

Data Virtual Data Movers (VDM) are assigned to nodes of the storage cluster and a backup node is assigned for each data VDM. A system VDM on each node collects node statistics including operational parameters of the node and activity levels of the data VDMs on the node. A cluster manager collects the node statistics from each of the system VDMs and uses weighted collected node statistics to assign a node score to each node in the storage cluster. The cluster manager uses the node scores to identify possible data VDM movement combinations within the storage cluster by applying a set of hard rules and a set of soft rules to evaluate the possible data VDM movement combinations. If a VDM movement combination is selected, it is implemented by moving at least some of the data VDMs within the cluster to attempt to equalize node scores within the cluster.

BACKGROUND

This disclosure relates to computing systems and related devices and methods, and, more particularly, to a method and apparatus for load balancing virtual data movers between nodes of a storage cluster.

SUMMARY

The following Summary and the Abstract set forth at the end of this application are provided herein to introduce some concepts discussed in the Detailed Description below. The Summary and Abstract sections are not comprehensive and are not intended to delineate the scope of protectable subject matter which is set forth by the claims presented below.

Data Virtual Data Movers (VDM) are assigned to nodes of the storage cluster and a backup node is assigned for each data VDM. A system VDM on each node collects node statistics including operational parameters of the node and activity levels of the data VDMs on the node. A cluster manager collects the node statistics from each of the system VDMs and uses weighted collected node statistics to assign a node score to each node in the storage cluster. The cluster manager uses the node scores to identify possible data VDM movement combinations within the storage cluster by applying a set of hard rules and a set of soft rules to evaluate the possible data VDM movement combinations. If a VDM movement combination is selected, it is implemented by moving at least some of the data VDMs within the cluster to attempt to equalize node scores within the cluster.

DETAILED DESCRIPTION

FIG. 1is a functional block diagram of an example storage environment100. As shown inFIG. 1, in storage environment100, a data client110may access storage resources provided by one or more storage systems120over a communication network130. In some embodiments, the communication network130is an Internet Protocol (IP) communication network130enabling transmission of IP data packets through the communication network130, although other forms of communication networks may be used to interconnect the data client110with storage systems120depending on the implementation.

Data from the data client110is stored in the storage resources of the storage systems120. Storage resources that are accessed by a data client110over a communication network130are referred to herein as Network Attached Storage (NAS). In some embodiments, the physical storage resources of a storage system120are abstracted to the data client110by software applications running on the storage systems120referred to herein as “Software Defined Network Attached Storage (SDNAS) applications.” A given SDNAS application may, in some embodiments, be implemented as a Virtual Network Attached Storage (VNAS) server140.

To provide enhanced reliability, data from data client110may be stored in more than one storage system120on the communication network130. In some embodiments, the data client110interacts with a file system maintained by a primary VNAS server140on a primary storage system120. If a failure occurs on the primary storage system120, on communication network130, or elsewhere, which renders the data client110unable to access the file system on the primary storage system120, the data client110is able to access the file system on the backup VNAS server140on the backup storage system120.

Two or more virtual NAS servers140that are logically associated to provide redundant access to one or more file systems will be referred to herein as a “cluster”. In some embodiments, a cluster may include multiple VNAS servers140, and each VNAS server140may be responsible for hundreds of file systems. A virtual NAS server140will also be referred to herein as a “node”300in the storage cluster330. In some embodiments one node300assumes responsibility for cluster management to specify which node in the cluster has primary responsibility for each file system, and which node(s) in the cluster are backup nodes for each respective file system. As used herein, the term “data Virtual Data Mover (VDM)” will be used to refer to software that is responsible for managing access to and replication of one or more file systems on a VNAS server140. A given node in a storage cluster330may have multiple SDNAS processes executing thereon, and each SDNAS process may have multiple data VDMs executing within it. The term “system Virtual Data Mover (VDM)” will be used to refer to software that is responsible for managing overall organization of the storage cluster330.

FIG. 2is a functional block diagram of an example storage system120for use in a storage environment100. As shown inFIG. 2, the storage system120has physical resources including a number of CPU processor cores142, local memory144, guest operating system145, storage resources146, and other physical resources. A hypervisor148abstracts the physical resources of the storage system120from emulations150, and allocates physical resources of storage system120for use by the emulations150. In some embodiments, a given storage system120may have storage resources146that are implemented using an array of discs160, which may be implemented using a number of different storage technologies.

Each emulation150has a base operating system152and one or more application processes running in the context of the operating system. As shown inFIG. 2, in some embodiments, one or more of the emulations150instantiated on storage system120has a Software Defined Network Attached Storage (SDNAS) process154instantiated thereon to enable the emulation150to implement a Virtual Network Attached Storage (VNAS) server140on the communication network130. In some embodiments, emulations150implementing network attached storage processes operate as nodes300in storage cluster330.

As used herein, the term “Virtual Data Mover” (VDM) will be used to refer to one or more software applications configured to execute in an emulation150to enable the emulation150to implement a VNAS server140on the communication network130. In the example shown inFIG. 1, emulations150A-150N include VDM applications162and, as such, are configured to implement VNAS servers140on the communication network130. As discussed below, each emulation150including an SDNAS process may support one or more VDMs and participate in managing data within a storage cluster330. A given storage system120may have emulations150functioning as nodes300in multiple storage clusters330. In some embodiments, the VDM applications are implemented in software and abstract the underlying data storage provided by the storage system120.

To provide enhanced reliability, data from data clients110may be replicated between storage nodes300. In this context, a given storage system120may be considered a storage node300. Likewise, a director board302within a storage system120may be considered a storage node300, such that each storage system120implements multiple storage nodes300in the storage environment100. A group of storage nodes300that are logically defined to cooperatively provide storage level redundancy will be referred to herein as a storage cluster330.

FIG. 3is a functional block diagram of an example node300of a storage cluster330. As shown inFIG. 3, in some embodiments node300is formed as a director board302having one or more CPUs304, memory306, and on which a guest operating system308is instantiated. A hypervisor310abstracts the physical resources of the director board302from emulations312to enable multiple emulations312to execute on a given director board302. In some embodiments, multiple emulations312on a given director board302are not allocated to the same storage cluster330, to provide physical redundancy within the storage cluster330. Each emulation312has its own base operating system314in which a SDNAS container316executes. A master system management process318and various other miscellaneous processes320are also instantiated to execute in the context of the base operating system314within emulation312.

In some embodiments, the SDNAS container316includes a system Virtual Data Mover (VDM)322that coordinates execution of the SDNAS processes154implemented by SDNAS container316with other SDNAS processes154being executed in other SDNAS containers316. Each node300also executes one or more Data VDMs324responsible for handling user file systems. Each data VDM324is executed on a node300that is part of a storage cluster330. Each data VDM324manages one or more file systems326. Data for the file systems326is maintained in primary node storage resources146. For redundancy, data of the file systems is also replicated to a backup node and maintained in backup node storage resources. During failover, the VDM is closed on the primary node and brought up on the backup node (optionally with the same VDM ID) to enable continued access to the file systems being managed by the VDM.

One of the SDNAS processes in a cluster of SDNAS processes executes a cluster manager328that controls the overall structure of the storage cluster330, such as defining which nodes300are included in the storage cluster330, which nodes and which SDNAS processes154executing on those nodes are to host particular data VDMs324, which nodes should be backup nodes for particular data VDMs324, and which user file systems236should be handled by each data VDM324.

FIG. 4shows an example storage cluster330including a plurality of nodes300. As shown inFIG. 4, each node300includes at least one SDNAS container316, within which data VDMs324execute. One of the nodes300in the storage cluster330(node300A inFIG. 4) is the master node and executes the cluster manager328in its SDNAS container316. The master node further executes master system management process318. Each of the other nodes300B,300C, in the storage cluster330run a slave system management process319. The management processes318,319collect statistics on their respective node300and are used to perform overall management of the storage cluster330.

In some embodiments, each VDM324in a cluster330is assigned a backup node300at the time of creation of the VDM324. This backup node300is responsible for failing over the VDM324when the VDM324or the node on which the VDM324is executing becomes unavailable. For example, inFIG. 4VDM_1324on node300A has node300B assigned as its backup node. Data VDM_2 on node300C has node300A assigned as its backup node. Data VDM_3 on node300B has node300C assigned as its backup node. InFIG. 4, the box that is shaded is the location of execution of the VDM, and the box that is not shaded is designated as the backup node for the VDM. As is clear, a given SDNAS container may host VDMs within the storage cluster330. Where the SDNAS container is executing on a node that has been designated as the backup node for one or more other VDMs in the cluster, the SDNAS container likewise may be required at a later point in time to assume responsibility for one or more of those other VDMs in the event of a failover.

The cluster manager328, in some embodiments, is responsible for determining which node300in the storage cluster330is provisioned to host the VDM324and which node300in the storage cluster330is the backup node for the VDM324. In some embodiments, a given cluster will have at most one SDNAS container316executing on a given node. Hence, if a node is a backup node for a VDM324in the cluster, upon failover of the VDM the SDNAS container executing on the given node will implement the VDM process to enable continued access to the file systems being maintained by the VDM.

In some embodiments, the cluster manager328collects usage statistics from the system management processes318,319, and uses the statistics to determine the optimum layout of the VDMs324and their backup nodes300in the storage cluster330. The cluster manager328then triggers VDM324movement within the storage cluster330to achieve improved overall performance by the storage environment100.

Load balancing of VDMs between nodes of a storage cluster330and attendant VDM324movement may be triggered manually, for example, from a user interface. Load balancing may also occur automatically, for example periodically within each storage cluster. Load balancing of VDMs between nodes of a storage cluster330may also occur automatically upon the occurrence of particular events. One example event that may cause load balancing of VDMs between nodes of a storage cluster330to occur may be a determination that performance of one or more of the file systems supported by the storage cluster has failed to achieve an intended service level objective. Another example event that may cause load balancing of VDMs between nodes of a storage cluster330is occurrence of an event affecting the topography of the storage cluster, such as a failure of a node of the storage cluster or a recovery of a node of the storage cluster.

In some embodiments, the system management processes318,319gathers statistics based in 10, CPU usage, and memory usage information of multiple monitored aspects of the storage environment100. Example statistics may be collected from the storage systems120, director boards302, nodes300, SDNAS containers316, and from the data VDMs324. Statistics may be collected natively by processes running on emulation150, from the various operating systems including storage system120guest operating system145, emulation host operating systems152, and from other aspects of the storage systems120having nodes300participating in the storage cluster330.

The cluster manager328gathers the complete statistics from all nodes300and the current layout of data VDMs324within the storage cluster330. Each node300in the storage cluster330is then assigned a score based on its statistics and the weighted statistics relative to other nodes300in the storage cluster330. The cluster manager reassigns data VDMs324between nodes300of the storage cluster330to balance workload between the nodes300such that all nodes300in the storage cluster330have similar workload scores, subject to the condition of minimizing VDM movement.

In some embodiments, each node300has a system VDM322that periodically, for example once per minute, collects statistics about the node300on which it is instantiated. In some embodiments, the system VDM322collects statistics for the node300as well as statistics per data VDM324instantiated on the node300. For example, where the node300is implemented on a director board302, the system VDM322may collect statistics relating to operation of the director board302including CPU304utilization levels, memory306access statistics, and other statistics available from guest operating system308. These statistics from the director board302provide information as to the usage levels of the physical resources of the underlying hardware relative to the capacity of the physical resources.

The entire cluster has controller, referred to herein as the cluster manager328, that collects information from each of the system VDMs322on each of the nodes300in the storage cluster330, and also maintains the current layout of which data VDM324in the cluster is responsible for each user file system326, and which node300is the backup node300for each VDM324. Accordingly, the cluster manager324knows the node responsible for each VDM324, which VDM324is handling each file system326, and which node300is assigned as the backup node for each VDM324.

In some embodiments, for each data VDM324, the cluster manager328determines multiple aspects associated with overall storage cluster management and workload distribution within the storage cluster330. In some embodiments, the cluster manager328examines the statistics for each node300in the storage cluster330to determine:the total number of data VDMs330on the node300;the number of user file systems326on each data VDM324;the total number of user file systems326supported by the set of data VDMs324on the node;activity levels of the file systems326; andoperational statistics of the node300.

In addition, when determining whether to move a VDM324from a current node300to a potential target node300, the cluster manager328examines a similar set of statistics for the target node, including:the number of total VDMs324on the potential target node;the number of user file systems326on the potential target node;the number of VDMs324for which the potential target node is currently assigned as the backup node, because if one or more of the VDMs324get failed over to the target node, the target node300must be able to continue to function and not be overloaded after the failover; andoperational statistics for the potential target node.

In some embodiments, the cluster manager328determines consumer statistics, such as the base operating system statistics of the node300. One reason to incorporate this level of usage information, for example, is because other processes such as background processes may affect performance of the director board302implementing node300. Example consumer statistics may include:front end (FA) statistics to look for high write pending conditions;events related to operation of the storage resources146and the underlying array of discs160, such as an event indicating the storage resource pool is full; andmemory usage indications and other thresholds.

Certain statistics may be more important than other statistics. For example, a statistic indicating large10on a file system may be more important than a large number of VDMs present on a given node. Likewise, combinations of statistics may be used to ascertain high workload levels, such as a combination of several VDMs on a given node each of which has high 10 access statistics. Evaluating the number of VDMs in the context of the workload of the VDMs may provide a better indication of high work load on a node, than simply looking at the number of VDMs assigned to the node.

Accordingly, in some embodiments, the statistics collected by the cluster manager328are weighted and analyzed in context to determine an overall workload score for each of the nodes300of the storage cluster330. The cluster manager328then attempts to distribute workload within the storage cluster330such that nodes300within the storage cluster330have similar workload scores.

In some embodiments, the cluster manager328identifies several sets of possible VDM movement combinations that enable VDMs to be more optimally distributed within the storage cluster330that will better balance the node score across the nodes of the storage cluster330. In some embodiments, the cluster manager uses a set of hard rules and a set of soft rules in connection with identifying combinations that may be used to distribute the VDMs within the cluster. If an identified combination violates one of the hard rules it is discarded from the set of possible VDM movements and not implemented by the cluster manager328. If an identified combination violates one or more of the soft rules, it remains a candidate combination and will be used if no other combination is suitable and it is the best combination. In some embodiments, a “best combination” is a combination that results in a better workload distribution within the storage cluster330than currently exists in the storage cluster330, and exhibits the fewest number of soft rule violations.

In some embodiments the set of hard rules dictates whether it is possible to move a VDM within the cluster.

In some embodiments, the set of hard rules includes a first hard rule that a node's score must be above a high threshold value for it to be considered overloaded.

In some embodiments the set of hard rules further includes a second hard rule that a node's score must be below a low threshold value for it to be considered underloaded.

In some embodiments, the set of hard rules includes a third hard rule that a node300must have at least more than one data VDM324assigned to it for it to be considered overloaded, such that a node300with zero or one data VDM324will never be considered overloaded.

In some embodiments, the set of hard rules includes a fourth hard rule that a potential target node for a data VDM324that is to be moved must not have a score (including the new VDMs statistics) that exceeds the VDM's original node's score. That is, the cluster manager will take the potential target node for a VDM that the cluster manager would like to move, add the statistics of the VDM that is the candidate to be moved to the potential target node, and recalculate the weighted score for the potential target node. This enables the cluster manager to determine a hypothetical score for the potential target node if the VDM were to be moved. If the hypothetical score for the potential target node exceeds the score of the node that is currently hosting the VDM, moving the VDM to the potential target node will not help distribute workload more evenly within the storage cluster330. Hence, the fourth hard rule prohibits moving a VDM to a potential target node where doing so would exacerbate workload inequality within the storage cluster330.

In some embodiments, a set of soft rules is used to optimize overall performance of the storage cluster by attempting to minimize VDM movement within a storage cluster330. VDM movement between nodes300of a storage cluster330requires the use of processor resources. For example, movement of a VDM from a first node300to a second node300within the storage cluster330may require a new emulation312to be started on a target node, and likewise may require a new system VDM322and data VDMs324to be started on the target node300. In some embodiments, where the VDM is to be moved from a node on one storage system120to a node300on a different storage system120, movement of the VDM324may also entail movement of data between storage systems120. Hence, movement of a VDM324within a storage cluster330can be resource intensive. Accordingly, when possible, it is often preferable to minimize VDM movement within the storage cluster330.

In some embodiments, the set of soft rules includes a first soft rule that attempts to assign VDMs to nodes in the cluster, at the original time of provisioning the storage cluster330, that will minimize VDM movement within the storage cluster330after provisioning. When a new cluster is to be provisioned, spreading the VDMs evenly across the nodes with a roughly equal number of VDMs on each of the nodes of the storage cluster may minimize VDM movement, at least initially. Likewise, distributing responsibility for file systems between VDMs, optionally also considering anticipated file system usage, may initially optimize workload distribution within the storage cluster330.

In some embodiments, the set of soft rules includes a second soft rule that seeks to minimize VDM movement by causing failover of a VDM from its current node to its backup node, rather than moving the VDM to a third node. During failover the backup node assumes responsibility for the VDM and node that previously had hosted the VDM becomes the backup node. Since the backup node has a replication of the file systems being handled by the VDM, failover of a VDM to its backup node can cause movement of workload within the cluster with a minimum of data movement. By using failover, accordingly, it is possible to move responsibility for the VDM, and hence the associated10load on the file systems serviced by the VDM, from the original node to the backup node. Where the backup node is underloaded and the original node is overloaded, this second soft rule enables distribution of the load within the cluster while expending a minimal amount of resources.

In some embodiments, the set of soft rules includes a third soft rule that seeks to proactively initiate VDM movement in connection with other actions affecting the structure of the cluster. For example, node failure or node recovery within the cluster may be triggered to perform VDM movement within the cluster to seek to balance node scores within the cluster in connection with an event where additional capacity is being added to the cluster (node recovery) or in connection with an event that otherwise will require at least some of the VDMs in the cluster to be moved anyway (node failure).

Although a particular set of hard and soft rules was described, it is possible that additional and/or different hard and soft rules may be used depending on the particular embodiment.

FIG. 5shows an example process of performing load balancing of VDMs between nodes300of a storage cluster330. As shown inFIG. 5, the system VDM322on each node300in the storage cluster330periodically collects node statistics from its node300(block500). Example statistics collected by the system VDM322are discussed in greater detail above.

The cluster manager328collects node statistics from each of the system VDMs322of the storage cluster330(block505). In some embodiments, node statistics are collected periodically by polling the system VDM processes322for node300statistics. In some embodiments the node statistics are collected by the cluster manager328on demand, for example in connection with initiation of a load balancing process.

After collecting node statistics, the cluster manager328weights the statistics and uses the weighted statistics to assign scores to each node (block510). The cluster manager328then identifies combinations of nodes300for sets of VDMs in a cluster balancing process, to balance node300scores while minimizing VDM movement between nodes (block515). In connection with some embodiments, consideration of node combinations for VDM placement uses the set of hard rules and the set of soft rules described above to determine whether one or more VDMs should be moved within the storage cluster330. In some embodiments, the cluster manager328also determines whether the storage cluster330has too many nodes, for example if the node scores within the cluster are all too low. In some embodiments, the cluster manager also determines whether the storage cluster330has too few nodes, for example if the node scores within the storage cluster330are all too high.

Once a determined set of VDM movements has been finalized, the cluster manager328reconfigures the topology of the storage cluster330by moving VDMs between the nodes300of the storage cluster330(block520). In connection with this, the cluster manager328will also commission new cluster nodes300and distribute VDMs to the new cluster nodes300where the cluster manager328has determined that the storage cluster330has too few nodes300. Likewise, if the cluster manager328determines that the storage cluster330has too many nodes300, the cluster manager328will decommission one or more of the cluster nodes300and redistribute the VDMs that were previously assigned to those nodes300to other nodes300of the storage cluster330.

The cluster manager328is responsible for determining which node in the cluster will host particular VDMs and which nodes in the storage cluster330are backup nodes for the particular VDMs. In some embodiments, the step of identifying combinations of nodes for the set of VDMs (block515) determines VDM placement for the VDMs and as well as the set of backup nodes. In other embodiments, the step of identifying combinations of nodes300for the set of VDMs (block515) initially only determines which nodes will host the VDMs within the storage cluster330. In embodiments where block515initially only determines VDM placement, backup node assignment is then implemented subsequently as shown in block525. Backup node assignment may be determined after VDM movement within the cluster (after block520inFIG. 5) or may be determined before VDM movement (before block520inFIG. 5).

In some embodiments, backup node assignment within the storage cluster330is at least partially based on assuming that the VDM will failover to the backup node at some point in time, and looking at the node scores based on the hypothetical failover of the VDM within the primary/backup node pair. This enables backup node assignment to look at what would happen within the storage cluster330if one or more of the VDMs were to experience failover, and ensure that VDM failover will not cause a node score of the backup node to increase during failover to an unacceptable level.

FIGS. 6A and 6Bshow an example distribution of VDMs within an example storage cluster330. The example storage cluster shown inFIG. 6Aincludes a set of three nodes300, designated as node300A, node300B, and node300C. The set of nodes are responsible for a set of six VDMs referred to as VDMs324A-F. Each VDM is assigned a backup node, where the node hosting the VDM is shaded and the backup node300has no shading. A black arrow indicates which node has been assigned as backup node for each VDM, to show where responsibility for the VDM will transfer in the event of failover. Node300A has instantiated thereon a cluster manager328that controls organization of the storage cluster330and distribution of workload within the storage cluster. In some embodiments, cluster manger328implements the statistics collection and load balancing processes described herein.

At a first point in time, as shown inFIG. 6A, storage node300A has been assigned to host five data VDMs (VDM324A, VDM324B, VDM324C, VDM324D, and VDM324E). Storage node300B has been assigned as the backup node for these five VDMs (VDM-A to VDM-E). Storage node300C has been assigned to host one data VDM (VDM-F), and storage node300B has been assigned as the backup node for VDM-F. In the hypothetical example, it will be assumed that the relative workload of each VDM is approximate, and accordingly the distribution shown inFIG. 6Arepresents a workload imbalance in which node300A is overloaded, node300C is underloaded, and node300B has the potential to become overloaded if there is a significant number of VDM failovers within the storage cluster330.

FIG. 6Bshows an example workload distribution within a storage cluster330after a load balancing process has been performed by cluster manager328. The following table (TABLE I) shows the changes that were made (VDM movements) relative to the original VDM and backup node assignments shown inFIG. 6Ato achieve the distribution shown inFIG. 6B:

The methods described herein may be implemented as software configured to be executed in control logic such as contained in a Central Processing Unit (CPU) or Graphics Processing Unit (GPU) of an electronic device such as a computer. In particular, the functions described herein may be implemented as sets of program instructions stored on a non-transitory tangible computer readable storage medium. The program instructions may be implemented utilizing programming techniques known to those of ordinary skill in the art. Program instructions may be stored in a computer readable memory within the computer or loaded onto the computer and executed on computer's microprocessor. However, it will be apparent to a skilled artisan that all logic described herein can be embodied using discrete components, integrated circuitry, programmable logic used in conjunction with a programmable logic device such as a Field Programmable Gate Array (FPGA) or microprocessor, or any other device including any combination thereof. Programmable logic can be fixed temporarily or permanently in a tangible computer readable medium such as random-access memory, a computer memory, a disk, or other storage medium. All such embodiments are intended to fall within the scope of the present invention.