Storage system with dynamic transfer of block file system ownership for load balancing

A storage system uses a block-level file system to manage physical storage of data blocks of logical units, the file system being mounted on an owning one of a set of storage processors. The storage processors redirect received storage requests to the owning storage processor, and periodically the amount of redirection is evaluated. If the amount of redirection is higher than a threshold, then the file system is dismounted at the owning storage processor and mounted at another storage processor to reduce the amount of redirection.

BACKGROUND

The present invention is related to the field of data storage.

Data storage systems may employ a block file system to manage the use of physical storage resources (e.g., areas of physical disks) in providing higher-level logical units of storage to applications or users in a data processing system. This is particularly the case for storage systems supporting mapped logical units (MLUs) used in “thin provisioning” schemes of storage allocation, where physical storage resources are assigned to logical units of storage dynamically as needed during operation rather than statically at an initial time of configuration. The block file system defines a corresponding file for each MLU and conducts block-level file-access operations in satisfaction of storage input/output requests (reads and writes) directed to the MLU. Here “block-level” refers to operations on fixed-size data units called “blocks”, a technique long known in the computing industry. One example of a block file system is the Common Block File System (CBFS) utilized in storage products manufactured and sold by EMC Corporation.

SUMMARY

In certain storage applications, it is desirable that a single block file system support multiple files for corresponding logical units of data, herein referred to as “LUNs” and which may be mapped logical units (MLUs). This need arises, for example, when features such as replication or data deduplication are provided by the storage system. In the case of deduplication, for example, a set of LUNs that form a domain for deduplication purposes preferably all share a single file system that creates and manages file-based mappings of LUN blocks to physical blocks for the LUNs. The file system recognizes duplicate data blocks across the LUNs and maps duplicated blocks to a single physical data block. In this kind of situation, all accesses to any of the LUNs that share a file system are necessarily processed by the shared file system.

Another feature of modern storage systems is the use of multiple storage processors (SPs) that have interfaces to external users and carry out storage requests received from such users. In some configurations, different LUNs may be “exposed” (presented for use) to external users via different SPs, while each block file system is at any given time owned by only one of the SPs. In the case that different LUNs that share a file system are exposed via different SPs, there is preferably a mechanism by which one SP can automatically forward or redirect a received storage request to another SP. This way, if a storage request is received at an SP that does not own the underlying block file system, the request can be forwarded for processing to the SP that owns the file system. However, this redirection functionality can create performance issues if the amount of redirection is very high, as latency is added and system resources are wastefully used in simply transferring requests internally from one SP to another. It is desirable to address this performance issue in order to achieve desired performance and efficiency in storage systems that employ block file systems in providing advanced storage features.

A method of operating a storage system is disclosed, where the storage system has a set of storage processors coupled to a set of physical storage devices that provide underlying physical storage for logical units of storage. The storage system uses a block-level file system to handle block-level physical data storage operations for data of two or more logical units of storage presented to external users of the storage system, where each logical unit of storage is organized into an array of data blocks and is associated with a respective distinct file in the block-level file system. The block-level file system is at any given time mounted on (or owned by) one of the storage processors.

Each storage processor continually performs a redirection operation by which storage requests for data of any of the logical units of storage are redirected to the owning storage processor from each other storage processor of the set receiving the storage requests. For requests received at the owning storage processor, the redirection operation allows the requests to be processed locally rather than being directed to another storage processor. Periodically, the amount of redirection that is occurring for the storage requests to a specific file system is evaluated. If the amount of redirection is higher than a predetermined threshold, then the file system is dismounted at the owning storage processor and mounted at another storage processor to reduce the amount of redirection. In a system having two storage processors, this transfer is to the other (currently non-owning) storage processor, which is indicated by the evaluation to be receiving a higher proportion of the storage requests to be processed by the file system. In systems having more than two storage processors, the evaluation and transfer may involve tracking and measuring redirection on a per-storage-processor basis (i.e., tracking redirections from each of multiple non-owning storage processors to the one owning storage processor).

By the above technique, as a pattern of storage requests changes in a way that increases the amount of redirection occurring in the storage system, the system automatically moves the target resource (block file system) to another storage processor to reduce the amount of redirection, improving overall performance and efficiency of operation. This can be viewed as a form of load balancing, dynamically assigning file system ownership among the storage processors to make more effective use of their computing resources.

DETAILED DESCRIPTION

FIG. 1shows a storage system from primarily a functional or logical perspective, with some hardware components also depicted. Among the hardware components are storage processors (SPs)10-A,10-B and physical disks12. The SPs10, which are also referred to as SP A and SP B herein, each include a device driver14having a driver upper layer (U.L.)16and redirector18, along with one or more file systems (F/S)20(identified as20-1through20-N for SP10-A, and20-O through20-Y for SP20-B). The file systems20have functional connections to the physical disks12. Each driver upper layer16has logical interfaces to separate host computers (hosts) that use the storage resources provided by the storage system, these resources being presented to the hosts as logical units (LUNs)22(identified as22-1through22-M for SP10-A, and22-N through22-Z for SP20-B). As shown, the driver upper layers16and redirectors18of the SPs10-A,10-B are coupled by respective communication channels24,26.

Each file system20is a so-called “block” file system, implementing the storage of one or more respective LUNs22as a collection of fixed-size data units (“blocks”) distributed among the physical disks12. Block file systems support sophisticated modern storage features such as mapped logical units (MLUs), which are logical units of storage whose underlying physical storage resources are allocated dynamically during operation rather than statically at an initial configuration time. The use of MLUs can significantly increase the efficiency of storage use in a data processing system. In the present context, a block file system also supports features such as replication (snapshots) and data de-duplication. A block file system maintains a low-level mapping between data units of the LUNs22and underlying blocks stored on the physical disks as known to those skilled in the art. As currently used, a file system20is available for use by only one SP10, such availability being referred to as being “mounted” on the SP10(or “owned” by the SP10). Thus inFIG. 1, file systems20-1through20-N are mounted on SP A, while file systems20-O through20-Y are mounted on SP B.

As generally known, a LUN is a unit of storage organized as a linear array of addressable storage locations. The storage system may be utilized in a direct-attached or storage area network (SAN) environment, in which case it typically exposes the LUNs22to external requestors as shown. Alternatively, it may be of a type commonly referred to as “network-attached storage” or NAS, in which case it typically implements a network file system and exposes storage resources as volumes of that file system (e.g., a network drive as commonly known in Windows® networks), using LUNs internally to represent units of storage assigned to the volumes. Such a network file system is not to be confused with the block file systems20. In either the NAS or SAN case, the storage system under control of operating software uses the physical disks12as underlying physical storage for the volumes or LUNs exposed externally.

FIGS. 2-4illustrate example relationships between the LUNs22and file systems20.FIG. 2shows a general case in which each of a set of LUNs22-I,22-J is associated with a single corresponding one of a set of file systems20-I,20-J, and each file system20manages the storage of physical blocks of the corresponding LUN22.

FIGS. 3 and 4illustrate slightly more complex relationships that may be applicable when features such as replication or de-duplication are in use.FIG. 3shows a case in which two LUNs22-Y and22-Z are both associated with one file system20-Y. In this case, the file system20-Y is responsible for maintaining the data mapping for both LUNs22-Y and22-Z, which presumably have some operational relationship that requires or takes advantage of the shared use of a single file system20. If the LUNs22-Y and22-Z form a de-duplication domain, for example, then the file system20-Y can map the data of these LUNs to three pieces of underlying physical storage, namely (1) a first piece for data blocks unique to LUN22-Y, (2) a second piece for data blocks unique to LUN22-Z, and (3) a third piece for data blocks that are identical between the two LUNs22-Y and22-Z, along with information mapping these shared blocks to corresponding blocks of the LUNs22-Y and22-Z. InFIG. 3, it is assumed that file system20-Y is mounted at SP-B, and that SP-B also presents the LUNs22-Y and22-Z to external users.

FIG. 4shows a similar case as that ofFIG. 3, except that LUN22-Y and LUN22-X share a file system20-X mounted on SP A. In this case, the redirectors18cooperate to direct storage requests for LUN22-Y received at SP B to SP A where the requests are handled by file system20-X. This operation is described in more detail below.

FIG. 5shows a storage processor10ofFIG. 1from a hardware perspective. It includes processing circuitry (PROCESSOR)28, network (NW) interface circuitry30for coupling the storage system10to an external network (not shown), and device (DEV) interface circuitry32for coupling the processing circuitry28to the physical disks12. The processing circuitry28contains memory and computer program instruction execution logic, and it stores and executes computer program instructions that correspond to the functional components shown inFIG. 1, i.e., to the driver upper layer16, redirector18, and file systems20. As indicated above, it may also execute a network file system or other higher-level storage-oriented application providing storage functionality to the external hosts.

As described above, the storage system provides a form of load balancing between the storage processors10, specifically in the form of transferring ownership of block file systems20between the SPs10to reduce the extent to which storage requests are redirected from one SP10to the other. This process has three essential components, including (1) maintaining counts of storage requests that are received locally (i.e., by the counting SP) versus via redirection (from another SP), (2) periodically analyzing the counts to determine whether a file system20should be transferred for load balancing reasons, and (3) when indicated, performing specific steps to effect such a transfer of a file system20. These processes are described below with reference toFIGS. 6-9.

FIG. 6shows the processing of a storage request (read or write) received by an SP10, specifically initial processing which begins at40with receipt of the request at a redirector18. The request specifies a particular target LUN22, whose underlying file system20may or may not be owned by (mounted on) the SP10that receives the request. At42, the redirector18of the receiving SP10determines whether this SP10is the “preferred” SP10for requests to this LUN22, which is the case if this SP10owns the underlying file system20. The tracking and querying of preferred SPs10may be accomplished through use of an indicator or flag variable associated with each LUN22. In this description, a set of LUNs22sharing a file system20is referred to as a “domain”. At the time of configuration of a domain and its underlying file system20, the per-LUN indicator on the owning SP10is set to a first value indicating that it is the preferred SP10for each LUN of the file system10, whereas this indicator is set to a second value on the other, non-owning SP(s)10indicating that the other SP(s) are the preferred SP10. The test at42thus involves examining the value of this indicator. If the file system20is owned by this SP10, then processing proceeds to step44, and otherwise it proceeds to step46.

At step44the storage request is processed locally, i.e., using the file system20which is owned by the SP10that received the request. As generally known in the art, such processing involves either or both reading data from or writing data to physical disks12that are mapped to the target LUN22, utilizing the services of the target block file system20. Once the request has been processed, the procedure ends at48.

At step46, the request is forwarded to the SP10that owns the file system20, referred to as a “peer” SP10, where the request will be processed. In an embodiment such as that ofFIG. 1having only two SPs10, the peer SP10is the other SP of the pair, and in this case a binary “Preferred” indicator is sufficient to inform an SP10that it should forward (redirect) the request to the other SP10. In other embodiments employing more than two SPs10, it may be necessary for each SP10to store an indication of which peer SP10is preferred. Once the request has been forwarded at step46, the procedure ends at48.

FIG. 7shows the processing of a storage request by a file system20, which begins at50with receiving the request from a redirector18. At52it is determined whether the request was received (redirected) from a peer SP10, as opposed to being received via the redirector18of the same SP10on which the file system20resides. If the request was not redirected, then processing continues to54where a “local serviced count” is updated (incremented), and then to56where file system operations are performed as necessary in satisfaction of the storage request56. The “local serviced count” is a count of storage requests to the target LUN22that were received directly from an external requestor (e.g., from a host) by the same SP10that owns the file system20.

If the test at52indicates that the storage request was redirected from a peer SP10, then at58a “peer serviced count” is updated (incremented), and then the operations of56are performed. The “peer serviced count” is a count of storage requests to the target LUN22that were received (redirected) from a peer SP10as described above with reference toFIG. 6. In an embodiment having only two SPs10such as shown inFIG. 1, only one peer serviced count for the one other SP10need be maintained, while in other embodiments having more than two SPs10it may be necessary to employ separate peer counts for different peer SPs10.

FIG. 8shows a process of periodically checking whether to initiate a transfer of ownership of a file system20to another SP10. It begins at70with an advance of a meter, as indicated by “count/tick” (referring to a repeating event (count) or a timer (tick) as the meter). In one embodiment, the procedure may be performed for each storage request processed by the file system20, and thus be an extension of the processing ofFIG. 7. At72, it is determined whether the meter (e.g., count of a succession of storage requests) has reached a “check threshold” at which a check for a need to transfer ownership of the file system20is done. The check threshold may be selected to provide a sufficiently long interval over which a sufficient number of storage requests are received, to provide meaningful information about the amount of redirection (current as well as trend, as explained below). If the check threshold has been reached, then a procedure including steps74-80is performed, and otherwise the procedure ends at82.

At74, the LUNs22that share the file system20are identified, and at76several steps are performed for each such LUN22. First, the values of the local count and peer serviced counts for the LUN22(refs.54and58ofFIG. 7) are obtained, then these are used to compute a LUN redirected count (which is the difference between the peer count and the local count). Note that the LUN redirected count may be negative, indicating that more local requests were processed than redirected requests over the most recent interval. The LUN redirected count is then added to an accumulated domain redirected count, which is the sum of the LUN redirected counts of all LUNs22sharing the target file system20.

At78, it is determined whether the domain redirected count is either above a threshold (indicated as “high”) or is increasing above some threshold rate (indicated as “increasing”). These thresholds are set so that they will be exceeded when there is a trend of storage requests being received at a non-preferred SP10. Exact values for these thresholds will depend on a variety of factors, including the length of the checking interval established by the check threshold. The “high” threshold is a value compared to the accumulated domain redirected count computed at76. The check for an increasing count may be done in a variety of ways. For example, in each check interval an indicator might be set if the count has increased by more than some predetermined amount over the interval, and the values of these indicators are saved over a recent succession of check intervals. The “increasing” threshold may specify a minimum number of these successive check intervals for which this indicator must be set to satisfy the “increasing” test.

As indicated above, the check threshold might be any of several types of values. Examples include:Time based (e.g. check every 5 minutes)I/O bandwidth based (e.g. check every 64 MB worth of reads/writes)Count on the number of I/Os received (e.g. check every 64K I/Os received)

If at78it is determined that the amount of redirection is too great or increasing too much, then at80a process of transferring ownership of the file system20is initiated. This procedure is described below with reference toFIG. 9. Upon initiation of the transfer at80, or in the event that the test at78is not satisfied, then the procedure ofFIG. 8ends at82.

FIG. 9illustrates the process by which the ownership of the file system20is transferred upon initiation by the procedure ofFIG. 8, which begins at90. At92, the owning SP10identifies all the LUNs22that share the file system20, and at94it sets the “Preferred” indicator for each LUN22to FALSE, indicating that this SP10is not the preferred SP10for receiving storage requests for these LUNs22. At96, the owning SP10dismounts the file system20and sends a message to the peer SP10indicating that it is to assume ownership of the file system20. Although not shown inFIG. 9, the peer SP10responds by mounting the file system20and setting the “Preferred” indicator for each of the subject LUNs22to TRUE, indicating that the peer SP10is now the preferred SP10for receiving storage requests for these LUNs22. Once the message is sent at96, the procedure ends at98.

Upon completion of the process ofFIG. 9, subsequent storage requests sent to the new owning SP10are processed locally rather than being redirected to the SP10that previously owned the SP10, so that the amount of redirection occurring in the storage system is reduced.