System and method for managing data portion ownership in a storage cluster

A method, computer program product, and computing system for dividing a logical address space into a first and at least a second set of data portions. Exclusive ownership of the first set may be assigned to a first storage node of a storage cluster. Exclusive ownership of the second set may be assigned to at least a second storage node of the storage cluster. One or more IO requests associated with one or more of the first set of data portions and the at least a second set of data portions may be processed without transferring data portions between the first the at least a second storage node based upon the exclusive ownership of the first set of data portions by the first storage node and the exclusive ownership of the at least a second set of data portions by the at least a second storage node.

BACKGROUND

Storing and safeguarding electronic content may be beneficial in modern business and elsewhere. Accordingly, various methodologies may be employed to protect and distribute such electronic content.

For example, consider a multi-node storage cluster where nodes are connected via a network link. Normally, both nodes serve IO requests, but if one node fails, the second storage node continues service, thus providing cluster high availability. The main challenge of this configuration is that both nodes generally should have access to the same metadata that maps from logical address (defined in IO) to a target data block. With conventional approaches, that means that the storage cluster should either enforce global/internode synchronization/locking mechanism, or engage asymmetric logical unit access (ALUA)-based approach (i.e., maintain volume ownership per node and configure the client multipath agent so that each IO request is always (or preferably) routed to the owner node).

The first option is very complex, has low reliability, and has a negative impact on cluster performance. The second, complicates cluster deployment and management for the client, and is unable to provide node load balancing. This means that the storage cluster suffers from overall performance degradation.

Summary of Disclosure

In one example implementation, a computer-implemented method executed on a computing device may include, but is not limited to, dividing a logical address space into a first set of data portions and at least a second set of data portions. Exclusive ownership of the first set of data portions may be assigned to a first storage node of a storage cluster. Exclusive ownership of the at least a second set of data portions may be assigned to at least a second storage node of the storage cluster. One or more IO requests associated with one or more of the first set of data portions and the at least a second set of data portions may be processed without transferring data portions between the first storage node and the at least a second storage node based upon, at least in part, the exclusive ownership of the first set of data portions by the first storage node and the exclusive ownership of the at least a second set of data portions by the at least a second storage node.

One or more of the following example features may be included. Non-exclusive ownership of one or more metadata objects associated with the first set of data portions and the at least a second set of data portions may be assigned based upon, at least in part, a type of IO request of the one or more IO requests. The one or more metadata objects associated with the first set of data portions and the at least a second set of data portions include one or more mapper metadata tree objects and one or more virtual layer blocks. Processing the one or more IO requests associated with one or more of the first set of data portions and the at least a second set of data portions may include: determining an exclusive ownership storage node associated with the one or more IO requests; and processing the IO request on the exclusive ownership storage node associated with the data portion without transferring data portions between the exclusive ownership storage node and the other storage nodes. Processing the IO request on the exclusive ownership storage node associated with the data portion may include processing an IO read request, wherein processing the IO read request may include: receiving the IO read request on the first storage node; determining that the exclusive ownership storage node associated with the data portion is a second storage node; sending a request to the second storage node to map the data portion to a virtual layer block; receiving a response from the second storage node with a virtual layer block address; and reading the data portion using the first storage node from a physical layer block associated with the virtual layer block address. Processing the IO request on the exclusive ownership storage node associated with the data portion may include processing an IO write request, wherein processing the IO write request may include: receiving the IO write request on the first storage node; determining that the exclusive owner node associated with the data portion is a second storage node; writing content for the data portion to a log memory system; sending a request to the second storage node with information concerning the IO write request; receiving a response from the second storage node acknowledging the IO write request; and sending an acknowledgement signal from the first storage node. Processing the IO request on the exclusive ownership storage node associated with the data portion may include flushing the data portion, wherein flushing the data portion may include: writing, from the first storage node, the data portion to a physical layer block; generating a virtual layer block corresponding to the physical layer block; updating a mapper metadata tree with information concerning the flushing of the data portion; sending a request to a second storage node with information concerning the flushing of the data portion; and receiving a response from the second storage node acknowledging the flushing of the data portion.

In another example implementation, a computer program product resides on a computer readable medium that has a plurality of instructions stored on it. When executed by a processor, the instructions cause the processor to perform operations that may include, but are not limited to, dividing a logical address space into a first set of data portions and at least a second set of data portions. Exclusive ownership of the first set of data portions may be assigned to a first storage node of a storage cluster. Exclusive ownership of the at least a second set of data portions may be assigned to at least a second storage node of the storage cluster. One or more IO requests associated with one or more of the first set of data portions and the at least a second set of data portions may be processed without transferring data portions between the first storage node and the at least a second storage node based upon, at least in part, the exclusive ownership of the first set of data portions by the first storage node and the exclusive ownership of the at least a second set of data portions by the at least a second storage node.

One or more of the following example features may be included. Non-exclusive ownership of one or more metadata objects associated with the first set of data portions and the at least a second set of data portions may be assigned based upon, at least in part, a type of IO request of the one or more IO requests. The one or more metadata objects associated with the first set of data portions and the at least a second set of data portions include one or more mapper metadata tree objects and one or more virtual layer blocks. Processing the one or more IO requests associated with one or more of the first set of data portions and the at least a second set of data portions may include: determining an exclusive ownership storage node associated with the one or more IO requests; and processing the IO request on the exclusive ownership storage node associated with the data portion without transferring data portions between the exclusive ownership storage node and the other storage nodes. Processing the IO request on the exclusive ownership storage node associated with the data portion may include processing an IO read request, wherein processing the IO read request may include: receiving the IO read request on the first storage node; determining that the exclusive ownership storage node associated with the data portion is a second storage node; sending a request to the second storage node to map the data portion to a virtual layer block; receiving a response from the second storage node with a virtual layer block address; and reading the data portion using the first storage node from a physical layer block associated with the virtual layer block address. Processing the IO request on the exclusive ownership storage node associated with the data portion may include processing an IO write request, wherein processing the IO write request may include: receiving the IO write request on the first storage node; determining that the exclusive owner node associated with the data portion is a second storage node; writing content for the data portion to a log memory system; sending a request to the second storage node with information concerning the IO write request; receiving a response from the second storage node acknowledging the IO write request; and sending an acknowledgement signal from the first storage node. Processing the IO request on the exclusive ownership storage node associated with the data portion may include flushing the data portion, wherein flushing the data portion may include: writing, from the first storage node, the data portion to a physical layer block; generating a virtual layer block corresponding to the physical layer block; updating a mapper metadata tree with information concerning the flushing of the data portion; sending a request to a second storage node with information concerning the flushing of the data portion; and receiving a response from the second storage node acknowledging the flushing of the data portion.

In another example implementation, a computing system includes at least one processor and at least one memory architecture coupled with the at least one processor, wherein the at least one processor is configured to divide a logical address space into a first set of data portions and at least a second set of data portions. Exclusive ownership of the first set of data portions may be assigned to a first storage node of a storage cluster. Exclusive ownership of the at least a second set of data portions may be assigned to at least a second storage node of the storage cluster. One or more IO requests associated with one or more of the first set of data portions and the at least a second set of data portions may be processed without transferring data portions between the first storage node and the at least a second storage node based upon, at least in part, the exclusive ownership of the first set of data portions by the first storage node and the exclusive ownership of the at least a second set of data portions by the at least a second storage node.

One or more of the following example features may be included. Non-exclusive ownership of one or more metadata objects associated with the first set of data portions and the at least a second set of data portions may be assigned based upon, at least in part, a type of IO request of the one or more IO requests. The one or more metadata objects associated with the first set of data portions and the at least a second set of data portions include one or more mapper metadata tree objects and one or more virtual layer blocks. Processing the one or more IO requests associated with one or more of the first set of data portions and the at least a second set of data portions may include: determining an exclusive ownership storage node associated with the one or more IO requests; and processing the IO request on the exclusive ownership storage node associated with the data portion without transferring data portions between the exclusive ownership storage node and the other storage nodes. Processing the IO request on the exclusive ownership storage node associated with the data portion may include processing an IO read request, wherein processing the IO read request may include: receiving the IO read request on the first storage node; determining that the exclusive ownership storage node associated with the data portion is a second storage node; sending a request to the second storage node to map the data portion to a virtual layer block; receiving a response from the second storage node with a virtual layer block address; and reading the data portion using the first storage node from a physical layer block associated with the virtual layer block address. Processing the IO request on the exclusive ownership storage node associated with the data portion may include processing an IO write request, wherein processing the IO write request may include: receiving the IO write request on the first storage node; determining that the exclusive owner node associated with the data portion is a second storage node; writing content for the data portion to a log memory system; sending a request to the second storage node with information concerning the IO write request; receiving a response from the second storage node acknowledging the IO write request; and sending an acknowledgement signal from the first storage node. Processing the IO request on the exclusive ownership storage node associated with the data portion may include flushing the data portion, wherein flushing the data portion may include: writing, from the first storage node, the data portion to a physical layer block; generating a virtual layer block corresponding to the physical layer block; updating a mapper metadata tree with information concerning the flushing of the data portion; sending a request to a second storage node with information concerning the flushing of the data portion; and receiving a response from the second storage node acknowledging the flushing of the data portion.

DETAILED DESCRIPTION

Referring toFIG.1, there is shown data ownership process10that may reside on and may be executed by storage system12, which may be connected to network14(e.g., the Internet or a local area network). Examples of storage system12may include, but are not limited to: a Network Attached Storage (NAS) system, a Storage Area Network (SAN), a personal computer with a memory system, a server computer with a memory system, and a cloud-based device with a memory system.

As is known in the art, a SAN may include one or more of a personal computer, a server computer, a series of server computers, a mini computer, a mainframe computer, a RAID device and a NAS system. The various components of storage system12may execute one or more operating systems, examples of which may include but are not limited to: Microsoft® Windows®; Mac® OS X®; Red Hat® Linux®, Windows® Mobile, Chrome OS, Blackberry OS, Fire OS, or a custom operating system. (Microsoft and Windows are registered trademarks of Microsoft Corporation in the United States, other countries or both; Mac and OS X are registered trademarks of Apple Inc. in the United States, other countries or both; Red Hat is a registered trademark of Red Hat Corporation in the United States, other countries or both; and Linux is a registered trademark of Linus Torvalds in the United States, other countries or both).

The instruction sets and subroutines of data ownership process10, which may be stored on storage device16included within storage system12, may be executed by one or more processors (not shown) and one or more memory architectures (not shown) included within storage system12. Storage device16may include but is not limited to: a hard disk drive; a tape drive; an optical drive; a RAID device; a random access memory (RAM); a read-only memory (ROM); and all forms of flash memory storage devices. Additionally/alternatively, some portions of the instruction sets and subroutines of data ownership process10may be stored on storage devices (and/or executed by processors and memory architectures) that are external to storage system12.

Various IO requests (e.g. IO request20) may be sent from client applications22,24,26,28to storage system12. Examples of IO request20may include but are not limited to data write requests (e.g., a request that content be written to storage system12) and data read requests (e.g., a request that content be read from storage system12).

The instruction sets and subroutines of client applications22,24,26,28, which may be stored on storage devices30,32,34,36(respectively) coupled to client electronic devices38,40,42,44(respectively), may be executed by one or more processors (not shown) and one or more memory architectures (not shown) incorporated into client electronic devices38,40,42,44(respectively). Storage devices30,32,34,36may include but are not limited to: hard disk drives; tape drives; optical drives; RAID devices; random access memories (RAM); read-only memories (ROM), and all forms of flash memory storage devices. Examples of client electronic devices38,40,42,44may include, but are not limited to, personal computer38, laptop computer40, smartphone42, notebook computer44, a server (not shown), a data-enabled, cellular telephone (not shown), and a dedicated network device (not shown).

Users46,48,50,52may access storage system12directly through network14or through secondary network18. Further, storage system12may be connected to network14through secondary network18, as illustrated with link line54.

Client electronic devices38,40,42,44may each execute an operating system, examples of which may include but are not limited to Microsoft® Windows®; Mac® OS X®; Red Hat® Linux®, Windows® Mobile, Chrome OS, Blackberry OS, Fire OS, or a custom operating system. (Microsoft and Windows are registered trademarks of Microsoft Corporation in the United States, other countries or both; Mac and OS X are registered trademarks of Apple Inc. in the United States, other countries or both; Red Hat is a registered trademark of Red Hat Corporation in the United States, other countries or both; and Linux is a registered trademark of Linus Torvalds in the United States, other countries or both).

In some implementations, as will be discussed below in greater detail, a data ownership process, such as data ownership process10ofFIG.1, may include but is not limited to, dividing a logical address space into a first set of data portions and at least a second set of data portions. Exclusive ownership of the first set of data portions may be assigned to a first storage node of a storage cluster. Exclusive ownership of the at least a second set of data portions may be assigned to at least a second storage node of the storage cluster. One or more IO requests associated with one or more of the first set of data portions and the at least a second set of data portions may be processed without transferring data portions between the first storage node and the at least a second storage node based upon, at least in part, the exclusive ownership of the first set of data portions by the first storage node and the exclusive ownership of the at least a second set of data portions by the at least a second storage node.

For example purposes only, storage system12will be described as being a network-based storage system that includes a plurality of electro-mechanical backend storage devices. However, this is for example purposes only and is not intended to be a limitation of this disclosure, as other configurations are possible and are considered to be within the scope of this disclosure.

The Storage System:

Referring also toFIG.2, storage system12may include storage processor100and a plurality of storage targets T 1-n (e.g., storage targets102,104,106,108). Storage targets102,104,106,108may be configured to provide various levels of performance and/or high availability. For example, one or more of storage targets102,104,106,108may be configured as a RAID 0 array, in which data is striped across storage targets. By striping data across a plurality of storage targets, improved performance may be realized. However, RAID 0 arrays do not provide a level of high availability. Accordingly, one or more of storage targets102,104,106,108may be configured as a RAID 1 array, in which data is mirrored between storage targets. By mirroring data between storage targets, a level of high availability is achieved as multiple copies of the data are stored within storage system12.

While storage targets102,104,106,108are discussed above as being configured in a RAID 0 or RAID 1 array, this is for example purposes only and is not intended to be a limitation of this disclosure, as other configurations are possible. For example, storage targets102,104,106,108may be configured as a RAID 3, RAID 4, RAID 5 or RAID 6 array.

While in this particular example, storage system12is shown to include four storage targets (e.g. storage targets102,104,106,108), this is for example purposes only and is not intended to be a limitation of this disclosure. Specifically, the actual number of storage targets may be increased or decreased depending upon e.g., the level of redundancy/performance/capacity required.

Storage system12may also include one or more coded targets110. As is known in the art, a coded target may be used to store coded data that may allow for the regeneration of data lost/corrupted on one or more of storage targets102,104,106,108. An example of such a coded target may include but is not limited to a hard disk drive that is used to store parity data within a RAID array.

While in this particular example, storage system12is shown to include one coded target (e.g., coded target110), this is for example purposes only and is not intended to be a limitation of this disclosure. Specifically, the actual number of coded targets may be increased or decreased depending upon e.g. the level of redundancy/performance/capacity required.

Examples of storage targets102,104,106,108and coded target110may include one or more electro-mechanical hard disk drives and/or solid-state/flash devices, wherein a combination of storage targets102,104,106,108and coded target110and processing/control systems (not shown) may form data array112.

The manner in which storage system12is implemented may vary depending upon e.g. the level of redundancy/performance/capacity required. For example, storage system12may be a RAID device in which storage processor100is a RAID controller card and storage targets102,104,106,108and/or coded target110are individual “hot-swappable” hard disk drives. Another example of such a RAID device may include but is not limited to an NAS device. Alternatively, storage system12may be configured as a SAN, in which storage processor100may be e.g., a server computer and each of storage targets102,104,106,108and/or coded target110may be a RAID device and/or computer-based hard disk drives. Further still, one or more of storage targets102,104,106,108and/or coded target110may be a SAN.

In the event that storage system12is configured as a SAN, the various components of storage system12(e.g. storage processor100, storage targets102,104,106,108, and coded target110) may be coupled using network infrastructure114, examples of which may include but are not limited to an Ethernet (e.g., Layer2or Layer3) network, a fiber channel network, an InfiniBand network, or any other circuit switched/packet switched network.

Storage system12may execute all or a portion of data ownership process10. The instruction sets and subroutines of data ownership process10, which may be stored on a storage device (e.g., storage device16) coupled to storage processor100, may be executed by one or more processors (not shown) and one or more memory architectures (not shown) included within storage processor100. Storage device16may include but is not limited to: a hard disk drive; a tape drive; an optical drive; a RAID device; a random access memory (RAM); a read-only memory (ROM); and all forms of flash memory storage devices. As discussed above, some portions of the instruction sets and subroutines of data ownership process10may be stored on storage devices (and/or executed by processors and memory architectures) that are external to storage system12.

As discussed above, various IO requests (e.g. IO request20) may be generated. For example, these IO requests may be sent from client applications22,24,26,28to storage system12. Additionally/alternatively and when storage processor100is configured as an application server, these IO requests may be internally generated within storage processor100. Examples of IO request20may include but are not limited to data write request116(e.g., a request that content118be written to storage system12) and data read request120(i.e. a request that content118be read from storage system12).

During operation of storage processor100, content118to be written to storage system12may be processed by storage processor100. Additionally/alternatively and when storage processor100is configured as an application server, content118to be written to storage system12may be internally generated by storage processor100.

Storage processor100may include frontend cache memory system122. Examples of frontend cache memory system122may include but are not limited to a volatile, solid-state, cache memory system (e.g., a dynamic RAM cache memory system) and/or a non-volatile, solid-state, cache memory system (e.g., a flash-based, cache memory system).

Storage processor100may initially store content118within frontend cache memory system122. Depending upon the manner in which frontend cache memory system122is configured, storage processor100may immediately write content118to data array112(if frontend cache memory system122is configured as a write-through cache) or may subsequently write content118to data array112(if frontend cache memory system122is configured as a write-back cache).

Data array112may include backend cache memory system124. Examples of backend cache memory system124may include but are not limited to a volatile, solid-state, cache memory system (e.g., a dynamic RAM cache memory system) and/or a non-volatile, solid-state, cache memory system (e.g., a flash-based, cache memory system). During operation of data array112, content118to be written to data array112may be received from storage processor100. Data array112may initially store content118within backend cache memory system124prior to being stored on e.g. one or more of storage targets102,104,106,108, and coded target110.

As discussed above, the instruction sets and subroutines of data ownership process10, which may be stored on storage device16included within storage system12, may be executed by one or more processors (not shown) and one or more memory architectures (not shown) included within storage system12. Accordingly, in addition to being executed on storage processor100, some or all of the instruction sets and subroutines of data ownership process10may be executed by one or more processors (not shown) and one or more memory architectures (not shown) included within data array112.

Further and as discussed above, during the operation of data array112, content (e.g., content118) to be written to data array112may be received from storage processor100and initially stored within backend cache memory system124prior to being stored on e.g. one or more of storage targets102,104,106,108,110. Accordingly, during use of data array112, backend cache memory system124may be populated (e.g., warmed) and, therefore, subsequent read requests may be satisfied by backend cache memory system124(e.g., if the content requested in the read request is present within backend cache memory system124), thus avoiding the need to obtain the content from storage targets102,104,106,108,110(which would typically be slower).

In some implementations, storage system12may include multi-node active/active storage clusters configured to provide high availability to a user. As is known in the art, the term “high availability” may generally refer to systems or components that are durable and likely to operate continuously without failure for a long time. For example, an active/active storage cluster may be made up of at least two nodes (e.g., storage processors100,126), both actively running the same kind of service(s) simultaneously. One purpose of an active-active cluster may be to achieve load balancing. Load balancing may distribute workloads across all nodes in order to prevent any single node from getting overloaded. Because there are more nodes available to serve, there will also be a marked improvement in throughput and response times. Another purpose of an active-active cluster may be to provide at least one active node in the event that one of the nodes in the active-active cluster fails.

In some implementations, storage processor126may function like storage processor100. For example, during operation of storage processor126, content118to be written to storage system12may be processed by storage processor126. Additionally/alternatively and when storage processor126is configured as an application server, content118to be written to storage system12may be internally generated by storage processor126.

Storage processor126may include frontend cache memory system128. Examples of frontend cache memory system128may include but are not limited to a volatile, solid-state, cache memory system (e.g., a dynamic RAM cache memory system) and/or a non-volatile, solid-state, cache memory system (e.g., a flash-based, cache memory system).

Storage processor126may initially store content118within frontend cache memory system126. Depending upon the manner in which frontend cache memory system128is configured, storage processor126may immediately write content118to data array112(if frontend cache memory system128is configured as a write-through cache) or may subsequently write content118to data array112(if frontend cache memory system128is configured as a write-back cache).

In some implementations, the instruction sets and subroutines of data ownership process10, which may be stored on storage device16included within storage system12, may be executed by one or more processors (not shown) and one or more memory architectures (not shown) included within storage system12. Accordingly, in addition to being executed on storage processor126, some or all of the instruction sets and subroutines of data ownership process10may be executed by one or more processors (not shown) and one or more memory architectures (not shown) included within data array112.

Further and as discussed above, during the operation of data array112, content (e.g., content118) to be written to data array112may be received from storage processor126and initially stored within backend cache memory system124prior to being stored on e.g. one or more of storage targets102,104,106,108,110. Accordingly, during use of data array112, backend cache memory system124may be populated (e.g., warmed) and, therefore, subsequent read requests may be satisfied by backend cache memory system124(e.g., if the content requested in the read request is present within backend cache memory system124), thus avoiding the need to obtain the content from storage targets102,104,106,108,110(which would typically be slower).

As discussed above, storage processor100and storage processor126may be configured in an active/active configuration where processing of data by one storage processor may be synchronized to the other storage processor. For example, data may be synchronized between each storage processor via a separate link or connection (e.g., connection130).

In the context of storage systems, metadata may generally include useful internal information managed by a storage array to describe and locate user data. All modern arrays abstract the physical media and present logical (virtualized) addresses to clients in the form of LUNs. The mapping between the logical address and physical address is a form of metadata that the array needs to manage. That is typically the most common form of metadata for SAN storage systems. Newer architectures manage additional metadata to implement additional capabilities. For example, snapshots, change tracking for efficient remote replication, deduplication pointers, and compression all involve managing some form of metadata.

The classic metadata structure of traditional storage systems directly links a Logical Address of a Block to the Physical Location of the Block. In this metadata structure, every logical block written, has a physical block linked directly to it. In addition, as most traditional storage systems were architected for a spinning disk storage medium optimized for sequential writes the address of the logical address affects the physical location that the data is stored. This can lead to an unbalanced storage array that can suffer from hot-spots as specific address space ranges may experience more performance/input-output operations per second (IOPs) than other address space ranges.

Embodiments of the present disclosure may support a flash/random access medium. For example, embodiments of the present disclosure may include a metadata structure that completely decouples the Logical Block Address space address from the physical one. This is done by leveraging a multi-layer architecture.

Referring also toFIG.3, a storage system may generally include a mapper layer which is structured as a file system with various layers of pages and blocks. In some implementations, the combination of various metadata layers mapper layers may be referred to as a mapper metadata tree. While the following example includes metadata “blocks”, it will be appreciated that other units of data storage may be used within the scope of the present disclosure. In some implementations, a top-level mapping page layer (e.g., top-level mapping page layer300) may include top-level mapping page pages (e.g., top-level mapping page302) with a plurality of entries (e.g., plurality of entries304) that map or point to a plurality of entries of one or more mid-level mapping pages. A mid-level mapping page layer (e.g., mid-level mapping page layer306) may include mid-level mapping page pages (e.g., mid-level mapping page308) with a plurality of entries (e.g., plurality of entries310) that map or point to a plurality of entries of one or more leaf mapping pages. A leaf mapping page layer (e.g., leaf mapping page layer312) may include leaf mapping page pages (e.g., leaf mapping page314) with a plurality of entries (e.g., plurality of entries316) that map or point to a plurality of entries of one or more virtual layer blocks. Leaf mapping page layer312may represent various ranges of Logical Block Addresses (LBAs). For example, each entry of the plurality of entries (e.g., plurality of entries316) of the leaf mapping page (e.g., leaf mapping page314) may be associated with a LBA range. In some implementations, the combination of top-level mapping page layer300, mid-level mapping page layer306, and leaf mapping page layer312may be organized in a “tree” data structure where each leaf mapping page is a “leaf” of the “tree” data structure that corresponds to a specific LBA range. Accordingly, each leaf mapping page (e.g., leaf mapping page314) may hold mapping of a LBA to a virtual layer block. It will be appreciated that other data structures may be used within the scope of the present disclosure to organize the first layer.

In some implementations, a virtual layer block layer (e.g., second layer318) may include virtual layer blocks (e.g., virtual layer block320) with a plurality of entries (e.g., plurality of entries322) that map to a plurality of entries of one or more physical data blocks. The virtual layer block layer (e.g., virtual layer block layer318) may generally isolate the logical address of a block from the physical location of the block. For example, a virtual layer block (e.g., virtual layer block308) may encapsulate the physical location of user data and allow relocation without updating leaf mapping pages (e.g., leaf mapping page314). Accordingly, the virtual layer block layer (e.g., virtual layer block layer318) may decouple the Logical Block Address space address from the physical one.

In some implementations, a physical data block layer (e.g., physical data block layer324) may include physical data blocks (e.g., physical data block326) with a plurality of entries or portions (e.g., plurality of entries328) that are configured to store user data. In this manner, physical data block layer324may describe the physical location of user data in a storage system. In some implementations, each physical data block (e.g., physical data block326) may have a predefined amount of storage capacity for storing data (e.g., user data).

The Data Ownership Process:

Referring also toFIGS.4A-8and in some implementations, data ownership process10may divide400a logical address space into a first set of data portions and at least a second set of data portions. Exclusive ownership of the first set of data portions may be assigned402to a first storage node of a storage cluster. Exclusive ownership of the at least a second set of data portions may be assigned404to at least a second storage node of the storage cluster. One or more IO requests associated with one or more of the first set of data portions and the at least a second set of data portions may be processed406without transferring data portions between the first storage node and the at least a second storage node based upon, at least in part, the exclusive ownership of the first set of data portions by the first storage node and the exclusive ownership of the at least a second set of data portions by the at least a second storage node.

In some implementations, data ownership process10may allow for management of data portions in a storage cluster without requiring global or cross-storage cluster locking mechanisms. For example, consider a multi-node storage cluster where nodes are connected via a network link. Normally, both nodes serve IO requests, but if one node fails, the second storage node continues service, thus providing cluster high availability. The main challenge of this configuration is that both nodes generally should have access to the same metadata that maps from logical address (defined in IO) to a target data block. With conventional approaches, the storage cluster should either enforce global/internode synchronization/locking mechanism, or engage asymmetric logical unit access (ALUA)-based approach (i.e., maintain volume ownership per node and configure the client multipath agent so that each IO request is always (or preferably) routed to the owner node).

The first option is very complex, has low reliability, and has a negative impact on storage cluster performance. The second, complicates storage cluster deployment and management for the client, and is unable to provide node load balancing. This means that the storage cluster suffers from overall performance degradation. As will be discussed in greater detail below, implementations of the present disclosure may provide exclusive or “strong” ownership of particular data portions to specific nodes in the storage cluster; provide non-exclusive or “weak” ownership for metadata objects associated with the data portions; and manage the processing of IO requests received on each storage node without transferring data portions between the storage nodes.

In some implementations, data ownership process10may divide400a logical address space into a first set of data portions and at least a second set of data portions. For example, storage system12may include a plurality of storage objects configured to store user data (i.e., data stored by and accessed by users of storage system12). Storage objects may generally include any container or storage unit configured to store data within a storage system (e.g., storage system12). For example, a storage object may be any one of the following: a volume (aka Logical Unit Number (LUN)), a file, or parts thereof that may be defined e.g. by offsets or address ranges (e.g., sub-LUNs, disk extents, and/or slices). Referring also toFIG.5and in some implementations, storage system12may include a plurality of storage objects (e.g., storage object500) that is accessible to both storage nodes of the storage cluster (e.g., storage nodes100,126). In this example, storage object500may include a plurality of logical data portions represented by logical address space (e.g., logical address space502). Logical address space is an abstracted, logical representation of storage space within the storage system.

For example, suppose storage object500is a volume or slice with a predefined amount of storage space. Storage object500may be organized or defined by a total amount of logical address space502where various portions of the logical address space may be separately addressable. For example and in some implementations, logical address space502may include a plurality of data portions (e.g., data portions504,506,508,510,512,514). Each data portion may include a predefined amount of storage capacity (e.g., two megabytes). For example, data portion504may include a capacity of e.g., two megabytes with a logical address of 0 megabytes to two megabytes. Similarly, data portion504may include two megabytes of logical address space with a logical address of 2 megabytes to four megabytes. In this manner, each of data portions504,506,508,510,512,514may be separately accessible. In some implementations, each data portion may be referred to as a “UXLA” or user data logical address where each data portion is defined by the combination of a storage object identifier and an address offset for the data portion within the storage object. For example, data portion504may include a UXLA defined by an identifier (e.g., name or assigned number) for storage object500and the offset within storage object500representing the logical address space that data portion504defines.

In some implementations, data ownership process10may divide400logical address space502into a first set of data portions (e.g., data portions504,508,512(shown inFIG.5with shading)) and at least a second set of data portions (e.g., data portions506,510,514(shown inFIG.5without shading)). As will be discussed in greater detail below, each set of data portions may correspond to a particular storage node within the storage cluster. In some implementations, dividing400logical address space502into two sets may include defining or receiving a data portion size and/or a total amount of data portions for each set. In one example, data ownership process10may divide logical address space502equally into the two sets of data portions. In another example, data ownership process10may divide logical address space502unequally into the two sets of data portions (i.e., where one set of data portions has more or fewer data portions). In some implementations, the amount of logical address space for each data portion may be equal for each set of data portions. For example and as shown inFIG.5, even logical address spaces are divided into the first set of data portions while odd logical address spaces are divided into the second set of data portions. However, data ownership process10may divide400logical address space502into any number of sets of data portions (i.e., at least a second data portion) for any number of storage nodes in the storage cluster. For example, for three storage nodes, data ownership process10may divide400logical address space502into three set of data portions. As such, it will be appreciated that any number of sets of data portions may be defined by dividing400logical address space502for any corresponding number of storage nodes in the storage cluster.

In some implementations, data ownership process10may assign402exclusive ownership of the first set of data portions to a first storage node of a storage cluster. Exclusive ownership may generally include a strong ownership or include the exclusive locking of data portions by a particular storage node within the first set of data portions that is not shared with other storage nodes. In this manner, a storage node assigned with exclusive ownership over a data portion may not yield access to that data portion to another storage node. Referring again toFIG.5, data ownership process10may assign402exclusive ownership of the first set of data portions (e.g., data portions504,508,512) to storage node100. In some implementations, data ownership process10may maintain a listing of exclusive ownership assignments for each data portion. In another example, the assignment may be derivable from the data portion itself. For example, data ownership process10may assign all even addressed data portions to one storage node. In this example, data ownership process10may determine which storage node has exclusive ownership over a particular data portion by determining whether the address of the data portion is odd or even.

In some implementations, data ownership process10may assign404exclusive ownership of the at least a second set of data portions to at least a second storage node of the storage cluster. Referring again toFIG.5, data ownership process10may assign404exclusive ownership of the second set of data portions (e.g., data portions506,510,514) to storage node126. In some implementations, the assigning of exclusive ownership of the second set of data portions may similarly be derivable from the data portion itself. For example, data ownership process10may assign all odd addressed data portions to one storage node. In this example, data ownership process10may determine which storage node has exclusive ownership over a particular data portion by determining whether the address of the data portion is odd or even. As noted above, the storage cluster may include any number of storage nodes and any number of corresponding sets of data portions within the scope of the present disclosure.

In some implementations, data ownership process10may assign408non-exclusive ownership of one or more metadata objects associated with the first set of data portions and the at least a second set of data portions based upon, at least in part, a type of IO request of the one or more IO requests. A non-exclusive ownership assignment may generally include a “weak” ownership assignment that may be shared by each storage node in the storage cluster. For example, data ownership process10may allow metadata objects that reference each set of data portions to be shared or assigned to a particular storage node with a non-exclusive ownership based upon, at least in part, a type of IO request being processed or other operation being performed on the metadata object.

In some implementations, the one or more metadata objects associated with the first set of data portions and the at least a second set of data portions may include one or more mapper metadata tree objects and one or more virtual layer blocks. For example, a metadata object may generally include a non-physical data container that is used to map the data portions to other non-physical data containers or physical data containers or blocks. As discussed above concerningFIG.3, data ownership process10may use various layers of metadata objects to abstract physical blocks or other portions of physical storage space into various logical representations for various purposes. For example, the one or more metadata objects may include mapper metadata tree objects such as top-level mapping page302, mid-level mapping page308, leaf mapping page314, and/or virtual layer block320.

Referring again toFIG.5, each data portion of logical address space502may map to or be associated with one or more metadata objects. For example, data portion504may map to or be associated with a combination of top-level mapping pages, mid-level mapping pages, and/or leaf mapping pages represented inFIG.5by mapper metadata tree objects516,518,520,522,524of mapper metadata tree526. Similarly, data portion504may map to mapper metadata tree object516; data portion506may map to mapper metadata tree object518; data portion508may map to mapper metadata tree object520; data portion510may map to mapper metadata tree object522; and data portion514may map to mapper metadata tree object524. In some implementations, each storage node may maintain its own mapper tree.

As will be described in greater detail below, mapper metadata tree object516may map to virtual layer block526; mapper metadata tree object518may map to virtual layer block528; mapper metadata tree object520may map to virtual layer block530; mapper metadata tree object522may map to virtual layer block532; and mapper metadata tree object524may map to virtual layer block534. Additionally, virtual layer block526may map to or point to physical layer block536; virtual layer block528may map to or point to physical layer block538; virtual layer block530may map to or point to physical layer block540; virtual layer block532may map to or point to physical layer block542; and virtual layer block534may map to or point to physical layer block544.

In some implementations, data ownership process10may assign408non-exclusive or weak ownership for the top-level mapping pages, mid-level mapping pages, leaf mapping pages, and/or virtual layer blocks based upon, at least in part, the type of IO request or other operation being performed. For example and as discussed above certain mapper tree objects may be associated with data portions assigned to a particular storage node with exclusive ownership. Accordingly, data ownership process10may assign non-exclusive ownership to storage nodes for these metadata objects depending upon the type of IO request. For example, for IO requests that read or provide updates to top-level mapping pages, mid-level mapping pages, leaf mapping pages, data ownership process10may assign non-exclusive ownership to the storage node for the mapper metadata tree objects that correspond to a data portion assigned to that storage node. In some implementations, non-exclusive ownership may be implemented with local locks on the mapper metadata tree objects during the processing of the read and/or update request. In another example, suppose an IO request includes reading, incrementing a reference count, or decrementing a reference count of a virtual layer block. In these examples, each storage node may be assigned non-exclusive ownership to these virtual layer blocks using local locks during processing. Accordingly, it will be appreciated that with exclusive ownership assigned to storage nodes for particular data portions and non-exclusive ownership assigned to storage nodes for particular mapper metadata tree objects and/or virtual layer blocks, data ownership process10may avoid global locks and the associated processing required to synchronize access to data portions, mapper metadata tree objects, and/or virtual layer blocks.

In some implementations, data ownership process10may process406one or more IO requests associated with one or more of the first set of data portions and the at least a second set of data portions without transferring data portions between the first storage node and the at least a second storage node based upon, at least in part, the exclusive ownership of the first set of data portions by the first storage node and the exclusive ownership of the at least a second set of data portions by the at least a second storage node. For example, when an IO request or other operation is received by or performed on the storage node that is owner of all the involved data portions (e.g., exclusive ownership assignment) and metadata objects, all the processing is done locally under local locks for the data portions and associated metadata objects. As such, no peer synchronization is required.

In another example, when an IO request is received by a storage node that is not owner of the corresponding data portion or some flow (flush, background operation, etc.) requires updating of metadata objects not owned by the storage node, data ownership process10may process406these IO requests without transferring data portions between the first storage node and the second storage node based upon, at least in part, the exclusive ownership of the first set of data portions by the first storage node and the exclusive ownership of the second set of data portions by the second storage node.

In some implementations, processing406the one or more IO requests associated with one or more of the first set of data portions and the at least a second set of data portions may include determining410an exclusive ownership storage node associated with the one or more IO requests; and processing412the IO request on the exclusive ownership storage node associated with the data portion without transferring data portions between the exclusive ownership storage node and the other ownership node(s). For example, with each IO request and/or other operation to be performed on data portions within the storage system, data ownership process10may determine410an “exclusive ownership node” (i.e., storage node that has been assigned exclusive ownership) for the relevant data portion(s). As will be discussed in greater detail below, data ownership process10may process412the one or more IO requests received at storage node without exclusive ownership of the relevant data portions without transferring the data portions between the exclusive ownership storage node and the other node.

In some implementations, processing412the IO request on the exclusive ownership storage node associated with the data portion may include processing an IO read request, wherein processing the IO read request includes: receiving414the IO read request on the first storage node; determining416that the exclusive ownership storage node associated with the data portion is a second storage node; sending418a request to the second storage node to map the data portion to a virtual layer block; receiving420a response from the second storage node with a virtual layer block address; and reading422the data portion using the first storage node from a physical layer block associated with the virtual layer block address. Referring also toFIG.6and in some implementations, suppose data ownership process10receives414an IO read request (e.g., IO read request600) from a host at storage node100. In this example, IO read request600may include a reference to a particular data portion (e.g., data portion510) of a specific storage object (e.g., storage object500). Receiving414IO read request600is shown inFIG.6as action “1”. Data ownership process10may determine416that the exclusive ownership storage node associated with data portion510is storage node126. For example, data ownership process10may search a listing of data portions and/or perform a calculation to determine which storage node is assigned exclusive ownership over data portion510. In this example, data portion510may be an evenly addressed data portion that is assigned, along with all other evenly addressed data portions, to storage node126.

In response to determining that storage node126is the exclusive ownership storage node for data portion510, data ownership process10may send418a request to storage node126to map data portion to a virtual layer block that is accessible to storage node100. In some implementations, data ownership process10may send a remote procedure call (RPC) request (e.g., RPC602) from storage node100to storage node126. As is known in the art, an RPC is when a computer program causes a procedure to execute in a different address space (i.e., on another storage node), which is coded as if it were a normal procedure call, without the programmer explicitly coding the details for the remote interaction. By using RPC requests, data ownership process10does not transmit the data portion (e.g., data portion510) between the storage nodes. Rather, storage node100is requesting that storage node126identify the location of the virtual layer block within the virtual layer block layer. This is represented inFIG.6as action “2”.

In response to receiving the request at storage node126, data ownership process10may, using storage node126, map data portion510to virtual layer block532to determine the address for virtual layer block532. This is represented inFIG.6as action “3” with one set of arrows from storage node126to virtual layer block532and a second set of arrows returning the virtual layer block address to storage node126. Data ownership process10may receive420a response from the second storage node with a virtual layer block address. For example, storage node126may send virtual layer block address604as a response to RPC602. This is shown inFIG.6as action “4”. With virtual layer block address604obtained from storage node126, storage node100may read422the data portion from a physical layer block associated with the virtual layer block address. For example, virtual layer block address604may reference virtual layer block532. Storage node100may read physical layer block542associated with virtual layer block532that includes the data from data portion510. This is shown inFIG.6as action “5”. In one example, data ownership process10may determine that the data of data portion510is stored in a log memory system (e.g., log memory system606). For example, log memory system606may be configured to log user data written to the storage cluster. Log memory system606may be accessible to both storage nodes and/or may be specific to storage node100with a corresponding, mirrored log memory system on storage node126. Accordingly, data ownership process10may read data portion510from log memory system606instead of from a physical layer block (e.g., physical layer block542). In response to reading422data portion510from physical layer block542or log memory system606, data ownership process10may return IO request600with content (e.g., content608).

In some implementations, processing412the IO request on the exclusive ownership storage node associated with the data portion may include processing an IO write request, wherein processing the IO write request includes: receiving424the IO write request on the first storage node; determining426that the exclusive owner node associated with the data portion is a second storage node; writing428content for the data portion to a log memory system; sending430a request to the second storage node with information concerning the IO write request; receiving432a response from the second storage node acknowledging the IO write request; and sending434an acknowledgement signal from the first storage node. Referring also toFIG.7and in some implementations, suppose data ownership process10receives424an IO write request (e.g., IO write request700) at storage node100. In this example, IO write request700may include a reference to write content to a particular data portion (e.g., data portion514) of a specific storage object (e.g., storage object500). Receiving424IO read request700is shown inFIG.7as action “1”. Data ownership process10may determine426that the exclusive ownership storage node associated with data portion514is storage node126. For example, data ownership process10may search a listing of data portions and/or perform a calculation to determine which storage node is assigned exclusive ownership over data portion514. In this example, data portion514may be an evenly addressed data portion that is assigned, along with all other evenly addressed data portions, to storage node126.

In response to determining that storage node126is the exclusive ownership storage node for data portion514, data ownership process10may write428content from IO write request700for the data portion to a log memory system (e.g., log memory system606). This is shown inFIG.7as action “2”. In response to writing428the content for data portion514to log memory system606, data ownership process10may send430a request to the second storage node with information concerning the IO write request. In some implementations, data ownership process10may send a remote procedure call (RPC) request (e.g., RPC702) from storage node100to storage node126. For example, RPC request702may include information concerning the IO write request associated with data portion514that is owned by storage node126. In some implementations, the information may describe the location of the data portion within log memory system606. This is represented inFIG.7as action “3”.

In response to receiving RPC request702, storage node126may update a log hash table (e.g., log hash table704) on storage node126with the information concerning the IO write request. For example, data ownership process10may update log hash table704with the location of the content for data portion514(i.e., the content written for data portion514) within log memory system606. This is shown inFIG.7as action “4”. As storage node126is assigned exclusive ownership over data portion514, storage node126may use this information to process content for data portion514from log memory system606.

In some implementations, data ownership process10may receive432a response from the second storage node acknowledging the IO write request. For example, in response to processing RPC request702on storage node126, storage node126may send a response (e.g., response706) acknowledging the IO write request700to storage node100. This is shown inFIG.7as action “5”. With response706from storage node126, storage node100may send434an acknowledgement signal (e.g., acknowledgement signal708) to the host that initially sent IO request700. This is shown inFIG.7as action “6”.

In some implementations, processing412the IO request on the exclusive ownership storage node associated with the data portion may include flushing the data portion, wherein flushing the data portion includes: writing436, from the first storage node, the data portion to a physical layer block; generating438a virtual layer block corresponding to the physical layer block; updating440a mapper metadata tree with information concerning the flushing of the data portion; sending442a request to the second storage node with information concerning the flushing of the data portion; and receiving444a response from a second storage node acknowledging the flushing of the data portion. In some implementations, data ownership process10may perform various internal or background operations using each storage node of the storage cluster. For example, data ownership process10may perform a flushing operation to flush data from a storage node to backend storage (e.g., storage array112). Referring also toFIG.8, suppose data ownership process10flushes a data portion from the first storage node by writing436the data portion (e.g., data portion514) to a physical layer block (e.g., physical layer block544). The physical layer block may be a partially utilized physical layer block or an empty physical layer block. This is shown inFIG.8as action “1”.

In response to writing436data portion514to physical layer block544, data ownership process10may generate438a virtual layer block (e.g., virtual layer block534) corresponding to physical layer block544. This is shown inFIG.8as action “2”. In response to generating438virtual layer block534, data ownership process10may update440a mapper metadata tree with information concerning the flushing of the data portion. For example, storage node100may update its mapper metadata tree with information concerning the flushing of data portion514to physical layer block544. This is shown inFIG.8as action “3”.

In some implementations, data ownership process10may send442a request (e.g., RPC request800) to storage node126with information concerning the flushing of data portion514. This is shown inFIG.8as action “4”. In response to sending442request800to storage node126, storage node126may process request800and update its mapper metadata tree with information concerning the flushing of data portion514. This is shown inFIG.8as action “5”. Data ownership process10may then receive, at storage node100, a response (e.g., RPC response802) from storage node126indicating that storage node126acknowledges the flushing of data portion514. This is shown inFIG.8as action “6”.