Active-passive configuration for synchronous remote replication in an active-active metro cluster

In one aspect, an example methodology implementing the disclosed techniques includes creating, by a first site of a volume that supports active-active bidirectional replication, a local copy of the volume, the local copy of the volume configured to be active. The method also includes enabling, by the first site of the volume, bidirectional write input/output (I/O) mirroring with a second site of the volume. The method further includes, by the second site of the volume, creating a remote copy of the volume, the remote copy of the volume configured to be passive, and enabling bidirectional write I/O mirroring with the first site of the volume.

BACKGROUND

Multi-site storage system configurations are becoming more common with the increasing use of public and private clouds. In such configurations, a multi-site storage system may include storage devices (e.g., storage arrays) into which data may be entered (e.g., written), and from which data may be obtained (e.g., read). These storage devices may be situated on the same physical location or in one or more physically different locations.

One particular example of a multi-site storage system is a metro cluster. In general, a metro cluster is a storage array deployment in which two storage arrays, typically deployed in two different data centers or two server rooms within the same data center, cooperate to expose a single “metro” storage resource (e.g., volume, volume group, or file system) to application hosts. Thus, the hosts and the applications running on the hosts perceive two storage resources hosted by the two storage arrays as a single storage resource. Examples of primary metro cluster use cases include high availability (HA) and disaster avoidance, resource balancing across data centers, and storage migration.

SUMMARY

In accordance with one illustrative embodiment provided to illustrate the broader concepts, systems, and techniques described herein, a method to configure a volume that supports active-active bidirectional replication in an active-passive mode includes, by a first site of the volume, creating a local copy of the volume, the local copy of the volume configured to be active, and enabling bidirectional write input/output (I/O) mirroring with a second site of the volume. The method also includes, by the second site of the volume, creating a remote copy of the volume, the remote copy of the volume configured to be passive, and enabling bidirectional write I/O mirroring with the first site of the volume.

In some embodiments, the method further includes exposing, by the first site of the volume, the local copy of the volume to one or more hosts.

In some embodiments, the method further includes exposing, by the second site of the volume, the remote copy of the volume to one or more hosts.

In some embodiments, the creating, by the second site of the volume, a remote copy of the volume is in response to a request from the first site to create the volume on the second site.

In some embodiments, the method further includes mapping, by the second site of the volume, the remote copy of the volume to a host with no write access.

In some embodiments, the method further includes, responsive to a request to reconfigure the active-passive volume to an active-active volume, reconfiguring, by the second site of the volume, the remote copy of the volume to be active.

In some embodiments, the method further includes, responsive to a request to configure the remote copy of the volume on the second site as preferred, by the first site of the volume, stopping acceptance of new host I/O requests, draining pending host I/O requests, and switching paths on the first site to disable write access. The method also includes switching, by the second site of the volume, paths on the second site to enable write access.

In some embodiments, the method further includes, responsive to a request to promote the remote copy of the volume on the second site, switching, by the second site of the volume, paths on the second site to enable write access.

In some embodiments, the method further includes, responsive to a request to configure the promoted remote copy of the volume on the second site as preferred, sending, by the second site of the volume to the first site of the volume, a request to copy accumulated delta on the first site. The method also includes, responsive to the request to copy the accumulated delta on the first site, by the first site of the volume, copying the accumulated delta to the volume on the first site and configuring paths on the first site to disable write access.

According to another illustrative embodiment provided to illustrate the broader concepts described herein, a computer program product includes one or more non-transitory machine-readable mediums encoding instructions that when executed by one or more processors cause a process to be carried out for configuring a volume that supports active-active bidirectional replication in an active-passive mode. The process includes, by a first site of the volume, creating a local copy of the volume, the local copy of the volume configured to be active, and enabling bidirectional write input/output (I/O) mirroring with a second site of the volume. The process also includes, by the second site of the volume, creating a remote copy of the volume, the remote copy of the volume configured to be passive, and enabling bidirectional write I/O mirroring with the first site of the volume.

In some embodiments, the process further includes exposing, by the first site of the volume, the local copy of the volume to one or more hosts.

In some embodiments, the process further includes exposing, by the second site of the volume, the remote copy of the volume to one or more hosts.

In some embodiments, the creating, by the second site of the volume, a remote copy of the volume is in response to a request from the first site to create the volume on the second site.

In some embodiments, the process further includes mapping, by the second site of the volume, the remote copy of the volume to a host with no write access.

In some embodiments, the process further includes, responsive to a request to reconfigure the active-passive volume to an active-active volume, reconfiguring, by the second site of the volume, the remote copy of the volume to be active.

In some embodiments, the process further includes, responsive to a request to configure the remote copy of the volume on the second site as preferred, by the first site of the volume, stopping acceptance of new host I/O requests, draining pending host I/O requests, and switching paths on the first site to disable write access. The process also includes switching, by the second site of the volume, paths on the second site to enable write access.

In some embodiments, the process further includes, responsive to a request to promote the remote copy of the volume on the second site, switching, by the second site of the volume, paths on the second site to enable write access.

In some embodiments, the process further includes, responsive to a request to configure the promoted remote copy of the volume on the second site as preferred, sending, by the second site of the volume to the first site of the volume, a request to copy accumulated delta on the first site. The process also includes, responsive to the request to copy the accumulated delta on the first site, by the first site of the volume, copying the accumulated delta to the volume on the first site and configuring paths on the first site to disable write access.

According to another illustrative embodiment provided to illustrate the broader concepts described herein, a system includes one or more non-transitory machine-readable mediums configured to store instructions and one or more processors configured to execute the instructions stored on the one or more non-transitory machine-readable mediums. Execution of the instructions causes the one or more processors to create, by a first site of the volume that supports active-active bidirectional replication, a local copy of the volume, the local copy of the volume configured to be active. Execution of the instructions also causes the one or more processors to enable, by the first site of the volume, bidirectional write input/output (I/O) mirroring with a second site of the volume. Execution of the instructions further causes the one or more processors to, by the second site of the volume, create a remote copy of the volume, the remote copy of the volume configured to be passive, and enable bidirectional write I/O mirroring with the first site of the volume.

In some embodiments, execution of the instructions further causes the one or more processors to expose, by the first site of the volume, the local copy of the volume to one or more hosts.

In some embodiments, execution of the instructions further causes the one or more processors to expose, by the second site of the volume, the remote copy of the volume to one or more hosts.

In some embodiments, execution of the instructions further causes the one or more processors to map, by the second site of the volume, the remote copy of the volume to a host with no write access.

In some embodiments, execution of the instructions further causes the one or more processors to, responsive to a request to reconfigure the active-passive volume to an active-active volume, reconfigure, by the second site of the volume, the remote copy of the volume to be active.

In some embodiments, execution of the instructions further causes the one or more processors to, responsive to a request to configure the remote copy of the volume on the second site as preferred, by the first site of the volume, stop acceptance of new host I/O requests, drain pending host I/O requests, and switch paths on the first site to disable write access. Execution of the instructions further causes the one or more processors to switch, by the second site of the volume, paths on the second site to enable write access.

In some embodiments, execution of the instructions further causes the one or more processors to, responsive to a request to promote the remote copy of the volume on the second site, switch, by the second site of the volume, paths on the second site to enable write access.

In some embodiments, execution of the instructions further causes the one or more processors to, responsive to a request to configure the promoted remote copy of the volume on the second site as preferred, send, by the second site of the volume to the first site of the volume, a request to copy accumulated delta on the first site. Execution of the instructions further causes the one or more processors to, responsive to the request to copy the accumulated delta on the first site, by the first site of the volume, copy the accumulated delta to the volume on the first site and configure paths on the first site to disable write access.

DETAILED DESCRIPTION

Before describing embodiments of the concepts, structures, and techniques sought to be protected herein, some terms are explained. The following description includes a number of terms for which the definitions are generally known in the art. However, the following glossary definitions are provided to clarify the subsequent description and may be helpful in understanding the specification and claims.

As used herein, the term “storage system” is intended to be broadly construed so as to encompass, for example, private or public cloud computing systems for storing data as well as systems for storing data comprising virtual infrastructure and those not comprising virtual infrastructure. As used herein, the terms “client,” “host,” and “user” refer, interchangeably, to any person, system, or other entity that uses a storage system to read/write data, as well as issue requests for configuration of storage units in the storage system. In some embodiments, the term “storage device” may also refer to a storage array including multiple storage devices. In certain embodiments, a storage medium may refer to one or more storage mediums such as a hard drive, a combination of hard drives, flash storage, combinations of flash storage, combinations of hard drives, flash, and other storage devices, and other types and combinations of computer readable storage mediums including those yet to be conceived. A storage medium may also refer both physical and logical storage mediums and may include multiple level of virtual to physical mappings and may be or include an image or disk image. A storage medium may be computer-readable, and may also be referred to herein as a computer-readable program medium. Also, a storage unit may refer to any unit of storage including those described above with respect to the storage devices, as well as including storage volumes, logical drives, containers, or any unit of storage exposed to a client or application. A storage volume may be a logical unit of storage that is independently identifiable and addressable by a storage system.

As used herein, the term “I/O request” or simply “I/O” refers, in addition to its plain and ordinary meaning, to an input or output request, such as a data read or data write request or a request to configure and/or update a storage unit feature. A feature may refer to any service configurable for the storage system.

As used herein, the term “storage device” refers, in addition to its plain and ordinary meaning, to any non-volatile memory (NVM) device, including hard disk drives (HDDs), solid state drivers (SSDs), flash devices (e.g., NAND flash devices), and similar devices that may be accessed locally and/or remotely (e.g., via a storage attached network (SAN) (also referred to herein as storage array network (SAN)).

As used herein, the term “storage array” (sometimes referred to as a disk array) refers, in addition to its ordinary meaning, to a data storage system that is used for block-based, file-based or object storage, where storage arrays can include, for example, dedicated storage hardware that contains spinning hard disk drives (HDDs), solid-state disk drives, and/or all-flash drives.

As used herein, the term “data storage entity” refers, in addition to its ordinary meaning, to any one or more of a file system, object storage, a virtualized device, a logical unit, a logical unit number, a logical volume, a logical device, a physical device, and/or a storage medium.

Asymmetric Logical Unit Access (ALUA) features of Small Computer System Interface (SCSI) (also known as SCSI Target Port Groups or Target Port Group Support) is a standard protocol for identifying optimized paths between a storage system and a host. ALUA enables an initiator to query a target about path attributes, such as primary path and secondary path. It also allows the target to communicate events back to the initiator. The SCSI and ALUA commands can be used across any of a number of transports, such as Fibre Channel (FC) or Internet SCSI (iSCSI), for example.

Non-Volatile Memory Express over Fabric (NVMe-oF) is a technology specification designed to enable nonvolatile memory express message-based commands to transfer data between a host computer and a target solid-state storage device or system over a network, such as Ethernet, Fibre Channel (FC) or InfiniBand.

Asymmetric Namespace Access (ANA) is an NVMe technology specification designed to enable access to a given namespace (e.g., a volume presented from the storage to a host).

While vendor-specific terminology may be used herein to facilitate understanding, it is understood that the concepts, techniques, and structures sought to be protected herein are not limited to use with any specific commercial products. In addition, to ensure clarity in the disclosure, well-understood methods, procedures, circuits, components, and products are not described in detail herein.

As noted above, a metro cluster is a storage array deployment that includes two storage arrays. In such configurations, the metro cluster storage arrays may be deployed in two different data centers or two server rooms within the same data center. These metro cluster storage arrays may operate in a “symmetric active-active” configuration or mode in which both storage arrays are equal in terms of input/output (I/O) processing and process all read I/O locally, while synchronously mirroring (copying) all write I/O to the peer (sometimes referred to as bidirectional write I/O mirroring or bidirectional remote replication). However, the bidirectional remote replication provided by a metro cluster may not be desired and/or needed for all use cases and/or applications. For instance, synchronous remote replication in a single direction may be sufficient for many use cases, such as disaster recovery where a first storage array located in a first data center is used to run the production copy of an application and synchronously replicates all incoming write I/O to a second storage array located in a second data center. Synchronous data replication allows for ensuring that the disaster recovery copy of the data on the second storage array is identical to the production copy of the data on the first storage array. Here, if the first storage array fails and an application is failed over to run from the second storage array, the application continues operating from the same data. This provides Recovery Point Objective (RPO) of zero (RPO=0) and means no data loss on application failover in case of a disaster, which is critical for some organizations to ensure their business continuity. Therefore, it may be desirable if a metro cluster that supports active-active bidirectional remote replication can be configurable for both use cases that need active-active bidirectional remote replication and for use cases that need single direction synchronous remote replication. Also, some applications and file systems may not support the active-active bidirectional remote replication configuration, but still require no data loss (i.e., RPO=0) and near-zero recovery (i.e., Recovery Time Objective (RTO) is near-zero). Thus, having the ability to specify an active-passive configuration with automatic recovery options in a metro cluster environment may be beneficial in that this will allow for providing a desired protection policy for a wider range of applications and file systems (e.g., applications and file systems that do not support the active-active configuration). Embodiments of the present disclosure provide solutions to these and other technical problems described herein.

Embodiments of the concepts, techniques, and structures disclosed herein are directed to implementing an active-passive synchronous remote replication mode in a metro cluster that supports active-active bidirectional replication. A metro cluster implementing the active-passive synchronous remote replication mode supports synchronous remote replication use cases and automatic recovery. In the active-passive synchronous remote replication mode, a preferred side of a metro volume or volume group (e.g., one of the two storage arrays) is active and services host I/O, and the non-preferred side of the metro volume or volume group (e.g., the other one of the two storage arrays) is passive, without write access mode and, thus, does not service host I/O. Note that a passive volume or volume group does not service host I/O even if mapped and the host issues (e.g., accidentally issues) read or write I/O requests. For example, in implementations that support SCSI ALUA, the non-preferred side of the metro volume or volume group can be set to passive, without write access mode and therefore does not service host I/O, by setting the paths on the non-preferred side to Unavailable. As another example, in implementations that support NVMe-oF ANA, the non-preferred side of the metro volume or volume group can be set to passive, without write access mode and therefore does not service host I/O, by setting ANA Group to Inaccessible. The metro cluster implemented to support the active-passive synchronous remote replication mode is able to satisfy a data protection policy of no data loss (i.e., RPO is 0) and near-zero recovery (i.e., RTO is near-zero).

Turning now to the figures,FIG.1illustrates a distributed storage environment100in which embodiments of the techniques disclosed herein may be implemented. Distributed storage environment100may be associated with an enterprise, a service provider, a storage vendor, or other type of organization. Storage environment100includes a first data center (data center 1)102-1and a second data center (data center 2)102-2. In accordance with various embodiments, storage environment100may represent a metro cluster configuration in which symmetric active-active access to storage resources (e.g., volume, volume group, file system, and the like) with bidirectional write I/O mirroring can be achieved. In such configurations, data center102-1and data center102-2may be located remotely from one another, for example, in different rooms of a building, different buildings, or other locations in which the distance between data center102-1and data center102-2is on the order of metropolitan distances (e.g., approximately 50 to 100 kilometers or 30 to 60 miles). While only two data centers are illustrated inFIG.1, it will be appreciated that storage environment100may include any number of data centers. The network that interconnects data center102-1and data center102-2may include any type of high speed network or combination of such networks, such as a storage area network (SAN), a local area network (LAN), and/or some other type of network, for example.

As shown, data center102-1includes a data storage system104-1, which further includes a first node (Node A)106-1, a second node (Node B)108-1, and storage (metro volume)110-1. Similarly, data center102-2includes a data storage system104-2, which further includes a first node (Node A)106-2, a second node (Node B)108-2, and storage (metro volume)110-2. An example of data storage system104includes Dell EMC PowerStore, though embodiments are not so limited. Data storage system104-1and data storage system104-2may be communicably coupled to one another via a network (not shown). The network that interconnects data storage systems104-1,104-2may include any type of high speed network or combination of such networks, such as a storage area network (SAN), a local area network (LAN), and/or some other type of network, for example. Storage110-1,110-2may be provided, for example, in the form of hard disk drives, solid state drives, flash drives, storage arrays (e.g., Redundant Array of Independent Disks (RAID)), and/or other storage devices.

In an example implementation, nodes106-1,108-1form one storage controller pair and work in a coordinated manner to provide the processing resources for performing storage operations and servicing input/output (I/O) between storage104-1and hosts. Similarly, nodes106-2,108-2form one storage controller pair and work in a coordinated manner to provide the processing resources for performing storage operations and servicing I/O between storage104-2and hosts. To this end, nodes106-1,108-1in data storage system104-1may be communicably coupled to one another via a high speed interconnect (e.g., PCI Express). Similarly, nodes106-2,108-2in data storage system104-2may be communicably coupled to one another via a high speed interconnect (e.g., PCI Express). Individual nodes106-1,108-1in data storage system104-1may be communicably coupled to the individual nodes106-2,108-2in data storage system104-2. While only one storage controller pair is illustrated in each data storage system inFIG.1, it will be appreciated that each of the data storage systems may include additional storage controller pairs.

Nodes106-1,108-1may be communicably coupled to storage110-1of data storage system104-1via, for example, iSCSI, Serial Attached SCSI (SAS), or NVMe-oF. Similarly, nodes106-2,108-2may be communicably coupled to storage110-2of data storage system104-2via, for example, iSCSI, SAS, or NVMe-oF. Nodes106-1,108-1realize the logical unit numbers (LUNs) (e.g., logical units such as volumes) provisioned from storage110-1, and nodes106-2,108-2realize the LUNs provisioned from storage110-2. Note that, since each data storage system (e.g., data storage system104-1and data storage system104-2) contains a copy of the LUNs in a symmetric active-active metro cluster configuration, all of the LUNs appear as local LUNs to the front ends of the individual nodes. In other words, the same LUN (e.g., metro volume) is available for host I/O out of the nodes of both data storage systems. To this end, the individual nodes may include one or more ports for establishing paths to the LUNs. The ports may be devices (e.g., iSCSI targets) on storage systems (e.g., data storage system104-1and data storage system104-2) that communicate with storage over Ethernet ports or FC target ports, for example. As shown inFIG.1, node106-1may include a target port group (TPG1A)112-1and node108-1may include a target port group (TPG1B)114-1. Similarly, node106-2may include a target port group (TPG2A)112-2and node108-2may include a target port group (TPG2B)114-2. Each target port group includes multiple ports that have the same access characteristics to a LUN (e.g., volume). In brief, the access characteristics may define a path to the LUN as Active/Optimized, Active/Non-Optimized, Unavailable, or In-transition.

In an example, as shown inFIG.1, data center102-1may include a host (Host 1)116-1and data center102-2may include a host (Host 2)116-2. Host116-1can connect to target port groups112-1,114-1,112-2,114-2via initiators118-1. Similarly, host116-2can connect to target port groups112-1,114-1,112-2,114-2via initiators118-2. Initiators118-1,118-2can be software iSCSI initiators that communicate with the data storage systems (e.g., data storage systems104-1,104-2) over Ethernet ports. In other implementations, initiators118-1,118-2can be FC initiators that communicate with the data storage systems (e.g., data storage systems104-1,104-2) over a FC infrastructure. In any case, host116-1can use initiator118-1to obtain a list of all the target port groups for a LUN (e.g., target port groups112-1,114-1,112-2,114-2), obtain a target port ID for a specific path, and use this information to organize the paths to the target port group to access the LUN. In this manner, initiators118-1allow the LUN (e.g., volume) be mapped to host116-1. Similarly, host116-2can use initiator118-2to obtain a list of all the target port groups for the LUN (e.g., target port groups112-1,114-1,112-2,114-2), obtain a target port ID for a specific path, and use this information to organize the paths to the target port group to access the LUN. As can be seen inFIG.1, host116-1can communicably couple to the individual target port groups112-1,114-1,112-2,114-2via respective paths, where one of the paths (i.e., path from host116-1to target port group112-1) is an Active/Optimized path and the remaining paths are Active/Non-optimized paths. Similarly, host116-2can communicably couple to the individual target port groups112-1,114-1,112-2,114-2via respective paths, where one of the paths (i.e., path from host116-2to target port group112-2) is an Active/Optimized path and the remaining paths are Active/Non-optimized paths. Once the host is connected to the target port group, the host can be mapped to the LUN using storage system management software.

In an example operation, any of the nodes (e.g., any of nodes106-1,108-1,106-2,108-2) may receive an I/O request directed to a LUN. For example, the I/O request, which may include block-based requests and file-based requests, may be issued by a host (e.g., host116-1or116-2). In the symmetric active-active metro cluster configuration, the specific node that receives the I/O request processes the I/O request it receives locally on its data storage system without the need for forwarding to another node on the peer data storage system. In the case of a write I/O, in addition to processing the write I/O locally on its data storage system, the node synchronously mirrors the write I/O to the peer data storage system to maintain consistency between the two data storage systems (e.g., metro volume) and, more generally, among all the nodes in the metro cluster. For example, assuming that node106-1(or node108-1) receives an write I/O request, node106-1(or node108-1) can process the write I/O locally on data storage system104-1and synchronously mirror the write I/O on data storage system104-2using either node106-2or node108-2. Similarly, assuming that node106-2(or node108-2) receives a write I/O request, node106-2(or node108-2) can process the write I/O locally on data storage system104-2and synchronously mirror the write I/O on data storage system104-1using either node106-1or node108-1. Note that, when the metro cluster is in the active-active bidirectional replication configuration or mode, a metro volume may likewise be considered an active-active metro volume. Likewise, when the metro cluster is in the active-passive synchronous remote replication configuration or mode, a metro volume may likewise be considered an active-passive metro volume.

As noted above, bidirectional remote replication provided by the metro cluster may not be desired and/or needed for all use cases and/or applications. Thus, and in accordance with certain of the embodiments disclosed herein, the individual nodes (e.g., nodes106-1,108-1,106-2,108-2) are programmed with or otherwise includes a storage system management application that is configured with a user interface (e.g., graphical user interface (GUI)) through which a user can configure the metro cluster for either active-active bidirectional remote replication or single direction (i.e., active-passive) synchronous remote replication as described herein. In some such embodiments, the storage system management application may also be configured to communicate via command line (CLI) of various network interfaces such as Representation State Transfer (REST), for example.

Referring toFIG.2, according to some embodiments, storage environment100ofFIG.1may be in a state in which no volume is mapped to a host. For example, environment100may be in such a state when configuring a metro volume between a first site (e.g., data storage system104-1) and a second site (e.g., data storage system104-2) in active-passive mode. In this state, an original volume is empty since the volume is not mapped to a host. In this case, since no volumes are mapped to a host, there is no initial synchronization between volumes on data storage system104-1and data storage system104-2. Moreover, when configuring the metro volume in active-passive mode, bi-directional write I/O mirroring can start after the volume is created. As can be seen inFIG.2, and as will be further described with respect toFIG.4, the metro volume can be configured in active-passive mode where the metro volume side on data storage system104-2is passive. When in the passive mode, the metro volume on data storage system104-2is in a write-access disabled state when it is mapped to a host.

Referring toFIG.3, according to some embodiments, storage environment100ofFIG.1may be in a state in which an existing volume on a data storage system (e.g., data storage system104-1) is mapped to a host (e.g., host116-1) as a metro volume. The metro volume may contain application data since it is mapped to host116-1. For example, similar to the state described previously with respect toFIG.2, environment100may be in such a state when configuring a metro volume between a first site (e.g., data storage system104-1) and a second site (e.g., data storage system104-2) in active-passive mode. In this case, when configuring the metro volume in active-passive mode, bi-directional write I/O mirroring can start after initial synchronization from data storage system104-1to data storage system104-2is completed and both sides of the volume contain identical data. As can be seen inFIG.3, and as will be further described with respect toFIG.4, the metro volume can be configured in active-passive mode where the metro volume side on data storage system104-2is passive. When in the passive mode, the metro volume on data storage system104-2is in a write-access disabled state when it is mapped to a host.

FIG.4is a flow diagram of an example process400for configuring a metro volume in active-passive mode, in accordance with an embodiment of the present disclosure. For example, as noted above, the metro volume may be associated with a storage environment (e.g., a metro cluster) that is in a state in which no volume is mapped to a host or a state in which an existing volume on a data storage system (e.g., data storage system104-1) is mapped to a host (e.g., host116-1) as a metro volume. With reference to process400, at402, a metro volume may be configured in active-passive mode in which the metro volume on data storage system104-1is active and the metro volume on its peer (e.g., data storage system104-2) is passive. For example, a user, such as a system administrator, can utilize a UI provided by a storage system management application of data storage system104-1(e.g., storage system management application running on one of node106-1or node108-1) to configure the metro volume in active-passive mode. In some embodiments of the active-passive metro cluster mode, the storage resource identity (SCSI or NVMe) may be the same on both sides, which allows applications to access the volume on both sides without any configuration changes.

At404, data storage system104-1may create a local copy of the metro volume. In an example implementation, a control path component of data storage system104-1can communicate with a local data path component to create the local copy of the metro volume on storage110-1of data storage system104-1. Here, the copy of the metro volume on storage110-1of data storage system104-1is being configured to be active. At406, data storage system104-1may enable bidirectional write I/O mirroring. The bidirectional write I/O mirroring is in effect single direction synchronous remote replication from data storage system104-1to data storage system104-2since the metro volume on data storage system104-1is active.

At408, data storage system104-1may expose the local copy of the metro volume to one or more hosts (e.g., host116-1). In an example implementation, the control path component of data storage system104-1can communicate with a local I/O front end component to expose the local copy of the metro volume to hosts. In the active-passive mode, exposing the local copy of the metro volume may include configuring or otherwise switching the paths associated with data storage system104-1to enable write access (e.g., write access mode enabled). For example, the control path component of data storage system104-1can communicate with the local I/O front end component to switch the paths to enable write access. As a result, a SCSI Report Target Port Groups command when invoked, for example, by a host (e.g., host116-1), may report an Active/Optimized ALUA state for target port group112-1of data storage system104-1and report an Active/Non-optimized ALUA state for target port group114-1of data storage system104-1. If the host is attached over NVMe-oF, an ANA Group Optimized state for target port group112-1and an ANA Group Non-optimized state for target port group114-1may be reported. Once attached, a host can support (i.e., host) one or more applications. For example, as can be seen inFIG.3, host116-1can host three applications, App1, App2, and App3. While only three applications are illustrated inFIG.3, it will be appreciated that host116-1may host any number of applications.

Referring again toFIG.4, at410, data storage system104-1may send or otherwise provide to its peer in the metro cluster (e.g., data storage system104-2) a request to create the metro volume as passive. In an example implementation, the control path component of data storage system104-1can communicate with a control path component of data storage system104-2to request the creation of the metro volume.

Upon receiving the request to create the metro volume, at412, data storage system104-2may create a remote copy of the metro volume. In an example implementation, the control path component of data storage system104-2can communicate with a local data path component to create the remote copy of the metro volume on storage110-2of data storage system104-2. Here, the copy of the metro volume on storage110-2of data storage system104-2is being configured to be passive. At414, data storage system104-2may enable bidirectional write I/O mirroring. The bidirectional write I/O mirroring is in effect single direction synchronous remote replication from data storage system104-1to data storage system104-2since the metro volume on data storage system104-1is active. That is, since the metro volume on data storage system104-2is passive, there is no write I/O mirroring is from data storage system104-2to data storage system104-1.

At416, data storage system104-2may expose the remote copy of the metro volume (i.e., the copy of the metro volume on storage110-2) to one or more hosts. In an example implementation, the control path component of data storage system104-2can communicate with a local I/O front end component to expose the remote copy of the metro volume to hosts. In the active-passive mode, exposing the remote copy of the metro volume may include configuring or otherwise switching the paths associated with data storage system104-2to disable write access (e.g., write access mode disabled). For example, the control path component of data storage system104-2can communicate with the local I/O front end component to switch the paths to disable write access. As a result, a SCSI Report Target Port Groups command may report an Unavailable ALUA state for target port group112-2of data storage system104-2once the volume is mapped to a host. If the host is attached over NVMe-oF, an ANA Group Inaccessible state may be reported.

Note that, if active-active mode was specified during configuration (e.g., the system administrator specified to configure the metro volume in active-active mode), a SCSI Report Target Port Groups command may report Active/Optimized or Active/Non-optimized ALUA states for target port group112-2of data storage system104-2once the volume is mapped to a host. If the host is attached over NVMe-oF, an ANA Group Optimized or ANA Group Non-optimized states may be reported.

Referring toFIG.5, according to some embodiments, a passive side of a metro volume (e.g., the active-passive metro volume ofFIG.3) may be mapped to a host. For instance, the passive side may be mapped to a host to have one less operation to perform during failover. As shown inFIG.5, when the active side of the metro volume (e.g., data storage system104-1) is mapped to a host (e.g., host116-1), the paths are in write access mode enabled. When in the enabled mode, the paths may be used for I/O requests directed to the LUN (i.e., metro volume). In SCSI implementations, the paths are ALUA Active/Optimized on one node (e.g., node106-1) and ALUA Active/Non-optimized on the other node (e.g., node108-1) of the active metro volume (e.g., data storage system104-1). In NVMe-oF implementations, the paths on one node may be ANA Group Optimized and the paths on the other node may be ANA Group Non-optimized. In contrast, when the passive side of the metro volume (e.g., data storage system104-2) is mapped to a host, the paths are in write access mode disabled (i.e., SCSI ALUA Unavailable or NVMe-oF ANA Group Inaccessible). When in the disabled mode, the platform front end (e.g., front ends of nodes106-2,108-2) can fail any I/O requests directed to the disabled write access mode volumes. Failing such I/O requests guarantees that a host cannot write to the passive side of the metro volume and corrupt the data. Since a host cannot write to the passive side of the metro volume, the passive side (e.g., data storage system104-2) may be mapped to a host (e.g., host116-2). Mapping a passive side of a metro volume to a host may be useful for reducing RTO during failover, for example.

FIG.6is a flow diagram of an example process600for mapping a passive side of a metro volume to a host, in accordance with an embodiment of the present disclosure. At602, a request may be received to map a passive side (e.g., data storage system104-2) of a metro volume to a host. For example, a user, such as a system administrator, can utilize a UI provided by a storage system management application of data storage system104-2(e.g., storage system management application running on one of node106-2or node108-2) to map data storage system104-2to host116-2. Upon receiving the request, at604, data storage system104-2may map the metro volume on storage110-2to host116-2. In an example implementation, a control path component of data storage system104-2can communicate with a local I/O front end component to map the metro volume on storage110-2to host116-2. As a result, a SCSI Report Target Port Groups command when invoked, for example, by host116-2, may report Unavailable ALUA states for target port groups112-2,114-2of data storage system104-2. If host116-2is attached over NVMe-oF, an ANA Group Inaccessible state may be reported for target port groups112-2,114-2of data storage system104-2.

Referring toFIGS.7A and7B, according to some embodiments, an active-passive metro volume may be reconfigured into an active-active metro volume. As shown inFIG.7A, a metro volume may be configured in active-passive mode in which the metro volume on data storage system104-1is active and the metro volume on its peer, data storage system104-2, is passive. As shown inFIG.7B, the active-passive metro volume may be reconfigured into an active-active metro volume in which the metro volume on data storage system104-1and the metro volume on data storage system104-2are both active. For example, as can be seen inFIG.7B, the metro volume on data storage system104-1may be active preferred and the metro volume on data storage system104-2may be active non-preferred. Active preferred and active non-preferred differ with respect to failure handling. While both storage systems are operating normally without failures of any kind and network connecting the two storage systems is also operational, the two storage systems replicate host write I/O to each other and may also perform periodic heartbeat messaging. If the network or one of storage systems fails, the failure manifests itself to the surviving storage system as absence of replicated write I/O and absence of heartbeat. Storage systems may not be able to distinguish between a network failure and other system failure on their own. Thus, to avoid “split brain” (i.e., both storage systems keeping their respective sides of a metro volume online while not being able to replicate writes to each other), if communication is lost, the preferred storage system keeps the volume online and the non-preferred storage system brings the volume off-line.

FIG.8is a flow diagram of an example process800for reconfiguring an active-passive metro volume into an active-active metro volume, in accordance with an embodiment of the present disclosure. At802, a request may be received to reconfigure an active-passive metro volume into an active-active metro volume. For example, a user, such as a system administrator, can utilize a UI provided by a storage system management application of data storage system104-1(e.g., storage system management application running on one of node106-1or node108-1) to reconfigure the active-passive metro volume into an active-active metro volume. In response to the request, data storage system104-1may send or otherwise provide to data storage system104-2a request to reconfigure the metro volume on data storage system104-2from its current passive mode to an active mode. In an example implementation, a control path component of data storage system104-1can communicate with a control path component of data storage system104-2to request the reconfiguration of the metro volume.

Upon receiving the request to reconfigure the metro volume, at804, data storage system104-2may switch the paths to enable write access (e.g., write access mode enabled). In an example implementation, the control path component of data storage system104-2can communicate with the local I/O front end component to switch the paths to enable write access. For example, as shown inFIG.7B, the path(s) on node106-2may be switched to ALUA Active/Optimized and the path(s) on node108-2may be switched to ALUA Active/Non-optimized. Note that bidirectional write I/O mirroring is already enabled on data storage system104-2. As described previously, when the metro volume on data storage system104-2was in the previous passive mode, no write I/O mirroring was being performed from data storage system104-2to data storage system104-1.

Referring again toFIG.8, at806, a SCSI Report Target Port Groups command, when invoked by host116-2, may report an Active/Optimized ALUA state for target port group112-2of data storage system104-2and report an Active/Non-optimized ALUA state for target port group114-2of data storage system104-2. If host116-2is attached over NVMe-oF, an ANA Group Optimized state for target port group112-2and an ANA Group Non-optimized state for target port group114-2may be reported.

At808, the active metro volume on data storage system104-2may start servicing host I/O requests from applications being hosted by host116-2. For example, as can be seen inFIG.7B, host116-2can host two applications, App4and App5. While only two applications are illustrated inFIG.7B, it will be appreciated that host116-2may host any number of applications.

Referring toFIGS.9A and9B, according to some embodiments, a planned failover for an active-passive metro volume group with preferred side switch is illustrated. As shown inFIG.9A, a metro volume may be configured in active-passive mode in which the metro volume on data storage system104-1is active and the metro volume on its peer, data storage system104-2, is passive. The planned failover changes the preferred side of the metro volume from data storage system104-1to data storage system104-2. In performing the planned failover, write access to the new passive side, data storage system104-1, is disabled, and write access to the new active side, data storage system104-2, is enabled.FIG.9Billustrates the metro volume after the planned failover of the active-passive volume group with preferred side switch. Note that, since hosts on the production side (e.g., data storage system104-1) and the disaster recovery side (e.g., data storage system104-2) are connected to their respective data storage systems, applications that are hosted on host116-1(e.g., App1, App2, and App3as shown inFIG.9A) need to be powered down before switching the preferred side and powered up after the switch on host116-2(e.g., App1, App2, and App3as shown inFIG.9B).

FIG.10is a flow diagram of an example process1000for performing a planned failover for an active-passive metro volume group with preferred side switch, in accordance with an embodiment of the present disclosure. At1002, a request may be received to configure the metro volume on data storage system104-2as preferred. For example, a user, such as a system administrator, can utilize a UI provided by a storage system management application of data storage system104-2(e.g., storage system management application running on one of node106-2or node108-2) to configure the currently passive metro volume on data storage system104-2as preferred. In response to the request, data storage system104-2may send or otherwise provide to data storage system104-1a request to perform the planned failover of data storage system104-1to data storage system104-2with preferred side switch. In an example implementation, a control path component of data storage system104-2can communicate with a control path component of data storage system104-1to request the planned failover.

Upon receiving the request for the planned failover, at1004, data storage system104-1may stop accepting new host I/O requests. In an example implementation, the control path component of data storage system104-1can communicate with a local I/O front end component to stop accepting new host I/O requests. At1006, data storage system104-1may drain all pending host I/O requests. In an example implementation, the control path component of data storage system104-1can communicate with the local I/O front end component to drain all pending host I/O requests. For example, the pending host I/O requests may be processed by data storage system104-1.

At1008, data storage system104-1may switch the paths to disable write access (e.g., write access mode disabled). In an example implementation, the control path component of data storage system104-1can communicate with the local I/O front end component to switch the paths to disable write access. As a result, as can be seen inFIG.9B, a SCSI Report Target Port Groups command when invoked, for example, by host116-1, may report Unavailable ALUA states for target port groups112-1,114-1of data storage system104-1. If host116-1is attached over NVMe-oF, an ANA Group Inaccessible state may be reported for target port groups112-1,114-1of data storage system104-1.

Referring again toFIG.10, at1010, data storage system104-2may switch the paths to enable write access (e.g., write access mode enabled). In an example implementation, the control path component of data storage system104-2can communicate with a local I/O front end component to switch the paths to enable write access. For example, as shown inFIG.9B, the path(s) on node106-2may be switched to ALUA Active/Optimized and the path(s) on node108-2may be switched to ALUA Active/Non-optimized. Note that bidirectional write I/O mirroring is already enabled on data storage system104-2. As described previously, when the metro volume on data storage system104-2was in the previous passive mode, no write I/O mirroring was being performed from data storage system104-2to data storage system104-1.

Referring again toFIG.10, at1012, a SCSI Report Target Port Groups command, when invoked by host116-2, may report an Active/Optimized ALUA state for target port group112-2of data storage system104-2and report an Active/Non-optimized ALUA state for target port group114-2of data storage system104-2. If host116-2is attached over NVMe-oF, an ANA Group Optimized state for target port group112-2and an ANA Group Non-optimized state for target port group114-2may be reported.

At1014, the active metro volume on data storage system104-2may start servicing host I/O requests from applications being hosted by host116-2. For example, as can be seen inFIG.9B, the applications (e.g., App1, App2, and App3) that were being hosted on data storage system104-1may be restarted on data storage system104-2(i.e., powered up on host116-2).

Referring toFIG.11, according to some embodiments, an unplanned failover for an active-passive metro volume with non-preferred side promote is illustrated. As shown inFIG.11, a metro volume may be configured in active-passive mode in which the metro volume on data storage system104-1is active and the metro volume on its peer, data storage system104-2, is passive. The unplanned failover with non-preferred side promote promotes the previously passive side of the metro volume (e.g., data storage system104-2) upon a failure of the active side of the metro volume (e.g., data storage system104-1). For example, the unplanned failover may be performed in the case of an unexpected data storage system104-1malfunction. Note that, in performing the unplanned failover, all applications running on host116-1likely need to be stopped (i.e., the system checks to ensure that no application is running on host116-1) since the unplanned failover does not make the metro volume on data storage system104-1read-only. Also note that the applications which were running on data storage system104-1may be powered up to run on host116-2after data storage system104-2is promoted.

FIG.12is a flow diagram of an example process1200for performing an unplanned failover for an active-passive metro volume with non-preferred side promote, in accordance with an embodiment of the present disclosure. At1202, a request may be received to promote the metro volume on data storage system104-2. For example, a user, such as a system administrator, can utilize a UI provided by a storage system management application of data storage system104-2(e.g., storage system management application running on one of node106-2or node108-2) to promote the currently passive metro volume on data storage system104-2.

Upon receiving the request, at1204, data storage system104-2may switch the paths to enable write access (e.g., write access mode enabled). In an example implementation, a control path component of data storage system104-2can communicate with a local I/O front end component to switch the paths to enable write access. For example, as shown inFIG.11, the path(s) on node106-2may be switched to ALUA Active/Optimized and the path(s) on node108-2may be switched to ALUA Active/Non-optimized. Note that no write I/O mirroring is configured between data storage system104-2and data storage system104-1since data storage system104-1is in a failed state.

Referring again toFIG.12, at1206, a SCSI Report Target Port Groups command, when invoked by host116-2, may report an Active/Optimized ALUA state for target port group112-2of data storage system104-2and report an Active/Non-optimized ALUA state for target port group114-2of data storage system104-2. If host116-2is attached over NVMe-oF, an ANA Group Optimized state for target port group112-2and an ANA Group Non-optimized state for target port group114-2may be reported.

At1208, the promoted metro volume on data storage system104-2may start servicing host I/O requests from applications being hosted by host116-2. For example, as can be seen inFIG.11, the applications (e.g., App1, App2, and App3) that were being hosted on data storage system104-1may be restarted on data storage system104-2(i.e., powered up on host116-2).

Referring toFIG.13, according to some embodiments, a re-protect of an active-passive volume after a failure is illustrated. As shown inFIG.13, a re-protect can be performed after a failed preferred side of a metro volume (e.g., data storage system104-1) has recovered. The re-protect of the metro volume from data storage system104-2to data storage system104-1brings data storage system104-1in synchronization with data storage system104-2. In embodiments, the re-protect process includes configuring the previous passive, non-preferred side of the metro volume to be the new active, preferred side of the metro volume, and re-protecting the new passive side of the metro volume from the new active side of the metro volume.

FIG.14is a flow diagram of an example process1400for re-protecting an active-passive volume after a failure, in accordance with an embodiment of the present disclosure. For example, process1400may be performed after an unplanned failover is performed (e.g., SeeFIG.11), when the data storage system104-1comes back online. After process1400is performed, data storage system104-1may be made active and data storage system,104-2may be made passive (or write-protected). As another example, process1400may be executed after a planned failover is performed (e.g., seeFIG.10). At1402, a request may be received to configure the metro volume on data storage system104-2as preferred. For example, a user, such as a system administrator, can utilize a UI provided by a storage system management application of data storage system104-2(e.g., storage system management application running on one of node106-2or node108-2) to configure the promoted passive metro volume on data storage system104-2as preferred. In an example implementation, a control path component of data storage system104-2can persist the change in mode in a management database, for example.

At1404, a request may be received to re-protect the metro volume on data storage system104-2For example, the system administrator can utilize the UI provided by the storage system management application of data storage system104-2(e.g., storage system management application running on one of node106-2or node108-2) to request re-protect of the metro volume to data storage system104-1. Upon receiving the request, at1406, data storage system104-2may send or otherwise provide to data storage system104-1a request to copy the accumulated delta. In an example implementation, a replication session component of data storage system104-2can communicate to data storage system104-1the accumulated delta.

Upon receiving the request to copy the accumulated delta, at1408, data storage system104-1may copy the accumulated delta to storage110-1, and synchronize the states of storage110-1and storage110-2as a result. Upon copying the accumulated delta, at1410, data storage system104-1may configure the paths to disable write access (e.g., write access mode disabled). In an example implementation, a control path component of data storage system104-1can communicate with the local I/O front end component to switch the paths to disable write access. As a result, as can be seen inFIG.13, a SCSI Report Target Port Groups command when invoked, for example, by host116-1, may report Unavailable ALUA states for target port groups112-1,114-1of data storage system104-1. If host116-1is attached over NVMe-oF, an ANA Group Inaccessible state may be reported for target port groups112-1,114-1of data storage system104-1.

Referring toFIG.15, according to some embodiments, an end of an active-passive metro volume is illustrated. As shown inFIG.15, once synchronous remote replication is no longer needed, the metro volume can be ended. Unlike the active-active configuration in which the metro volume can be ended from either side (i.e., from the active side or the passive side), an active-passive metro volume can only be ended from the active (preferred) side to avoid unwanted failover to the passive side. In embodiments, the volume on the passive side may be maintained with a different identity (e.g., a different LUN).

FIG.16is a flow diagram of an example process1600for ending of an active-passive metro volume, in accordance with an embodiment of the present disclosure. For example, and without limitation, process1600may be performed after a planned failover is performed (e.g., SeeFIG.10), when it is not desired to re-protect data storage system140-2(e.g., by executing process1400). At1602, a request may be received to end the metro volume on data storage system104-2. For example, a user, such as a system administrator, can utilize a UI provided by a storage system management application of data storage system104-2(e.g., storage system management application running on one of node106-2or node108-2) to request the end of the metro volume. Upon receiving the request, at1604, data storage system104-2may send or otherwise provide to data storage system104-1a request to end the metro volume. In an example implementation, a control path component of data storage system104-2can communicate with a control path component of data storage system104-1to end the metro volume.

Upon receiving the request to end the metro volume, at1606, data storage system104-1may stop the bidirectional write I/O mirroring. In an example implementation, the control path component of data storage system104-1can communicate with a local data path component to disable or otherwise stop the bidirectional write I/O mirroring.

At1608, data storage system104-2may send or otherwise provide to data storage system104-1a request to change the metro volume copy identity (e.g., the SCSI identity). In an example implementation, the control path component of data storage system104-2can communicate with the control path component of data storage system104-1to change the metro volume copy identity. Upon receiving the request to change the metro volume copy identity, at1610, data storage system104-1may change the identity of the metro volume on storage110-1of data storage system104-1.

FIG.17schematically shows selective components of an illustrative computer system1700that may be used in accordance with an embodiment of the concepts, structures, and techniques disclosed herein. As shown, computer system1700includes a processor1702, a volatile memory1704, a communication module1706(e.g., network chip or chipset which allows for communication via a network, a bus, an interconnect, etc.), and a non-volatile memory1708(e.g., hard disk or flash). Non-volatile memory1708stores an operating system1710, computer instructions1712, and data1714. In one example, computer instructions1712are executed by processor1702out of volatile memory1704to perform all or part of the processes described herein (e.g., processes illustrated and described in reference toFIGS.1through16).

These processes are not limited to use with particular hardware and software; they may find applicability in any computing or processing environment and with any type of machine or set of machines that is capable of running a computer program. The processes described herein may be implemented in hardware, software, or a combination of the two. The processes described herein may be implemented in computer programs executed on programmable computers/machines that each includes a processor, a non-transitory machine-readable medium or other article of manufacture that is readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and one or more output devices. In embodiments, the processor can include ASIC, FPGA, and/or other types of circuits. Program code may be applied to data entered using an input device to perform any of the processes described herein and to generate output information.

The system may be implemented, at least in part, via a computer program product, (e.g., in a non-transitory machine-readable storage medium such as, for example, a non-transitory computer-readable medium), for execution by, or to control the operation of, data processing apparatus (e.g., a programmable processor, a computer, or multiple computers)). Each such program may be implemented in a high level procedural or object-oriented programming language to communicate with a computer system. However, the programs may be implemented in assembly or machine language. The language may be a compiled or an interpreted language and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network. A computer program may be stored on a non-transitory machine-readable medium that is readable by a general or special purpose programmable computer for configuring and operating the computer when the non-transitory machine-readable medium is read by the computer to perform the processes described herein. For example, the processes described herein may also be implemented as a non-transitory machine-readable storage medium, configured with a computer program, where upon execution, instructions in the computer program cause the computer to operate in accordance with the processes. A non-transitory machine-readable medium may include but is not limited to a hard drive, compact disc, flash memory, non-volatile memory, volatile memory, magnetic diskette and so forth but does not include a transitory signal per se.

The processes described herein are not limited to the specific examples described. For example, the processes ofFIGS.1through16are not limited to the specific processing order illustrated. Rather, any of the processing blocks of the Figures may be re-ordered, combined or removed, performed in parallel or in serial, as necessary, to achieve the results set forth above.

The processing blocks associated with implementing the system may be performed by one or more programmable processors executing one or more computer programs to perform the functions of the system. All or part of the system may be implemented as, special purpose logic circuitry (e.g., an FPGA (field-programmable gate array) and/or an ASIC (application-specific integrated circuit)). All or part of the system may be implemented using electronic hardware circuitry that include electronic devices such as, for example, at least one of a processor, a memory, a programmable logic device, or a logic gate. It is understood that embodiments of event synchronization are applicable to a variety of systems, objects and applications.

In the foregoing detailed description, various features of embodiments are grouped together for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claims require more features than are expressly recited. Rather, inventive aspects may lie in less than all features of each disclosed embodiment.

As will be further appreciated in light of this disclosure, with respect to the processes and methods disclosed herein, the functions performed in the processes and methods may be implemented in differing order. Additionally or alternatively, two or more operations may be performed at the same time or otherwise in an overlapping contemporaneous fashion. Furthermore, the outlined actions and operations are only provided as examples, and some of the actions and operations may be optional, combined into fewer actions and operations, or expanded into additional actions and operations without detracting from the essence of the disclosed embodiments.

In the description of the various embodiments, reference is made to the accompanying drawings identified above and which form a part hereof, and in which is shown by way of illustration various embodiments in which aspects of the concepts described herein may be practiced. It is to be understood that other embodiments may be utilized, and structural and functional modifications may be made without departing from the scope of the concepts described herein. It should thus be understood that various aspects of the concepts described herein may be implemented in embodiments other than those specifically described herein. It should also be appreciated that the concepts described herein are capable of being practiced or being carried out in ways which are different than those specifically described herein.

As used in this application, the words “exemplary” and “illustrative” are used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” or “illustrative” is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, use of the words “exemplary” and “illustrative” is intended to present concepts in a concrete fashion.

In the description of the various embodiments, reference is made to the accompanying drawings identified above and which form a part hereof, and in which is shown by way of illustration various embodiments in which aspects of the concepts described herein may be practiced. It is to be understood that other embodiments may be utilized, and structural and functional modifications may be made without departing from the scope of the concepts described herein. It should thus be understood that various aspects of the concepts described herein may be implemented in embodiments other than those specifically described herein. It should also be appreciated that the concepts described herein are capable of being practiced or being carried out in ways which are different than those specifically described herein.

Terms used in the present disclosure and in the appended claims (e.g., bodies of the appended claims) are generally intended as “open” terms (e.g., the term “including” should be interpreted as “including, but not limited to,” the term “having” should be interpreted as “having at least,” the term “includes” should be interpreted as “includes, but is not limited to,” etc.).

In addition, even if a specific number of an introduced claim recitation is explicitly recited, such recitation should be interpreted to mean at least the recited number (e.g., the bare recitation of “two widgets,” without other modifiers, means at least two widgets, or two or more widgets). Furthermore, in those instances where a convention analogous to “at least one of A, B, and C, etc.” or “one or more of A, B, and C, etc.” is used, in general such a construction is intended to include A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together, etc.

All examples and conditional language recited in the present disclosure are intended for pedagogical examples to aid the reader in understanding the present disclosure, and are to be construed as being without limitation to such specifically recited examples and conditions. Although illustrative embodiments of the present disclosure have been described in detail, various changes, substitutions, and alterations could be made hereto without departing from the scope of the present disclosure. Accordingly, it is intended that the scope of the present disclosure be limited not by this detailed description, but rather by the claims appended hereto.