Patent Description:
This technology relates to managing distributed snapshot for low latency storage.

As high performance storage class memory SSD become available, it becomes increasingly difficult to provide high level storage functions with high levels of data management (snapshot in particular) at the required latency levels due to the amount of overhead associated with managing the snapshots. Typical overheads for systems in the existing technologies deploy advanced data management features such as snapshots in the <NUM>-<NUM> microsecond range. However, adding these latencies to low latency SSDs significantly reduces the performance of the storage systems.

Documents <CIT>, <CIT>, <CIT> and <CIT> disclose backup/replication among various storage devices including the usage of a snapshot of the primary device to perform a non-disruptive replication to the secondary device.

Document <CIT> also concerns data replication among SSD storage volumes using snapshots.

A clustered network environment <NUM> that implement one or more aspects of the technology described and illustrated herein is shown in <FIG>. The clustered network environment <NUM> includes data storage apparatuses <NUM>(<NUM>)-<NUM>(n) that are coupled over a cluster fabric <NUM> facilitating communication between the data storage apparatuses <NUM>(<NUM>)-<NUM>(n) (and one or more modules, components, etc. therein, such as, node computing devices <NUM>(<NUM>)-<NUM>(n)), although any number of other elements or components can also be included in the clustered network environment <NUM> in other examples.

Node computing devices <NUM>(<NUM>)-<NUM>(n) can be primary or local storage controllers or secondary or remote storage controllers that provide client devices <NUM>(<NUM>)-<NUM>(n), with access to data stored within data storage devices <NUM>(<NUM>)-<NUM>(n). The data storage apparatuses <NUM>(<NUM>)-<NUM>(n) and/or node computing device <NUM>(<NUM>)-<NUM>(n) of the examples described and illustrated herein are not limited to any particular geographic areas and can be clustered locally and/or remotely. The data storage apparatuses <NUM>(<NUM>)-<NUM>(n) and/or node computing device <NUM>(<NUM>)-<NUM>(n) can be distributed over a plurality of storage systems located in a plurality of geographic locations or a clustered network can include data storage apparatuses <NUM>(<NUM>)-<NUM>(n) and/or node computing device <NUM>(<NUM>)-<NUM>(n) residing in a same geographic location (e.g., in a single onsite rack).

One or more of the client devices <NUM>(<NUM>)-<NUM>(n), which are personal computers (PCs), computing devices used for storage (e.g., storage servers), and other computers or peripheral devices, are coupled to the respective data storage apparatuses <NUM>(<NUM>)-<NUM>(n) by storage network connections <NUM>(<NUM>)-<NUM>(n). Network connections <NUM>(<NUM>)-<NUM>(n) include a local area network (LAN) or wide area network (WAN)that utilizes Network Attached Storage (NAS) protocols, such as a Common Internet File System (CIFS) protocol or a Network File System (NFS) protocol to exchange data packets, a Storage Area Network (SAN) protocol, such as Small Computer System Interface (SCSI) or Fiber Channel Protocol (FCP), an object protocol, such as S3, etc..

Illustratively, the client devices <NUM>(<NUM>)-<NUM>(n) are general-purpose computers running applications, and interact with the data storage apparatuses <NUM>(<NUM>)-<NUM>(n) using a client/server model for exchange of information. That is, the client devices <NUM>(<NUM>)-<NUM>(n) request data from the data storage apparatuses <NUM>(<NUM>)-<NUM>(n) (e.g., data on one of the data storage devices <NUM>(<NUM>)-<NUM>(n) managed by a network storage control configured to process I/O commands issued by the client devices <NUM>(<NUM>)-<NUM>(n)), and the data storage apparatuses <NUM>(<NUM>)-<NUM>(n) return results of the request to the client devices <NUM>(<NUM>)-<NUM>(n) via the storage network connections <NUM>(<NUM>)-<NUM>(n).

The node computing devices <NUM>(<NUM>)-<NUM>(n) of the data storage apparatuses <NUM>(<NUM>)-<NUM>(n) include network or host nodes that are interconnected as a cluster to provide data storage and management services, such as to an enterprise having remote locations, cloud storage (e.g., a storage endpoint is stored within a data cloud). Such a node computing device <NUM>(<NUM>)-<NUM>(n) is a device attached to the fabric <NUM> as a connection point, redistribution point, or communication endpoint. One or more of the node computing devices <NUM>(<NUM>)-<NUM>(n) are capable of sending, receiving, and/or forwarding information over a network communications channel, and could comprise any type of device that meets any or all of these criteria.

The node computing device <NUM>(<NUM>) is located on a first storage site and the node computing device <NUM>(n) is located at a second storage site. The node computing devices <NUM>(<NUM>) and <NUM>(n) are configured according to a disaster recovery configuration whereby a surviving node provides switchover access to the storage devices <NUM>(<NUM>)-<NUM>(n) in the event a disaster occurs at a disaster storage site (e.g., the node computing device <NUM>(<NUM>) provides client device <NUM>(n) with switchover data access to storage devices <NUM>(n) in the event a disaster occurs at the second storage site). Additionally, while two node computing devices are illustrated in <FIG>, any number of node computing devices or data storage apparatuses can be included.

As illustrated in the clustered network environment <NUM>, node computing devices <NUM>(<NUM>)-<NUM>(n) include various functional components that coordinate to provide a distributed storage architecture. The node computing devices <NUM>(<NUM>)-<NUM>(n) include network modules <NUM>(<NUM>)-<NUM>(n) and disk modules <NUM>(<NUM>)-<NUM>(n). Network modules <NUM>(<NUM>)-<NUM>(n) are configured to allow the node computing devices <NUM>(<NUM>)-<NUM>(n) (e.g., network storage controllers) to connect with client devices <NUM>(<NUM>)-<NUM>(n) over the storage network connections <NUM>(<NUM>)-<NUM>(n), allowing the client devices <NUM>(<NUM>)-<NUM>(n) to access data stored in the clustered network environment <NUM>.

Further, the network modules <NUM>(<NUM>)-<NUM>(n) provide connections with one or more other components through the cluster fabric <NUM>. The network module <NUM>(<NUM>) of node computing device <NUM>(<NUM>) access the data storage device <NUM>(n) by sending a request via the cluster fabric <NUM> through the disk module <NUM>(n) of node computing device <NUM>(n). The cluster fabric <NUM> include one or more local and/or wide area computing networks embodied as Infiniband, Fibre Channel (FC), or Ethernet networksalthough other types of networks supporting other protocols can also be used.

Disk modules <NUM>(<NUM>)-<NUM>(n) are configured to connect data storage devices <NUM>(<NUM>)-<NUM>(<NUM>), such as SSDs to the node computing devices <NUM>(<NUM>)-<NUM>(n). Often, disk modules <NUM>(<NUM>)-<NUM>(n) communicate with the data storage devices <NUM>(<NUM>)-<NUM>(n) according to the SAN protocol, such as SCSI, FCP, SAS, NVMe, NVMe-oF, although other protocols can also be used. Thus, as seen from an operating system on node computing devices <NUM>(<NUM>)-<NUM>(n), the data storage devices <NUM>(<NUM>)-<NUM>(n) can appear as locally attached. In this manner, different node computing devices <NUM>(<NUM>)-<NUM>(n), etc. access data blocks through the operating system, rather than expressly requesting abstract files.

While the clustered network environment <NUM> illustrates an equal number of network modules <NUM>(<NUM>)-<NUM>(<NUM>) and disk modules <NUM>(<NUM>)-<NUM>(n), it can include a differing number of these modules. There may be a plurality of network and disk modules interconnected in a cluster that do not have a one-to-one correspondence between the network and disk modules. That is, different node computing devices can have a different number of network and disk modules, and the same node computing device can have a different number of network modules than disk modules.

Further, one or more of the client devices <NUM>(<NUM>)-<NUM>(n) and server devices <NUM>(<NUM>)-<NUM>(n) can be networked with the node computing devices <NUM>(<NUM>)-<NUM>(n) in the cluster, over the storage connections <NUM>(<NUM>)-<NUM>(n). Respective client devices <NUM>(<NUM>)-<NUM>(n) that are networked to a cluster request services (e.g., exchanging of information in the form of data packets) of node computing devices <NUM>(<NUM>)-<NUM>(n) in the cluster, and the node computing devices <NUM>(<NUM>)-<NUM>(n) return results of the requested services to the client devices <NUM>(<NUM>)-<NUM>(n). The client devices <NUM>(<NUM>)-<NUM>(n) exchange information with the network modules <NUM>(<NUM>)-<NUM>(n) residing in the node computing devices <NUM>(<NUM>)-<NUM>(n) (e.g., network hosts) in the data storage apparatuses <NUM>(<NUM>)-<NUM>(n).

The storage apparatuses <NUM>(<NUM>)-<NUM>(n) host aggregates corresponding to physical local and remote data storage devices, such as local flash or disk storage in the data storage devices <NUM>(<NUM>)-<NUM>(n). One or more of the data storage devices <NUM>(<NUM>)-<NUM>(n) include mass storage devices, such as disks of a disk array. The disks comprise SSDs adapted to store information, including, data (D) and/or parity (P) information.

The aggregates include volumes <NUM>(<NUM>)-<NUM>(n) in this example, although any number of volumes can be included in the aggregates. The volumes <NUM>(<NUM>)-<NUM>(n) are virtual data stores that define an arrangement of storage and one or more file systems within the clustered network environment <NUM>. Volumes <NUM>(<NUM>)-<NUM>(n) span a portion of a disk or other storage device, a collection of disks, or portions of disks, for example, and typically define an overall logical arrangement of file storage. Volumes <NUM>(<NUM>)-<NUM>(n) include stored data as one or more files or objects that reside in a hierarchical directory structure within the volumes <NUM>(<NUM>)-<NUM>(n). Volumes <NUM>(<NUM>)-<NUM>(n) are typically configured in formats that are associated with particular storage systems, and respective volume formats typically comprise features that provide functionality to the volumes <NUM>(<NUM>)-<NUM>(n), such as providing an ability for volumes <NUM>(<NUM>)-<NUM>(n) to form clusters.

To facilitate access to data stored on the disks or other structures of the data storage device <NUM>(<NUM>)-<NUM>(n), a file system (e.g., write anywhere file system) is implemented that logically organizes the information as a hierarchical structure of directories and files. Respective files are implemented as a set of disk blocks configured to store information, whereas directories may be implemented as specially formatted files in which information about other files and directories are stored.

Data are stored as files or objects within a physical volume and/or a virtual volume, which can be associated with respective volume identifiers, such as file system identifiers (FSIDs). The physical volumes correspond to at least a portion of physical storage devices, such as the data storage device <NUM>(<NUM>)-<NUM>(n) (e.g., a Redundant Array of Independent (or Inexpensive) Disks (RAID system)) whose address, addressable space, location, etc. does not change. Typically the location of the physical volumes does not change in that the (range of) address(es) used to access it generally remains constant.

Virtual volumes, in contrast, are stored over an aggregate of disparate portions of different physical storage devices. Virtual volumes may be a collection of different available portions of different physical storage device locations, such as some available space from disks. It will be appreciated that since the virtual volumes are not "tied" to any one particular storage device, virtual volumes can be said to include a layer of abstraction or virtualization, which allows them to be resized and/or flexible in some regards.

Further, virtual volumes can include one or more logical unit numbers (LUNs), directories, Qtrees, and/or files. Among other things, these features, but more particularly the LUNS, allow the disparate memory locations within which data is stored to be identified, and grouped as a data storage unit. As such, the LUNs are characterized as constituting a virtual disk or drive upon which data within the virtual volumes is stored within an aggregate LUNs are often referred to as virtual disks, such that they emulate a hard drive, while they actually comprise data blocks stored in various parts of a volume.

The data storage devices <NUM>(<NUM>)-<NUM>(n) have one or more physical ports, wherein each physical port is assigned a target address (e.g., SCSI target address). To represent respective volumes, a target address on the data storage devices <NUM>(<NUM>)-<NUM>(n) is used to identify one or more of the LUNs. Thus, when one of the node computing devices <NUM>(<NUM>)-<NUM>(n) connects to a volume, a connection between the one of the node computing devices <NUM>(<NUM>)-<NUM>(n) and one or more of the LUNs underlying the volume is created.

Respective target addresses can identify multiple of the LUNs, such that a target address can represent multiple volumes. The I/O interface, which is implemented as circuitry and/or software in a storage adapter or as executable code residing in memory and executed by a processor connect to volumes by using one or more addresses that identify the one or more of the LUNs.

Referring to <FIG>, node computing device <NUM>(<NUM>) includes processor(s) <NUM>, a memory <NUM>, a network adapter <NUM>, a cluster access adapter <NUM>, and a storage adapter <NUM> interconnected by a system bus <NUM>. The node computing device <NUM> also includes a storage operating system <NUM> installed in the memory <NUM> that can implement a Redundant Array of Independent (or Inexpensive) Disks (RAID) data loss protection and recovery scheme to optimize a reconstruction process of data of a failed disk or drive in an array. The node computing device <NUM>(n) is substantially the same in structure and/or operation as node computing device <NUM>(<NUM>), although the node computing device <NUM>(n) can include a different structure and/or operation in one or more aspects than the node computing device <NUM>(<NUM>).

The storage operating system <NUM> also manage communications for the node computing device <NUM>(<NUM>) among other devices that may be in a clustered network, such as attached to a cluster fabric <NUM>. Thus, the node computing device <NUM>(<NUM>) respond to client device requests to manage data on one of the data storage devices <NUM>(<NUM>)-<NUM>(n) (e.g., or additional clustered devices) in accordance with the client device requests.

The storage operating system <NUM> also establish one or more file systems including software code and data structures that implement a persistent hierarchical namespace of files and directories. When a new data storage device (not shown) is added to a clustered network system, the storage operating system <NUM> is informed where, in an existing directory tree, new files associated with the new data storage device are to be stored. This is often referred to as "mounting" a file system.

In the node computing device <NUM>(<NUM>), memory <NUM> include storage locations that are addressable by the processor(s) <NUM> and adapters <NUM>, <NUM>, and <NUM> for storing related software application code and data structures. The processor(s) <NUM> and adapters <NUM>, <NUM>, and <NUM> include processing elements and/or logic circuitry configured to execute the software code and manipulate the data structures.

The storage operating system <NUM>, portions of which are typically resident in the memory <NUM> and executed by the processor(s) <NUM>, invokes storage operations in support of a file service implemented by the node computing device <NUM>(<NUM>).

The executable instructions are configured to perform one or more steps of a method, described and illustrated later with reference to <FIG>.

The network adapter <NUM> includes the mechanical, electrical and signaling circuitry needed to connect the node computing device <NUM>(<NUM>) to one or more of the client devices <NUM>(<NUM>)-<NUM>(n) over storage network connections <NUM>(<NUM>)-<NUM>(n), which comprise, among other things, a point-to-point connection or a shared medium, such as a local area network. The network adapter <NUM> further communicates (e.g., using TCP/IP) via the fabric <NUM> and/or another network (e.g. a WAN) (not shown) with cloud storage devices to process storage operations associated with data stored thereon.

The storage adapter <NUM> cooperates with the storage operating system <NUM> executing on the node computing device <NUM>(<NUM>) to access information requested by one of the client devices <NUM>(<NUM>)-<NUM>(n) (e.g., to access data on a data storage device <NUM>(<NUM>)-<NUM>(n) managed by a network storage controller). The information are stored on SSDs adapted to store information.

In the data storage devices <NUM>(<NUM>)-<NUM>(n), information are stored in data blocks on disks. The storage adapter <NUM> include input/output (I/O) interface circuitry that couples to the disks over an I/O interconnect arrangement, such as a storage area network (SAN) protocol (e.g., Small Computer System Interface (SCSI), iSCSI, hyperSCSI, Fiber Channel Protocol (FCP)). The information is retrieved by the storage adapter <NUM> and, if necessary, processed by the processor(s) <NUM> (or the storage adapter <NUM> itself) prior to being forwarded over the system bus <NUM> to the network adapter <NUM> (and/or the cluster access adapter <NUM> if sending to another node computing device in the cluster) where the information is formatted into a data packet and returned to a requesting one of the client devices <NUM>(<NUM>)-<NUM>(<NUM>) and/or sent to another node computing device attached via the cluster fabric <NUM>. A storage driver <NUM> in the memory <NUM> interfaces with the storage adapter to facilitate interactions with the data storage devices <NUM>(<NUM>)-<NUM>(n), as described and illustrated in more detail later with reference to <FIG>.

Referring to <FIG>, a method for managing distributed snapshot for low latency storage will now be described. In step <NUM> the node computing device <NUM>(<NUM>) receiving a request to create a snapshot of the primary and the secondary storage, wherein the primary and the secondary storage includes primary and secondary solid state devices (SSD) respectively, although the node computing device <NUM>(<NUM>) can receive other types or number of requests. The node computing device <NUM>(<NUM>) receives the request to create the snapshot when it is determined that a point in time snapshot image is required to allow a full data or a delta update to be made from the primary storage to the secondary storage.

Next in step <NUM>, the node computing device <NUM>(<NUM>) quiesces the associated logical volumes requiring the snapshot creation on both the primary storage and the secondary storage. Quiesces relates to queueing all the new input/output requests and allowing all outstanding input/output requests to be completed for all associated LBA extents to all involved SSD,.

In step <NUM>, the node computing device <NUM>(<NUM>) issues a command to create a snapshot to all the SSDs present in both the primary and the secondary storage that is involved with the specified aggregator or volumes. The command to create the snapshot includes the addressing technique to be used to access the snapshot entity on each of the SSD. Additionally, the SSDs using the flash translation layer (FTL) creates a snapshot allocated capacity as a response to the request and maintains the old data in that region. The snapshot allocated capacity as illustrated in <FIG> is a virtual region. The SSD flash translation layer (FTL) presents a primary LUN to the host by presenting the LBA extents that have been written, along with the LBA extents that has not been written. Further, the SSD FTL utilizes the over provisioned capacity and the unused capacity in the logical division of the SSD to allow efficient management of the flash media when writes and data scrubs of the SSD are required. Additionally, the SSD FTL also presents to the host a second snap LUN that provides data to the host reflected at the time the snapshot is going to be created. Once the snapshot allocated capacity is created, each of the SSDs creates a snapshot and stores the snapshot in the created snapshot allocated capacity space.

Next in step <NUM>, the node computing device <NUM>(<NUM>) acquires the access point for the new snapshot volume provided by each of the SSDs.

In step <NUM>, the node computing device <NUM>(<NUM>) performs a quiesce release operation by releases the input/output queues that were being held in step <NUM>. By releasing the input/output queues all the new input/output operations can resume the operations on the primary and the secondary storage.

Next in step <NUM>, the node computing device <NUM>(<NUM>) utilizes the snapshots accessed from the SSDs to perform the full backup or the delta backup from the primary storage to the secondary storage. Node computing device <NUM>(<NUM>) moves data from the snapshot LUN presented by the primary storage to the original LUN on the secondary storage provided by the secondary storage volume. This data movement is done by reading some appropriate number of LBA extents from the snapshot LUN on the primary storage volume and writing these same LBA extents to the original LUN on the secondary storage volume. Additionally, writing is completed to the original LUN on the secondary storage to allow the secondary storage to continue to access the original LUNs during the update. The snapshot is access by the local application as required and the data that is being copied or updated is written to the original remote volume. Once the original LUN has been updated by the backup operation, the secondary storage will typically delete the snapshot LUN and access the now updated original LUN. Additionally, all LBA reads to the original LUN are performed using normal FTL function with the original LUN's metadata from the utilized capacity region illustrated in <FIG>. With respect to writes to the original LUN, for the extents that have not been written, the SSD's FTL allocates data from the unused or over provisioned capacity regions and writes the new data to this allocated capacity. The original LUN's metadata is updated by the SSD's FTL to reflect this new LBA extent in the original LUN's metadata. Furthermore, the SSD's FTL updates the snap LUN's metadata reflecting the previous LBA extent. Now with respect to the LBA extents that have been written previously and are located in the utilized capacity region illustrated in <FIG>, are simply updated per normal FTL function for the original LUN and there is no impact to the snap LUN.

With respect to the snap LUN, the reads to the snap LUN is provided using the following technique: LBA Extents associated with the original LUN that has not been written since the point in time (PIT) snapshot was created is read from the utilized capacity region. LBA extents that have been written since the PIT snapshot was created, is read from the snapshot allocated capacity region. With respect to the writes to the snap LUN, extents that have not been written: the SSD's FTL allocates data from the unused or over provisioned capacity regions and writes the new data to this allocated capacity. The new data is associated with or is part of the snapshot allocation capacity region. The snapshot LUN's metadata is updated by the SSD's FTL to reflect this new LBA extent in the snap LUN's metadata. Furthermore, LBA extents that have been written previously to the original LUN indicates that the snap LUN's LBA extent already exists in the snapshot allocated capacity and are updated per normal FTL function operating on the snap LUN's metadata within the snapshot allocated capacity region illustrated in <FIG>.

In step <NUM>, the node computing device <NUM>(<NUM>) determines when the data backup is complete. The data backup is complete when either the full data or the delta data has been completely transferred from the primary storage to the secondary storage. Accordingly, when the node computing device <NUM>(<NUM>) determines that the data backup is not complete, a No branch is taken back to step <NUM>. However, when the node computing device <NUM>(<NUM>) determines that the data backup is complete, then the Yes branch is taken to step <NUM>.

In step <NUM>, the node computing device <NUM>(<NUM>) deletes the snapshot that was stored in the snapshot allocated capacity of the SSDs to free up the snapshot allocated capacity in the SSDs and the exemplary method ends in step <NUM>. With this technology, normal snapshot processing can be accelerated by using low latency SSDs.

Claim 1:
A method, comprising:
receiving, by a computing device, a request to create a snapshot of a primary storage and a secondary storage, wherein the primary and the secondary storage includes primary and secondary solid state devices (SSDs), respectively;
quiescing, by the computing device, prior to creating the snapshot, all logical volumes requiring the snapshot on the primary storage and the secondary storage by allowing one or more outstanding input/output requests to be completed and queueing all the new input/output requests associated with the logical volumes;
issuing, by the computing device, a command to create a snapshot to all the SSDs present in both the primary and the secondary storage, the SSDs using a flash translation layer (FTL) to create, as a response to the command, a snapshot allocated capacity space in a virtual region, create and store a snapshot in the snapshot allocated capacity space, present a first original LUN to the computing device by presenting first LBA extents that have been written, along with second LBA extents that have not been written and a second snapshot LUN that provides data to the computing device reflected at the time the snapshot is going to be created;
accessing, by the computing device the snapshot created by the SSDs;
performing, by the computing device, a quiesce release operation by releasing the input/output queues that were being held such that all the new input/output operations can resume the operations on the primary and the secondary storage;
initiating, by the computing device, a data transfer operation from the primary storage to the secondary storage, wherein the data transfer operation moves data from the snapshot LUN presented by the primary storage volume by reading third LBA extents from the snapshot LUN on the primary storage volume and writing these same LBA extents to the original LUN on the secondary storage volume, wherein, during the data transfer operation, the snapshot on the secondary device is accessed by local applications as required;
determining, by the computing device, when the data transfer operation is completed; and
deleting, by the computing device, the snapshot that was stored in the snapshot allocated capacity of the SSDs of both the primary and the secondary storage volumes , when the determination indicates the data transfer operation is completed.