Delta information volumes to enable chained replication of data by uploading snapshots of data to cloud

Method and apparatus for managing data in a distributed data storage system, such as but not limited to a cloud computing environment. In some embodiments, snapshots of a data set are uploaded from a source storage subsystem to a cloud store, along with intervening difference information volumes (DIVs). The DIVs are data structures that identify a set of updated data blocks that have been changed by the source storage subsystem between each successive pair of the snapshots. A reader subsystem requests and uses the latest DIV to request the latest set of changed data blocks from the cloud store, and uses the latest set of changed data blocks to update a previous snapshot to generate a copy of the most recent snapshot. The source and reader subsystems can comprise Internet of Things (IoT) devices, client devices, edge computing devices, etc. from different vendors and which utilize different protocols.

SUMMARY

Various embodiments of the present disclosure are generally directed to a method and apparatus for managing data in a distributed data storage system, such as but not limited to a cloud computing environment.

In some embodiments, snapshots of a data set are uploaded from a source storage subsystem to a cloud store, along with intervening difference information volumes (DIVs). The DIVs are data structures that identify a set of updated data blocks that have been changed by the source storage subsystem between each successive pair of the snapshots. The latest DIV is used to identify and transfer the latest set of changed data blocks from the cloud store to a reader subsystem. The reader subsystem uses the latest set of changed data blocks to update a previous snapshot to generate a copy of the most recent snapshot. In some embodiments, the source and reader subsystems can comprise Internet of Things (IoT) devices, client devices, edge computing devices, etc. from different vendors and which utilize different protocols.

These and other features and advantages which characterize the various embodiments of the present disclosure can be understood in view of the following detailed discussion and the accompanying drawings.

DETAILED DESCRIPTION

The present disclosure generally relates to systems and methods for storing data in a distributed data storage environment, such as but not limited to a cloud computing environment.

Cloud computing environments generally operate to store and process data across a geographically distributed network. Network services such as computational resources, software and/or data are made available to various user devices (clients) via a wide area network, such as but not limited to the Internet.

A cloud computing network is generally arranged as an object storage system whereby storage volumes (e.g., files) from the clients are stored in, and replicated to, various storage locations throughout the system in the form of snapshots (copies at a particular point in time). Cloud computing often includes the use of cloud stores (e.g., large scale data centers, etc.) to provide industrial-scale storage and processing resources.

Edge computing refers to a cloud storage strategy where at least certain types of storage and processing resources are located at the “edge” of the network. Instead of having the main cloud stores perform most of the data processing, intermediary type devices can interface between the client devices and the main cloud stores. This can reduce the bandwidth requirements on the system and provide better response times and other efficiencies. Edge computing has particular applicability to so-called IoT (Internet of Things) devices, since IoTs tend to generate massive amounts of data. It makes sense to decentralize certain data storage and replication efforts to locations that are logically and/or physically closer to the sources of the data generation activities.

While operable, a limitation with this approach relates to the use of equipment from different storage system vendors to access and replicate data sets. Due to differences in architecture and data storage protocols, as well as the general undesirability (or unwillingness) of different vendors to share proprietary algorithms and information, a data replication operation across different vendor platforms may require the transfer of an entire volume of data, even if only a few changes have been made since the last recorded snapshot. This can have undesirable effects on network performance.

Accordingly, there is a continual need for improvements in the manner in which data volumes can be efficiently accessed and duplicated among numerous reader subsystems without consuming significant network resources. Various embodiments of the present disclosure are directed to a method and apparatus to address these and other needs.

In some embodiments, a computer network is provided. The network may be a cloud computing environment, may utilize edge computing devices, and may involve the processing of IoT style data, but other configurations and environments can be utilized so these characterizations are merely exemplary and are not limiting.

A distributed data set from a source storage subsystem is stored as a first volume in a cloud storage system. Without limitation, the source storage subsystem (“source”) may take the form of a client device, an edge computing device, etc. Regardless, the source has the capability of generating and updating data sets from a user. The cloud storage system (“cloud store”) may take the form of a network server or some other processing/storage element in the network configured to store the data sets from the source.

At such time that a replication operation is desired, a reader storage subsystem (“reader”) requests and receives the DIV from the cloud store. The reader may take any number of suitable forms such as another network server, client device, edge computing device, etc. The reader uses the DIV in combination with a prior snapshot to replicate the latest distributed data set in a local memory of the reader. In this way, only the most recently changed data blocks can be requested and retrieved from the cloud store, rather than requiring a complete transfer of the latest snapshot. Metadata can be generated and tracked to indicate the replication activities among various readers, and this metadata can be accessed by other readers as well as the cloud store.

These and other features and advantages of various embodiments can be understood beginning with a review ofFIG. 1which shows an exemplary data storage device100. The device100is characterized as a solid-state drive (SSD), although other configurations can be used.

The device100includes a controller circuit102which provides top-level control and communication functions as the device interacts with a host device (not shown) to store and retrieve host user data. A memory module104provides a non-volatile memory (NVM) store for data from the host device. The controller102may take the form of one or more programmable CPU processor(s) and the memory may include 3D NAND flash memory.

FIG. 2represents a network110in which a number of storage devices such as100fromFIG. 1may be implemented. The network110is characterized as a cloud computing environment and includes a number of client devices112, edge devices114, cloud servers116and cloud storage devices118.

The client devices112can take a number of user device forms including computers, laptops, smart phones, tablets, gaming systems, IoT devices, etc. The edge devices114can provide localized data processing and computation services for the client devices including servers, routers, computers, etc.

Data centers may be used to implement the cloud servers116and the cloud storage devices to provide high capacity computation and storage capabilities. It will be appreciated that the cloud computing environment110ofFIG. 2is merely illustrative and the various embodiments presented herein can be adapted for any number of different environments as required.

FIG. 3shows another network120similar to the network110ofFIG. 2. The network120includes a source storage subsystem122, a cloud storage system124and a number of reader storage subsystems126. As explained below, data provided by the source storage subsystem122(“source”) to the cloud storage system124(“cloud store”) is desired to be replicated onto the respective reader storage subsystems126(“readers”). While not required, it is contemplated that the various elements may be provided by different vendors and operate using different protocols, proprietary information, etc.

In one example, the source122corresponds to one of the client devices112inFIG. 2, the cloud store124corresponds to one of the cloud servers116inFIG. 2, and the various readers126may correspond to some other element of the system including an edge device114, another client device112, another cloud server116, another set of cloud storage devices118, or some other device in the network.

As noted above, one problem that may be encountered is that, for each of the readers desiring to pull data from the cloud store, the reader does not have immediate knowledge of which data has changed between each snapshot. In order to replicate the data, the reader subsystems would have to request the entire volume image, which can be prohibitive at cloud speeds (or even local attached speeds).

Since the cloud store may enact a third party cloud infrastructure, proprietary code may be operating in the cloud store and so it may not be feasible to access this code in order to determine the differences between the snapshots. Bypassing the cloud store and having the reader subsystems contact the source subsystem directly is not a viable option due to communication, authentication and compatibility limitations, as well as on the basis that this would tend to run counter to the goal of easy inter-vendor interoperability among the various cloud attached devices.

Accordingly, as discussed below, each source such as122operates to generate difference information in a new volume in the cloud store124after the source has used that same information in transferring the data. This difference information is referred to herein as a “delta information volume” (DIV), and represents a storage volume in the cloud that includes metadata identifying the changed portions between two successive snapshots that are also stored in the cloud. When multiple successive snapshots are uploaded, a corresponding set of intervening DIVs will be provided, with each DIV identifying a set of updated data blocks between each successive pair of the snapshots (e.g., a former or most previously generated snapshot and a latter or latest snapshot, etc.).

The contents of the DIV can take a variety of forms. In some embodiments, the DIV includes metadata that identifies information associated with both of the respective bounding snapshots. These can be referred to as snapshots RSn and RSn+1. The DIV may also identify information associated with the source, such as address, timestamp data, unique data associated with the snapshots (e.g., hashes, etc.), and so on. In some cases, a bitmap data structure can be included which indicates which blocks are different between the respective snapshots. Other data can be included in the DIV as desired, including in some cases some or all of the actual data blocks that are changed from one snapshot to the next.

Each reader (e.g.,126) that scans and reads the cloud store124can identify the two latest snapshot volumes and also access the intervening DIV. From this information, the reader can perform a reverse volume copy. For example, the reader system can access (either locally or retrieve) a base volume corresponding to the previous snapshot, and then request, as required, the necessary changed blocks to provide the most up-to-date system volume. In some cases, a tree structure can be used to track all changes, and as many DIV volumes can be accessed as required to recreate and update the local data by the local reader.

If the respective readers are endeavoring to maintain up-to-date replicas of the data volume, the first replication operation may require a full transfer of the first snapshot. From that point forward, however, only the DIV and updated data blocks (as well as any other information as required) need to be transferred for subsequent replication operations. Depending on operability, it may be possible for a second reader (reader storage subsystem2) to request and obtain data (e.g., the DIV, etc.) from a first reader (reader storage subsystem1) to further reduce the data requirements upon the cloud store so that chained replication may occur among the various readers. This is represented by an optional dotted line arrow inFIG. 3.

FIG. 4is a functional block representation of the source storage subsystem122and the cloud storage system124fromFIG. 3in accordance with some embodiments. The source122includes various elements including an intelligent replication module130A operable to carry out the various tasks discussed above. The module130A can take a variety of forms, including one or more programmable processors that use program instructions in the form of firmware (FW) which are loaded to a local memory for execution. Other configurations can be used, including hardware based circuits, etc.

The source122further includes memory in which various data sets and structures are stored, including a storage volume132A, a first snapshot134A, a second snapshot136A, a delta information volume (DIV)138A, and an optional first tree structure140A. Other configurations can be used, including configurations with fewer or other data sets and structures. For example, the source122may generate these various elements but not necessarily retain these over time during normal operation.

Generally, the storage volume132A represents the most up-to-date localized version of a data set owned/controlled by a user of the source122. The data set can take any number of forms, but generally represents data that is periodically uploaded/saved to the cloud via transfer to the cloud store124.

The first snapshot134A represents a first (prior version) of the storage volume that was taken at a first point in time. The second snapshot136A is a second (current version) of the storage volume that was taken at a subsequent second point in time. The DIV138A indicates the differences between the first and second snapshots. The optional tree structure140A shows history metadata that tracks these other elements.

The cloud store124receives periodic updates from the source122for cloud storage of the data. The cloud store124is shown inFIG. 4to include a second intelligent replication module130B. This may take a similar form to the first module130A, or may take some other form. Various data sets and structures that correspond to those in the source122include a storage volume copy132B, a first snapshot copy134B, a second snapshot copy136B, a DIV copy138B and a second tree structure140B. As before, other formats may be used.

In one example, the source122operates, via user interface, a generation operation to generate an initial storage volume. At some point in time, this may be uploaded to the cloud store124. Updates may be generated in the form of snapshots which are also uploaded. In some cases, the entire updated data set may be uploaded. In other cases, the DIV information along with the updated data blocks may be uploaded so that the cloud store assembles and stores the updated storage volumes.

It will thus be appreciated that the DIV information, which is stored as a separate volume in the cloud store, can be also used by the cloud store as well as by the readers126. The tree structure140B may be generated by the cloud store124in order to track the various hierarchical changes made to each successive generation of the data. Generally, it is contemplated that the various data sets and structures may be initially generated locally and provided by the source122to the cloud124, but such is not necessarily required. In other embodiments, the user (or a different user) may be authorized to make further changes to the storage volume copy at the cloud storage level.

FIG. 5is an exemplary format for the DIV copy138B maintained by the cloud store124. Other formats can be used. The DIV includes a first snapshot descriptor field142, a second snapshot descriptor field144, a source identification (ID) information field146and a bitmap of changed data blocks field148. The field142generally provides descriptive information that describes the first snapshot (e.g., copy134A). This can include location, datestamp, authentication, security (e.g., hash, etc.) and other types of appropriate information to enable location and accessing of the first snapshot copy134A. The field144provides similar information for the second snapshot copy136B.

The field146provides relevant information regarding the source. This may include information relating to the originating device, datestamp information, a list of authorizations/security features and requirements, etc. The field148may identify a bitmap as a data structure that identifies which data blocks or other portions of the storage volume have been modified in the second snapshot as compared to the first snapshot. In some cases, the actual changed portions (e.g., the updated data blocks) may be included in the DIV, although it is contemplated that providing a listing of the changed blocks will be sufficient for the reader to request those blocks from the second snapshot. The blocks can be identified in any number of suitable ways including the use of logical block addresses (LBAs), virtual block addresses (VBAs), key-values, object names, etc. The blocks may be identified using the same, or a different, nomenclature as used by the source122.

FIG. 6shows aspects of the cloud store124and a selected reader126fromFIG. 3in accordance with further embodiments. Other configurations can be used. As with the source122and the cloud store124, the reader126includes an intelligent replication module130C, which may be configured as described above. The reader126inFIG. 6generally operates to periodically generate a local replica of the latest storage volume from the source122.

In some cases, the replicas will be generated as soon as new updates are provided by the source to the cloud store, so that the replicas at the reader subsystem level are nominally maintained in near real-time fashion (clearly, replicas cannot be generated instantaneously but they can be generated as soon as possible). In this approach, the cloud store may use the tree structure to identify those reader subsystems that are maintaining up-to-date replicas, and may provide a notification that updated data have been received from the source.

In other cases, the reader may only periodically require copies of up-to-date replica data to perform certain functions. In this case, the reader may initiate the request, and one (or more) intervening DIVs may be used in order to build the latest replica at the reader level. If multiple DIVs are used to build the latest replica set, a backward search process may be used starting with the latest DIV and working backwards, with the knowledge that once a latest version of a data block is located, the previous intervening changes to the data block can be skipped.

In one example, an initial copy of the storage volume (e.g., snapshot1) is transferred to the reader126to provide a locally stored (baseline) copy. At such time that an updated copy is required, the reader126requests the latest DIV, which is returned and evaluated. A request for the updated blocks is issued, and these are returned and combined with the first snapshot. As desired, completion data such as in the form of a second DIV or other form of metadata may be generated and transferred to the cloud store. In this way, the cloud store may be able to track what replicas are present in the system along with datecode and other suitable information.

It is contemplated although not necessarily required that the reader126actually request a copy of the DIV be transferred from the cloud store124. In other embodiments, the system may be configured to automatically provide a notification and have the cloud store124pull and broadcast the updated blocks to each active reader, which in turn can perform the necessary updates to generate locally an associated replica copy.

FIG. 7shows a sequence diagram150to illustrate a source side uploading of data to the cloud store in accordance with the foregoing discussion. This is merely exemplary and other steps and operations may be used as desired. Initially, a storage volume is generated by a source device at block152, and the storage volume is uploaded to the cloud at block154.

At some point in the future, the user of the source updates the volume at block156, generates a first snapshot and a first DIV at block158, and uploads these data sets and structures to the cloud at block160. As noted above, the entire snapshot can be uploaded, or only those blocks that were changed over the initial storage volume may be uploaded. An advantage of uploading the entire snapshot relates to reducing the processing required at the cloud store level, which is likely more configured to simply receive and process new data sets rather than performing significant data processing/updating of existing data sets. Nevertheless, either approach may be carried out as required.

The cloud store proceeds at block162to generate an initial tree structure at the cloud level to identify the relationships and storage locations of the respective storage volume, first snapshot and DIV. Depending on retention policies, the original storage volume (and all intervening snapshots after that) may be retained by the cloud store, even if such earlier versions are not made visible/accessible to the user of the source. The various data sets and structures may be subjected to replication for cloud store use, such as by providing copies to other cloud store servers, etc. in a known manner to ensure the stored data can be retrieved and accessed quickly and reliably.

The source continues to provide another update to the local storage volume at block164, and the previously described actions are repeated at blocks166,168and170. This process continues so long as the source continues to generate and upload new, updated copies of the storage volume to the cloud. At any given time, the cloud store is maintained in a condition ready to service requests for the latest volume data from one or more readers, which will now be discussed inFIG. 8.

FIG. 8shows a sequence diagram180to illustrate a reader side replication operation to replicate the data uploaded inFIG. 7. At block182, a command is received by the cloud store to replicate the data. As noted above, this is not necessarily required; in other embodiments, the replication may take place automatically as soon as the updated DIV is received by the cloud store from the source.

The cloud store operates at block184to access the tree structure to identify the latest DIV as well as the latest snapshots described thereby. As noted above, access to the latest snapshot (RSn+1) is likely going to be carried out, while access to the next latest snapshot (RSn) may or may not be, depending on the requirements of a particular application. The appropriate data sets and structures are transferred to the reader at block186, which uses this information to generate a request for the appropriate updated data blocks from the latest snapshot (RSn+1) at block188.

The reader proceeds to use the retrieved data to generate a copy of the latest snapshot (e.g., local replica) of the storage volume at block190, after which updated DIV information associated with the reader operation is transferred to the cloud store at block192. As before, the foregoing steps are repeated as required to maintain the local copy of the storage volume in an up-to-date state.

FIG. 9shows another network200similar to the network discussed above inFIG. 2. The network includes a number of client devices202which are connected to a cloud network204to various elements including a cloud store206and various replication devices208.

As provided previously, the client devices202can take a number of forms including IoT devices, edge devices, local computers, etc. It is contemplated that the client devices correspond to the various source devices and are configured to generate/aggregate/process new data volumes as required. The cloud store206similarly provides main cloud storage processing and may represent servers in one or more geographically distributed data centers. The replication devices208are various types of readers that endeavor to replicate copies of the data from the clients.

FIG. 10shows a functional block representation of aspects of the network200fromFIG. 9in some embodiments. The client device202includes a controller210and memory212. The cloud store206similarly has a controller214and memory216, and the replication device has yet another controller218and memory220.

It will be appreciated that the memory212in the client202may include both local memory (e.g., DRAM) as well as a main NVM store, such as flash memory or the like as described inFIG. 1. The controller210, also sometimes referred to as a source storage controller circuit, similarly may comprise hardware and/or programmable processor circuits to enable the generation/processing of the storage volumes uploaded to the cloud store206.

The cloud store206may be a server so that the controller214, also referred to as a cloud storage controller circuit, is one or more banks of various forms of communication and storage processors with associated programming. The memory may include racks of storage devices (e.g., SSDs, HDDs, etc.) to provide a combined cloud storage space (e.g., a large main NVM store), as well as local volatile or non-volatile cache to cache the data transferred between the storage devices and the other elements in the network.

The replication device208is contemplated as also including one or more programmable processors as the controller218, also sometimes referred to as a reader storage controller circuit. The memory220may be arranged in the form of both cache and NVM to store and utilize the replicated data sets.

FIGS. 11 and 12show aspects of a storage node230that can form a portion of the cloud store206ofFIG. 10in some embodiments. Other architectures can be used.

The storage node230inFIG. 11includes a storage assembly232and a storage controller (computer/server)234. The storage assembly232may include one or more server cabinets (racks)236with a plurality of modular storage enclosures238. In some cases, multiple zones may be defined in a single rack, multiple racks may constitute the same zone, etc. Moreover, the controller234can be incorporated into the rack or elsewhere as required.

The storage rack is a 42 U server cabinet with 42 units (U) of storage, with each unit extending about 1.75 inches (in) of height. The width and length dimensions of the cabinet can vary but common values may be on the order of about 24 in.×36 in. Each storage enclosure238can have a height that is a multiple of the storage units, such as 2 U (3.5 in.), 3 U (5.25 in.), etc.

FIG. 12shows a selected enclosure238to incorporate36(3×4×3) data storage devices240. Other numbers of data storage devices can be incorporated into each enclosure. The data storage devices240can take a variety of forms, such as hard disc drives (HDDs), solid-state drives (SSDs), hybrid drives (Solid State Hybrid Drives, SDHDs), etc. The storage devices240provide a portion of the main NVM store of the memory216(FIG. 10). Additional elements in the storage enclosure can include power supplies242, fans244and a controller board246with processor (CPU)248.

The various storage volumes, snapshots, DIV structures and tree structures can be managed by the storage node, with copies maintained in the storage devices240. In some cases, appropriate levels of controller functionality can be selected to provide the DIV and data block management as described above. The various storage volumes, snapshots and DIV volumes may be replicated and stored using parallel channel access paths to allow efficient, concurrent transfer operations.

It will be appreciated that using the next to last snapshot (RSn) plus the DIV information to form the chained replicas can be advantageous for a number of reasons. Not only does this reduce the need to send a complete copy of each snapshot to the various reader devices, but it is likely that the previous snapshot will have already been replicated and stored to the local storage devices in the cloud store, while the new snapshot (RSn+1) may still be in cache pending transfer/replication.

In some cases the data blocks requested/needed by the reader device(s) may be present in the local cloud store cache, and can be serviced using a cache hit mechanism so that it is not necessarily required to access the storage devices. Stated another way, the requested data blocks can both transferred to the readers while the data blocks are also being written to the storage devices at the cloud level. This can further reduce the overhead processing required to obtain the replicated copies, and can in some cases result in faster replication of the data. Write cache retention may be enacted by the cloud store controller214to temporarily retain the affected data blocks so that the blocks can be forwarded prior to being jettisoned from the cache. The visibility afforded by the DIV enables cache manager aspects of the controller214to quickly and easily identify these blocks.