Reducing the amount of data transferred to remote storage locations for modified objects

A computer-implemented method, according to one embodiment, includes: receiving, by a computer, a data access request; sending, by the computer, a recall request to a remote storage location for data which corresponds to the data access request; and receiving, by the computer, a copy of an existing object which includes blocks. The data which corresponds to the data access request is stored in at least one of the blocks. The data access request is satisfied, by the computer, by providing the copy of the existing object. Moreover, a sparse object, which only includes ones of the blocks which contain data that has been modified, is received by the computer. The sparse object is sent, by the computer, to the remote storage location; and one or more instructions to use the blocks included in the sparse object to update the existing object are also sent by the computer.

BACKGROUND

The present invention relates to data storage systems, and more specifically, this invention relates to maintaining updated copies of data across different storage locations in a distributed storage system.

Distributed storage systems attempt to offer the advantages of centralized storage with the scalability and cost base of local storage. Moreover, a distributed storage system can relate to block-level storage, file-based storage, or object-based storage (also referred to herein as “object store” or “object storage”). In case of block-level storage systems “distributed data storage” typically relates to one storage system in a tight geographical area, usually located in one data center, since performance demands are typically high. However, object-based storage systems can be located in one or more locations. Accordingly, geographically distributed storage systems are achievable.

For example, a distributed object store is made up of many individual object stores, each of which typically include a small number of physical storage disks. These object stores operate using commodity server hardware, which may include compute nodes or separate servers configured to provide storage services. As such, the hardware is relatively inexpensive. Moreover, a storage network is used to communicate between the various object stores.

Currently, regulatory bodies mandate the retention of critical data for significant periods of time for compliance purposes. Data is also retained for substantial amounts of time for other business purposes, such as historical analysis. In order to satisfy these requirements, organizations typically move data that is no longer being actively used (e.g., “cold” data) to storage types and/or locations which are less expensive, e.g., such as cloud-based storage. Object store is often the preferred storage format in cloud storage environments, as it offers significant scalability as well as allowing for the storage and retrieval of large amounts of data.

One of the ways which object storage derives its high scalability characteristics stems from the fact that it implements a simplified set of permitted operations (“Retrieval” and “Creation”). Object storage also imposes restrictions that objects are fundamentally immutable. Certain applications allow customers to migrate data from an on-premise file system to a cloud storage location. Moreover, these applications support the migration of “cold data” to the cloud location and also provide recall capabilities, both of which are done transparently to the on-premise applications.

SUMMARY

A computer-implemented method, according to one embodiment, includes: receiving, by a computer, a data access request; sending, by the computer, a recall request to a remote storage location for data which corresponds to the data access request; and receiving, by the computer, a copy of an existing object which includes blocks. The data which corresponds to the data access request is stored in at least one of the blocks. The data access request is satisfied, by the computer, by providing the copy of the existing object. Moreover, a sparse object, which only includes ones of the blocks which contain data that has been modified, is received by the computer. The sparse object is sent, by the computer, to the remote storage location; and one or more instructions to use the blocks included in the sparse object to update the existing object are also sent by the computer.

A computer-implemented method, according to another embodiment, includes: receiving, by the computer, a recall request from a remote location for data which corresponds to a data access request; and sending, by the computer, a copy of an existing object which includes more than one block to the remote location. The data which corresponds to the data access request is stored in at least one of the blocks. A sparse object is also received, by the computer, from the remote location. The sparse object only includes ones of the blocks which contain data that has been modified. Accordingly, the modified data in the blocks of the sparse object is used, by the computer, to update the existing object.

A computer program product, according to yet another embodiment, includes a computer readable storage medium having program instructions embodied therewith. The computer readable storage medium is not a transitory signal per se. Moreover, the program instructions readable and/or executable by a processor to cause the processor to perform a method which includes: sending, by the processor, a data access request; and receiving, by the processor, a copy of an existing object which includes blocks. Data which corresponds to the data access request is stored in at least one of the blocks. A snapshot of the received copy of the existing object is captured by the processor, and the received copy of the existing object is also used, by the processor, to perform a data operation. The snapshot is further used, by the processor, to determine ones of the blocks which include data that has been modified by the data operation. A sparse object which only includes the ones of the blocks determined as including modified data is thereby sent by the processor.

DETAILED DESCRIPTION

The following description discloses several preferred embodiments of systems, methods and computer program products which are able to achieve significant improvements to the efficiency by which storage systems are able to manage data stored therein. Particularly, some of the embodiments described herein reduce the amount of data which is transferred between storage locations, thereby reducing the amount of network bandwidth, computational resources, system throughput, etc. consumed in order to satisfy received data requests and/or operations, e.g., as will be described in further detail below.

In one general embodiment, a computer-implemented method includes: receiving, by a computer, a data access request; sending, by the computer, a recall request to a remote storage location for data which corresponds to the data access request; and receiving, by the computer, a copy of an existing object which includes blocks. The data which corresponds to the data access request is stored in at least one of the blocks. The data access request is satisfied, by the computer, by providing the copy of the existing object. Moreover, a sparse object, which only includes ones of the blocks which contain data that has been modified, is received by the computer. The sparse object is sent, by the computer, to the remote storage location; and one or more instructions to use the blocks included in the sparse object to update the existing object are also sent by the computer.

In another general embodiment, a computer-implemented method includes: receiving, by the computer, a recall request from a remote location for data which corresponds to a data access request; and sending, by the computer, a copy of an existing object which includes more than one block to the remote location. The data which corresponds to the data access request is stored in at least one of the blocks. A sparse object is also received, by the computer, from the remote location. The sparse object only includes ones of the blocks which contain data that has been modified. Accordingly, the modified data in the blocks of the sparse object is used, by the computer, to update the existing object.

In yet another general embodiment, a computer program product includes a computer readable storage medium having program instructions embodied therewith. The computer readable storage medium is not a transitory signal per se. Moreover, the program instructions readable and/or executable by a processor to cause the processor to perform a method which includes: sending, by the processor, a data access request; and receiving, by the processor, a copy of an existing object which includes blocks. Data which corresponds to the data access request is stored in at least one of the blocks. A snapshot of the received copy of the existing object is captured by the processor, and the received copy of the existing object is also used, by the processor, to perform a data operation. The snapshot is further used, by the processor, to determine ones of the blocks which include data that has been modified by the data operation. A sparse object which only includes the ones of the blocks determined as including modified data is thereby sent by the processor.

Now referring toFIG. 3, a storage system300is shown according to one embodiment. Note that some of the elements shown inFIG. 3may be implemented as hardware and/or software, according to various embodiments. The storage system300may include a storage system manager312for communicating with a plurality of media and/or drives on at least one higher storage tier302and at least one lower storage tier306. The higher storage tier(s)302preferably may include one or more random access and/or direct access media304, such as hard disks in hard disk drives (HDDs), nonvolatile memory (NVM), solid state memory in solid state drives (SSDs), flash memory, SSD arrays, flash memory arrays, etc., and/or others noted herein or known in the art. The lower storage tier(s)306may preferably include one or more lower performing storage media308, including sequential access media such as magnetic tape in tape drives and/or optical media, slower accessing HDDs, slower accessing SSDs, etc., and/or others noted herein or known in the art. One or more additional storage tiers316may include any combination of storage memory media as desired by a designer of the system300. Also, any of the higher storage tiers302and/or the lower storage tiers306may include some combination of storage devices and/or storage media.

The storage system manager312may communicate with the drives and/or storage media304,308on the higher storage tier(s)302and lower storage tier(s)306through a network310, such as a storage area network (SAN), as shown inFIG. 3, or some other suitable network type. The storage system manager312may also communicate with one or more host systems (not shown) through a host interface314, which may or may not be a part of the storage system manager312. The storage system manager312and/or any other component of the storage system300may be implemented in hardware and/or software, and may make use of a processor (not shown) for executing commands of a type known in the art, such as a central processing unit (CPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc. Of course, any arrangement of a storage system may be used, as will be apparent to those of skill in the art upon reading the present description.

According to some embodiments, the storage system (such as300) may include logic configured to receive a request to open a data set, logic configured to determine if the requested data set is stored to a lower storage tier306of a tiered data storage system300in multiple associated portions, logic configured to move each associated portion of the requested data set to a higher storage tier302of the tiered data storage system300, and logic configured to assemble the requested data set on the higher storage tier302of the tiered data storage system300from the associated portions.

As previously mentioned, regulatory bodies often mandate the retention of certain data for significant periods of time for compliance purposes. Data is also retained for substantial amounts of time for other business purposes, such as historical analysis. In order to satisfy these requirements, organizations typically move data that is no longer being actively used (e.g., “cold” data) to storage types and/or locations which are less expensive, e.g., such as cloud-based storage. Object store is often the preferred storage format in cloud storage environments, as it offers significant scalability as well as allowing for the storage and retrieval of large amounts of data.

One of the ways which object storage derives its high scalability characteristics stems from the fact that it implements a simplified set of permitted operations (“Retrieval” and “Creation”). Object storage also imposes restrictions that objects are fundamentally immutable. Certain applications allow customers to migrate data from an on-premise file system to a cloud storage location. Moreover, these applications support the migration of “cold data” to the cloud location and also provide recall capabilities, both of which are done transparently to the on-premise applications.

Though the cold data is migrated to the cloud, there exist use cases which involve recalling the data back to the on-premise environment. Often times, this recalled data is then processed and updated before finally being archived back to the cloud storage location when it is no longer actively being used (becomes cold again). However, given the immutable nature of object store, these use cases have resulted in the entire object being recalled and subsequently archived back to the cloud storage location, even if only small portions of the entire object have been amended. This causes significant issues in conventional storage implementations, particularly when network bandwidths are constrained and large amounts of data is being transitioned between the different storage locations.

In sharp contrast to the aforementioned shortcomings, various ones of the embodiments included herein achieve significant improvements to the efficiency by which storage systems are able to manage data stored therein. Particularly, some of the embodiments described herein reduce the amount of data which is transferred between storage locations, thereby reducing the amount of network bandwidth, computational resources, system throughput, etc. consumed in order to satisfy received data requests and/or operations, e.g., as will be described in further detail below.

Looking toFIG. 4, a distributed storage system400is illustrated in accordance with one embodiment. As an option, the present storage system400may be implemented in conjunction with features from any other embodiment listed herein, such as those described with reference to the other FIGS. However, such storage system400and others presented herein may be used in various applications and/or in permutations which may or may not be specifically described in the illustrative embodiments listed herein. Further, the storage system400presented herein may be used in any desired environment. ThusFIG. 4(and the other FIGS.) may be deemed to include any possible permutation.

As shown, the distributed storage system400includes an on-premise object store location402which is connected to a remote object store location404by a network406. In preferred approaches, the remote object store location404is a cloud-based storage environment where data is archived. As such, the cloud-based storage environment may be a read-only environment which prevents data modification requests from being performed therein. However, the cloud-based storage environment is preferably able to update the data stored therein such that any updates, deletes, overwrites, new writes, etc. performed at the on-premise object store location402and/or elsewhere in the distributed storage system400are reflected at the cloud-based storage environment. Moreover, the remote object store location404may be managed and/or structured according to any desired type of cloud-based storage environment, e.g., as would be appreciated by one skilled in the art after reading the present description.

The network406may be of any type, e.g., depending on the desired approach. For instance, in some approaches the network406is a WAN, e.g., such as the Internet. However, an illustrative list of other network types which network406may implement includes, but is not limited to, a LAN, a PSTN, a SAN, an internal telephone network, etc. Accordingly, the on-premise object store location402and the remote object store location404are able to communicate with each other regardless of the amount of separation which exists therebetween, e.g., despite being positioned at different geographical locations.

Referring now to the on-premise object store location402, a number of data access nodes408are in communication with a server410. Depending on the approach, the data access nodes408may be coupled to the server410using a wireless connection, e.g., WiFi, Bluetooth, a cellular network, etc.; a wired connection, e.g., a cable, a fiber-optic link, a wire, etc.; etc., or any other type of connection which would be apparent to one skilled in the art after reading the present description. Moreover, the process of forming a communication link between one or more of the data access nodes408and the server410may implement any protocols and/or processes which would be apparent to one skilled in the art after reading the present description.

In some approaches, one or more of the data access nodes408serve as virtual machines which provide access to data stored in the distributed storage system400. Thus, various ones of the data access nodes408may implement (e.g., run) any number of applications. According to an illustrative approach, which is in no way intended to limit the invention, one or more of the data access nodes408serve as a migration client which issues read operations, write operations, update operations, etc. received from a user and/or application being run thereby.

Looking to the server410, a queue412is used in some approaches to manage data operations (e.g., requests) received from the various data access nodes408. The queue412may manage the operations according to any desired structure. For instance, in some approaches the queue412processes the operations received in a first-in-first-out (FIFO) manner. However, in other approaches the412processes the operations received in a last-in-first-out (LIFO) manner.

The server410further includes a controller414which is coupled to the queue412as well as internal memory416. According to some approaches, the internal memory416is used by the server410to temporarily store data being transitioned between one or more of the various data access nodes408and the remote object store location404. It follows that the internal memory416may include any desired type of memory, e.g., such as RAM.

The remote object store location404also includes a controller418which is coupled to an array420of storage drives422(e.g., a memory array). As mentioned above, the remote object store location404is a read-only, cloud-based storage environment in some approaches, which prevents data modification requests from being performed therein. However, the remote object store location404is preferably able to update the data stored in the array420of storage drives422, e.g., such that any updates, deletes, overwrites, new writes, etc. performed at the on-premise object store location402and/or elsewhere in the distributed storage system400are reflected at the remote object store location404. Accordingly, the controller418is able to update the data stored in the array420of storage drives422in addition to being able to read the data stored therein.

Again, various ones of the embodiments included herein are able to reduce the amount of data which is transferred between the different storage locations of a distributed storage network, e.g., as a part of a data re-archival. For instance, cold data is typically migrated to a remote storage location (e.g., a remote cloud storage location) as part of a data archival procedure. Data archived at a remote storage location may be recalled to an on-premise location to perform updates thereto, access data therefrom, etc. However, once recalled data has become cold again, it is preferably returned to the remote storage location as part of a re-archival procedure. Various ones of the embodiments herein are able to reduce the amount of network bandwidth, computational resources, system throughput, etc. consumed in order to satisfy received data requests and/or operations, particularly during the re-archival procedure, e.g., as will soon become apparent.

For instance, looking toFIG. 5A, a method a flowchart of a computer-implemented method500is shown according to one embodiment. The method500may be performed in accordance with the present invention in any of the environments depicted inFIGS. 1-4, among others, in various embodiments. Of course, more or less operations than those specifically described inFIG. 5Amay be included in method500, as would be understood by one of skill in the art upon reading the present descriptions.

Each of the steps of the method500may be performed by any suitable component of the operating environment. For example, each of the nodes501,502,503shown in the flowchart of method500may correspond to one or more processors positioned at a different location in a distributed data storage system. Moreover, each of the one or more processors are preferably configured to communicate with each other.

As mentioned above,FIG. 5Aincludes different nodes501,502,503, each of which represent one or more processors, controllers, computer, etc., positioned at a different location in a multi-tiered data storage system. For instance, node501may include one or more processors which are electrically coupled to, and/or included in, a data access node of a distributed data storage system (e.g., see data access nodes408ofFIG. 4above). Node503may include one or more processors which are electrically coupled to, and/or included in, a remote object store location of a distributed data storage system (e.g., see remote object store location404ofFIG. 4above). Furthermore, node502may include one or more processors which are electrically coupled to, and/or included in, a migration server of a distributed data storage system (e.g., see server410ofFIG. 4above), the migration controller being in communication with the one or more processors at each of nodes501and503. Accordingly, commands, data, requests, etc. may be sent between each of the nodes501,502,503depending on the approach. Moreover, it should be noted that the various processes included in method500are in no way intended to be limiting, e.g., as would be appreciated by one skilled in the art after reading the present description. For instance, data sent from node502to node503may be prefaced by a request sent from node503to node502in some approaches.

As shown, operation504of method500includes sending a data access request to node502. Accordingly, operation504includes sending a data access request from a data access node (e.g., migration client) to a migration server in some approaches. The type of data access request sent differs depending on the situation. For instance, in some approaches the data access request includes a read request, while in other approaches the data access request includes a write operation. Moreover, the type of write operation may also differ. For instance, a write request may be a “new” write request which involves writing a new object, file, etc. in memory, an “update” write request which involves overwriting at least some data already stored in memory, etc. depending on the approach.

The different types of data access requests effect the data in different ways. For instance, read requests typically involve accessing the requested data without actually making any changes thereto. However, modification related write requests (e.g., data updates) typically involve making changes to the data stored in memory. Thus, while more than one read request can be performed on the same data without issue, only one application at a time can be given access to modify a given portion of data (e.g., file) without causing corruption. Accordingly, different lock types are used in some approaches to identify the type of access request being performed and/or actually lock the given data in certain situations, e.g., such as those involving a write operation as would be appreciated by one skilled in the art after reading the present description.

It follows that in some approaches, a requested lock type which corresponds to the data access request is sent to node502along with the data access request. As mentioned above, the requested lock type may be a read lock or a write lock corresponding to the respective data.

Referring still toFIG. 5A, node502receives the data access request sent from node501. Moreover, decision506includes determining whether the data which corresponds to the data access request is stored in memory at a remote storage location. In other words, decision506includes determining whether the requested data is stored locally (e.g., in memory which is directly accessible by the one or more processors at node502), or located remotely (e.g., accessible by the one or more processors at node503). In response to determining that the requested data is stored locally, method500proceeds to operation508which includes accessing the data from the local storage location. Furthermore, operation510includes satisfying the data access request by sending a copy of the requested data to node501.

Although not shown inFIG. 5A, a lock type sent along with the data access request in some approaches may be used to protect the data from which the copy of the requested data was created. Moreover, for approaches in which the data access request sent in operation504involves actually making modifications to the requested data, a modified copy of the requested data is returned to node502as shown in operation512. The modified copy of the requested data may thereby be used to update the data stored locally in memory. See operation514.

Returning to decision506, method500proceeds to operation516in response to determining that the requested data is not stored locally, but rather is stored remotely in memory at node503. Accordingly, operation516includes sending a recall request to a remote storage location at node503, the recall request being for the data which corresponds to the data access request. As mentioned above, in some approaches a requested lock type is received at node502along with the data access request. Thus, in some approaches the recall request is sent to node503along with the requested lock type. In other approaches, a requested lock type may be sent to node503along with the recall request regardless of whether a recall request was originally received from node501along with the original data access request. It follows that the one or more processors at node502may be able to determine the type of data access request received, as well as the lock type associated therewith.

Looking to node503, the recall request is received from remote node502. Again, the recall request is for data which corresponds to the data access request originally issued by node501(e.g., by a host, client, user, etc.). Moreover, operation518includes storing metadata which corresponds to the data access request. The metadata that is stored preferably indicates that the data which corresponds to the data access request has been (e.g., is currently) recalled. In some approaches, the metadata is correlated with the requested lock type and may be received along with the recall request. In other words, at least a portion of the metadata stored in operation518includes the requested lock type in some approaches. Moreover, the metadata may be stored in a central memory and/or any other storage location at node503, e.g., depending on the desired approach.

Operation520further includes accessing an existing object in memory which includes the data indicated in the received recall request, while operation522includes creating a copy of the existing object. As would be appreciated by one skilled in the art, the existing object accessed in operation520includes one or more blocks of storage space therein. It follows that the requested data is included in at least one of the one or more blocks of storage space in the existing object.

Furthermore, operation524includes sending the copy of the existing object to node502. Accordingly, the copy of the existing object which includes one or more blocks containing the requested data is received at node502. This received copy of the existing object is thereafter used to satisfy the data access request received in operation504by providing (e.g., sending) the copy of the existing object to node501. See operation526.

Looking now to node501, the copy of the existing object which includes the originally requested data is received. Moreover, a snapshot of the received copy of the existing object is captured in operation528. This captured snapshot provides node501with a simple copy of the existing object as it was received, which also represents the object as it actually exists in memory at the remote storage location associated with node503. Thus, the snapshot may be used to deduce which portions of the copy of the existing object are modified (if any), e.g., as will soon become apparent.

Operation530further includes using the received copy of the existing object to perform one or more data operations (e.g., read and/or write operations). At least some of the one or more data operations may correspond to the initial data access request. However, it should be noted that in some approaches a data operation which is different than that represented in the initial data access request may be performed using the received copy. In other words, a requested lock type sent along with the original data access request may not accurately reflect all of the data operations which are ultimately performed on the copy of the existing object. For example, the data access request originally sent in operation504may have been a read request. Accordingly, the requested lock type sent along with the data access request would indicate that a read type lock should be placed on the corresponding extent in the remote storage, thereby allowing other read requests to access the same object. However, upon receiving the requested data, a write request may be triggered for any number of reasons. In such situations it is preferred that an updated notification (e.g., lock type) is sent to the remote storage at node503which indicates that the data in the copy of the existing object will be, or has been, updated. Node503is thereby able to update the lock type currently issued on the existing object such that the integrity of the data is not compromised. Moreover, a snapshot of the existing object is preferably taken in response to detecting that an originally unanticipated write operation has been triggered.

In situations where the one or more operations do not include write operations or other types of operations which may otherwise cause any of the received data to be deleted, amended, and/or added to, the one or more operations may be performed on the received copy of the existing object without making any changes to the existing object in memory at remote node503. However, approaches in which the one or more operations include making amendments to the data in the copy of the existing object involve additional processes. For instance, once the data in the copy of the existing object becomes sufficiently cold, it is preferably re-archived back to the remote storage location at node503. Accordingly, operation532includes using the captured snapshot to determine ones of the blocks in the copy of the existing object which include data that has been modified by the one or more data operations. This determination is made in some approaches by comparing each block of data in the snapshot with the corresponding block in the modified object and noting any differences therebetween. However, any processes which would be apparent to one skilled in the art after reading the present description may be implemented in order to perform operation532, e.g., such as a logical XOR process. It is also preferred that operation532is performed in response to determining that the one or more operations have completed, e.g., such that the determination does not need to be performed an unnecessary number of times, thereby conserving computational resources.

The blocks determined as including data which has been modified by the one or more data operations are used to form a sparse object. See operation534. According to the present description, the “sparse object” is a partial version of the modified object which only includes the ones of the blocks which are determined as including modified or new data. In other words, the sparse object only includes the differences between the copy of the existing object received in operation526and the modified object resulting from the one or more data operations being performed thereon. It follows that a number of blocks included in the sparse object is fewer than a number of blocks included in the previously received copy of the existing object, and therefore a size of the sparse object is noticeably smaller than a size of the copy of the existing object.

Referring still toFIG. 5A, the sparse object is sent to node502. See operation536. The fewer number of blocks included in the sparse object reduces an amount of system resources consumed by the one or more processors at node501during the sending. Moreover, upon receiving the sparse object from node501, node502sends the sparse object to the remote storage location at node503. See operation538. Again, the fewer number of blocks included in the sparse object (compared to the complete modified object) reduces an amount of the network bandwidth consumed by the one or more processors at node502during the sending. The reduced size of the sparse object in comparison to a size of the complete modified object also results in reduced system delays, faster update times, improved system efficiency, etc.

Furthermore, operation540includes sending one or more instructions to node503which specify that the blocks included in the sparse object are to be used to update the existing object. This update to the existing object is preferably performed by replacing blocks in the existing object which correspond to the modified blocks. As a result, the efficiency by which the existing object is updated is also improved as only the modified blocks are replaced, rather than the entire object.

Accordingly, operation542includes using the modified data in the blocks of the sparse object to update the corresponding blocks in the existing object, e.g., in accordance with the one or more instructions received in operation540. Any processes of updating the blocks in the existing object which would be apparent to one skilled in the art after reading the present description may be implemented, e.g., depending on the desired approach. For instance, in some approaches an in-place modification is made to implement the modified data in the blocks of the sparse object.

For instance, referring momentarily toFIG. 5B, exemplary sub-processes of sending the sparse object to the remote storage location at node503are illustrated in accordance with one embodiment, one or more of which may be used to perform operation538ofFIG. 5A. However, it should be noted that the sub-processes ofFIG. 5Bare illustrated in accordance with one embodiment which is in no way intended to limit the invention.

As shown, sub-operation550is performed at node502and includes separating the sparse object into a number of portions. Separating the sparse object into portions further reduces an amount of data which is transferred across the network which connects one or more of the nodes501,502,503at a given time. Accordingly, the separating is preferably performed such that a size of each of the number of portions is based on a network bandwidth which exists between nodes502and503. Each of the portions are about the same size (include about the same amount of data) in some approaches, but in other approaches the size of some portions may be different than others. For example, the total size of the object may not be evenly divisible depending on the number of portions and/or size of each portion.

Sub-operation552includes asynchronously sending each of the number of portions to the remote storage location at node503. The multiple arrows included inFIG. 5Bare included to represent sending each of the number of portions. In preferred approaches each of the portions are sent in a temporally staggered manner. This reduces an amount of network bandwidth that is consumed by the process of sending the sparse object to the remote storage location at node503. As a result, efficiency of the storage system is increased as network and computing resources are conserved. In some approaches a computer control timing protocol of a type known in the art is used to determine when each of the number of portions are sent to node503. For example, a portion of the object is sent to node503in response to receiving an indication (e.g., signal) that the preceding portion has been successfully received.

Upon receiving one or more of the number of portions, node503uses the modified data in each of the portions to update the existing object in memory (e.g., see operation542inFIG. 5A). However, referring momentarily toFIG. 5C, exemplary sub-processes of using the modified data in the blocks of the sparse object to update the existing object. Accordingly, one or more of the sub-processes included inFIG. 5Cmay be used to perform operation542ofFIG. 5A. However, it should be noted that the sub-processes ofFIG. 5Care illustrated in accordance with one embodiment which is in no way intended to limit the invention.

Following the receipt of the sparse object, or portions thereof, sub-operation560includes retrieving a copy of the existing object from memory. Moreover, sub-operation562includes using the sparse object to update blocks in the copy of the existing object which correspond to the blocks that are included in the sparse object. In other words, sub-operation562includes using the updated data in the blocks of the sparse object to rewrite (e.g., replace) the data included in the corresponding blocks of the existing object. According to some approaches, sub-operation562is performed in a sequential manner such that each block of the sparse object is progressively used to update the existing object. However, in other approaches any desired process of using the modified data in the sparse object to update the corresponding data in the existing object may be used.

Following sub-operation562, the flowchart includes storing the updated copy of the existing object in the memory. See sub-operation564. Metadata corresponding to the organization of the objects in memory at node503may also be updated to reflect that the updated copy of the existing object has been stored to a new location in memory. See sub-operation566. Furthermore, the existing object is deleted from the memory. See sub-operation568.

Returning toFIG. 5A, from operation542, the flowchart proceeds to operation544, whereby method500may end. However, it should be noted that although method500may end upon reaching operation544, any one or more of the processes included in method500may be repeated in order to process additional data access requests. In other words, any one or more of the processes included in method500may be repeated for data access requests subsequently received from one or more data access nodes (e.g., migration clients) at node501.

Referring momentarily back to operations536and538, although it is preferred that the sparse object is sent to remote node503in the interest of conserving system resources, some approaches may permit the transmission of the complete modified object. For example,FIG. 5Dillustrates an exemplary set of optional sub-processes which may be performed in certain approaches to determine how much, or little, of the modified object should be sent to the remote storage location at node503. Thus, any of the optional sub-processes included inFIG. 5Dmay be performed by the one or more processors at node501prior to performing operations536and538inFIG. 5A, thereby potentially changing the flow of method500, e.g., as would be appreciated by one skilled in the art after reading the present description.

As shown, sub-operation570includes determining network bandwidth which exists between node501and the remote storage location at node503. The network bandwidth may be determined a number of different ways depending on the approach. For instance, in some approaches the network bandwidth may be determined by querying a network management module. In other approaches, the network bandwidth may be determined by sampling one or more servers used to transfer data across the network.

Moving to decision572, the network bandwidth determined in sub-operation570is used to determine whether to send the sparse object or a complete object to the remote storage location. According to an illustrative approach, which is in no way intended to limit the invention, the determination made in decision572depends on whether the current network bandwidth is below a threshold which may be selected by a user, preconfigured at a time that the network is established, updated in real-time, etc. In other words, the outcome of decision572is determined based on whether the network has the available bandwidth to handle transferring (e.g., sending) the larger object itself, or whether the current conditions of the network would benefit from the reduced amount of network bandwidth, computing power, system resources, etc. consumed by sending the sparse object. In some approaches, a current network quality of service is also considered in making the determination of decision572, thereby desirably leading to a consistent model, e.g., as would be appreciated by one skilled in the art after reading the present description. As mentioned above, impact on the network may be even further reduced by separating the sparse object into a number of portions, each of which may be sent across the network asynchronously.

Accordingly, the flowchart ofFIG. 5Dproceeds to sub-operations574aand574bin response to determining that sending the complete object is justified in the present situation based on available network bandwidth. However, the flowchart proceeds to sub-operation576aand576bin response to determining that sending the sparse object (or portions thereof) is justified in the present situation based on available network bandwidth. Regardless of whether the complete object or the sparse object is sent to node502, and in turn to the remote storage location, node503is able to use the modified data included therein to update the existing object, e.g., as described above with respect toFIG. 5A.

Thus, the sub-processes included inFIG. 5Dmay be used in some approaches to optimize the workflow by determine the network bandwidth between remote and on-premise locations. Moreover, the size of an object may be used to make determinations of most efficient data transfers which also satisfy given optimization criteria, e.g., such as bandwidth optimization, transfer rate optimization, etc.

It follows that various ones of the embodiments included herein are able to reduce the amount of data that is actually transferred between an on-premise location and a remote (e.g., cloud) storage location while also maintaining successful archival/recall procedures. This functionality can also be applied across remote storage tiers of a network in some approaches, thereby extending the effective application of these improvements. Moreover, actual network bandwidth is taken into consideration when performing some of the various processes included herein. Thus, some of the embodiments included herein achieve significant improvements to the efficiency by which storage systems are able to manage data stored therein. Particularly, some of the embodiments described herein reduce the amount of data which is transferred between storage locations, thereby reducing the amount of network bandwidth, computational resources, system throughput, etc. consumed in order to satisfy received data requests and/or operations.

It should be noted that although a number of the embodiments and approaches herein have been described in the context of object-based storage, this has been done by way of example only, and should not be deemed limiting on the invention defined in the claims. Rather, any desired type of data storage architecture may be implemented in conjunction with any of the embodiments and/or approaches included herein, e.g., as would be appreciated by one skilled in the art after reading the present description.

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

Referring now toFIG. 6, illustrative cloud computing environment650is depicted. As shown, cloud computing environment650includes one or more cloud computing nodes610with which local computing devices used by cloud consumers, such as, for example, personal digital assistant (PDA) or cellular telephone654A, desktop computer654B, laptop computer654C, and/or automobile computer system654N may communicate. Nodes610may communicate with one another. They may be grouped (not shown) physically or virtually, in one or more networks, such as Private, Community, Public, or Hybrid clouds as described hereinabove, or a combination thereof. This allows cloud computing environment650to offer infrastructure, platforms and/or software as services for which a cloud consumer does not need to maintain resources on a local computing device. It is understood that the types of computing devices654A-N shown inFIG. 6are intended to be illustrative only and that computing nodes610and cloud computing environment650can communicate with any type of computerized device over any type of network and/or network addressable connection (e.g., using a web browser).

Hardware and software layer760includes hardware and software components. Examples of hardware components include: mainframes761; RISC (Reduced Instruction Set Computer) architecture based servers762; servers763; blade servers764; storage devices765; and networks and networking components766. In some embodiments, software components include network application server software767and database software768.

Virtualization layer770provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers771; virtual storage772; virtual networks773, including virtual private networks; virtual applications and operating systems774; and virtual clients775.

Workloads layer790provides examples of functionality for which the cloud computing environment may be utilized. Examples of workloads and functions which may be provided from this layer include: mapping and navigation791; software development and lifecycle management792; virtual classroom education delivery793; data analytics processing794; transaction processing795; and using sparse objects to reduce an amount of network bandwidth consumed by re-archival procedures796.