Patent Publication Number: US-2016224638-A1

Title: Parallel and transparent technique for retrieving original content that is restructured in a distributed object storage system

Description:
CROSS-REFERENCE TO RELATED APPLICATION(S) 
     The present application is a continuation-in-part of U.S. patent application Ser. No. 14/998,320, filed Dec. 24, 2015. U.S. patent application Ser. No. 14/998,320 claims the benefit U.S. Provisional Patent Application No. 62/098,727, filed Dec. 31, 2014, and is a continuation-in-part of U.S. patent application Ser. No. 14/832,075, filed Aug. 21, 2015. The present application is also a continuation-in-part of U.S. patent application Ser. No. 14/832,075, filed Aug. 21, 2015. U.S. patent application Ser. No. 14/832,075 claims the benefit of U.S. Provisional Patent Application No. 62/040,962, filed Aug. 22, 2014. 
    
    
     BACKGROUND 
     1. Technical Field 
     The present disclosure relates generally to data storage systems and data communication systems. 
     2. Description of the Background Art 
     With the increasing amount of data being created, there is increasing demand for data storage solutions. Storing data using a cloud storage service is a solution that is growing in popularity. A cloud storage service may be publicly-available or private to a particular enterprise or organization. Popular public cloud storage services include Amazon S3™, the Google File System, and the OpenStack Object Storage (Swift) System™. 
     Cloud storage systems provide “get” and “put” access to objects, where an object includes a payload of data being stored. The payload of an object may be stored in parts referred to as “chunks”. Using chunks allows the payload of a single large object to be spread over multiple storage servers and enables the parallel transfer of the payload from the servers to the reading user or application. 
     SUMMARY 
     The present disclosure provides a parallel and transparent technique for retrieval of a restructured (for instance, compressed or erasure coded) object in a distributed object storage system. The content retrieval technique disclosed herein may be implemented with parallel operations by multiple storage servers in the system. The retrieval is transparent: an original chunk reference, referred to as a original chunk identifier token or CHIT, may still be used to retrieve the original content, such that a client requesting the original content need not be aware that the original content has been restructured. Other embodiments, aspects, and features are also disclosed. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  depicts a distributed object storage system in accordance with an embodiment of the invention. 
         FIG. 2  illustrates an exemplary architecture for a storage server that implements persistent storage of key-value tuples in accordance with an embodiment of the invention. 
         FIG. 3  depicts three forms of KVT entries in accordance with an embodiment of the invention. 
         FIG. 4A  depicts a hierarchical structure for the storage of an object into chunks in accordance with embodiment of the invention. 
         FIG. 4B  depicts KVT entries that are used to implement the hierarchical structure of  FIG. 4A  in accordance with an embodiment of the invention. 
         FIG. 4C  depicts KVT entries for tracking back-references from a chunk to objects in accordance with an embodiment of the invention. 
         FIG. 5A  depicts a payload chunk that has been restructured transparently into a compressed payload chunk in accordance with an embodiment of the invention. 
         FIG. 5B  depicts an alt-index KVT entry and an alternate-payload chunk KVT entry which are used to implement the restructuring shown in  FIG. 5A  in accordance with an embodiment of the invention. 
         FIG. 5C  is a flow chart of a method of restructuring an original-format payload chunk to a compressed payload chunk in accordance with an embodiment of the invention. 
         FIG. 6A  depicts a payload chunk that has been restructured transparently into erasure encoded slices in accordance with an embodiment of the invention. 
         FIG. 6B  depicts an alt-index KVT entry, an erasure encoded content-manifest (EECM) chunk KVT entry and multiple erasure encoded (EE) slice KVTs which are used to implement the restructuring shown in  FIG. 6A  in accordance with an embodiment of the invention. 
         FIG. 6C  is a flow chart of a method of restructuring an original-format payload chunk to a set of erasure encoded slices in accordance with an embodiment of the invention. 
         FIG. 7A  depicts a payload chunk that has been restructured transparently into a base payload chunk and a delta payload chunk in accordance with an embodiment of the invention. 
         FIG. 7B  depicts an alt-index KVT entry, an delta-CM chunk KVT entry and a base-payload KVT entry and a delta-payload KVT entry which are used to implement the restructuring shown in  FIG. 7A  in accordance with an embodiment of the invention. 
         FIG. 7C  is a flow chart of a method of restructuring an original-format payload chunk to a delta payload relative to a base payload in accordance with an embodiment of the invention. 
         FIG. 8A  depicts a version-manifest chunk that has been restructured transparently into an alternate version-manifest chunk in accordance with an embodiment of the invention. 
         FIG. 8B  depicts an alt-index KVT, an alt-VM chunk and payload chunks and/or content manifest chunks which are used to implement the restructuring shown in  FIG. 8A  in accordance with an embodiment of the invention. 
         FIG. 8C  is a flow chart of a method of restructuring the version-manifest chunk in accordance with an embodiment of the invention. 
         FIG. 8D  depicts an example of a change in boundaries from an original version manifest to an alternate version manifest in accordance with an embodiment of the invention. 
         FIG. 9A  depicts a content-manifest chunk that has been restructured transparently into an alternate content-manifest chunk in accordance with an embodiment of the invention. 
         FIG. 9B  depicts an alt-index KVT, an alt-CM chunk and payload chunks and/or content manifest chunks which are used to implement the restructuring shown in  FIG. 9A  in accordance with an embodiment of the invention. 
         FIG. 9C  is a flow chart of a method of restructuring the content-manifest chunk in accordance with an embodiment of the invention. 
         FIG. 9D  depicts an example of a change in boundaries from an original content manifest to an alternate content manifest in accordance with an embodiment of the invention. 
         FIG. 10  is a flow chart of a method of parallel restructuring a chunk in a distributed object storage system in accordance with an embodiment of the invention. 
         FIG. 11  is a flow chart showing that the restructuring of a chunk using the presently-disclosed technique may be performed by a live system because the restructuring does not interfere with normal operations of the system relating to the chunk. 
         FIG. 12  illustrates the sequence of steps to perform a retrieval of a chunk when the gateway has no prior knowledge of the formats currently stored for the chunk in the storage servers in accordance with an embodiment of the invention. 
         FIG. 13  is a flow chart of processing steps performed by a storage server to indicate available original and/or alternate formats of a requested chunk in accordance with an embodiment of the invention. 
     
    
    
     DETAILED DESCRIPTION 
     Trade-Offs with Different Structures 
     When storing a vast number of objects, it may be desirable for a storage cluster to restructure stored data with a different encoding structure to be optimal for current retrieval needs. This is because the optimal format for storing the same information can change over time for an object. 
     For example, at a first time when there is a high frequency of retrieval requests, storing the data as whole replicas, in the same form for the data as when it is delivered to the application layer, may be desirable for lower-latency retrieval. Retrieving whole replicas may have the lowest latency, depending on the relative speed on the network versus decompression. On the other hand, at a second time when there is a low frequency of retrieval requests, storing the data using erasure encoded slices may be desirable for efficient storage. Using an erasure encoding algorithm can reduce the amount of raw storage capacity required to reliably hold an object. However, as a trade-off, such encoding typically increases the latency for retrieving data so encoded. 
     Such trade-offs between conserving storage space and storing or retrieving data with low latency are likely to vary as networking, storage and processing costs change over the lifespan of what is supposed to be a single object. Hence, it is desirable for the storage cluster to restructure stored data so that it can minimize the overall cost of being able to faithfully and reliably deliver that data on demand. 
     Restructuring of Immutable Content 
     In order to restructure stored data, an object storage cluster needs the capability to create derivatives of existing data in new formats at various times. This capability to create derivatives of stored data may appear to be potentially at conflict with the “immutable” (unchangeable) nature of chunks created by certain object storage clusters, such as, for example, the object storage cluster described in U.S. Pat. No. 8,533,231 (“Cloud Storage System with Distributed Metadata,” inventors Alexander Aizman and Caitlin Bestler), and the object storage cluster using multicast transport described in U.S. Patent Application Publication No. 2014/0204941 (“Scalable Transport System for Multicast Replication”). In these previous object storage clusters, when a chunk is put to the storage cluster, an immutable chunk reference (chunk identifier) is returned. The chunk identifier is globally unique and never re-used to identify a different chunk payload. Such a chunk identifier prevents accidental or deliberate alteration of chunk payload. This is true even when the physical storage of the chunk is not under control of the storage cluster. The capability to attest to data being intact from accidental corruption or even deliberate alteration is vital to providing document archiving. 
     Fortunately, using the techniques disclosed herein, an object storage cluster may restructure the encoding of already stored data with such pre-existing references that are immutable. The techniques disclosed herein enable the restructuring to be performed while still supporting the pre-existing references to this content. In accordance with an embodiment of the invention, the techniques disclosed herein provide the capability to use the original reference to drive the validation of the restructured content. This enables content that is considered to be “immutable” by the storage cluster user to be restructured in a transparent manner in numerous advantageous ways. 
     Transparent Restructuring 
     In a previous object storage cluster, while the application layer may be aware that a given chunk of data is a derivation from another chunk, the storage layer itself is generally unaware of the derivation relationship between the two chunks. In other words, unless the derivation generates a simple copy of the chunk, the previous storage layer is not aware of the fact that one chunk is derived from another chunk. 
     In contrast, the presently-disclosed storage cluster utilizes a storage layer that has information on derivation relationships between data chunks (including both payload data chunks and cluster metadata chunks). The derivation relationship information enables the presently-disclosed storage cluster to restructure data transparently. The transparent restructuring enables a storage cluster to re-encode chunks or whole objects in alternate formats while still supporting both the original references to the content and allowing the original references to drive validation of the restructured content. 
     The presently-disclosed techniques for transparent restructuring are compatible with a distributed object cluster that uses cluster metadata that specifies payload as uniquely identified chunks, without recording the specific locations of any replica. These techniques are further compatible with totally decentralized processing, as featured by certain distributed object clusters. 
     Parallel Restructuring 
     Conventional storage clusters that lack location and format independence in their metadata can only restructure storage through a process which restructures the payload and replaces the existing metadata. This requires a single process for any given object that is being restructured. Any requirement for a single process limits the scalability of the network. 
     The presently-disclosed restructuring technique avoids these restrictions. Each storage server is free to choose which key-value tuples (KVTs) it stores for a given chunk. Hence, the restructuring may proceed in parallel with independent processes running on multiple storage servers at the same time. 
     Parallel Transparent Restructuring of Immutable 
     The present disclosure describes systems and methods for parallel transparent restructuring of immutable content (PTRIC). The PTRIC technology taught in this section may be applied within a distributed object storage cluster, especially for object storage systems that allow payload references to be cached extensively. For such systems, being able to honor existing chunk references, even after the underlying content has been restructured, is of considerable value. 
     Further, the PTRIC technology disclosed herein is fully compatible with a fully-distributed object storage cluster. That is, the restructuring may be implemented without requiring any central point of processing. Advantageously, the presently-disclosed PTRIC technology allows storage servers (also referred to herein as storage nodes) to encode facets of the information about a chunk to enable optimized handling of derivate data and re-encoding of payload to alternate formats (such as erasure encoding) without requiring any modifications to the cluster metadata referencing the chunk. 
       FIG. 1  is a high-level system diagram showing various components of a distributed object storage system  100  in accordance with an embodiment of the invention. As shown, users  102  may access the storage servers (storage nodes)  108  of the distributed object storage system  100  via gateway servers  104  and a network of switches  106 . The users  102  may be clients or proxies operating on behalf of clients and may send requests to get and put chunks to the storage system  100  via the gateway servers  104 . 
     A gateway server  104  may be defined as a server in the set of servers responsible for making special replications of chunks that do not get added to the chunk&#39;s replication count. A gateway server may be used as the front-end or gateway to either archival storage or as gateways to a remote cluster that shares knowledge of assets. 
     In an exemplary implementation, the switches  106  may be a non-blocking switch. A switch can be considered to be non-blocking if it is capable of running every one of its links at full capacity without dropping frames, as long as the traffic was distributed such that it did not exceed the capacity on any one of its links. For example, each of the eight ports of a non-blocking 8-port switch is capable of sending 1/7th of the wire speed to each of the other ports. A non-blocking switch has sufficient internal buffering so it can queue the output frames to any one of its ports. The other ports can “share” this output without having to synchronize their transmissions. If they each have a sustained rate of 1/7th of the wire capacity then the output queue for the target port may grow temporarily, but it will not grow indefinitely. 
     The storage servers  108  may act as chunk servers that store and provide access to chunks of objects. The storage servers  108  may also act as manifest servers that store and provide access to version manifests and content manifests. 
     Note that each component of the storage system need not be on a separate computer system. For example, a gateway server  104  may be implemented on a same computer system as storage server  108 . 
       FIG. 2  illustrates an exemplary architecture for a storage server  220  that implements persistent storage of key-value tuples in accordance with an embodiment of the invention. As illustrated, the storage server  220  may include a storage server packet processing module  222 ; a persistent storage module  224 ; fast-access storage devices  225  and storage devices  227 . 
     The storage server packet processing module  222  is the primary module that process and transmits packets to the other members of the distributed object storage system. 
     The persistent storage module  224  is a module that implements a key-value application programming interface (key-value API). The key-value API provides access to the local KVT index  226 . The local KVT index  226  may be stored on a fast-access storage device or devices  225 . The fast-access storage devices  225  may be random access memories (RAMs) or solid-state drives (SSDs), for example. 
     blobs  228  may be stored locally on the storage devices  227  that are accessible by the persistent storage module  224 . The storage devices  227  may be hard drives, for example. 
     The local KVT index  226  stores KVTs, each KVT consisting of a key and an associated value. The KVTs include chunk KVTs and index KVTs. 
     A chunk KVT has a key having a content hash identifying token (CHIT) that identifies a content blob, and a value that points to the storage location of the content blob (where “blob” stands for “binary large object”). Together, a chunk KVT and its associated content blob may be referred to as a chunk. The content blob may store payload or a type of metadata. The metadata may be, for example, a version manifest, a content manifest, a set of back-references, or other metadata. 
     An index KVT may be associated with a chunk KVT and provides further information associated with the content blob. The further information may be, for example, an object name associated with a version manifest, or a chunk associated with a set of back-references. 
       FIG. 3  depicts three forms of KVT entries in accordance with an embodiment of the invention. The three forms include two chunk KVT forms ( 310 -A and  310 -B) and an index KVT form ( 320 ). Each form includes a key that is associated with a value so as to constitute a key-value tuple (KVT). This association may be implemented by an inline arrangement of the key and value. 
     The chunk KVT  310 -A is a KVT structure that provides access to a content blob (binary large object) via a self-verifying content hash identifying token (content-CHIT). A self-verifying CHIT may be defined to be an identifying token for a chunk formed by applying a cryptographic hash on the content blob. The full CHIT includes preferably both the cryptographic hash value and an enumerator identifying the cryptographic hash algorithm used. Together, the chunk KVT  310 -A and the referenced content blob may be referred to as simply a chunk. 
     The key of chunk KVT  310 -A includes a &lt;Blob-Category&gt;, a &lt;Content-CHIT&gt; and a &lt;Table&gt;. The &lt;Blob-Category&gt; field of the key indicates the category of the content blob and may be the most significant portion of the bits of the key. For example, the Blob-Category may indicate that the content blob contains payload or that the content blob contains a type of metadata, such as, for example, a version manifest or a content manifest. The &lt;Content-CHIT&gt; field of the key provides the CHIT of the content blob and may be a next most significant portion of the bits of the key, where the CHIT serves as a fingerprint that is used to verify the content blob. The &lt;Table&gt; field of the key may provide additional information regarding the content, such as type-related information and may be a least significant portion of the bits of the key. 
     The value of chunk KVT  310 -A provides the location and length of the content blob. The content blob may contain payload or metadata. 
     The chunk KVT  310 -B is an alternative KVT structure that provides access to a content blob (binary large object) via a non-verifying CHIT (content-CHIT). The CHIT is non-verifying in that the CHIT itself is not useable to verify the Content. Chunk KVT  310 -B is similar to chunk  310 -A. However, since chunk KVT  310 -B has a non-verifying CHIT, error detection data is included in the value of chunk KVT  310 -B. The error detection data may be, for example, a cyclic redundancy check (CRC) code, a cryptographic hash, or other error detection code generated from the content blob. While the present disclosure primarily describes implementations that use the chunk KVT  310 -A form (with verifying CHIT), alternative implementations may use the chunk KVT  310 -B form (with non-verifying CHIT). 
     The usage of error detection data is optional for KVTs when the ultimately referenced payload provides its own error detection. For example, the payload of a version manifest contains a metadata field specifying the fully-qualified object name. If this name does not match the Name Hash that started the search, then this can be treated the same as an inconsistent error detection field, that is the entry is invalid and should be expunged. 
     An index KVT  320  may be used to provide supplementary information related to a chunk KVT (of either form  310 -A or  310 -B). For example, a name-index KVT provides an object name related to a version-manifest chunk KVT. Other uses of an index KVT  320  are disclosed herein. Of particular interest, the present disclosure describes innovative uses of an index KVT to provide parallel transparent restructuring of immutable content. 
     The key of index KVT  320  contains an &lt;Index-Category&gt; field, a &lt;Cryptohash&gt; field, and a &lt;Table&gt; field. In an exemplary arrangement, the &lt;Index-Category&gt; is provided in a most-significant portion of the bits of the key, the &lt;Cryptohash&gt; is provided in a next-most significant portion of the bits of the key, and the &lt;Table&gt; is provided in a least-most significant portion of the bits of the key. Other arrangements may be utilized instead. 
     The &lt;Index-Category&gt; indicates a high-level category (which may be referred to as “major type” data) of the supplementary information. For example, the category may indicate that the supplementary information relates to an object name, in which case the index KVT may be referred to as a Name-index KVT. The &lt;Cryptohash&gt; field may be used for various purposes, depending on the category of index KVT. For example, the “cryptohash” (i.e. cryptographic hash) for a name-index KVT provides a name hash identifying token (NHIT). The &lt;Table&gt; field of the key may provide further information, such as finer category information that may be referred to “minor type” data. 
     The value of the index KVT entry comprises the content-CHIT and may include error detection data. The content-CHIT provides an index to the key of the chunk KVT entry (i.e. points to the chunk KVT entry) with which this index KVT entry is associated. In other words, the content-CHIT provides a pointer from the index KVT entry to the associated chunk KVT entry. The error detection data is useable to validate the index KVT entry. The error detection data may be, for example, a cyclic redundancy check (CRC) code, a cryptographic hash, or other error detection code. In some cases, the error detection data may not be needed in the index KVT entry, such as, for example, a name-index KVT entry does not need error detection data when the object name and name hash identifying token (NHIT) are included in the version manifest Blob. 
       FIG. 4A  depicts a hierarchical structure for the storage of an object into chunks in accordance with embodiment of the invention. The top of the structure is a version manifest that may be associated with a current version of an object. The version manifest holds the root of metadata for an object and has a name hash identifying token (NHIT). As shown, the version manifest may reference content manifests, and each content manifest may reference payload chunks. Note that a version manifest may also directly reference payload chunks and that a content manifest may also reference further content manifests. 
     In an exemplary implementation, a version manifest contains a list of tokens (i.e. CHITs) that identify payload chunks and/or content manifests and information indicating the order in which they are combined to reconstitute the object payload. The ordering information may be inherent in the order of the tokens or may be otherwise provided. Each content manifest chunk contains a list of tokens (i.e. CHITs) that identify payload chunks and/or further content manifest chunks (and ordering information) to reconstitute a portion of the object payload. 
       FIG. 4B  depicts KVTs that are used to implement the hierarchical structure of  FIG. 4A  in accordance with an embodiment of the invention. Depicted in  FIG. 4B  are a version-manifest chunk  410 , a content-manifest chunk  420 , and a payload chunk  430 . Also depicted is a name-index KVT  415  that relates an NHIT to a version manifest  415 . 
     The version-manifest chunk  410  includes a version-manifest chunk KVT and a referenced version manifest Blob. The key of the version-manifest chunk KVT has a &lt;Blob-Category=Version-Manifest&gt; that indicates that the of this chunk is a version manifest. The key also has a &lt;VerM-CHIT&gt; that is a CHIT of the version manifest blob. The value of the version-manifest chunk KVT points to the version manifest blob. The version manifest blob contains CHITs that reference payload chunks and/or content manifest chunks, along with ordering information to reconstitute the object payload. The version manifest blob may also include the object name and the NHIT. 
     The content-manifest chunk  420  includes a content-manifest chunk KVT and a referenced content manifest Blob. The key of the content-manifest chunk KVT has a &lt;Blob-Category=Content-Manifest&gt; that indicates that the of this chunk is a content manifest. The key also has a &lt;ContM-CHIT&gt; that is a CHIT of the content manifest blob. The value of the content-manifest chunk KVT points to the content manifest Blob. The content manifest blob contains CHITs that reference payload chunks and/or further content manifest chunks, along with ordering information to reconstitute a portion of the object payload. 
     The payload chunk  430  includes the payload chunk KVT and a referenced payload blob. The key of the payload chunk KVT has a &lt;blob-category=payload&gt; that indicates that the of this chunk is a payload blob. The key also has a &lt;Payload-CHIT&gt; that is a CHIT of the payload blob. The value of the payload chunk KVT points to the payload blob. 
     Finally, a name-index KVT  415  is also shown. The key of the name-index KVT has an &lt;Index-Category=Object Name&gt; that indicates that this index KVT provides name information for an object. The key also has a &lt;NHIT&gt; that is a name hash identifying token. The NHIT is an identifying token of an object formed by calculating a cryptographic hash of the fully-qualified object name. The NHIT includes an enumerator specifying which cryptographic hash algorithm was used as well as the cryptographic hash result itself. 
     While  FIG. 4B  depicts the KVT entries that allow for the retrieval of all the payload chunks needed to reconstruct an object payload,  FIG. 4C  depicts KVT entries that allow tracking of all the objects to which a payload chunk belongs. The tracking is accomplished using back-references from a payload chunk back to objects to which the payload chunk belongs. 
     A back-reference chunk  440  is shown that includes a back-references chunk KVT and a back-references blob. The key of the back-references chunk KVT has a &lt;Blob-Category=Back-References&gt; that indicates that this chunk contains back-references. The key also has a &lt;Back-Ref-CHIT&gt; that is a CHIT of the back-references blob. The value of the back-references chunk KVT points to the back-references blob. The back-references blob contains NHITs that reference the name-index KVTs of the referenced objects. 
     A back-references index KVT  445  is also shown. The key has a &lt;Payload-CHIT&gt; that is a CHIT of the payload to which the back-references belong. The value includes a back-ref CHIT which points to the back-reference chunk KVT. 
     Restructuring to Compressed Payload 
       FIG. 5A  depicts the hierarchical structure for the storage of an object into chunks after a payload chunk has been restructured by applying data compression in accordance with an embodiment of the invention. The restructured payload chunk is referred to as the comp-payload chunk. 
     A local KVT Index is stored at, and accessed by, the storage server that stores the payload chunk. Depicted in  FIG. 5A  are two KVT entries that are used to implement the restructuring of the payload chunk: an alt-index KVT and a comp-payload chunk KVT. The alt-index KVT  515  and comp-payload chunk  510  are depicted in detail in  FIG. 5B . 
     The payload chunk that is restructured includes a payload-chunk KVT (depicted separately in  FIG. 5A , though it is part of the payload chunk) and a payload blob. The key of the payload chunk KVT has a payload-CHIT (=0xf28a . . . , for example) that identifies the payload blob. In addition, the key of the payload chunk KVT may have a “ ” (i.e. blank) in the type field (i.e. in the table portion of the key) to designate that the payload blob is in its original format. 
     The comp-payload chunk  510  (which is the payload chunk after compression) includes a comp-payload chunk KVT and a comp-payload blob. The key of the comp-payload chunk KVT has a comp-payload-CHIT (=0xd123 . . . , for example) that identifies the comp-payload Blob. In addition, the key of the comp-payload chunk KVT may have a type field (i.e. in the table portion of the key) that indicates the compression algorithm (“Compress Algo”) used in generating the comp-payload blob. The value of the comp-payload chunk KVT points to the comp-payload blob. 
     The present disclosure provides an alt-index KVT  515  that effectively links the comp-payload chunk to the payload chunk in an advantageous way. The key of the alt-index KVT has an index-category of payload and includes the payload-CHIT that identifies the payload blob. In other words, the CHIT of the alt-index KVT is the same as the CHIT of the payload chunk KVT. The value of the alt-index KVT points to the comp-payload blob. Together, these features enable the retrieval of the comp-payload chunk when the payload chunk is requested. In addition, the key of the alt-index KVT may have a “COMP” in the type field (i.e. in the table portion of the key) to indicate that the alternate (i.e. restructured) format is a compressed format. 
       FIG. 5C  is a flow chart of a method  550  of restructuring the payload chunk to a compressed payload chunk in accordance with an embodiment of the invention. The method  550  may include the following steps. 
     Per step  552 , a determination may be made that the payload chunk is to be compressed. For example, the back-references from the payload chunk may only be to objects that are rarely accessed. As such, it may be determined that saving space by compression outweighs the performance penalty that would occur due to the need to perform decompression when retrieving the payload chunk. 
     Per step  554 , a compressed payload (comp-payload) blob may be derived from the payload blob. This step may be performed using any of various conventional compression procedures. 
     Per step  556 , the comp-payload blob may be fingerprinted to create a verifying comp-payload CHIT. This step may be performed using a cryptographic hash procedure, for example. (Alternatively, a non-verifying comp-payload CHIT may be created, and the fingerprint may be used for error detection data to verify the comp-payload blob.) 
     Per step  558 , the comp-payload chunk KVT may be created with a key that includes the comp-payload CHIT and a value that points to the comp-payload Blob. (If the comp-payload CHIT is non-verifying, then the value may include the fingerprint as error detection data.) Together, the comp-payload chunk KVT and comp-payload Blob form the comp-payload chunk. 
     Per step  560 , the alt-index KVT may be created. The alt-index KVT has a key including the payload CHIT, and a value with the comp-payload CHIT so as to point to the comp-payload chunk. As described in detail in the present disclosure, the alt-index KVT enables a search for the payload chunk to return the comp-payload chunk. 
     Per step  562 , the need for this storage server to retain the payload chunk is eliminated after the comp-payload KVT is created. Hence, the payload chunk KVT entry at the storage server may be marked as removable. 
     Per step  564 , the payload chunk KVT entry at the storage server may be removed. The removal may occur at a future time, such as, for example, after the storage server receives confirmation of the successful creation of the comp-payload chunk. 
     Restructuring to Erasure Encoded Slices 
       FIG. 6A  depicts the hierarchical structure for the storage of an object into chunks after a payload chunk has been restructured to erasure encoded slices in accordance with an embodiment of the invention. Erasure encoding is a label for a set of encoding schemes where the data is spread across N slices, where at most M of them are needed to reconstruct the lost slices. There are multiple algorithms that can be described as erasure encoding. In the context of the present disclosure, any specific erasure encoding algorithm can be one of the alternate formats supported by the parallel transparent restructuring of immutable content (PTRIC).  FIG. 6B  depicts an alt-index KVT  615 , an erasure encoded content-manifest (EECM) chunk  610  and multiple erasure encoded (EE) slices which are used to implement the restructuring shown in  FIG. 6A  in accordance with an embodiment of the invention. 
     The payload chunk that is restructured includes a payload-chunk KVT (depicted separately in  FIG. 6A , though it is part of the payload chunk) and a payload blob. The key of the payload chunk KVT has a payload-CHIT (=0x26ab . . . , for example) that identifies the payload blob. In addition, the key of the payload chunk KVT may have a “ ” (i.e. blank) in the type field (i.e. in the table portion of the key) to designate that the payload blob is in its original format. 
     The EECM chunk  610  includes an EECM chunk KVT (depicted separately in  FIG. 6A , though it is part of the EECM chunk) and an EECM blob. The key of the EECM chunk KVT has a EECM-CHIT (=0xe2a4 . . . , for example) that identifies and validates the EECM blob. In addition, the key of the EECM chunk KVT may have a type field (i.e. in the table portion of the key) that indicates the erasure encoding algorithm (“EE Algo”) used in generating the erasure encoded slices. The value of the EECM chunk KVT points to the EECM blob. 
     The present disclosure provides an alt-index KVT  615  that effectively links the EECM chunk to the payload chunk in an advantageous way. The key of the alt-index KVT has an index-category of payload and includes the payload-CHIT that identifies the payload blob. In other words, the CHIT of the alt-index KVT is the same as the CHIT of the payload chunk KVT. The value of the alt-index KVT points to the EECM chunk. Together these features enable the retrieval of the EECM-payload chunk when the payload chunk is requested. In addition, the key of the alt-index KVT may have an “EE” in the type field (i.e. in the table portion of the key) to indicate that the alternate (i.e. restructured) format is an erasure encoded format. 
     The EECM blob contains EE-slice-CHITs (1 to N). EE-slice-CHIT-1 points to EE slice 1, EE-slice-CHIT-2 points to EE slice 2, . . . , EE-slice-CHIT-N points to EE slice N. The &lt;table&gt; in the key of each EE slice n may indicate that the slice is n of N total slices. Note that, while the alt-index KVT  615  and the EECM chunk  610  are preferably stored locally at the storage server, the EE slices (1 to N) are preferably stored at different storage servers for robustness of the data storage. 
       FIG. 6C  is a flow chart of a method  650  of restructuring the payload chunk to a set of erasure encoded slices in accordance with an embodiment of the invention. The method  650  may be performed by a background process so as to minimize impact on “live” performance of the distributed object storage system. The method  650  may include the following steps. 
     Per step  652 , a determination may be made that the payload chunk is to be erasure encoded. This determination may be made, for example, by a background process so as to minimize impact on live performance of the system. 
     Per step  654 , the set of erasure encoded (EE) slices may be derived from the payload blob. The EE slices in the set may be generated at (and/or transferred to) different storage servers so as to protect the data from failure of a single storage server. The set may include a total of N EE slices. 
     Per step  655 , the erasure-encoding content manifest (EECM) blob may be written. As discussed above, the EECM blob contains EE-slice-CHITs (1 to N). EE-slice-CHIT-1 points to EE slice 1, EE-slice-CHIT-2 points to EE slice 2, . . . , EE-slice-CHIT-N points to EE slice N. The &lt;table&gt; in the key of each EE slice n may indicate that the slice is n of N total slices. 
     Per step  656 , the EECM blob may be fingerprinted to create a verifying EECM CHIT. This step may be performed using a cryptographic hash procedure, for example. (Alternatively, a non-verifying EECM CHIT may be created, and the fingerprint may be used for error detection data to verify the EECM blob.) 
     Per step  658 , the EECM chunk KVT may be created with a key that includes the EECM CHIT and a value that points to the EECM blob. (If the EECM CHIT is non-verifying, then the value may include the fingerprint as error detection data for verifying the EECM blob.) Together, the EECM chunk ENT and EECM Blob form the EECM chunk. 
     Per step  660 , the alt-index KVT may be created. The alt-index KVT has a key including the payload CHIT, and a value with the EECM CHIT so as to point to the EECM chunk. As described in detail in the present disclosure, the alt-index KVT enables a search for the payload chunk to return the EECM chunk, along with the EE slices. 
     Per step  662 , the retention requirement for the payload chunk may be reduced. This may be implemented, for example, by marking the payload chunk KVT entry at the storage server as releasable. 
     Per step  664 , the payload chunk KVT entry at the storage server may be released. This release may occur at a future time, such as, for example, after the storage server receives confirmation of the successful creation of the complete set of EE slices. 
     Note that the present disclosure contemplates two methods of encoding erasure encoded slices of chunks. The first method, described above in relation to  FIGS. 6A-6C , uses an erasure encoded content manifest which enumerates the erasure encoded slices made from the original chunk, as well as the algorithm used (for example, Reed-Solomon) and the specific cardinality of the encoding (i.e. M of N). 
     In the second (alternative) method, the alternate content comprises an erasure encoded slice, and an erasure encoded slice KVT encodes a type that specifies a cardinality for the erasure encoding and a specific slice that is encoded. For example, a value X can specify that this is slice 2 of a Reed-Solomon 7 of 9 encoding. In this format, whichever server is gathering roll call inventory responses is expected to understand the total size of the encoding (from the cardinality data) and check for an adequate set is present in the responses. In contrast, with the first method that uses an EECM, the set to be collected is explicitly stated in the EECM. 
     Erasure encoding encodes storage assets in slices across N devices or servers, where the original content can be restored from a lesser number M (M&lt;N) of the data slices. For example, if M=8 and N=10 (“8 of 10” encoding), then each of N=10 slices contains a different 1/M=⅛ of the contents of the chunk. The N=10 slices in total store content the size of N/M=10/8 times the chunk size. This example of 8 of 10 encoding protects against the loss of two slices. 
     Erasure encoding reduces both network traffic (compared with unicast replication) and raw storage capacity required to store an asset at the cost of greater computational work being required on both put and get. Without erasure encoding, a conventional object cluster can only protect against the loss of two servers by creating three replicas. This requires that the content be transmitted over the network three times. With erasure encoding, a minimum of M slices, each holding at least 1/Nth of the total content, must be transmitted, resulting in M/Nths of the payload size. This represents a considerable savings in network bandwidth. 
     Restructuring to Base and Delta Payload 
       FIG. 7A  depicts the hierarchical structure for the storage of an object into chunks after a payload chunk has been restructured to a base (reference) payload chunk and a delta payload chunk in accordance with an embodiment of the invention. The delta payload is the difference between the payload before restructuring and the base payload. In other words, the payload before restructuring may be regenerated by adding the delta payload to the base payload.  FIG. 7B  depicts an alt-index KVT  715 , a delta content manifest (delta-CM) chunk  710 , a base-payload chunk  720  and a delta-payload chunk  722  which are used to implement the restructuring shown in  FIG. 7A  in accordance with an embodiment of the invention. 
     The payload chunk that is restructured includes a payload-chunk KVT (depicted separately in  FIG. 7A , though it is part of the payload chunk) and a payload blob. The key of the payload chunk KVT has a payload-CHIT (=0x2e9e . . . , for example) that identifies the payload blob. In addition, the key of the payload chunk KVT may have a “ ” (i.e. blank) in the type field (i.e. in the table portion of the key) to designate that the payload blob is in its original format. 
     The delta-CM chunk  710  includes a delta-CM chunk KVT (depicted separately in  FIG. 7A , though it is part of the delta-CM chunk) and a delta-CM blob. The key of the delta-CM chunk KVT has a delta-CM-CHIT (=0xab12 . . . , for example) that identifies and validates the delta-CM blob. In addition, the key of the delta-CM chunk KVT may have a type field (i.e. in the table portion of the key) that may indicate a delta (difference) algorithm (“delta algo”) used in generating the delta payload. The value of the delta-CM chunk KVT points to the delta-CM blob. 
     The present disclosure provides an alt-index KVT  715  that effectively links the delta-CM chunk to the payload chunk in an advantageous way. The key of the alt-index KVT has an index-category of payload and includes the payload-CHIT that identifies the payload blob. In other words, the CHIT of the alt-index KVT is the same as the CHIT of the payload chunk KVT. In addition, the value of the alt-index KVT points to the delta-CM chunk. The combination of these features enables the retrieval of the delta-CM chunk (along with the delta and base payload chunks) when the payload chunk is requested. In addition, the key of the alt-index KVT may have a “DELTA” in the type field (i.e. in the table portion of the key) to indicate that the alternate (i.e. restructured) format is a delta format. 
     The delta-CM blob contains a base-payload CHIT and a delta-payload CHIT. The base-payload CHIT points to the base-payload chunk  720 , and the delta-payload-CHIT points to the delta-payload chunk  722 . 
       FIG. 7C  is a flow chart of a method of restructuring an original-format payload chunk to a delta payload relative to a base payload in accordance with an embodiment of the invention. The method  750  may include the following steps. 
     Per step  752 , a determination may be made that the payload chunk is to be encoded into a delta payload chunk relative to a base payload chunk. The base payload chunk may be, for example, a “previous version” of the payload chunk such that the difference between them is small. 
     Per step  754 , the delta payload blob may be derived from the payload blob and the base payload blob. This step may be performed using a conventional difference algorithm. 
     Per step  755 , the delta content manifest (delta-CM) blob may be written. As discussed above, the delta-CM blob contains the delta-payload-CHIT and the base-payload-CHIT. 
     Per step  756 , the delta-CM blob may be fingerprinted to create a verifying delta-CM-CHIT. This step may be performed using a cryptographic hash procedure, for example. (Alternatively, a non-verifying delta-CM-CHIT may be created, and the fingerprint may be used for error detection data to verify the delta-CM blob.) 
     Per step  758 , the delta-CM chunk KVT may be created with a key that includes the delta-CM-CHIT and a value that points to the delta-CM blob. (If the delta-CM-CHIT is non-verifying, then the value may include the fingerprint as error detection data for verifying the delta-CM blob.) Together, the delta-CM chunk KVT and delta-CM blob form the delta-CM chunk. 
     Per step  760 , the alt-index KVT may be created. The alt-index KVT has a key including the payload CHIT, and a value with the delta-CM-CHIT so as to point to the delta-CM chunk. As described in detail in the present disclosure, the alt-index KVT enables a search for the payload chunk to return the delta-CM chunk, along with the base and delta payload chunks. 
     Per step  762 , the retention requirement for the payload chunk may be reduced. This may be implemented, for example, by marking the payload chunk KVT entry at the storage server as releasable. 
     Per step  764 , the payload chunk KVT entry at the storage server may be released. This release may occur at a future time, such as, for example, after the storage server receives confirmation of the successful creation of the delta-format version of the payload chunk. 
     Restructuring Manifests 
     The restructuring techniques disclosed herein may be applied to a chunk storing a manifest, such as a version manifest, as well as to a chunk storing a payload. In general, the mechanisms described above for restructuring payload chunks are applicable for restructuring manifest chunks. Note that the content ultimately reached through an alternate (i.e. restructured) manifest is the same as the content ultimately reached from the original manifest. 
     By using the above-described technique to create an alternate encoding of a version manifest, the original encoding of the version manifest remains valid. This prevents expungement of the original encoding of the version manifest, although at a cost of storage space used. 
     While  FIGS. 5A, 6A and 7A  each depict restructuring of a Payload chunk,  FIGS. 8A and 9A  depict restructuring of a manifest chunk. 
       FIG. 8A  depicts the restructuring of a version-manifest chunk into an alternate version manifest (alt-VM) chunk in accordance with an embodiment of the invention.  FIG. 8B  depicts an alt-index KVT, an alt-VM chunk and payload chunks and/or content manifest chunks which are used to implement the restructuring shown in  FIG. 8A  in accordance with an embodiment of the invention.  FIG. 8C  is a flow chart of a method of restructuring the version-manifest chunk in accordance with an embodiment of the invention. Finally,  FIG. 8D  illustrates example original and alternate chunkings (i.e. original and alternate divisions into chunks) of the original and alternate version manifests, respectively. 
     The version-manifest chunk that is restructured includes a version-manifest chunk KVT (depicted separately in  FIG. 8A , though it is part of the version-manifest chunk) and a version-manifest Blob. The key of the version-manifest chunk KVT has a VerM-CHIT (=0xa16f . . . , for example) that identifies the version-manifest Blob. In addition, the key of the version-manifest chunk KVT may have a “ ” (i.e. blank) in the type field (i.e. in the table portion of the key) to designate that the version-manifest blob is in its original format. 
     The alt-VM chunk  810  includes an alt-VM chunk KVT (depicted separately in  FIG. 8A , though it is part of the alt-VM chunk) and an alt-VM (Alternate VM) blob. The key of the alt-VM chunk KVT may have a blob-category that indicates that the chunk contains a content manifest (CM). (Note that the blob-category indicates a content manifest, not a version manifest, for purposes of maintaining transparency of the restructuring.) In addition, the key of the alt-VM chunk KVT has an alt-VM-CHIT (=0xbe88 . . . , for example) that identifies and validates the alt-VM blob. The value of the alt-VM chunk KVT points to the alt-VM blob. 
     The present disclosure provides an alt-index KVT  815  that effectively links the alt-VM chunk to the version manifest chunk in an advantageous way. The key of the alt-index KVT has an index-category of “VM” (which indicates a version manifest). In addition, the key of the alt-index KVT includes the VerM-CHIT that identifies the version manifest blob. In other words, the alt-index KVT includes the same CHIT as the CHIT of the version manifest. The value of the alt-index KVT includes the alt-VM-CHIT that points to the alt-VM chunk. Together, these features enable the retrieval of the alt-VM chunk when the version manifest chunk is requested. 
       FIG. 8C  is a flow chart of a method  850  of restructuring a version manifest in accordance with an embodiment of the invention. In step  852 , a determination is made to restructure a version manifest. The version manifest being restructured may be referred to below as the original (or target) version manifest. As discussed above, the contents of the original version manifest may be stored in an original version-manifest chunk. 
     In step  854 , the original chunks referenced by the original version manifest may be obtained. These original chunks may be combined in an indicated order to regenerate the original object payload. A small example set of original chunks (original chunk  1 , original chunk  2 , original chunk  3  and original chunk  4 ) is depicted in  FIG. 8D . Note that these original chunks in  FIG. 8D  may be either payload chunks or content manifest chunks (which may themselves point to payload chunks and/or content manifest chunks). 
     Also depicted in  FIG. 8D  are the original boundaries between the original chunks (original boundary  1  between original chunks  1  and  2 , original boundary  2  between original chunks  2  and  3 , and original boundary  3  between original chunks  3  and  4 ). The original boundaries depend on the original chunking (original division) performed when the object payload is first put to the distributed object storage system. 
     In step  856 , an alternate chunks are generated. A small example set of alternate chunks (alternate chunk  1 , alternate chunk  2 , and alternate chunk  3 ) generated by an alternate chunking of the original object payload depicted in  FIG. 8D . Note that these alternate chunks in  FIG. 8D  may be either payload chunks or content manifest chunks (which may themselves point to payload chunks and/or content manifests). The alternate chunking may have boundaries that differ from the boundaries of the original chunking. 
     Also depicted in  FIG. 8D  are the alternate boundaries between the alternate chunks (alternate boundary  1  between alternate chunks  1  and  2 , and alternate boundary  2  between alternate chunks  2  and  3 ). The alternate boundaries depend on the alternate chunking (i.e. the alternate division) performed on the object payload. 
     Per step  858 , the alt-VM KVT that points to the alt-VM Blob may be written to the local KVT index at the storage server. In addition, per step  860 , the alt-index KVT (with key including the VerM-CHIT and value pointing to the alt-VM chunk) may be written to the local KVT index at the storage server. These steps are discussed in further detail above in relation to  FIG. 8B . 
     Finally, per step  860 , verified back-references may be issued from the original version manifest to the alternate chunks. The back-references indicate that the alternate chunks belong to the object associated with the original version manifest. 
       FIG. 9A  depicts a content-manifest chunk that has been restructured transparently into an alternate content-manifest chunk in accordance with an embodiment of the invention.  FIG. 9B  depicts an alt-index KVT, an alt-CM chunk and payload chunks which are used to implement the restructuring shown in  FIG. 9A  in accordance with an embodiment of the invention.  FIG. 9C  is a flow chart of a method of restructuring the content-manifest chunk in accordance with an embodiment of the invention. Finally,  FIG. 9D  illustrates example original and alternate chunkings of the original and alternate content manifests, respectively. 
     The content-manifest chunk that is restructured includes a content-manifest chunk KVT (depicted separately in  FIG. 9A , though it is part of the content-manifest chunk) and a content-manifest blob. The key of the content-manifest chunk KVT has a ContM-CHIT (=0xb26f . . . , for example) that identifies the content-manifest blob. In addition, the key of the content-manifest chunk KVT may have a “ ” (i.e. blank) in the type field (i.e. in the table portion of the key) to designate that the content-manifest blob is in its original format. 
     The alt-CM chunk  910  includes an alt-CM chunk KVT (depicted separately in  FIG. 9A , though it is part of the alt-CM chunk) and an alt-CM blob. The key of the alt-CM chunk KVT may have a blob-category that indicates that the chunk contains a content manifest (CM). In addition, the key of the alt-CM chunk KVT has an alt-CM-CHIT (=0xaf32 . . . , for example) that identifies and validates the alt-CM blob. The value of the alt-CM chunk KVT points to the alt-CM blob. 
     The present disclosure provides an alt-index KVT  915  that effectively links the alt-CM chunk to the content manifest chunk in an advantageous way. The key of the alt-index KVT has an index-category of CM (content manifest). In addition, the key of the alt-index KVT includes the ContM-CHIT that identifies the content manifest blob. In other words, the alt-index KVT includes the same CHIT as the CHIT of the content manifest. The value of the alt-index KVT includes the alt-CM-CHIT that points to the alt-CM chunk. Together, these features enable the retrieval of the alt-CM chunk when the content manifest chunk is requested. 
       FIG. 9C  is a flow chart of a method  950  of restructuring a content manifest in accordance with an embodiment of the invention. In step  952 , a determination is made to restructure a content manifest. The content manifest being restructured may be referred to below as the original (or target) content manifest. As discussed above, the contents of the original content manifest may be stored in an original content-manifest chunk. 
     In step  954 , the original chunks referenced by the original content manifest may be obtained. These original chunks may be combined in an indicated order to regenerate the original content manifest payload. A small example set of original chunks (original chunk  1 , original chunk  2 , original chunk  3  and original chunk  4 ) is depicted in  FIG. 9D . Note that these original chunks in  FIG. 9D  may be either payload chunks or content manifest chunks (which may themselves point to payload chunks and/or content manifest chunks). 
     Also depicted in  FIG. 9D  are the original boundaries between the original chunks (original boundary  1  between original chunks  1  and  2 , original boundary  2  between original chunks  2  and  3 , and original boundary  3  between original chunks  3  and  4 ). The original boundaries depend on the original chunking (original division) performed when the content manifest payload is first put to the distributed object storage system. 
     In step  956 , an alternate chunks are generated. A small example set of alternate chunks (alternate chunk  1 , alternate chunk  2 , and alternate chunk  3 ) generated by an alternate chunking of the original object payload depicted in  FIG. 9D . Note that these alternate chunks in  FIG. 9D  may be either payload chunks or content manifest chunks (which may themselves point to payload chunks and/or content manifests). The alternate chunking may have boundaries that differ from the boundaries of the original chunking. 
     Also depicted in  FIG. 9D  are the alternate boundaries between the alternate chunks (alternate boundary  1  between alternate chunks  1  and  2 , and alternate boundary  2  between alternate chunks  2  and  3 ). The alternate boundaries depend on the alternate chunking (i.e. the alternate division) performed on the content manifest payload. 
     Per step  958 , the alt-CM KVT that points to the alt-CM Blob may be written to the local KVT index at the storage server. In addition, per step  960 , the alt-index KVT (with key including the ContM-CHIT and value pointing to the alt-CM chunk) may be written to the local KVT index at the storage server. These steps are discussed in further detail above in relation to  FIG. 9B . 
     Finally, per step  960 , verified back-references may be issued from the original content manifest to the alternate chunks. The back-references indicate that the alternate chunks belong to the object associated with the original content manifest. 
     Parallel Restructuring Method 
       FIG. 10  is a flow chart of a method  1000  of parallel restructuring of a chunk in a distributed object storage system using iterative collaboration in accordance with an embodiment of the invention. Multicast messaging is used to collect the current state of how a specific chunk is stored in the object cluster. Multicast messaging is then used to iterate towards a desired state through a series of steps. Note that the method  1000  advantageously does not require (or use) any single node to drive the restructuring process. 
     i) Initiating Roll Call 
     Per step  1002 , an initiating node multicasts a “roll call” request to the group responsible for the chunk to be restructured (i.e. for the target chunk). In the above-described distributed object storage system, the initiating node may be a storage server of the distributed object storage system. 
     The group responsible for the target chunk may be referred to as a negotiating group for the target chunk. In an exemplary implementation, the negotiating group is identified using a cryptographic hash identifier of the chunk. The negotiating group for a target chunk is a group of the storage servers (nodes) in the storage system that is assigned to store and provide access to the target chunk. However, alternate methods of identifying an existing multicast group for the target chunk may be used instead. 
     In other words, in accordance with an embodiment of the invention, when a target chunk has been identified as a candidate for restructuring, a multicast “roll call” message may be sent to the negotiating group for that target chunk. This message preferably includes an identifier of the negotiating group, an identifier for the target chunk, the specific restructuring algorithm to be used, and a unique identifier of the roll call. One embodiment of the roll call identifier is formed by concatenating a timestamp and the IP address of the requester. Alternately, the roll call identifier may be a concatenation of a sequence number and the source. IP address. 
     ii) Roll Call Inventory Response 
     Per step  1004 , each node in the group receiving the request multicasts a “roll call inventory” response to the “roll call” request to all the other nodes in the group. The roll call inventory response message identifies: a) the roll call message being responded to; b) the specific storage node that is responding; and c) which, if any, format(s) this storage node has (or has begun building) for this specified chunk. In other words, each roll call inventory response from a node enumerates the encodings of the target chunk (including original and/or alternate formats) that are stored at that node. 
     In other words, in accordance with an embodiment of the invention, each recipient of the restructuring roll call request responds with a restructuring chunk inventory multicast message which it sends to the same negotiating group. This roll call inventory response may preferably include: 
     the echoed identifier of the roll call request; 
     the identity of the responding object storage server instance; 
     the identifier of the target chunk being queried; and 
     how the identified target chunk is encoded on local storage by this instance. 
     In an exemplary implementation, “how the target chunk is encoded on local storage” may be: not at all; as a whole copy of the target chunk in the original format; as a compressed version of the target chunk; as an erasure encoded content manifest for the target chunk; an erasure encoded slice derived from the target chunk; as a delta content manifest for the target chunk; as a delta chunk derived from the target chunk (and a base chunk); as a base chunk related to a delta chunk; or as a combination of the foregoing. In other implementations, other alternate encodings may be used. 
     iii) Collection of Roll Call Inventory Responses 
     Per step  1006 , every node in the group collects “roll call inventory” responses that respond to the “roll call” request. By having every node in the group collect the roll call inventories, every node has the information needed to evaluate the collective inventories within the group to formulate a same set of actions. While every member of the group collects the full set of inventory responses that it receives for any given roll call request, a dropped message may be simply ignored without a retransmission request. 
     Note that a storage server may receive roll call inventory responses to a roll call request that it did not hear. Embodiments are free to either ignore such messages, or to retroactively infer reception of the roll call request by not only collecting the response but responding to it as well. 
     iv) Evaluation of Roll Call Inventory Responses and Determination of Desired Storage State and Subset of Actions to be Performed by Individual Node 
     Per steps  1007  and  1008 , every node in the group evaluates, in parallel, the collected roll call inventory messages to determine the desired state of storage for the chunk amongst the nodes in the group (step  1007 ) and the set of actions that are required to be performed by the nodes of the group in order to achieve the desired state (step  1008 ). The logical evaluation by a node may begin once roll call inventory response messages have been received from all members of the group, or after a predetermined time has elapsed that is sufficient such that no more responses can be reasonably expected. 
     In an exemplary implementation, the desired state of storage may be one of various states. A first state may store original-format whole-replica chunks at different nodes in the group. A second state may store compressed chunks at different nodes in the group. A third state may store erasure encoded slices at different nodes in the group. A fourth state may store delta and base chunks at different nodes in the group. Other states may store the chunk in various different formats (encodings) in the group. 
     The desired state of storage for a target chunk may change over time. For example, the desired state may begin in the first state where original-format whole replicas are stored at different nodes. Subsequently, if storage space becomes scarce, then the desired state may change to the second state where compressed chunks are stored at different nodes. If storage space becomes more scarce, then the desired state may change to the third state where a sufficient number of erasure encoded slices are stored at different nodes. As another example, if a target chunk is a small update to an existing chunk, then the desired state may change from the first state where whole replicas are stored to the fourth state where a delta chunk is stored to indicate changes relative to the existing chunk. 
     In an exemplary implementation, the set of actions determined may include, for example: 
     (i) unicasting or multicasting the whole chunk in its original format to create new replicas at other nodes in the group; 
     (ii) creating and potentially transferring a compressed copy of the whole chunk; creating and potentially transferring an erasure encoded content manifest and/or erasure encoded slices of the chunk; and 
     (iii) creating and potentially transferring a delta content manifest and/or a delta payload chunk relative to a base chunk. 
     The evaluation of roll call inventory responses depends upon the collective roll call inventories from the nodes in the group. The evaluation further depends on which nodes in the group already stores encodings of the target chunk and what those encodings are. 
     In other words, each member of the group evaluates the collected responses, determines the collective action required (if any) and assigns itself a portion (if any) of that work. This assignment is based upon parallel evaluation that orders both the tasks to be done and the available storage nodes. In order for an ordering algorithm to be useable, all members of the group should determine the same assignment of work to storage nodes without the need for active collaboration between the members. 
     Per step  1009 , every node in the group determines a subset of the set of actions to be performed by itself. The subset of actions assigned to a node is determined in a same agreed-upon manner at each node in the group. The subset of actions for a particular node may depend upon the particular node&#39;s place in the ordering of the nodes in the group. The ordering may be, for example, a sorted order based on a node identifier. 
     v) Performance of Subsets of Actions by Nodes in the Group 
     Per steps  1010 - n , for n=1 to Ng, where Ng is the number of nodes in the group, the nth node in the group performs an nth subset of actions. The subset of actions to be performed by a node being determined in step  1009  as discussed above. As mentioned above, the subset of actions may include: unicasting or multicasting the whole chunk in its original format to create new replicas at other nodes in the group; creating and potentially transferring a compressed copy of the whole chunk: creating and potentially transferring an erasure encoded content manifest and/or erasure encoded slices of the chunk; creating and potentially transferring a delta content manifest and/or a delta payload chunk relative to a base chunk. Note that it is possible that a subset of actions may be empty, in which case the node corresponding to that subset is not assigned to perform any action. 
     vi) Selection of Pseudo-Random Times for New Roll Call Request 
     Per steps  1012 - n , for n=1 to Ng, when a node completes its assigned subset of actions, a time may be selected for a new roll call request. The time may be selected pseudo-randomly to distributed the selected times amongst the nodes in the group. At the selected time, the new roll call request may be multicast from the node to the group. However, if a restructuring roll call for the same chunk is received before the selected time, then the multicasting of the new roll call request at the selected time will be pre-empted for the node. 
     In other words, after completing its work pursuant to a prior roll call, a storage node will pseudo-randomly select a time to issue a new roll call for the chunk. The new roll call for the chunk will be pre-empted should a roll call for the same chunk be received before the node&#39;s own new roll call is issued. 
     The above-described pseudo-random mechanism that distributes the issuance of new roll call requests prevents multiple nodes from flooding the available network at the same time. 
     vii) Next Iteration Initiated by New Roll Call Request 
     Upon a new roll call request being sent by any one of the nodes, then the method  1000  loops back to step  1014  and a next iteration is performed. In this way, the method collaboratively iterates in a dynamic manner to a desirable state. In accordance with an embodiment of the invention, the procedures of the presently-disclosed solution converge upon a stable encoding of the chunk, either as whole replicas in the original format, or as chunks in an alternate format, or the combination of the foregoing. Lost chunks or slices may be replaced through background scanning of the multicast group. 
     Note that the method  1000  depicted in  FIG. 10  uses a pseudo-random mechanism to select in an ad-hoc manner the executor of the next roll call. An alternate embodiment may utilize a deterministic way of, effectively, determining the master of the next iteration. For example, one deterministic technique would be to issue the next roll call from the first node ID in a sorted list of node IDs of the nodes that take part in the iteration. Other pseudo-random or deterministic techniques may be used in other implementations. 
       FIG. 11  is a flow chart showing that the restructuring of a chunk using the presently-disclosed technique may be performed by a live system because the restructuring does not interfere with normal operations of the system relating to the chunk. In particular, the flow chart shows a method  1100  performed by the live system. In this method  1100 , restructuring of a chunk does not interfere with processing a get request for the chunk, and processing the get request for the chunk does not interfere with the restructuring of the chunk. 
     Per step  1112 , the original chunk is stored in the distributed object storage system. Subsequently, both restructuring (steps  1122  and  1124 ) and retrieval (steps  1132  and  1134 ) of the chunk may be performed by the live system in parallel. Note that the chunk being restructured may be a payload chunk or a manifest chunk. 
     Per step  1122 , the restructuring of the original chunk is initiated within the live system. For example, a roll-call request for the restructuring may be issued, as described above in relation to  FIG. 10 , for example. 
     Per step  1124 , the live system performs the restructuring process (for example, as described above in relation to  FIG. 10 ). Because the restructuring process leaves the original chunk unchanged while adding the alternate chunk, a get request for the original chunk may be processed in parallel without interfering with the restructuring process. 
     Per step  1132 , a get request to retrieve the original chunk may be received by a gateway to the live system. Per step  1134 , because the retrieval process may be performed without the alternate chunk, the live system may process the get request and send the original chunk in a response to the gateway. Meanwhile, the restructuring of the original chunk may proceed in parallel without interfering with the processing of the get request. 
     Transparent Technique for Retrieving Original Content that is Restructured 
       FIG. 12  illustrates the processing steps of the gateway and a Storage Server to fetch a target chunk without the gateway having prior knowledge of which formats are currently stored. The storage server is a member of a negotiating group for the target chunk. Note that the processing steps performed by the particular storage server are also performed by other storage servers in the negotiating group for the chunk. 
     After the gateway receives a get request for the target chunk from a user, the first step  1202  is for the gateway to send a get request for the target chunk using the CHIT originally specified for the target chunk. The get request may also indicate a category for the target chunk (such as a payload category, for example). The get request may be sent to all the storage servers in the negotiating group for the chunk. 
     In the next step  1204 , upon receiving the get request, a storage server searches locally-stored key-value tuples (KVTs)  1220  to find all the KVTs which reference this original CHIT and are of the indicated category. The KVTs may correspond to different formats of the requested (target) content that may be delivered from the storage server. This step  1204  is described further below in relation to step  1304  in  FIG. 13 . 
     In a next step  1205 , the storage server determines a bid for each format that indicates when the target content in that format could be delivered by the storage server. In one implementation, the bid(s) depends on existing get reservations  1230  at the storage server. Each get reservation represents rights to use of the internal IO bandwidth of the storage server and the transmit bandwidth of the network interface. In step  1206 , based on the bid(s), a new get reservation may be made that does not conflict with the existing get reservations  1230  at the storage server. 
     Next, in step  1207 , the storage server may assemble a get response that references each option available locally, with the bid on when that format of the target content could be delivered. In step  1208 , the storage server sends its get response to the gateway. The other storage servers in the negotiating group for the target chunk may also send their get responses to the gateway. 
     In step  1209 , after receiving the get responses (or after a predetermined time is elapsed during which get responses are to be received), the gateway selects a KVT (or multiple KVTs, depending on the format) from the best target (or targets). The best target(s) is (are) generally the storage server(s) that provides the earliest possible delivery of the requested content. 
     In step  1210 , the gateway multicasts a get accept message to the storage servers of the negotiating group for the chunk. The get accept message indicates the selected KVT (or KVTs) and also indicates a specific target server and scheduled time for each selected KVT. 
     In step  1212 , each storage server processes the get accept message by either cancelling their get reservation (if it is not accepted as a target server) or adjusting it to match the specific KVT desired at the scheduled time (if it is accepted as a target server). The processing by a storage server may also involve scheduling internal IO as required to have the desired payload ready to transmit at the scheduled time. 
     In step  1213 , at the scheduled time specified in the get accept message, the storage servers that were accepted each initiates a rendezvous transfer to the gateway. Each rendezvous transfer obtains the selected content from the locally-stored content blobs  1225  and delivers the selected content to the gateway. When the rendezvous transfer is initiated, the get reservations  1230  may be adjusted by removing the corresponding reservation. 
     In step  1214 , the gateway collects the content delivered by the rendezvous transfer (or transfers). 
     Per step  1215 , further iterations are performed, if needed, to fetch referenced chunks so as to obtain all the content needed to reconstruct the target chunk. For example, if the content delivered may be a content manifest that references chunks, then further iterations would be performed to obtain those referenced chunks. 
     Finally, in step  1216 , if needed, the gateway will assemble or otherwise reconstruct the target chunk from the collected content. Subsequently, the gateway may send the target chunk to the requesting user. 
     Storage Server Processing of a Get Request 
       FIG. 13  is a flow chart of a method performed by a storage server to indicate available original and/or alternate formats of a requested chunk in accordance with an embodiment of the invention. This method  1300  may be performed by each storage server in the negotiating group for the chunk requested. The chunk requested may be referred to below as the original chunk or the target chunk, and the content of the original (target) chunk may be referred to below as the original-format content. 
     The content of the chunk may be of a specified category. In one embodiment, the content of the chunk may be payload, and the specified category may be a payload category. In another embodiment, the content of the chunk may be a version manifest, and the specified category may be a version-manifest category. In another embodiment, the content of the chunk may be a content manifest, and the specified category may be a content-manifest category. 
     Per step  1302 , the get request for the original (target) chunk is received by the storage server. The get request may be received from a gateway (or user client) of the object storage system. For example, the original (target) chunk may be one chunk of a set of chunks of an object that is being retrieved by the gateway server from the distributed object storage system. 
     Per step  1304 , the storage server may search among the KVT entries stored locally to find all local KVT entries which are of the specified category and which specify the CHIT of the original chunk. In one embodiment, the KVT entries may be stored locally at the storage server in a sorted tree structure, and the category of each KVT entry may be specified by a most significant portion of the key. Such a sorted tree structure facilitates rapid searching for a particular category (for example, the payload category, or the version-manifest category, or the content-manifest category) of local KVT entries. 
     The local KVT entries found by the search includes any local chunk KVT entry for a whole replica of the original-format content, if such a whole replica is stored locally at the storage server. 
     The local KVT entries found by the search also includes any local alternate-index (alt-index) KVT entry of the specified category that is associated with the CHIT of the original (target) chunk. Such an alt-index KVT entry is an index KVT entry that has a value that points to a chunk KVT entry for an alternate-format content, where the alternate-format content was derived from the original-format content. Examples of the alternate-format content are described above (compressed content, erasure encoded content, and delta content). A chunk KVT entry associated with the alt-index KVT entry is also found by the search. 
     If no local KVT entries are found by the search, then a response with no KVT entries are returned. Such a response means that this storage server stores neither the original (target) chunk nor any alternate chunk derived from the original (target) chunk. 
     Per step  1306 , a get response is generated. The get response includes all the KVT entries that were found by the search. If the original (target) chunk is stored at the storage server, then the returned KVT entries includes the chunk KVT entry for the original (target) chunk. If an alternate chunk (derived from the original chunk) is stored at the storage server, then the returned KVT entries includes both the chunk KVT entry for the alternate chunk and the alternate-index (alt-index) KVT entry that indicates the relationship between the alternate chunk and the original chunk. 
     Generating the get response per step  1306  may also involve determining a bid for each format, making a new get reservation, and assembling the get response. These further steps are described above in relation to steps  1205 - 1207  in  FIG. 12 . 
     Per step  1308 , the get response is returned to the gateway. The get response indicates which formats, if any, of the target chunk are stored at the storage server and provides a bid indicating a time that the storage server can deliver each indicated format. 
     CONCLUSION 
     In the above description, numerous specific details are given to provide a thorough understanding of embodiments of the invention. However, the above description of illustrated embodiments of the invention is not intended to be exhaustive or to limit the invention to the precise forms disclosed. One skilled in the relevant art will recognize that the invention can be practiced without one or more of the specific details, or with other methods, components, etc. 
     In other instances, well-known structures or operations are not shown or described in detail to avoid obscuring aspects of the invention. While specific embodiments of, and examples for, the invention are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the invention, as those skilled in the relevant art will recognize. These modifications may be made to the invention in light of the above detailed description.