Patent ID: 8838541

Claim:
An apparatus comprising: a chunk store having a plurality of chunk sections each storing specimen data chunks; a manifest store for containing a manifest representing at least part of a data set and having references to said chunk sections; at least one processor configured to: process an input data set into input data chunks; identify, using the manifest, a specimen data chunk in a given one of said chunk sections which corresponds to a first of the input data chunks; identify a second of the input data chunks not corresponding to a specimen data chunk in the chunk store; store the second input data chunk as a specimen data chunk in deliberate proximity to the identified specimen data chunk, wherein the storing in deliberate proximity results in selecting the given chunk section rather than another of said chunk sections to store the second input data chunk as a specimen data chunk; associate a specimen data chunk in at least one chunk section with a back-reference to a manifest referencing that specimen data chunk; determine when a given specimen data chunk is not associated with a back-reference to a manifest; delete the given specimen data chunk from a particular chunk section after a predetermined time period or number of iterations in response to determining that the given specimen data chunk is not associated with a back-reference to a manifest; and after the deleting, reduce fragmentation of the particular chunk section by rearranging chunks remaining in the particular chunk section.