Patent ID: 8332404

Claim:
A data processing apparatus comprising: a chunk store containing specimen data chunks, a manifest store containing a plurality of manifests, each of which represents at least a part of a data set and each of which comprises at least one reference to at least one of said specimen data chunks, a sparse chunk index containing information on only some specimen data chunks, at least one processor to: process input data into input data chunks; identify manifests having at least one reference to one of said specimen data chunks that corresponds to one of said input data chunks and on which there is information contained in the sparse chunk index, wherein for a particular specimen data chunk indexed by the sparse chunk index, the sparse chunk index contains information on the most recent predetermined number of manifests added to the manifest store that comprise a reference to said particular specimen data chunk, and wherein the sparse chunk index further contains information for manifests that reference other specimen data chunks; and prioritize the identified manifests for subsequent operation.