Patent ID: 8447740

Claim:
A system for processing data, comprising: a stream locality system having a processor to select a previously stored stream indicated locale, the locale including a selection of a plurality of previously stored data segments satisfying a locality criterion with respect to a portion of a stream currently being processed for storage, wherein in the event that a first data segment included in the portion of the stream currently being processed for storage is determined to be identical to a previously stored data segment, sketches for the plurality of previously stored data segments satisfying the locality criterion are loaded into a limited sketch index; and a delta compression system having a processor to determine that the first data segment included in the portion of the stream currently being processed for storage is similar to a similar data segment among the data segments in the locale by satisfying a similarity criterion, wherein in the event that the first data segment was determined to be not identical to a previously stored data segment, the similar data segment is determined based at least in part on the limited sketch index, and a reference to the similar data segment and one or more sequences in the first data segment that are not in the similar data segment are stored; wherein a master sketch index is accessed only when the sketch of the incoming data segment is not found in the limited sketch index.