Patent ID: 7631144

Claim:
A method for storing data comprising: receiving an input data stream comprising a plurality of data segments; preliminarily checking in a memory having a relatively low latency whether a duplicate of a segment of the plurality of data segments has been stored previously; in the event that the preliminary check does not conclusively determine whether the duplicate of the segment of the plurality of data segments has been stored previously, determining whether or not to check in a memory having a relatively high latency to conclusively determine whether a duplicate of the segment of the plurality of data segments has been previously stored in the relatively high latency memory, wherein the determination of whether to check is based at least in part on rate limiting access to the memory having the relatively high latency memory, and based at least in part on qualifying data, wherein qualifying data selects a subset of data segments for checking; and in the event that the determination is made not to check in the memory having a relatively high latency, storing the segment of the plurality of data segments in the relatively high latency memory without determining whether a duplicate of the segment of the plurality of data segments has been previously stored in the relatively high latency memory.