Patent ID: 8751448

Claim:
A system for directing for storage, comprising: a processor configured to: segment input data comprising a data stream, a data block, or a data file into a plurality of segments; group at least some of the plurality of segments into a super segment including by: determining, for each of at least some of the plurality of segments, whether a hash value computed for a segment of the plurality of segments that is associated with a segment boundary and for the segment boundary meet one or more super segment boundary criteria; and in the event that the one or more super segment boundary criteria are met, setting a super segment boundary at the segment boundary that is associated with the segment; and determine, for a first node of a plurality of nodes, a segment overlap for the super segment by determining whether each of the at least some of the plurality of segments comprising the super segment is stored on the first node based at least in part on a probabilistic data structure, wherein the segment overlap comprises a value representative of a number of the plurality of segments comprising the super segment that have been previously stored on the first node of the plurality of nodes; determine a selected node of the plurality of nodes including by comparing the segment overlap determined for at least the first node to a threshold, wherein the threshold is a value that is based at least in part on a total number of the plurality of segments comprising the super segment and a total number of nodes in the plurality of nodes; and a memory coupled to the processor and configured to provide the processor with instructions.