Patent ID: 8234250

Claim:
A storage system for processing data blocks of a file for deduplication, the storage system comprising: computer hardware configured for: determining a plurality of sub-portions of the file, each sub-portion comprising a predetermined number of data blocks of the file; assigning a scanner thread to each of the plurality of sub-portions; and storing status data of each scanner thread in a checkpoint data structure, the status data of a scanner thread indicating the sub-portion currently assigned to the scanner thread; and the plurality of scanner threads, each scanner thread configured for: processing the data blocks of the assigned sub-portion to produce and store content identifiers for the data blocks to a content identifier database; upon a crash and restart of the storage system, reading its corresponding status data in the checkpoint data structure to determine its currently assigned sub-portion; and re-starting processing of data blocks from the currently assigned sub-portion, wherein a maximum number of data blocks for re-processing by the scanner thread is equal to the predetermined number of data blocks of a sub-portion.