Patent ID: 8620877

Claim:
A method of tunable data fingerprinting for optimizing data deduplication performance of a storage system, comprising: receiving a file to be stored by the storage system; determining data fingerprinting parameters for turning a rolling-hash algorithm for chunking data in the file, said parameters are based on at least one of: file type usage in the storage system; a type of content in the file; and the file's size, and said parameters include at least one of: a minimum chunk size; average expected chunk size; maximum chunk size; and chunking window size; tuning the rolling-hash algorithm based on said parameters; chunking data in the file using said tuned algorithm; generating a content identifier for each of the file's data chunks by applying a collision-resistant hashing algorithm to each of the chunks of data; and processing the chunks of data that are unique, the content identifier for each of the chunks of data that are unique, and references to the chunks of data that are unique, said processing including: when an index of the storage system includes the content identifier, then: identifying the chunk of data corresponding to the content identifier as being a duplicated chunk of data; removing the duplicated chunk of data from the system; and adding a reference to the identified chunk of data, and when an index of the system does not include the content identifier, then: identifying the chunk of data corresponding to the content identifier as being a unique chunk of data; identifying the content identifier as a unique content identifier; adding the unique content identifier to the index; storing the identified chunk of data in the storage system; and storing said parameters for subsequent verification of parameters used to generate a content identifier.