Patent ID: 9678974
Date: 2017-06-13
CPC Classifications: G06F

Claim:
1. A method, comprising: receiving a request at a host to deduplicate a plurality of files, the host and a plurality of other hosts connected to a target system and a plurality of other target systems over a network and communicate using file level protocol; determining segment boundaries for the plurality of files using data type information associated with the plurality of files and determine segment size using file type information; determining whether segments in the plurality of files have already been deduplicated at a target system in a deduplication system, wherein determining whether segments in the plurality of files have already been deduplicated includes comparing hashes of data segments stored at the target system with hashes in a deduplication dictionary at a source device, wherein ranges for the hashes of the data segments are assigned to different nodes within a cluster, each node corresponding to a separate deduplication dictionary, wherein if a first node is processing a data segment which has a hash value which maps to a second node, the first node will contact the second node that owns the range to find out if the data segments already exist in a datastore on the second node, wherein a work portioning scheme is applied based the hash ranges; transmitting a first plurality of segments to the target system, the first plurality of segments not already deduplicated at the target system, the first plurality of segments added to a datastore suitcase, and not transmitting a second plurality of segments to the target system, the second plurality of segments already deduplicated at the target system; modifying references and reference counts including filemaps and the datastore suitcases at the target system for the first plurality of segments and a second plurality of segments.