Patent ID: 8407193

Claim:
A method for data deduplication compression in a streaming storage application, comprising compressing fully sequential data stored in a data repository to a sequential streaming storage, by: splitting fully sequential data into data blocks; hashing content of each data block and comparing each hash to an in-memory lookup table for a match, the in-memory lookup table storing all hashes that have been encountered during the compression of the fully sequential data; for each data block without a hash match, adding the data block as a new data block for compression of fully sequential data; encoding duplicate data blocks using the in-memory lookup table into data segments such that both sequential and random access decompression of the data repository are enabled; compressing partially sequential data and data from a random access storage stored in a data repository to the sequential streaming storage; and decompressing fully and partially sequential data stored on the sequential streaming storage to the data repository wherein reconstruction metadata references previous data blocks and new data blocks, the previous data blocks read and stored in a decompressed output, the new data blocks contained in a current data segment.