using checksums to speed up deduping
only storing checksums in memory