Many optimizations to segmented global dedupe.
Use chunk hash based cumulative similarity matching instead of chunk content.