fix bugs in deduplication algorithm