dedupe is case insensitive and ignores trailing hashes