deal with duplicates well