new deduping algorithm in place