Fix a mistake with removing duplicates

By not setting removed_duplicate_position_for_same_person we
might remove both positions with the same person.