remove duplicates and add a new unique index
cleaning