cleanup duplicates and sorting