removing duplicates and cleanup