Revert "Revert "deleting duplicates""

This reverts commit 4c935c5ebea13ec20a8d5483014180e1874e1adf.