added support to remove duplicates