add deduplication in O(n2) and tests