deduplicator refactoring