working version of generic deduplicator