do some deduplication