added deduplication for queue and archive