to deduplicate, or not to deduplicate... that is the option