Deduper and bucket manager