fix all the bugs with dedup