finish the dedup function during snapshot