Improve speed

With big map, Set::merge is slow and sometime not working. So shouldn't
use this function.