Clone the hashset before looping to prevent concurrent modification when removing