Avoid set size changed during iteration error, in case one of the sockets gets
discarded