Fix missing invalidation of listening socket

When we run out of ports and can't bind, we would close the socket but
never invalidate it, causing an infinite loop situation later trying to
accept on the socket.

Thanks to @dctrwatson for finding this!