Invalidate listening sockets when closing