Don't cleanup server sockets that have not been created in the first place