only remove the socket from the list if it's still in there