Do not die if we try to close a port that was not opened previously.