Avoid trying to remove an invalid IP