Safer removePort (when remoteAddr is empty)