Don't add colon if there is no port.