Don't add the port to the site config unless explicitly set.