Only add port if requested scheme doesn't match