I see the `withXXX` modifiers as simple helpers for changing *one* property of the `Uri`.
I wouldn't expect the `withScheme` modifier to also change the port. So I think the current implementation is fine.