Don't append port to host when it's a socket path