master: do not use -p $port if unset in ssh url

If the port part is not specified in the address url, we should not
substitute it to 22.

This enables the choice to be made in /etc/ssh/ssh_config