Only specify port number if it's neither 80 nor 443

Closes #557.