Add port in urls unless port 80 or 443

Should probably add some tests around this.

Addresses #191 
build_url should abstract this logic