Add port in urls unless port 80 or 443

Should probably add some tests around this.