Add port to host if it's not 80 or 443

Closes #511.