Only look for SRV records if necessary

Vanilla client implementation only looks for SRV records when the server port isn't specified, so let's mirror that behavior.