Do not test default port by default

Closes #760
Review 560 by @mlucy