Fixes random port testing problem

Fixes random port testing problem (#685).
Agree, just want to make it nice. @whirm Shall we do IPv4 only?