Fixes random port testing problem

Fixes random port testing problem (#685).
I don't think we need the socket_familiy stuff (just do ipv4).