If server can't bind, try to connect anyway

The reason we can't bind might be because something is already
listening, in which case we can test the connectivity anyway.