Do not destroy ability to use unix socket

Change that allowed us to use ipv6 introduced a regression, which disabled using unix sockets.