Avoid unnecessary getsockname when listen port is specified