test-server-utils: Fixed to wait for clients to finalize

Previously we would get sporadic errors from tests when clients
do not finalize synchronously in response to g_object_unref().

However this syncrhonous finalization is not really mandated by
the API, so instead we give them a grace period to finalize and
error out if they fail to finalize eventually.