Force-disable Xlib in cairo builds

On some host environments, this can incorrectly be detected as available
during a cross build, so let's force-disable it.