If we're calling as we're deleting the app, we should be passing in disable=True, so we can use that to know whether to check if it's public.