don't remove apps from app_runner until they're deleted