Make sure we delete apps in the right way