Remove the permission check file after we've verified we can add it.