Remove whitelists on delete