Protect permission removal against bad plugins