Disallow access removal for yourself always (unless su)