Delete permissions if we have too many