Remove allowed users correctly