Only remove a List if all its Privileges were removed