Massive improvement to group-rule-member recalculating: only remove and add as needed, do not recalculate the entire list