Don't remove when there are no AttributeModifiers