Remove variantIds from VariantSet