Change:
We were using cel-go v0.12.7 in Kubernetes 1.27 release and cel-go v0.16.0 in Kubernetes 1.28 release. There are many changes from cel-go side between those two releases and some may break backward compatible of CRD validation rules. An example would be the estimated cost for CRD validation rules(Thanks to who brought this to our attention with the following example). In the example CRD validation rules below: It will pass perfect in Kubernetes 1.27 version but fail in Kubernetes 1.28 with error: This is due to the estimator fix happened in cel-go(, ) The changes should not cause backward incompatible of CRD validation rules. OR a stability test suite should be in place to prevent the changes causing backward compatible issue before GA. Apply the above CRD in Kubernetes 1.27 and 1.28, 1.27 cluster should pass and 1.28 cluster should fail. Since the fix in cel-go side makes sense to me and the feature is in Beta now, I am not quite sure how strict we should be on backward compatible. cc As a followup, a stability test suite for estimate cost is worth adding before GA to make sure any future changes be caught before the feature promoted to GA. <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi I agree that the changes in cel-go should not break the backward compatibility of CRD validation rules, especially for users who have already defined and deployed their custom resources using the previous version of Kubernetes. One suggestion I have is to use a version flag for the CEL library, so that users can specify which version of cel-go they want to use for their CRD validation rules. This way, they can avoid unexpected errors or behavior changes when upgrading their Kubernetes cluster. For example, they can add something like in their CRD schema to indicate that they want to use the old version of cel-go.
Another suggestion I have is to use a cost limit parameter for the CEL library, so that users can adjust the budget for their CRD validation rules. This way, they can fine-tune the trade-off between performance and complexity of their rules. For example, they can add something like in their CRD schema to indicate that they want to allow more expensive rules.
I agree this is the type of bug fix that is legitimate to make during beta. when the size changes came in, did our existing size estimation test coverage notice this cost increased? If not, is there a systemic change we could make to exercise each type of estimation combination so we have visibility to any future changes. Existing persisted CRD objects should continue to work as-is on upgrade, but the CEL rules and/or schema would be required to fit within fixed cost estimates before making future updates/changes to the CEL rules there should be unit tests ensuring updates to a CRD that don't change CEL rules are allowed even if the existing CEL rules exceed cost estimates, but can you verify that is the case? That's not workable, both from an implementation perspective and from a fragmentation perspective. We explicitly don't want to allow opting into loopholes / bugs in cost estimation that allow CRD authors to make API servers do excessive amounts of work. Similar to above, the responsibility for API server resource consumption lies with the kube-apiserver admin, not the CRD author, so exposing this in the API is not something I would expect.
/assign
Thank you! I guess the next step would be: the existing test suite regarding with the cel cost evaluation. Add estimated cost tests for every single operation, function, type, etc if it's not existing now. updates to a CRD that don't change CEL rules are allowed even if the existing CEL rules exceed cost estimates. Currently it is not working like this but would definitely help here. cc , please correct me if I'm wrong :) Those should be done before the feature goes to GA. I am not sure about the policy of a GA feature relying on alpha/beta feature is acceptable or not. is going to Beta in this release. The worst case is we have to wait further until the feature is ready.
I don't think that helps... CRD ratcheting is about validation of the custom resources. This is about validation of the CRD itself.
thanks for clarifying! Are you suggesting while updating CRD, if the rules matching what's in the existing object, we should skip cost estimation check? That makes sense. I could start a PR if my understanding aligns :)
something along those lines, though now that I think about it, other changes in the schema could affect the number of times a CEL expression could be run (dropping or increasing maxItems, maxProperties, etc) we already collect preexisting expressions when updating a CRD in order to allow already-persisted expressions (possibly persisted via a newer API server) making use of new CEL features: I think that is what I was thinking of. I would probably focus on: coverage of the edges of the cost estimation so we don't pick up changes there without knowing coarse-grained skipping of schema/cel validation for a particular version in a CRD for updates which don't change anything under that version's schema (would let someone make unrelated updates to the CRD and leave an existing version's schema / CEL rules alone)
Thank you! We have those work planned for the current cycle. /assign /assign

Discussion:
requireMapListKeysMapSetValidation bool // preexistingExpressions tracks which CEL expressions existed in an object before an update. May be nil for create. preexistingExpressions preexistingExpressions <ins> // versionsWithUnchangedSchemas tracks schemas of which versions are unchanged when updating a CRD. // Does not apply to creation or deletion. // Some checks use this to avoid rejecting previously accepted versions due to a control plane upgrade/downgrade. versionsWithUnchangedSchemas sets.Set[string] // suppressPerExpressionCost indicates whether CEL per-expression cost limit should be suppressed. // It will be automatically set during Versions validation if the version is in versionsWithUnchangedSchemas. suppressPerExpressionCost bool </ins> celEnvironmentSet *environment.EnvSet }

}) } <ins> // findVersionsWithUnchangedSchemas finds each version that is in the new CRD object and differs from that of the old CRD object. // It returns a set of the names of mutated versions. // This function does not check for duplicated versions, top-level version not in versions, or coexistence of // top-level and per-version schemas, as further validations will check for these problems. func findVersionsWithUnchangedSchemas(obj, oldObject *apiextensions.CustomResourceDefinition) sets.Set[string] { versionsWithUnchangedSchemas := sets.New[string]() for _, version := range obj.Spec.Versions { newSchema, err := apiextensions.GetSchemaForVersion(obj, version.Name) if err != nil { continue } oldSchema, err := apiextensions.GetSchemaForVersion(oldObject, version.Name) if err != nil { continue } if apiequality.Semantic.DeepEqual(newSchema, oldSchema) { versionsWithUnchangedSchemas.Insert(version.Name) } } return versionsWithUnchangedSchemas } // suppressExpressionCostForUnchangedSchema returns a copy of opts with suppressPerExpressionCost set to true if // the specified version's schema is unchanged. func suppressExpressionCostForUnchangedSchema(opts validationOptions, version string) validationOptions { if opts.versionsWithUnchangedSchemas.Has(version) { opts.suppressPerExpressionCost = true } return opts } </ins> // ValidateCustomResourceDefinitionUpdate statically validates // context is passed for supporting context cancellation during cel validation when validating defaults func ValidateCustomResourceDefinitionUpdate(ctx context.Context, obj, oldObj *apiextensions.CustomResourceDefinition) field.ErrorList {

requireAtomicSetType:                     requireAtomicSetType(&oldObj.Spec), requireMapListKeysMapSetValidation:       requireMapListKeysMapSetValidation(&oldObj.Spec), preexistingExpressions:                   findPreexistingExpressions(&oldObj.Spec), <ins> versionsWithUnchangedSchemas:             findVersionsWithUnchangedSchemas(obj, oldObj), </ins> celEnvironmentSet:                        environment.MustBaseEnvSet(environment.DefaultCompatibilityVersion()), } return validateCustomResourceDefinitionUpdate(ctx, obj, oldObj, opts)

for _, err := range validateDeprecationWarning(version.Deprecated, version.DeprecationWarning) { allErrs = append(allErrs, field.Invalid(fldPath.Child("deprecationWarning"), version.DeprecationWarning, err)) } <ins> opts = suppressExpressionCostForUnchangedSchema(opts, version.Name) </ins> allErrs = append(allErrs, validateCustomResourceDefinitionValidation(ctx, version.Schema, statusEnabled, opts, fldPath.Child("schema"))...) allErrs = append(allErrs, ValidateCustomResourceDefinitionSubresources(version.Subresources, fldPath.Child("subresources"))...) for i := range version.AdditionalPrinterColumns {

} allErrs = append(allErrs, ValidateCustomResourceDefinitionNames(&spec.Names, fldPath.Child("names"))...) <del> allErrs = append(allErrs, validateCustomResourceDefinitionValidation(ctx, spec.Validation, hasAnyStatusEnabled(spec), opts, fldPath.Child("validation"))...) </del> <ins> allErrs = append(allErrs, validateCustomResourceDefinitionValidation(ctx, spec.Validation, hasAnyStatusEnabled(spec), suppressExpressionCostForUnchangedSchema(opts, spec.Version), fldPath.Child("validation"))...) </ins> allErrs = append(allErrs, ValidateCustomResourceDefinitionSubresources(spec.Subresources, fldPath.Child("subresources"))...) for i := range spec.AdditionalPrinterColumns {

} else { for i, cr := range compResults { expressionCost := getExpressionCost(cr, celContext) <del> if expressionCost > StaticEstimatedCostLimit { </del> <ins> if !opts.suppressPerExpressionCost && expressionCost > StaticEstimatedCostLimit { </ins> costErrorMsg := getCostErrorMessage("estimated rule cost", expressionCost, StaticEstimatedCostLimit) allErrs.CELErrors = append(allErrs.CELErrors, field.Forbidden(fldPath.Child("x-kubernetes-validations").Index(i).Child("rule"), costErrorMsg)) }

allErrs.CELErrors = append(allErrs.CELErrors, field.Invalid(fldPath.Child("x-kubernetes-validations").Index(i).Child("messageExpression"), schema.XValidations[i], cr.MessageExpressionError.Detail)) } else { if cr.MessageExpression != nil { <del> if cr.MessageExpressionMaxCost > StaticEstimatedCostLimit { </del> <ins> if !opts.suppressPerExpressionCost && cr.MessageExpressionMaxCost > StaticEstimatedCostLimit { </ins> costErrorMsg := getCostErrorMessage("estimated messageExpression cost", cr.MessageExpressionMaxCost, StaticEstimatedCostLimit) allErrs.CELErrors = append(allErrs.CELErrors, field.Forbidden(fldPath.Child("x-kubernetes-validations").Index(i).Child("messageExpression"), costErrorMsg)) }

forbidden("spec", "validation", "openAPIV3Schema", "properties[bar]", "items", "nullable"), }, }, <ins> { name: "suppress per-expression cost limit in pre-existing versions", old: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "plural.group.com", ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Group: "group.com", Scope: apiextensions.ResourceScope("Cluster"), Names: apiextensions.CustomResourceDefinitionNames{Plural: "plural", Singular: "singular", Kind: "Plural", ListKind: "PluralList"}, Versions: []apiextensions.CustomResourceDefinitionVersion{ { Name:    "v1", Served:  true, Storage: true, Schema: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", Properties: map[string]apiextensions.JSONSchemaProps{ "f": { Type: "array", Items: &apiextensions.JSONSchemaPropsOrArray{ Schema: &apiextensions.JSONSchemaProps{ Type: "string", }, }, XValidations: apiextensions.ValidationRules{ { Rule:              "true", MessageExpression: `self[0] + self[1] + self[2] + self[3] + self[4] + self[5] + self[6] + self[7]`, }, }, }, }, }, }, }, { Name: "v2", Schema: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", Properties: map[string]apiextensions.JSONSchemaProps{ "f": { Type: "array", Items: &apiextensions.JSONSchemaPropsOrArray{ Schema: &apiextensions.JSONSchemaProps{ Type: "string", }, }, XValidations: apiextensions.ValidationRules{ { Rule:              "true", MessageExpression: `self[0] + self[1] + self[2] + self[3] + self[4] + self[5] + self[6] + self[7]`, }, }, }, }, }, }, }, }, }, Status: apiextensions.CustomResourceDefinitionStatus{StoredVersions: []string{"v1"}}, }, resource: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "plural.group.com", ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Group: "group.com", Scope: apiextensions.ResourceScope("Cluster"), Names: apiextensions.CustomResourceDefinitionNames{Plural: "plural", Singular: "singular", Kind: "Plural", ListKind: "PluralList"}, Versions: []apiextensions.CustomResourceDefinitionVersion{ { Name:    "v1", // unchanged Served:  true, Storage: true, Schema: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", Properties: map[string]apiextensions.JSONSchemaProps{ "f": { Type: "array", Items: &apiextensions.JSONSchemaPropsOrArray{ Schema: &apiextensions.JSONSchemaProps{ Type: "string", }, }, XValidations: apiextensions.ValidationRules{ { Rule:              "true", MessageExpression: `self[0] + self[1] + self[2] + self[3] + self[4] + self[5] + self[6] + self[7]`, }, }, }, }, }, }, }, { Name: "v2", // touched Schema: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", Properties: map[string]apiextensions.JSONSchemaProps{ "f": { Type: "array", Items: &apiextensions.JSONSchemaPropsOrArray{ Schema: &apiextensions.JSONSchemaProps{ Type: "string", }, }, XValidations: apiextensions.ValidationRules{ { Rule:              "true", MessageExpression: `self[0] + self[1] + self[2] + self[3] + self[4] + self[5] + self[6] + self[7] + self[8]`, }, }, }, }, }, }, }, { Name: "v3", // new Schema: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", Properties: map[string]apiextensions.JSONSchemaProps{ "f": { Type: "array", Items: &apiextensions.JSONSchemaPropsOrArray{ Schema: &apiextensions.JSONSchemaProps{ Type: "string", }, }, XValidations: apiextensions.ValidationRules{ { Rule:              "true", MessageExpression: `self[0] + self[1] + self[2] + self[3] + self[4] + self[5] + self[6] + self[7]`, }, }, }, }, }, }, }, }, }, Status: apiextensions.CustomResourceDefinitionStatus{StoredVersions: []string{"v1"}}, }, errors: []validationMatch{ // versions[0] is exempted because it existed in oldObject forbidden("spec", "versions[1]", "schema", "openAPIV3Schema", "properties[f]", "x-kubernetes-validations[0]", "messageExpression"), forbidden("spec", "versions[2]", "schema", "openAPIV3Schema", "properties[f]", "x-kubernetes-validations[0]", "messageExpression"), }, }, { name: "suppress per-expression cost limit in new object during top-level schema to Versions extraction", old: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "plural.group.com", ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Group:   "group.com", Scope:   apiextensions.ResourceScope("Cluster"), Names:   apiextensions.CustomResourceDefinitionNames{Plural: "plural", Singular: "singular", Kind: "Plural", ListKind: "PluralList"}, Version: "v1", Validation: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", Properties: map[string]apiextensions.JSONSchemaProps{ "f": { Type: "array", Items: &apiextensions.JSONSchemaPropsOrArray{ Schema: &apiextensions.JSONSchemaProps{ Type: "string", }, }, XValidations: apiextensions.ValidationRules{ { Rule:              "true", MessageExpression: `self[0] + self[1] + self[2] + self[3] + self[4] + self[5] + self[6] + self[7]`, }, }, }, }, }, }, }, Status: apiextensions.CustomResourceDefinitionStatus{StoredVersions: []string{"v1"}}, }, resource: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "plural.group.com", ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Group: "group.com", Scope: apiextensions.ResourceScope("Cluster"), Names: apiextensions.CustomResourceDefinitionNames{Plural: "plural", Singular: "singular", Kind: "Plural", ListKind: "PluralList"}, Versions: []apiextensions.CustomResourceDefinitionVersion{ { Name:    "v1", // unchanged, was top-level Served:  true, Storage: true, Schema: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", Properties: map[string]apiextensions.JSONSchemaProps{ "f": { Type: "array", Items: &apiextensions.JSONSchemaPropsOrArray{ Schema: &apiextensions.JSONSchemaProps{ Type: "string", }, }, XValidations: apiextensions.ValidationRules{ { Rule:              "true", MessageExpression: `self[0] + self[1] + self[2] + self[3] + self[4] + self[5] + self[6] + self[7]`, }, }, }, }, }, }, }, { Name: "v2", // new Schema: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", Properties: map[string]apiextensions.JSONSchemaProps{ "f": { Type: "array", Items: &apiextensions.JSONSchemaPropsOrArray{ Schema: &apiextensions.JSONSchemaProps{ Type: "string", }, }, XValidations: apiextensions.ValidationRules{ { Rule:              "true", MessageExpression: `self[0] + self[1] + self[2] + self[3] + self[4] + self[5] + self[6] + self[7] + self[8]`, }, }, }, }, }, }, }, }, }, Status: apiextensions.CustomResourceDefinitionStatus{StoredVersions: []string{"v1"}}, }, errors: []validationMatch{ // versions[0] is exempted because it existed in oldObject as top-level schema. forbidden("spec", "versions[1]", "schema", "openAPIV3Schema", "properties[f]", "x-kubernetes-validations[0]", "messageExpression"), }, }, </ins> } for _, tc := range tests {