Project: StackRox (Provisioning / Security & Compliance)
File: EXTENDING_CRDS.md

This document describes the required steps for extending the CRDs (for StackRox Central and StackRox Secured Cluster) supported by our Kubernetes Operator alongside some general rules and best practices for modifying the CRDs. Let us assume that there is a new (fictitious) feature "Low Energy Consumption Mode", which we would like to implement support for. Prerequisite is that support for this new feature has already been implemented in the relevant Helm charts (`stackrox-central-services` and/or `stackrox-secured-cluster-services`). See . Add a new setting for the feature to the appropriate structs within `operator/apis/platform/<VERSION>/securedclustertypes.go` and/or `operator/apis/platform/<VERSION>/centraltypes.go`. For example: ```go // Set this to 'true' to enable low energy consumption mode. //+operator-sdk:csv:customresourcedefinitions:type=spec,order=42 LowEnergyConsumption *bool `json:"lowEnergyConsumption,omitempty"` ``` The `operator-sdk`-marker in the comment is used for exposing the setting as a user-visible configuration option during operator installation. See for more information on this. For a description of the `kubebuilder`-markers we use in `centraltypes.go` and `securedclustertypes.go`, see the . In particular they are used for configuring default values, as in e.g. ```go //+kubebuilder:default=false ``` Run the following command: ```sh make -C operator generate manifests bundle ``` to update all auto-generated files if required. For example, after adding a new field to `central_types.go` the following files are updated by the above `make` command: ``` operator/apis/platform/v1alpha1/zz_generated.deepcopy.go operator/bundle/manifests/platform.stackrox.io_centrals.yaml operator/bundle/manifests/rhacs-operator.clusterserviceversion.yaml operator/config/crd/bases/platform.stackrox.io_centrals.yaml operator/config/manifests/bases/rhacs-operator.clusterserviceversion.yaml ``` In order for the new setting to be effective, the new field added to a CRD needs to be translated into the appropriate Helm chart configuration. This translation needs to be added to `operator/pkg/central/values/translation/translation.go` and/or `operator/pkg/securedcluster/values/translation/translation.go`. Tests related to the translation of the new setting need to be added to the corresponding `translation_test.go` files. For example, assuming that the corresponding Helm chart setting is named `lowEnergyConsumption`, use something like ```go v.SetBoolValue("lowEnergyConsumption", sc.Spec.LowEnergyConsumption) ``` Regarding defaulting, note that there exist different kinds of defaults: Schema-level defaults: These are set via `+kubebuilder` directives, and if not set by the user, will be inserted automatically upon object creation. These values will be visible during translation, but only if the enclosing struct field is already present. Changing a schema-level default counts as a breaking API change, but it is treated as such only semantically, nothing will fail at runtime (see ) Translation logic-level defaults: The translation logic will recognize an absent (`nil`) value, decide on its meaning, and will set a corresponding value in the Helm values (see ). Propagating chart-level defaults: The translation logic will set the corresponding Helm values field only for explicitly set values; for absent values, it will do nothing, thus deferring to the chart's defaulting logic (see ). The CR of an operator is the public API of the ACS

Additionally we need to keep in mind that CRs are often managed in CI/CD pipelines which would break existing automations. Never remove a CR field from a CRD Never remove an enum field from a CRD Defined values must continue to stay valid values, e.g.: `KERNELMODULE` defaults to `EBPF`, but `KERNELMODULE` remains a valid value as an alias for `EBPF` `central-db` is still a valid config section in the Central CR even though users must use postgres Defaults can change when the change is not going to break things Introducing a breaking change: This is possible by promoting a CR, i.e. from `v1alpha1` to `v1beta`. Technically this is equal to introducing moving from a `/v1` to a `/v2` API. Using conversion webhooks from v1alpha1 to v1beta1 (introducing a CRD with a new version) Kubernetes deprecation notices are 2 years on their resources is a tool for finding breaking changes and violations of best practices in CRDs. crd-schema-checker has limitations. It cannot find all types of breaking changes and violations of best practices. A passing grade from crd-schma-checker does not mean that there are no breaking changes or violations of best practices. For example crd-schema-checker cannot find the breaking change caused by the removal of an enum. However, it can find the breaking change caused by the removal of a field. crd-schema-checker can be run on one CRD, to check for violations of best practices, or it can be run on two CRDs to find breaking changes and violations of best practices introduced into the new CRD. When running crd-schema-checker on ACS CRDs, many violations of best practices are reported. These include use of booleans and maps. Most if not all of these violations cannot be fixed without introducing breaking changes. The following shows how to run crd-schema-checker. ``` crd-schema-checker check-manifests [--existing-crd-filename=] --new-crd-filename= ``` Example ``` cd operator/config/crd/bases git show release-4.4:./platform.stackrox.iosecuredclusters.yaml > platform.stackrox.iosecuredclusters-4.4.yaml crd-schema-checker check-manifests --existing-crd-filename=platform.stackrox.iosecuredclusters-4.4.yaml --new-crd-filename=platform.stackrox.iosecuredclusters.yaml ``` See the and . To summarize: Use declarative names, not imperative. Do not use an `is` prefix. Do not use abbreviations, be careful with acronyms. Use camelCase field names. For modelling units, see . For certain use-cases some data types are recommended, in particular: `resource.Quantity`: See . `metav1.Time`: provides a stable serialization format for timestamps, see . `ObjectReference` to refer to specific objects, see . Use ints with specific width, see . Some data types are discouraged, in particular: Avoid floats in spec, avoid floats in status if possible. Avoid unsigned ints. Avoid iota-based enums, prefer named string constants instead. Think twice about booleans: See . Maps: See . Optional/Required values: See . Strings, regex-based validation Constants/Enums: See . Unions: See . Defaulting: See and . Nullability: See . Late initialization: See . Labels, Selector and Annotations: See .