Project: cert-manager (Provisioning / Security & Compliance)
File: 20220118.server-side-apply.md

title: Server-Side Apply authors: "@joshvanl" reviewers: @jetstack/team-cert-manager approvers: @jetstack/team-cert-manager editor: "@joshvanl" creation-date: 2022-01-18 last-updated: 2022-01-18 status: implementable <!-- toc --> * + * * * <!-- /toc --> Server-Side Apply is a [Kubernetes feature](https://kubernetes.io/docs/reference/using-api/server-side-apply/) whereby clients writing to a resource that is managed by more than one client can declare what fields that client manages, and make decisions on what to do if there is a conflict with another client on what a field value should be. Server-Side Apply works by the client sending a PATCH API request with a `Content-Type` header with the value `application/apply-patch+yaml`. The `fieldManager=<my-field-manager>` URL query can be optionally sent which instructs which field manager to use (or alternatively, will be derived from the client's user agent). This API call tells the API server that the client wishes to manage and set the fields and values of the resource (or alternatively remove managed fields which are omitted) as they appear in the body. A client's managed fields are defined as the fields that are labelled with the client's manager name `<my-field-manager>` (or client's user agent equivalent). Server-Side Apply is useful for cert-manager for 2 reasons: cert-manager controllers, namely the certificates controllers, have multiple controllers that are writing to the same resource type. This leads to cases where UPDATE operations result in resource version conflicts. This in turn, results in error logs that regularly confuse users and miss-attribute the problem they are having, as well as needless re-queuing churn through the controller, slowing cert-manager down. Without Server-Side Apply, it is difficult for cert-manager to reason about some fields that should be deleted on an UPDATE operation during reconciliation. One such example are Certificate SecretTemplate's Annotations and Labels. If a key was removed from either the Annotation or Label field on the SecretTemplate, the next reconciliation would not know which keys should be preserved or removed (as they may have been annotated or labelled from a third party). This problem occurs in several places where copying or transforming state from one resource to another happens. Managed fields define ownership of applied fields. cert-manager is therefore able to observe a discrepancy occurring from, for example, a previous SecretTemplate state and performing another apply (omitting those keys), resulting in those fields being removed on the Secret. Without managed fields, cert-manager would have to create a third state store in either annotations or status fields on that resource. The Field Manager is a string denoting the manager who owns a field or set of fields on a

This string is set by the client at API call time and is derived by either the client setting the `fieldManager` URL query explicitly, or calculated from the client's user agent ([characters preceding the first "/", quote unprintable characters and then trim what's beyond the 128 limit](https://github.com/kubernetes/kubernetes/blob/9a75e7b0fd1b567f774a3373be640e19b33e7ef1/staging/src/k8s.io/apiserver/pkg/endpoints/handlers/create.go#L252)). cert-manager will use a consistent naming scheme for both the user agent prefix and field manager across all components. Each component will have a human readable name that describes that component, and is used for both its user agent and field manager. Using the same name enables better auditing trails that helps debugging and improves telemetry. This is the list of known field managers. This might not be an exhaustive list as they will be discovered during the changes. ```text cert-manager # base field manager of cert-manager controller, used when no distinct component cert-manager-leader-election cert-manager-acme-[orders,challenges] cert-manager-certificates-[issuing,trigger,keymanager,readiness] cert-manager-certificaterequests-[acme,approver,ca,selfsigned,vault,venafi] cert-manager-clusterissuers-[acme,ca,selfsigned,vault,venafi] cert-manager-issuers-[acme,ca,selfsigned,vault,venafi] cert-manager-cainjector # base field manager of cert-manager cainjector cert-manager-webhook # base field manager of cert-manager webhook cert-manager-cmctl ``` The field manager is derived from the component in cert-manager, prefixed by the string `cert-manager`: ```text cert-manager-<cert-manager component> ``` e.g. ```text cert-manager-certificates-issuing ``` The user agent is inspired by the [client-go default user agent](https://github.com/kubernetes/client-go/blob/664b1a6c8ce9c92ce65bef3f9833b402449c98d2/rest/config.go#L499), and is defined with the following: ```text <component field manager>/<cert-manager version> (<Operating System>/<Architecture>) cert-manager/<git commit> ``` e.g: ```text cert-manager-certificates-issuing/v1.7.0 (Linux/amd64) cert-manager/0b686b8f38c8c7442744c9224d18e780ee7f244a ``` All CREATE and UPDATE operations in cert-manager controllers are to be replaced with Apply. For this change, controllers will need to be modified so that for these calls, only the fields in which they manage should be included in the PATCH API call. This means each controller will need to define exactly which fields they are concerned with. A package will be created which all API calls will use. This package will implement the logic for checking the feature gate for consistency across the project whilst minimising disruption to existing code paths. Some fields, such as the Certificate Issuing Condition are managed by more than one controller (issuing and trigger Certificate controllers, and cmctl), and as such, will need to make use of the `force` parameter in their API calls. This option tells the API server to revoke management of that field from the previous owner, overwrite the field, and change owner ship to the new client. Since some fields, such as the Issuing Condition, may have an undefined number of potential managers (both internal and external to the cert-manager controller), using the same manager for things is not a possibility. You can read more about the `force` paramerter on the Kubernetes documentation on , and in particular the

Use of the `force` parameter will not overwrite other controllers progress since, 1 controllers only update the fields that they manage, 2 controllers who manage the same fields should never be writing to the same field at the same time (as they should be gated by those or other fields presence and values, such as the Issuing condition for the issuing and trigger controllers). There is likely no Apply call that cert-manager does not set the `force` parameter to true since it, [never wants to give up ownership claim, and always wants to overwrite values](https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts). See . The does not support the Apply PATCH call for mocking API calls and events. This means that significant controller unit-testing will either need to moved to testing against a real API server as integration tests, the controller , or a new testing framework should be developed. We can also PR this upstream but will take time to be released so a stop gap would always be needed. Some API fields will need to have some metadata updated to function better under Server-Side Apply. One such example is adding `x-kubernetes-list-type=map` and `x-kubernetes-list-map-keys=Type` to the [Certificates Status Condition slice](https://github.com/jetstack/cert-manager/blob/0ca1ce9a6a1d7c311afd4b3e786975759249132a/pkg/apis/certmanager/v1/types_certificate.go#L385), so that controllers are able to apply distinct condition types, without conflicting with other controller conditions (i.e. the Ready and Issuing conditions). Integration tests will be able to ensure cert-manager have set these API type tags correctly. Placing the Apply functionality behind a feature gate should be required. Placing this functionality behind a feature gate would allow the cert-manager authors gain confidence about its correctness, and ensure there are no regressions in the stability of controller reconciliation. To reach graduation, the cert-manager authors should consider the Server-Side Apply implementation for cert-manager to be safe for end users. One strategy for gaining confidence that it is a safe change is to solicit feedback from end users who have the feature enabled. Graduation should ideally be done over no more than one or two releases, however the project shouldn't proceed with removing the gate until the authors are confident. For an exiting cluster with cert-manager and resources installed, a user should be able to safely enable the feature or upgrade cert-manager with the feature enabled by default without any down time or loss of data. Although testing is needed, it should be the case that an upgrade/feature enable that all controllers on the next reconcile that applies, will take ownership of that component's fields and set the correct values. Fields will not be deleted that should be however (i.e. Annotation or Label from a previous SecretTemplate) as the field manager has changed (though this is the same behaviour as today).