Kustomize explained; an MLOps Use Case | by Tibor Fabian | Towards Data Science

Kustomize is a tool to customize YAML files like Kubernetes (K8s) manifests, template free. Meanwhile, it became a built-in kubectl operation to apply K8s object definitions from YAML files stored in a hierarchical directory structure.
There are some great examples in the documentation of Kustomize. Nevertheless, sharing my experience from a journey in the field of Machine Learning Operations (MLOps) might allow you to gain some practical knowledge from a real use case.
K8s object definition is thought to be a manifest of highly reproducible value. Therefore, templates or environment variables, to make content definitions alterable, are not supported. This might lead to a conflict if multiple deployments share many similarities because common parts need a lot of copy and paste and parallel versioning. Also, copies are hard to maintain and source for errors if changes have repeatedly been made across similar definitions.
To dissolve such conflicts, Kustomize comes really in handy.
General concept ☍
Kustomize features☍- bases ☍- nameSuffix ☍- commonLabels ☍- images ☍- configMapGenerator ☍- patchesStrategicMerge ☍
Comparison between customizations ☍
Wrap up ☍
Appendix: Complete customization output ☍
The example in this article is about to run Dataiku Data Science Studio (DSS) on K8s. DSS provides multiple variants — node types — with similar installation configurations to build a complete MLOps pipeline — Design, Automation, and API Deployer. I’m using Kustomize to manage the similarities of the K8s pod configuration definitions to avoid duplicates. See details of this architecture in my post, “MLOps w/ Dataiku DSS on Kubernetes”.
The K8s configurations of the DSS nodes differ only slightly and have a significant amount of overlapping parts. It would be tough to maintain them correctly if the manifests for the node deployments were completely separated. Due to similarities, common details needed to be updated everywhere in all node configurations each time they are subject to changes.
Consider the directory structure of the deployment configuration below. The code to the example use case is available in my GitHub repository https://github.com/tibfab/dss-at-k8s.git.
-- deployment   |-- apideployer   |   |-- kustomization.yaml   |   |-- resources.yaml   |   `-- volumes.yaml   |-- automation   |   |-- kustomization.yaml   |   |-- resources.yaml   |   `-- volumes.yaml   |-- base   |   |-- dss-deployment.yaml   |   |-- kustomization.yaml   |   `-- service.yaml   `-- design       |-- kustomization.yaml       |-- resources.yaml       `-- volumes.yaml
There are three deployment directories deployment/design, deployment/automation, and deployment/apideployer for the three corresponding DSS node types — Design node, Automation node, and API Deployer node. The configuration part common to all deployments is stored in the directory deployment/base.
The main idea behind Kustomize is to build the K8s configuration based on two parts, special to each deployment and common to all. The directory deployment/base consists of the common configuration in the above tree, and the special parts are defined in the node-specific directories.
To apply the entire deployment configuration for the DSS Design node, for example, the following kubectl command needs to be performed.
% kubectl apply --kustomize deployment/designconfigmap/dss-node-type-design-node-982cct889d createdservice/dss-design-node createddeployment.apps/dss-design-node created
Let me walk you through this example, step by step. As a reference and for further study, the complete configuration output is inserted at the end of this post, given its lengthy nature.
First, Kustomize looks up the file kustomization.yaml in the target directory, deployment/design, and based on its content, it merges different parts of the definitions into a whole. The resources specified by the keyword bases are included as the common/shared part, and the customization features (nameSuffix, commonLabels, etc.) are performed on the content of those shared resources.
This article elaborates on some Kustomize features in detail explaining the above configuration in full.
With the keyword bases, Kustomize looks at the file kustomize.yaml in the additional configuration directory specified by the path ../base. In this example, that is to be considered as the common/shared part.
As the content of base/kustomization.yaml above shows, the keyword resources includes two additional configuration files from the same ../base directory:
dss-deployment.yaml☍ with the common parts of the K8s DSS pod deployment definition
service.yaml☍ with the configuration on how to reach the DSS service API
Customization, defined in the node-specific kustomization.yaml files, is performed on these resources. Below is an excerpt from base/dss-deployment.yaml, which serves as the basis for modifications by additional customization steps defined for each DSS cluster node separately.
The Kustomize feature nameSuffix appends a specific suffix, -design-node in my example, to all name definition occurrences shown in file base/dss-deployment.yaml below.
The diff above highlights changes to the original common definition (on the left) after configuration names were updated (on the right) based on the nameSuffix given in file design/kustomization.yaml.
To give you the full example, below is base/service.yaml updated with the same nameSuffix automatically by Kustomize.
Note: The file automation/kustomization.yaml, which uses the same common deployment basis, defines -automation-node at this place and apideployer/kustomization.yaml defines -apideployer-node respectively.
Like name suffixes, labels are updated in base/dss-deployment.yaml according to the configuration in design/kustomization.yaml. This works slightly differently as it replaces labels’ values for a specific keyword, app in this case, instead of extending it — even if it’s not that obvious from the diff below.
The configuration in base/service.yaml is also updated respectively.
Kustomize allows, in this concrete use case, to create three different services for all DSS node types — Design, Automation, and API Deployer — with a single service definition file.
This feature replaces the placeholder image dss-node-image in the configuration base/dss-deployment.yaml. The placeholder is specified in the container configuration section template → spec → containers → image. Its value is replaced by the customized image name defined in file design/kustomization.yaml with the keywords newName/newTag. The placeholder dss-node-image marks the exact spot where to replace.
Note: If we look at the GitHub repo in the file deployment/automation/kustomization.yaml, which uses the same common deployment basis, we find a different image but with the same place holder to replace dss-node-image. As a result, a very similar deployment with different images is generated by Kustomize if we specify the target directory deployment/automation instead of deployment/design.
Kustomize’s configMapGenerator creates K8s ConfigMap resources allowing the usage of customized ConfigMaps as environment variables in the pod definition. To achieve that, configMapGenerator is defined in design/kustomization.yaml and configMapKeyRef with the same name dss-node-type in file base/dss-deployment.yaml.
To bring these two parts together, Kustomize first generates a ConfigMap with a specific name extension automatically.
The name of the new ConfigMap is then embedded in the deployment configuration to be referenced as an environment variable.
Note: The value dss-node-type is the same for all DSS nodes, but the literals in the node-specific kustomization.yaml are different:
DSS_NODE_TYPE=design in this example
DSS_NODE_TYPE=automation in the file automation/kustomization.yaml
As a consequence, the environment variable DSS_NODE_TYPE becomes different values depending on which customization directory is applied.
As its name suggests, patchesStrategicMerge allows the merging of specific configuration segments from design/kustomization.yaml into to the common configuration base/dss-deployment.yaml. In this example, the merge is applied for the configuration of resources and volumes.
resources: The content of design/resources.yaml is merged into the pod definition base/dss-deployment.yaml along with the configuration path template → spec → containers → -name: dss(-design-node) → resources.
volumes: Similarly to resources but on two places this time, the content of design/volumes.yaml is merged into the pod definition base/dss-deployment.yaml along the specified configuration paths:
template → spec → containers → -name: dss(-design-node) → volumeMounts
template → spec → volumes
As mentioned already above, the contents of the other DSS node configuration directories deployment/automation and deployment/apideployer differ only slightly from the above example of deployment/design. Names and labels now contain automation or apideployer instead of design. Images, resources, and volumes are configured slightly differently. See the example automation/kustomization.yaml below.
Having different contents in automation/resources.yaml and automation/volumes.yaml allows to define different K8s resource requests and the usage of installation-specific volumes as necessary.
The main part of the deployment configuration, especially the two additional sidecar containers and their volumes for each pod, remains the same and can be maintained easily in one place.
Kustomize saves us from multiplicating the common part: The contents of base/dss-deploymnet.yaml and base/service.yaml with over a hundred lines of YAML code.
Kustomize features, described in this article, can be a tremendous help to maintain multiple Kubernetes pods with similar definitions. The advantage is undeniable. I hope that the detailed description of the features above can help you adapt to your actual situation.
Note: Kustomize output does not maintain the original order of the configuration, making it a little bit difficult to compare with the above examples.