Project: Grafana Mimir (Observability and Analysis / Observability)
File: contributing-to-helm-chart.md

Please see the for reference. Changelog is in the chart itself . If you made any changes to the , run `make doc` and commit the changed files to update the . If your changes impact the test configurations in the directory, see . We keep a compiled version of the helm chart committed in the repository for each values file in the `operations/helm/charts/mimir-distributed/ci` directory and its sub-directories. This makes it easy to see how a given PR impacts the final output. For each values file the helm chart is compiled into intermediate YAML manifests, then some frequently changing version information is removed from the manifests and the result is committed. A PR check will fail if you forget to update the compiled manifests, and you can use `make build-helm-tests` to update them and `make check-helm-tests` to verify them after commit. The PR check will also fail if on the intermediate YAML manifests fail. To lint, verify, and execute static analysis tests, use the make targets `conftest-fmt`, `conftest-verify` and `check-helm-tests`. The tests verify that the policies defined in `operations/helm/policies` are met for all Kubernetes manifests that are generated from configurations that are defined in the `operations/helm/charts/mimir-distributed/ci` directory and its sub-directories. Normally contributors need not bump the version. The chart will be released with a beta version weekly by maintainers (unless no changes were made) and also regular stable releases will be released by cherry picking commits from the `main` branch to a release branch (e.g. `release-2.1`). If version increase is need, the version is set in the chart itself . On the `main` branch, versions should be pre-release only, meaning that the version should be suffixed by `-beta.<n>`, for example `2.1.0-beta.1` for a pre-release of the next stable `2.1.0` chart version. Versioning should follow the which is . Once a PR that updates the chart version is merged to `main`, it takes a couple of minutes for it to be published in Helm repository. Update the Helm repository to refresh the list of available charts, by using the command `helm repo update`. In order to search, template, install, upgrade, etc beta versions of charts, Helm commands require the user to specify the `--devel` flag. This means that checking for whether the beta version is published should be done with `helm search repo --devel`. Run with the `docker` command: ```bash docker run --rm -u $(id -g):$(id -u) -e HOME=/tmp -v $(pwd):/data quay.io/helmpack/chart-testing:latest sh -c "ct lint --all --debug --chart-dirs /data/operations/helm/charts --check-version-increment false --config /data/operations/helm/ct.yaml" ``` In order to prevent configuration drift between the Mimir jsonnet library and the Mimir helm chart, an automated diff is performed against every pull request. This diff makes extensive use of to remove unimportant or known differences between the two sets of manifests. A custom kustomize function is used to extract the Mimir configuration from kubernetes manifests so that it can also be compared. The end goal is to ensure that only "useful" differences appear in the diff

Deciding which differences are useful is a complicated topic, but at a high level we use the following heuristics: Differences in Kubernetes annotations and labels are typically not useful, since changes in those will appear in other tests (ie golden record, functionality, etc). Differences in configuration parameters related to urls and file paths are typically not interesting, since they don't change what the cluster does, only where it happens. Differences in performance or scale related properties are typically useful, since these often have difficult-to-test implications on the cluster. In order to keep the kustomize configuration manageable, it is divided into overlays, each named based on the order they are applied. For example, at the time of writing, the Helm chart is passed through the following overlays: ``` $ ls -1 operations/compare-helm-with-jsonnet/helm 00-base 01-ignore 02-configs-and-k8s-defaults 03-set-namespace 04-labels 05-memberlist 06-memcached 07-services 08-pods 09-config ``` Each directory contains a `kustomize.yaml` file that describes the transformations made in that overlay. A full explanation of kustomize is outside the scope of this document, but generally the overlays do one or all of the following: Remove properties that are not useful for diffing Modify property values so that they match between Helm and Jsonnet Remove entire objects that only exist in either Helm or Jsonnet You can use the `make check-helm-jsonnet-diff` target to perform an automatic diff of the Helm and Jsonnet templates. This target requires the following: `yq`, `kubectl`, and `kustomize` installed and on the PATH A running kubernetes API server selected as the currently active `kubectl` context. The API server is only used to perform dry-runs of server-side apply. No resources are actually created in kubernetes, but API calls will be made against the server. It is recommended to use kind, k3d, or some other local development cluster. This allows the final diff to ignore any fields that match kubernetes defaults. Ideally, there will be no differences between helm and jsonnet, so it's expected that the kustomize configuration will shrink over time rather than grow. Achieving perfect parity is difficult, so this process allows us to incrementally fix differences while also preventing new differences from being added. If CI reports a difference, you have several options: Modify the Helm chart or Jsonnet library such that the differences are no longer present Modify the Helm values file or Jsonnet configuration such that the differences are no longer present If the difference is not useful, modify the kustomize configuration to remove that field or otherwise patch the manifests such that the differences are no longer present There is another script, `operations/compare-helm-with-jsonnet/compare-kustomize-outputs.sh`, which can be used to compare any two overlays in the diffing process. This can be a useful tool when debugging kustomize configuration. For example, the following invocation will show only changes that have been applied by the 9th helm overlay. ``` cd operations/compare-helm-with-jsonnet ./compare-kustomize-outputs.sh ./helm/08- ./helm/09- ``` The output can be filtered further by supplying a `select` expression. This is useful to limit otherwise noisy output to only show objects of a certain kind, name, or other property. ``` cd operations/compare-helm-with-jsonnet ./compare-kustomize-outputs.sh ./helm/09- ./jsonnet/09- 'select(.kind == "StatefulSet")' ```