Project: KubeStellar (Orchestration & Management / Scheduling & Orchestration)
File: packaging.md

The following is a graph of the GitHub repositories in the `kubestellar` GitHub organization and the dependencies among them. The repo at the tail of an arrow depends on the repo at the head of the arrow. These are not just build-time dependencies but any reference from one repo to another. ```mermaid flowchart LR kubestellar --> kubeflex kubestellar --> ocm-status-addon ocm-status-addon --> kubestellar ocm-transport-plugin --> kubestellar kubestellar --> ocm-transport-plugin ``` The references from ocm-status-addon to kubestellar are only in documentation and are in the process of being removed (no big difficulty is anticipated). See . The repo is the source of a RedHat-style operator. ```mermaid flowchart LR subgraph "ocm-status-addon@GitHub" osa_code[OSA source code] osahcsrc[OSA Helm chart source] end osactrimage[OSA container image] --> osa_code osahcrepo[published OSA Helm Chart] --> osahcsrc osahcsrc -.-> osactrimage osahcrepo -.-> osactrimage ``` The dashed dependencies are at run time, not build time. "OSA" is OCM Status Addon. There is a container image at . The container image is built and published by that repo's release process, which is documented at . By our development practices and not doing any manual hacks, we maintain the association that a container image tagged with `$VERSION` is built from the Git commit that has the Git tag `v$VERSION`. To support testing, `make ko-local-build` will build a single-platform image and not push it, only leave it among your Docker images. The single platform's OS is Linux. The single platform's ISA is defined by the `make` variable `ARCH`, which defaults to what `go env GOARCH` prints. The operator is delivered by a Helm chart at . The chart references the container image. By our development practices and doing doing any manual hacks, we maintain the association that the OCI image tagged `v$VERSION` contains a Helm chart that declares its `version` and its `appVersion` to be `v$VERSION` and the templates in that chart include a Deployment for the OCM Status Add-On Agent using the container image `ghcr.io/kubestellar/ocm-status-addon:$VERSION`. The source is the GitHub repo This appears at . TODO: document how the image is built and published, including explain versioning. Literal KubeStellar release numbers appear here, and are historical. The version of this document in a given release does not mention that release. See for more details on what self-references are and are not handled. ```mermaid flowchart LR subgraph cladm_repo["ocm/clusteradm@GitHub"] cladm_src["clusteradm source"] end subgraph ks_repo["kubestellar@GitHub"] kcm_code[KCM source code] kcmhcsrc[KCM Helm chart source] ks_pch[kubestellar PostCreateHook] ocm_pch["ocm PostCreateHook"] ksscripts -.-> ocmpch ksscripts -.-> kspch end cladmimage["ks/clusteradm image"] --> cladmsrc ocmpch -.-> cladmimage kcmctrimage[KCM container image] --> kcm_code kcmhcrepo[published KCM Helm Chart] --> kcmhcsrc kcmhcsrc -.-> kcmctrimage kcmhcrepo -.-> kcmctrimage kspch -.-> kcmhc_repo ksscripts -.-> osahc_repo[published OSA Helm Chart] ksscripts -.-> otpsomething["published OTP ??"] ks_scripts -.-> KubeFlex ``` The dashed dependencies are at run time, not build time. "KCM" is the KubeStellar

"ks_scripts" are the user-facing instructions and the end-to-end tests that use published artifacts (container images, Helm charts). (There are also e2e tests that do not use previously published artifacts, their temporary local artifacts are beyond the scope of this document.) NOTE: among the references to published artifacts, some have a version that is maintained in Git while others have a placeholder in Git that is replaced in the publishing process. See [the release document](release.md) for more details. This is an on-going matter of development. NOTE: Because of [a restriction in one of the code generators that we use](https://github.com/kubernetes/code-generator/blob/v0.28.2/kube_codegen.sh#L394-L395), a contributor needs to have their local copy of the git repo in a directory whose pathname ends with the Go package name that is, ends with `/github.com/kubestellar/kubestellar`. Some files in the kubestellar repo are derived from other files there. Contributors are responsible for invoking the commands to (re)derive the derived files as necessary. Some of these derived files are derived by standard generators from the Kubernetes milieu. A contributor can use the following command to make all of those, or use the individual `make` commands described in the following subsubsections to update particular subsets. ```shell make all-generated ``` The following command, which we aspire to check in CI, checks whether all those derived files have been correctly derived. It must be invoked in a state where the `git status` is clean, or at least the dirty files are irrelevant; the current commit is what is checked. This command has side-effects on the filesystem like `make all-generated`. ```shell hack/verify-codegen.sh ``` `make manifests` generates the CustomeResourceDefinition files, which exist in two places: `config/crd/bases` and `pkg/crd/files`. `make generate` generates the deep copy code, which exists in `zz_generated.deepcopy.go` next to the API source. The files in `pkg/generated` are generated by . This generation is done at development time by the command `make codegenclients`. KubeStellar has one container image, for what is called the KubeStellar controller-manager. For each WDS, KubeStellar has a pod running that image. It installs the needed custom resource definition objects if they are not already present, and is a controller-manager hosting the per-WDS controllers ( and ) from the kubestellar repo. The image repository is `ghcr.io/kubestellar/kubestellar/controller-manager`. By our development practices and not doing any manual hacking we maintain the association that the container image tagged `$VERSION` is built from the Git commit having the Git tag `v$VERSION`. The builds and publishes that container image. `make ko-build-local` will make a local image for just the local platform. This is used in local testing. There is a Helm chart that is designed to be instantiated in a KubeFlex hosting cluster, once per WDS. The focus of the chart is getting the KubeStellar controller-manager installed. The source for the Helm chart is in the `chart/` directory. `make chart` (re)derives it from local sources. This is not included in `make all-generated`. This chart creates (among other things) a `Deployment` object that runs a container from the

The chart is published at the OCI repository `ghcr.io/kubestellar/kubestellar/controller-manager-chart`. A GitHub Actions workflow (in `.github/workflows/goreleaser.yml`) specializes and publishes this chart as part of . By our development practices and not doing any manual hacking, we maintain the association that the OCI image tagged `$VERSION` contains a Helm chart that declares its `version` and its `appVersion` to be `$VERSION` and that chart has a Deployment that uses the kubestellar-controller-manager container image tagged `$VERSION`. The kubestellar GitHub repository has a script, `hack/build-clusteradm-image.sh`, that creates and publishes a container image holding the `clusteradm` command from OCM. The source of the container image is read from the latest release of , unless a command line flag says to use a specific version. This script also pushes the built container image to using a tag that equals the ocm/clusteradm version that the image was built from. There are two `PostCreateHook` objects defined in the `config/postcreate-hooks/` directory. The PostCreateHook defined in `ocm.yaml` gets used on an ITS and adds the hub side of OCM there, using the image `quay.io/kubestellar/clusteradm:0.8.2`. This image includes the `clusteradm` CLI release `v0.8.2` which is bundled with the OCM release `v0.13.2`. See about the source of that. Currently this PostCreateHook is used in the E2E tests but this is a problem because of its fixed reference to container image previously built from sources in this same Git repository. The PostCreateHook defined in `kubestellar.yaml` is intended to be used in the hosting cluster, once per WDS, and runs container image `quay.io/kubestellar/helm:v3.14.0` (which is built from [the Helm source](https://github.com/helm/helm/tree/v3.14.0) by a process that we need to document) to instantiate the chart from `oci://ghcr.io/kubestellar/kubestellar/controller-manager-chart`; the chart version appears as a literal in the PostCreateHook, currently "0.20.0". Currently the only reference to any copy of this PostCreateHook is from the , which references the copy in the Git commit tagged `v0.20.0`. Currently only showing kubestellar and ocm-status-addon. TODO: finish this ```mermaid flowchart LR subgraph osa_repo["ocm-status-addon@GitHub"] osa_code[OSA source code] osahcsrc[OSA Helm chart source] end osactrimage[OSA container image] --> osa_code osahcrepo[published OSA Helm Chart] --> osahcsrc osahcsrc -.-> osactrimage osahcrepo -.-> osactrimage subgraph cladm_repo["ocm/clusteradm@GitHub"] cladm_src["clusteradm source"] end subgraph ks_repo["kubestellar@GitHub"] kcm_code[KCM source code] kcmhcsrc[KCM Helm chart source] ks_pch[kubestellar PostCreateHook] ocm_pch["ocm PostCreateHook"] ksscripts -.-> ocmpch ksscripts -.-> kspch end osarepo -.-> ksrepo cladmimage["ks/clusteradm image"] --> cladmsrc ocmpch -.-> cladmimage kcmctrimage[KCM container image] --> kcm_code kcmhcrepo[published KCM Helm Chart] --> kcmhcsrc kcmhcsrc -.-> kcmctrimage kcmhcrepo -.-> kcmctrimage kspch -.-> kcmhc_repo ksscripts -.-> osahc_repo[published OSA Helm Chart] ksscripts -.-> otpsomething["published OTP ??"] ks_scripts -.-> KubeFlex ``` Every dotted line is a reference that must be versioned. How do we keep all those versions right? Normally a git tag is an immutable reference to an immutable git commit. Let's not violate that. Can/should we say that an OCI image (or whatever) tag equals the tag of the commit that said image (or whatever) was built from? While keeping `main` always a working system?