Project: kpt (Provisioning / Automation & Configuration)
File: 02-oci-support.md

Author(s): Louis Dejardin, @loudej Approver: \<kpt-maintainer\> Every feature will need design sign off an PR approval from a core maintainer. If you have not got in touch with anyone yet, you can leave this blank and we will try to line someone up for you. Systems that deal with packaging software or bundling configuration often have an atomic, versionable artifact. This artifact can exist as a source of truth independant from the source controlled content from which it was built. It is also very common for those packages to have an associated feed or repository which can receive those packages as they are published, and make them available for download as needed. In many companies, using `git` as the as the repository and source of truth for production configuration comes with challenges. This design document proposes to add `OCI` as an alternative to `git` for publishing and distributing Kpt config packages. As a packaging format, it is well understood and documented. As a repository format, it leverages existing container registries for pushing and pulling config as image content. For security and production configuration management, if a company has practices for managing Docker container images in private registries, then the same practices and security model can be applied to config package images in private registries. https://github.com/GoogleContainerTools/kpt/issues/2300 The first stage of OCI support comes from adding support for `oci` in the places where `git` appears today. An image tag is used in the same way a git branch or tag would be used. An image digest is used in the same way a git commit would be used. The scope of a single image is one root package, with any number of optional sub-packages. The structure of the image is a single tar layer. The root `Kptfile` is in the base directory from the tar layer's point of view. It contains only the Kpt package files, no entrypoint or executables. A package image should not be confused with a container image. Container images are executable by software, like `Docker`, and package images are purely configuration data. The `Kptfile` structures for `upstream` and `upstreamLock` have `oci` in addition to `git` properties. The `type` property also has the string `oci` added as an accepted value. ```yaml upstream: type: oci oci: image: 'IMAGE:TAG' upstreamLock: type: oci oci: image: 'IMAGE:DIGEST' ``` New verions of `kpt` will support existing `Kptfile`. The file structures and `git` functionality is unchanged. Existing versions of `kpt` will support `Kptfile` with `upstream` based on `git` for the same reason. The structure and meaning of existing fields is not changed. Existing versions of `kpt` will not support `Kptfile` with `upstream` based on `oci`. The `type` value, and missing `git` information will fail validation. The `kpt` binary used will need to be upgraded. The argument that determines upstream today is parsed into `repo`, `ref`, and `path`, and is implicitly a `git` location. To support `oci`, it will be necessary to extract different values in a way that's unambiguous. Unfortunately, OCI image names have no Uri prefix, and are indistinguishable from a valid path or file

To solve this, using as an example, the prefix `oci://` can be used. This ensures that selecting `oci` protocol isn't accidental, and it won't collide with other location formats that may be added. ```shell kpt pkg get oci://us-docker.pkg.dev/the-project-id/the-repo-name/the-package:v3 my-package ``` Because OCI image reference already has a convention for `image:tag` references, using `:v3` should be used instead of `@v3` for version. It will be more intuitive how it relates to the registry, and easier to cut and paste values. It is possible to use `kpt pkg get` to add sub-packages to a target location. Syntax for a sub-package target location is unchanged, it's a normal filesystem path. Syntax for an OCI sub-package source location requires the ability to tell when an image name ends and a sub-package path inside that image begins. In `git` this requires an explicit `.git` extension at the transition, and in `.oci` this requires double slash. ```shell kpt pkg get oci://us-docker.pkg.dev/the-project-id/the-repo-name/the-package//simple/example:v3 my-package/simple/my-example ``` The command for update is not changed, but when the `upstream` is `oci` then the `@VERSION` is used to change the `upstream` image's `tag` or `digest` value. To update to an image tag, `kpt pkg update @v14` and `kpt pkg update DIR@v14` will assign the `:v14` tag onto the upstream image. ```yaml upstream: type: oci oci: image: us-docker.pkg.dev/the-project-id/the-repo-name/the-package:v14 ``` To update to an upstream digest, `kpt pkg update @sha256:{SHA256HEX}` and `kpt pkg update DIR@sha256:{SHA256HEX}` will assign `@sha256:{SHA256_HEX}` as the new upstream image digest. ```yaml upstream: type: oci oci: image: us-docker.pkg.dev/the-project-id/the-repo-name/the-package@sha256:8815143a333cb9d2cb341f10b984b22f3b8a99fe ``` Calling `kpt pkg update` and `kpt pkg update DIR` will perform an update without changing the upstream image name. At that point, if the `upstream` is an `image:tag` that is to discover the current `image:digest` for tag, otherwise the `upstream` value for `image:digest` is used. In either case, the `upstreamLock` is changed to point at that new `image:digest`. The package contents of the old and new `upstreamLock` image digest are fetched to temp folders, and are the basis of the 3-way merge to update the target package. The `kpt pkg diff` command is identical to `kpt pkg update` in the way that `[PKG_PATH@VERSION]` argument is mapped to OCI concepts. Although it is possible to create and push an OCI image using a combination of commands like `tar` and `gcrane`, that doesn't provide a very complete end to end experience. Because kpt would already be built with the same OCI go module used by `gcrane`, it is not difficult to support additional commands to move pull and push package contents from local folders to remote images and back. ``` Usage: kpt pkg pull oci://{IMAGE[:TAG|@sha256:DIGEST]} [DIR] DIR Destination folder for image contents. Default folder name is the last part of the IMAGE path. IMAGE[:TAG|@sha256:DIGEST] Name of image to pull contents from, with optional TAG or DIGEST. Default TAG is `Latest` ``` This command is the reverse of push. An image can be pulled from a repository to a local folder, modified, and pushed back to the same location, same location with different TAG, or entirely different location. The target DIR is optional, following the conventions of `kpg pkg get`, and will default to the final image/path

`kpt pkg pull` works on git uri as well. This may be used, for example, to mirror a set of known blueprints into a private oci registry. ``` Usage: kpt pkg push [DIR@VERSION] [--origin oci://{IMAGE[:TAG]}] [--increment] DIR@VERSION Folder containing package root Kptfile. Default is current directory. Optional @VERSION changes tag or branch to push onto. Default is most recently pulled or pushed tag. --origin Name of image to push contents onto, with optional TAG to assign to resulting commit. Default is to use most recently pulled/pushed image. Required if Kptfile does not have an origin. --increment Increase the version by 1 while pushing. Default is to leave the origin's TAG or DIR@VERSION unchanged. The Kptfile's image TAG is also updated to the new value. ``` This command will `tar` the contents of the package into a single image layer, and push it into the OCI repository. For Google Artifact Registry and Google Container Registry, the current `gcloud auth` SSO credentials are used. The simplest form of the command is `kpt pkg push` or `kpt pkg push DIR` which will push the current contents back to the IMAGE:TAG location that was saved when `kpt pkg pull` was run. The synxax `kpt pkg push @VERSION` or `kpt pkg push DIR@VERSION` will push back to the image location it came from, but with a new TAG name or version. Examples are `kpt pkt push @draft` or `kpt pkg push @v4` If the Kptfile was not obtained by `kpt pkg pull` - for example it's a new package from `kpt pkg init` or `kpt pkg get` - then the first `kpt pkg push` will require an `--origin IMAGE:TAG` option to provide the target location. It is only necessary on the first call. Finally, if the IMAGE's TAG value is a valid version number, the `--increment` switch can be used to add 1 to the current value before pushing. In the simplest case a `v1` is changed to `v2`, and `1` is changed to `2`, but any TAG that is a valid semver (with optional leading 'v') will have the smallest part of the number incremented. So `v1.0` becomes `v1.1`, `v1.0.0` becomes `v1.0.1`, and `v4.1.9-alpha` becomes `v4.1.10-alpha` Starting with a simple root package, and an orange variant with root as the upstream: ``` -- root \-- orange {upstream: root} ``` The purpose of `kpt get` is to create a new leaf node. This is done by creating the initial copy of the new leaf package in a local folder. This has the side-effects of altering the kptfile name, the upstream values to point at the source, and makes appropriate changes to sub-package metadata. As an example, after running `kpt pkg get scheme://repo/root green` and `kpt pkg get scheme://repo/orange blue` the `green` and `blue` local folder packages are appended to the inheritance tree like this: ``` -- root \-- orange {upstream: root} | `-- blue {upstream: orange} working copy in ./blue \-- green {upstream: root} working copy in

``` By comparison, `kpt pkg pull` does not create a new package node or identity - it only extracts a copy of existing package contents to a working directory. In this example, if the user additionally ran `kpt pkg pull scheme://repo/root root` and `kpt pkg pull scheme://repo/orange orange` the overall state would be this: ``` -- root working copy in ./root \-- orange {upstream: root} working copy in ./orange | `-- blue {upstream: orange} working copy in ./blue \-- green {upstream: root} working copy in ./green ``` There are several ways that pull and push could appear as commands. Those two names are very conventional, but alternatives to consider could be: ``` Usage: kpt pkg copy {SOURCE} {DEST} SOURCE Package source location: a local DIR, or `oci://` image, or git repo and path DEST Package destination: a local DIR, or `oci://` image. ``` Puts a copy of the SOURCE package at the DEST location. The package contents would be entirely unchanged by this operation (unlike `kpt pkg get`). To pull from remote image to local folder: ``` kpt pkg copy \ oci://us-docker.pkg.dev/the-project-id/the-repo-name/the-package:v14 \ the-package ``` To push from local folder to new remote image tag: ``` kpt pkg copy \ the-package \ oci://us-docker.pkg.dev/the-project-id/the-repo-name/the-package:v15 ``` To copy a package image from one OCI repo to another: ``` kpt pkg copy \ oci://us-docker.pkg.dev/the-project-id/dev-blueprints/the-package:v25 \ oci://us-docker.pkg.dev/the-project-id/prod-blueprints/the-package:v25 ``` To copy a package from a git location to an OCI repo: ``` kpt pkg copy \ https://github.com/GoogleCloudPlatform/blueprints.git/catalog/gke@main \ oci://us-docker.pkg.dev/the-project-id/gcp-catalog/gke:latest ``` Before kpt packages can be pushed and pulled as OCI images, a suitable repository must be created. Google Artifact Registry and Google Container Registry are both excellent choices. ```shell LOCATION="us" PROJECT_ID="kpt-demo-73823" REPOSITORY_NAME="blueprints" REPOSITORY="${LOCATION}-docker.pkg.dev/${PROJECTID}/${REPOSITORYNAME}" gcloud artifacts repositories create --location="${LOCATION}" --repository-format=docker --project="${PROJECTID}" "${REPOSITORYNAME}" ``` Creating a new package is no different. But when ready to publish, the `kpt pkg push` command is used instead of source control operations. ```shell mkdir hello-world kpt pkg init hello-world --description="A simple blueprint" kpt pkg push hello-world --image=${REPOSITORY}/hello-world:v1 rm -r hello-world ``` Because the package folder was discarded earlier, the `kpt pkg pull` command is used to place the contents of a particular version at a location. These set of commands may be run in `Cloud Build` steps as well, if you are automating the publication of packages as part of a CI/CD process. ```shell kpt pkg pull hello-world --image=${REPOSITORY}/hello-world:v1 kpt pkg get https://github.com/GoogleCloudPlatform/blueprints.git/catalog/bucket hello-world/my-bucket kpt pkg render hello-world kpt pkg push hello-world --image=${REPOSITORY}/hello-world:v2 ``` Similar to container image tags, the package image tags like `:v1` and `:v2` above may be used any number of ways based on your preferred workflows. The tag `:latest` is used by default if all pulls and pushes should read from and overwrite the same location. Semantic tags like `:draft` and environmental tags like `:dev`, `:qa`, and `:prod` may be also used. No matter what tags are used, the image repository and `kpt` cli will treat them as an alphanumeric label. In addition to providing storage for packages, an OCI registry may also be used as a source of upstream images to

The `oci://` prefix on this command is required to ensure ```shell kpt pkg get oci://${REPOSITORY}/hello-world:v1 greetings-planet kpt pkg push greetings-planet --image=${REPOSITORY}/greetings-planet ``` Looking in the `greetings-planet/Kptfile` at this point will show that the `hello-world:v1` image is the `upstream`, and the `upstreamLock` will show exactly the digest that this clone is up-to-date with. ```yaml apiVersion: kpt.dev/v1 kind: Kptfile metadata: name: greetings-planet upstream: type: oci oci: image: us-docker.pkg.dev/kpt-demo-73823/blueprints/hello-world:v1 updateStrategy: resource-merge upstreamLock: type: oci oci: image: us-docker.pkg.dev/kpt-demo-73823/blueprints/hello-world@sha256:1632e00af3fe858c5e3b3f9e75c16e6327449155 ``` Often a folder inside a package is meant to be used as a way to create "more of the same". To use an OCI image subfolder as the source of a subpackage, the path is added in a way that's distinct from the image itself. ```shell kpt pkg get oci://${REPOSITORY}/hello-world//my-bucket:v1 greetings-planet/another-bucket ``` The `greetings-planet` package will now contain both a `greetings-planet/my-bucket` as well as a `greetings-planet/another-bucket` folder. The contents in locations will now both receive changes when the upstream `hello-world/my-bucket` is updated. The value of the upstream image tag is used to `kpt pkg update` to a specific version. This works no matter if the tag appears to look like a version number or not. ```shell kpt pkg update greetings-planet@v2 kpt pkg push greetings-planet --image=${REPOSITORY}/greetings-planet ``` The Kptfile will now show that the `upstream` and `upstreamLock` have both been changed. ```yaml apiVersion: kpt.dev/v1 kind: Kptfile metadata: name: greetings-planet upstream: type: oci oci: image: us-docker.pkg.dev/kpt-demo-73823/blueprints/hello-world:v2 updateStrategy: resource-merge upstreamLock: type: oci oci: image: us-docker.pkg.dev/kpt-demo-73823/blueprints/hello-world@sha256:a6f1ed69c6ab51e2a148f6d4926bccb24c843887 ``` Just like with a git upstream, it is also possible to `kpt pkg update` without providing a different tag or version value. This is similar to pulling from a remote branch where the name of the branch does change but the latest commit on that branch is a different hash. In that case the Kptfile `upstream` tag will not change, but if that tag has been overwritten with new contents then the differences between `upstreamLock` and current contents will be applied to the local copy, and the `upstreamLock` will be changed tot he current digest. Much like you can `kpt pkg update` to a specific commit hash in git, you can update to an exact image digest with OCI. This is an even more precise reference than by a version number because, like git commits, the image digest is based on the package file contents and cannot be forged or altered. ```shell kpt pkg update greetings-planet@sha256:3b42daa41102fa83bce07bd82a72edcd691868d6 ``` The resulting Kptfile in the local folder will look like this. ```yaml apiVersion: kpt.dev/v1 kind: Kptfile metadata: name: greetings-planet upstream: type: oci oci: image: us-docker.pkg.dev/kpt-demo-73823/blueprints/hello-world@sha256:3b42daa41102fa83bce07bd82a72edcd691868d6 updateStrategy: resource-merge upstreamLock: type: oci oci: image: us-docker.pkg.dev/kpt-demo-73823/blueprints/hello-world@sha256:3b42daa41102fa83bce07bd82a72edcd691868d6 ``` Please list any open questions here in the following format: ### \<Question\> Resolution: Please list the resolution if resolved during the design process or specify Not Yet Resolved The protocol and information is the same. It would mainly be a question of how the credentials for the call are provided. It may be possible `kpt` commands that to not process `upstream` structures may not require update to work correctly. `kpt fn` and `kpt live` commands should be tested to see how they behave. If there is an industry precedent or alternative approaches please list them here as well as citing why you decided not to pursue those paths. Links and description of the approach, the pros and cons identified during the design.