Project: kpt (Provisioning / Automation & Configuration)
File: 01-live-invenstory-to-rg.md

Author(s): Yuwen Ma <yuwenma@google.com> Contributor(s): Sean Sullivan <seans@google.com> Sunil Arora <sunilarora@google.com> Mortent Torkildsen <mortent@google.com> Mengqi Yu <mengqiy@google.com> Natasha Sarkar <natashasarkar@google.com> Approver: Sunil Arora <sunilarora@google.com> This design is aimed at making `kpt live` be hydration-tool-neutral so that `kpt live` applicable use cases can be expanded to `kustomize build`, `helm templates` and raw kubernetes manifests. This doc is not aimed at designing a unified hydration between kpt and kustomize This doc is not aimed at changing the kpt in-place hydration models. We propose to make `kpt live` support reading configurations from standard input, detach the Inventory from Kptfile and simplify the Inventory-ResourceGroup mechanism to use ResourceGroup only. We will change the existing `kpt live apply` from STDIN to not require the existence of Kptfile ( to be more specific, the `inventory` file from Kptfile). As long as the STDIN contains one and only one valid ResourceGroup, `kpt live apply` should be able to create/match the ResourceGroup in cluster. `kpt live init` will create a ResourceGroup CR in resourcegroup.yaml. By default, the ResoureGroup will have name and namespace assigned as below. And users can override via existing flags "--name" and "--namespace". metadata.name: A client-provided valid RFC 1123 DNS subdomain name. Default value has prefix inventory- with 8 digit numbers. E.g. "inventory-02014045" metadata.namespace: a valid namespace. default to value "default" If users want to reuse an existing inventory (from Kptfile) or ResourceGroup (which has been deployed to the cluster), they shall provide the value of the inventory's inventory-id or the ResourceGroup's "metadata.labels[0].cli-utils.sigs.k8s.io/inventory-id" via "--inventory-id" flag. The client-side Inventory is considered as the identifier of the cluster-side ResourceGroup CR. Right now, the Inventory is stored in the Kptfile to guarantee the ResourceGroup is unique. This requires Kptfile to exist to use `kpt live apply`. To split the Inventory from Kptfile to resourcegroup.yaml and convert the Inventory to ResourceGroup, `kpt live migrate` should be extended to map the inventory.name, inventory.namespace, inventory.inventoryID to ResourceGroup CR metadata.name, metadata.namespace, metadata.labels.cli-utils.sigs.k8s.io/inventory-id correspondingly. Inventory in Kptfile ```yaml apiVersion: kpt.dev/v1 kind: Kptfile inventory: name: <INVENTORY_NAME> namespace: <INVENTORY_NAMESPACE> inventoryID: <INVENTORY_ID> ``` resoucegroup.yaml ```yaml apiVersion: kpt.dev/v1alpha1 kind: ResourceGroup metadata: name: <INVENTORY_NAME> namespace: <INVENTORY_NAMESPACE> labels: cli-utils.sigs.k8s.io/inventory-id: <INVENTORY_ID> ``` Current inventory contains inventory-id which is required to match the label `cli-utils.sigs.k8s.io/inventory-id`. For new users, they should no longer need to be exposed to the inventory-id, but kpt will build one composed by `name-namespace` on the fly. For existing users, the inventory-id is still required in the standalone ResourceGroup file to guarantee the adoption matches, unless they use "--inventory-policy=adopt" to override the label. This flag is only required as a one-off via `kpt live apply -` to override the label to `name-namespace`. `kpt live apply [--rg-file] -` from STDIN accepts and only accepts a single ResourceGroup, including the ResourceGroup provided by the flag. It detects If more than one ResourceGroup is found, raise errors and display all the ResourceGroup

If no ResourceGroup is found in STDIN and and Kptfile inventory does not exists, raise errors and suggest users to run kpt live init If no ResourceGroup is found in STDIN and Kptfile inventory exists, raise errors and suggest users to run kpt live migrate description: The file path to the ResourceGroup CR, default to `resourcegroup.yaml` short form `--rg` This flag will be added to `kpt live init`, `kpt live migrate` and `kpt live apply` description: The name for the ResourceGroup This flag will continue to be used by `kpt live init`. Rather than overriding the inventory.name in Kptfile, it will override the default metadata.name in the standalone ResourceGroup file. description: The namespace for the ResourceGroup This flag will continue to be used by `kpt live init`, Rather than overriding the inventory.namespace in Kptfile, it will override the default metadata.namespace in the standalone ResourceGroup file. description: Inventory identifier. This is used to detect overlap between two sets of ResourceGroup managed resources that might use the same name and namespace. This flag will continue to be accepted by `kpt live init` for backward compatibility reasons. If given, ResourceGroup will store the inventory-id value in "metadata.labels[0].cli-utils.sigs.k8s.io/inventory-id" of the ResourceGroup. If not given, the ResourceGroup labels will be empty and the value of "<name>-<namespace>" will be used as the "cli-utils.sigs.k8s.io/inventory-id" label in `kpt live apply` from STD. All `kpt fn` commands should treat `resoucegroup.yaml` as a special resource and not modify it in any scenario. It should not be considered as a meta resource and `include-meta-resources` flag should not include the `ResourceGroup` resource in the `ResourceList`. In the future, we can add a new flag `--include-rg` if there are valid use-cases to modify or include `ResourceGroup` resource in `ResourceList`. User can run `kpt live init [--rg-file=resourcegroup.yaml]` to create a ResourceGroup object and store it in a resourcegroup.yaml file. Users can customize the file path with the flag --rg-file. Users run `kpt live migrate [--rg-file=resourcegroup.yaml]` to convert the Inventory object from Kptfile to a standalone resourcegroup.yaml file. Users can customize the file path with the flag --rg-file. If the ResourceGroup is expected to be shared in the Gitops workflow, users can add the resourcegroup.yaml file path to the .resources field in kustomization.yaml. This simplifies the Day N deployment by omitting the rg-file flag. Users can configure the hydration rules in kustomization.yaml Users can run `kustomize build <DIR> | kpt live apply -` to hydrate and deploy the kustomize-managed configurations. If resourcegroup.yaml is not added to kustomize .resources field, users should provide the rg-file flag. `kustomize build <DIR> | kpt live apply rg-file <resourcegroup.yaml> -` User can run `kpt live init --rg-file=<DIR>/resourcegroup.yaml` to create a ResourceGroup object and store it in the helm template <DIR>. Users run `kpt live migrate --rg-file=<DIR>/resourcegroup.yaml` to convert the Inventory object from Kptfile to a standalone resourcegroup.yaml file. Users can run `helm templates <DIR> | kpt live apply -` to hydrate and deploy the helm-managed resources. See kpt issue #2399 for expected Inventory usage in package publisher/consumer. Kpt still supports inventory in Kptfile and it is not required to migrate to the standalone rg-path. In fact, users who do not use STDIN in `kpt live apply` will still have inventory read from Kptfile by default unless the inventory-path flag is

We originally propose to use a single command kpt live apply -k (similar idea as kubectl apply -k ) to kustomize-hydrate and kpt-deploy the resources, storing the inventory in the annotation fields of the kustomization.yaml (See below). The inventory can be auto added to kustomization.yaml by kpt live init ```yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization metadata: annotations: kpt-inventory-id: kpt-inventory-name: kpt-inventory-namespace: ``` Extreme simple user steps. Kustomize users only need to install kpt and run kpt live init. Then they can start using kpt live apply to deploy their kustomize resources. No manual changes to any configurations. Easy to understand. The command is similar to kubectl apply -k which makes it easier to be accepted and remembered. There are two main reasons to abandon this approach: Making Kpt a hydration-neutral deployment tool helps it better hook up with different hydration tools. kpt is not planning to treat kustomize differently and thus it does not make sense to provide kustomize a shortcut. Store the Kptfile inventory to a standalone Inventory object. Since the inventory is the metadata of the cluster side ResourceGroup, hiding the ResourceGroup behind inventory provides users simpler syntax and mitigates the probability of overriding the ResourceGroup unintentionally (e.g. via kubectl apply). inventory.yaml ```yaml apiVersion: kpt.dev/v1 kind: Inventory metadata: name: <INVENTORY_NAME> namespace: <INVENTORY_NAMESPACE> inventoryID: <INVENTORY_ID> ``` Simpler resource syntax Mitigate unintentional resourceGroup override. The client-side inventory requires users to manage. This new resource increases the overall complexity of the kpt actuation config. We frequently receive user confusions about the difference between ResourceGroup and Inventory and why they cannot configure inventory like other kubernetes resource (adoption mismatch errors). inventory.yaml ```yaml apiVersion: kpt.dev/v1 kind: Inventory spec: resourceGroup: name: <INVENTORY_NAME> Namespace: <INVENTORY_NAMESPACE> labels: cli-utils.sigs.k8s.io/inventory-id: <INVENTORY_ID> ``` Obey the kubernetes convention and make it clearer that users are expected to provide the resourceGroup kind with name, namespace and labels. Mitigate unintentional resourceGroup override. More complex syntax than ResourceGroup or Inventory metadata solution Inventory is a used to store the . It is stored in Kptfile .Inventory and required by `kpt live apply`. In lieu of a user configuration, it is more of a data for kpt to identify the corresponding ResourceGroup for pruning. This adds non-trivial overheads for users to understand, maintain and retrieve the inventory. The alternative proposal is to remove the inventory but use a new controller to prune the resource. The controller will be installed together with ResourceGroup CRD (in each `kpt live apply`). It will find the previous ResourceGroup CR from current resources, three-way merging and deploying the new resources and update the ResourceGroup. Improve the user experience on using kpt live and minimize the confusions around inventory and ResourceGroup Eliminate the pain points on retrieving the missing Inventory (list all ResourceGroup, check the .spec.resources and try one by one until not encounter the missing adoption error). The controller needs multiple requests to find out the previous ResourceGroup CR. This causes heavy traffic in a production cluster. The controller cannot 100% guarantee the resourceGroup reverse lookup are accurate. Since the pruning happens on the cluster side, kpt can no longer provide real-time status updates.