Project: Kapitan (Provisioning / Automation & Configuration)
File: references.md

One of the motivations behing Kapitan's design is that we believe that everything about your setup should be tracked, and Kapitan takes this to the extreme. Sometimes, however, we have to manage values that we do not think they belong to the Inventory: perhaps they are either too variable (for instance, a Git commit sha that changes with every build) or too sensitive, like a password or a generic secret, and then they should always be encrypted. Kapitan has a built in support for References, which you can use to manage both these use cases. Kapitan References supports the following backends: | Backend | Description | Encrypted | |-||| | `plain` | Plain text, (e.g. commit sha) | :material-close: | | `base64` | Base64, non confidential but with base64 encoding | :material-close: | | `gpg` | Support for <https://gnupg.org/> | :material-check: | | `gkms` | GCP KMS | :material-check: | | `awskms` | AWS KMS | :material-check: | | `azkms` | Azure Key Vault | :material-check: | | `env` | Environment | :material-check: | | `vaultkv` | Hashicorp Vault (RO) | :material-check: | | `vaulttransit` | Hashicorp Vault (encrypt, decrypt, updatekey, rotatekey) | :material-check: | Some reference backends require configuration, both in the Inventory and to configure the actual backend. !!! tip "Get started" If you want to get started with references but don't want to deal with the initial setup, you can use the `plain` and `base64` reference types. These are great for demos, but we will see they are extremely helpful even in Production environments. !!! danger Both `plain` and `base64` references do not support encryption: they are intended for development or demo purposes only. DO NOT use `plain` or `base64` for storing sensitive information! !!! info "Backend configuration" Configuration for each backend varies, and it is perfomed by configuring the inventory under `parameters.kapitan.secrets`. === "plain" !!! note "No configuration needed" === "base64" !!! note "No configuration needed" === "gpg" ```yaml parameters: kapitan: secrets: gpg: recipients: name: example@kapitan.dev fingerprint: D9234C61F58BEB3ED8552A57E28DC07A3CBFAE7C ``` === "gkms" ```yaml parameters: kapitan: secrets: gkms: key: 'projects/<project>/locations/<location>/keyRings/<keyRing>/cryptoKeys/<key>' ``` === "awskms" ```yaml parameters: kapitan: secrets: awskms: key: 'alias/nameOfKey' ``` === "azkms" ```yaml parameters: kapitan: secrets: azkms: key: 'https://<keyvault-name>.vault.azure.net/keys/<object-name>/<object-version>' ``` === "env" ```yaml parameters: ... mysql: rootpassword: ?{env:targets/${targetname}/mysql/root_password} ... ``` === "vaultkv" ```yaml parameters: kapitan: secrets: vaultkv: VAULT_ADDR: http://127.0.0.1:8200 auth: token mount: secret ``` === "vaulttransit" ```yaml parameters: kapitan: secrets: vaulttransit: VAULT_ADDR: https://vault.example.com VAULT_TOKEN: s.mqWkI0uB6So0aHH0v0jyDs97 VAULTSKIPVERIFY: "False" # Recommended auth: token mount: mytransit key: 2022-02-13-test ``` !!! tip "Organize your configuration in classes" Just like any other inventory parameters, these configurations can be inherited from a common class or defined per target. !!! example "`inventory/classes/common.yml`" ```yaml classes: security.backend ... ``` !!! example "`inventory/classes/security/backend.yml`" ```yaml parameters: kapitan: secrets: <backend>: <configuration> ``` ??? tip "ADVANCED: Mix-and-Match backends" Remember that you can use multiple backends at the same time, and also use variable interpolation for an even greater flexibility. In a multi-cloud setup, for instance, you could configure both GKMS !!! quote "GCP configuration" !!! example "`inventory/classes/cloud/gcp.yml`" ```yaml classes: security.backends.gkms

``` !!! example "`inventory/classes/security/backends/gkms.yml`" ```yaml parameters: backend: gkms kapitan: secrets: gkms: <configuration> ``` !!! quote "AWS configuration" !!! example "`inventory/classes/security/backends/awskms.yml`" ```yaml parameters: backend: awskms kapitan: secrets: awskms: <configuration> ``` !!! example "`inventory/classes/cloud/aws.yml`" ```yaml classes: security.backends.awskms ... ``` Now because they both set the `parameters.backend` variable, you can define a reference whose backend changes based on what class is assigned to the target !!! example "`inventory/targets/cloud/gcp/acme.yml`" ```yaml classes: cloud.aws parameters: ... mysql: rootpassword: ?{${backend}:targets/${targetname}/mysql/root_password} ... ``` !!! info "" References can be defined in the inventory following the syntax *spaces added for clarity*: `?{` `<backend_id>` `:` `<reference_path>` `}` ??? tip "expand for advanced features" The syntax also supports for process functions and create_functions which we will discuss later, which brings the full syntax to !!! example "" `?{` `<backend_id>` `:` `<reference_path>` `}` |`<process_function>` ||`<create_function>` === "plain" ```yaml parameters: ... mysql: rootpassword: ?{plain:targets/${targetname}/mysql/root_password} ... ``` !!! danger "not encrypted" This reference type does not support encryption: it is intended for non sensitive data only. DO NOT use `plain` for storing sensitive information! === "base64" ```yaml parameters: ... mysql: rootpassword: ?{base64:targets/${targetname}/mysql/root_password} ... ``` !!! danger "not encrypted" This reference type does not support encryption: it is intended for non sensitive data only. DO NOT use `base64` for storing sensitive information! === "gpg" ```yaml parameters: ... mysql: rootpassword: ?{gpg:targets/${targetname}/mysql/root_password} ... ``` === "gkms" ```yaml parameters: ... mysql: rootpassword: ?{gkms:targets/${targetname}/mysql/root_password} ... ``` === "awskms" ```yaml parameters: ... mysql: rootpassword: ?{awskms:targets/${targetname}/mysql/root_password} ... ``` === "azkms" ```yaml parameters: ... mysql: rootpassword: ?{azkms:targets/${targetname}/mysql/root_password} ... ``` === "env" ```yaml parameters: ... mysql: rootpassword: ?{env:targets/${targetname}/mysql/root_password} ... ``` === "vaultkv" read-only ```yaml parameters: ... mysql: rootpassword: ?{vaultkv:targets/${targetname}/mysql/root_password} ... ``` read-write: ```yaml parameters: ... mysql: rootpassword: ?{vaultkv:targets/${targetname}/mysql/root_password:mount:path/in/vault:mykey} ... ``` === "vaulttransit" ```yaml parameters: ... mysql: rootpassword: ?{vaulttransit:targets/${targetname}/mysql/root_password} ... ``` You can assign values to your reference using the command line. Both reading from a file and pipes are supported. !!! warning "Please Note" Kapitan will fail compilation if a reference is not

Please see how to assign a value in the next section !!! info "" === "plain" ```shell kapitan refs --write plain:refs/targets/${TARGETNAME}/mysql/rootpassword -t ${TARGET_NAME} -f <input file> ``` which also works with pipes ```shell cat inputfile | kapitan refs --write plain:refs/targets/${TARGETNAME}/mysql/rootpassword -t ${TARGETNAME} -f - ``` === "base64" ```shell kapitan refs --write base64:refs/targets/${TARGETNAME}/mysql/rootpassword -t ${TARGET_NAME} -f <input file> ``` which also works with pipes ```shell cat inputfile | kapitan refs --write base64:refs/targets/${TARGETNAME}/mysql/rootpassword -t ${TARGETNAME} -f - ``` === "gpg" ```shell kapitan refs --write gpg:refs/targets/${TARGETNAME}/mysql/rootpassword -t ${TARGET_NAME} -f <input file> ``` which also works with pipes ```shell cat inputfile | kapitan refs --write gpg:refs/targets/${TARGETNAME}/mysql/rootpassword -t ${TARGETNAME} -f - ``` === "gkms" ```shell kapitan refs --write gkms:refs/targets/${TARGETNAME}/mysql/rootpassword -t ${TARGET_NAME} -f <input file> ``` which also works with pipes ```shell cat inputfile | kapitan refs --write gkms:refs/targets/${TARGETNAME}/mysql/rootpassword -t ${TARGETNAME} -f - ``` === "awskms" ```shell kapitan refs --write vaulttransit:refs/targets/${TARGETNAME}/mysql/rootpassword -t ${TARGET_NAME} -f <input file> ``` which also works with pipes ```shell cat inputfile | kapitan refs --write vaulttransit:refs/targets/${TARGETNAME}/mysql/rootpassword -t ${TARGETNAME} -f - ``` === "azkms" ```shell kapitan refs --write azkms:refs/targets/${TARGETNAME}/mysql/rootpassword -t ${TARGET_NAME} -f <input file> ``` which also works with pipes ```shell cat inputfile | kapitan refs --write azkms:refs/targets/${TARGETNAME}/mysql/rootpassword -t ${TARGETNAME} -f - ``` === "env" !!! note "Setting default value only" The `env` backend works in a slightly different ways, as it allows you to reference environment variables at runtime. For example, for a reference called `{?env:targets/envs_defaults/mysql_port_${target_name}}`, Kapitan would look for an environment variable called `KAPITAN_ENV_mysql_port_${TARGET_NAME}`. If that variable cannot be found in the Kapitan environment, the default will be taken from the `refs/targets/envs_defaults/mysql_port_${TARGET_NAME}` file instead. ```shell kapitan refs --write env:refs/targets/envsdefaults/mysqlport${TARGETNAME} -t ${TARGET_NAME} -f <input file> ``` which also works with pipes ```shell cat inputfile | kapitan refs --write env:refs/targets/envsdefaults/mysqlport${TARGETNAME} -t ${TARGETNAME} -f - ``` === "vaultkv" ```shell kapitan refs --write vaultkv:refs/targets/${TARGETNAME}/mysql/rootpassword -t ${TARGET_NAME} -f <input file> ``` which also works with pipes ```shell cat inputfile | kapitan refs --write vaultkv:refs/targets/${TARGETNAME}/mysql/rootpassword -t ${TARGETNAME} -f - ``` === "vaulttransit" This backend expects the value to be stored as a `key:value` pair. ```shell echo "akey:avalue" | kapitan refs --write vaulttransit:refs/targets/${TARGETNAME}/mysql/rootpassword -t ${TARGET_NAME} -f - ``` When reading from disk, the input file should be formatted accordingly. Kapitan has built in capabilities to initialise its references on creation, using an elegant combination of primary and secondary functions. This is extremely powerful because it allows for you to make sure they are always initialised with sensible values. To automate the creation of the reference, you can add one of the following primary functions to the reference tag by using the syntax `||primary_function:param1:param2` For instance, to automatically initialise a reference with a *random string* with a lenght of 32 characters, you can use the `random` primary function ```yaml parameters: ... mysql: rootpassword: ?{${backend}:targets/${targetname}/mysql/root_password||random:str:32} ... ``` !!! note "Initialise non existent references" The first operator here `||` is more similar to a *logical OR*. If the reference file does not exist, Kapitan* will use the function to initialise it If the reference file exists, no functions will run. !!! tip "Automate secret rotation with ease" You can take advantage of it to implement easy rotation of secrets. Simply delete the reference files, and run `kapitan compile`: let Kapitan do the rest. === "random" === "str" !!! quote "" Generator function for alphanumeric characters, will be url-token-safe ```yaml ?{${backend}:targets/${targetname}/mysql/rootpassword||random:str} ``` === "int" !!! quote "" generator function for digits (0-9) ```yaml ?{${backend}:targets/${targetname}/mysql/rootpassword||random:int} ``` === "loweralpha" !!! quote "" generator function for lowercase letters (a-z) ```yaml ?{${backend}:targets/${targetname}/mysql/rootpassword||random:loweralpha} ``` === "upperalpha" !!! quote "" generator function for uppercase letters (A-Z) ```yaml ?{${backend}:targets/${targetname}/mysql/rootpassword||random:upperalpha} ``` === "loweralphanum" !!! quote "" generator function for lowercase letters and numbers (a-z and 0-9) ```yaml ?{${backend}:targets/${targetname}/mysql/rootpassword||random:loweralphanum} ``` === "upperalphanum" !!! quote "" generator function for uppercase letters and numbers (A-Z and 0-9) ```yaml ?{${backend}:targets/${targetname}/mysql/rootpassword||random:upperalphanum} ``` === "special" !!! quote "" generator function for alphanumeric characters and given special characters ```yaml ?{${backend}:targets/${targetname}/mysql/rootpassword||random:special} ``` === "private keys" === "rsa" !!! quote "" Generates an RSA 4096 private key (PKCS#8). You can optionally pass the key size ```yaml ?{${backend}:targets/${targetname}/privatekey||rsa} ``` === "ed25519" !!! quote "" Generates a ed25519 private

(PKCS#8) ```yaml ?{${backend}:targets/${targetname}/privatekey||ed25519} ``` === "publickey" !!! quote "" Derives the public key from a revealed private key ```yaml ?{${backend}:targets/${targetname}/privatekey||rsa} ?{${backend}:targets/${targetname}/publickey||reveal:targets/${targetname}/privatekey|publickey} ``` === "rsapublic" !!! quote "" DEPRECATED: use `||publickey` === "basicauth" !!! quote "" Generates a base64 encoded pair of `username:password` ```yaml ?{${backend}:targets/${targetname}/apachebasicauth||basicauth:username:password} ``` === "reveal" !!! quote "" Reveals the content of another reference, useful when deriving public keys or a reference requires a different encoding or the same value. ```yaml ?{${backend}:targets/${target_name}/secret||random:str} ?{${backend}:targets/${targetname}/base64secret||reveal:targets/${target_name}/secret|base64} ``` !!! danger "attention when rotating secrets used with `reveal`" If you use reveal to initialise a reference, like `myreference||reveal:sourcereference` the `myreference` will not be automatically updated if `sourcereference` changes. Please make sure you also re-initialise `my_reference` correctly === "base64" !!! quote "" base64 encodes your reference ```yaml ?{${backend}:targets/${target_name}/secret||random:str|base64} ``` === "sha256" !!! quote "" sha256 hashes your reference `param1`: `salt` ```yaml ?{${backend}:targets/${target_name}/secret||random:str|sha256} ``` You can reveal the secrets referenced in the outputs of `kapitan compile` via: ```shell kapitan refs --reveal -f path/to/rendered/template ``` For example, `compiled/minikube-mysql/manifests/mysql_secret.yml` with the following content: ```yaml apiVersion: v1 data: MYSQLROOTPASSWORD: ?{gpg:targets/minikube-mysql/mysql/password:ec3d54de} MYSQLROOTPASSWORDSHA256: ?{gpg:targets/minikube-mysql/mysql/passwordsha256:122d2732} kind: Secret metadata: annotations: {} labels: name: example-mysql name: example-mysql namespace: minikube-mysql type: Opaque ``` can be revealed as follows: ```shell kapitan refs --reveal -f compiled/minikube-mysql/manifests/mysql_secret.yml ``` This will substitute the referenced secrets with the actual decrypted secrets stored at the referenced paths and display the file content. You can also use: ```shell kapitan refs --reveal --ref-file refs/targets/all-glob/mysql/password ``` or ```shell kapitan refs --reveal --tag "?{base64:targets/all-glob/mysql/password}" kapitan refs --reveal --tag "?{base64:targets/all-glob/mysql/password:3192c15c}" ``` for more convenience. Please refer to the Kapitan is also able to use access specific keys in YAML content by using subvars. For instance given a reference `plain:larder` with content: ```yaml food: apples: 1 ``` I could now have an inventory variable like: ```yaml parameters: numberofapples: ?{plain:larder@food.apple} ``` Subvars can have a very practical use for storing YAML outputs coming straight from other tools. For instance, I could use the GCP `gcloud` command to get all the information about a cluster, and write it into a reference ```shell gcloud container clusters describe \ --project ${TARGET_NAME}-project \ gke-cluster --zone europe-west1 --format yaml \ | kapitan refs --write plain:clusters/${TARGETNAME}/cluster -t ${TARGETNAME} -f - ``` knowing the output of `gcloud` to produce yaml that contain the following values: ```yaml ... name: gke-cluster releaseChannel: channel: REGULAR selfLink: https://container.googleapis.com/v1/projects/kapicorp/locations/europe-west1/clusters/gke-cluster ... ``` I can not reference the link to the cluster in the inventory using: ```yaml parameters: cluster: name: ?{plain:clusters/${target_name}/cluster@name} releasechannel: ?{plain:clusters/${targetname}/cluster@releaseChannel.channel} link: ?{plain:clusters/${target_name}/cluster@selfLink} ``` Combined with a Jinja template, I could write automatically documentation containing the details of the clusters I use. ```text {% set p = inventory.parameters %} Cluster has release channel {{p.cluster.release_channel}} ``` Considering a key-value pair like `mykey`:`mysecret` in the path `secret/foo/bar` in a kv-v2(KV version 2) secret engine on the vault server, to use this as a secret use: ```shell echo "foo/bar:mykey" | kapitan refs --write vaultkv:path/to/secretinsidekapitan -t <targetname> -f - ``` To write a secret in the vault with kapitan use a ref tag with following structure: ```yaml parameters: ... secret: mysecret: ?{vaultkv:targets/${targetname}/mypath:mount:path/in/vault:mykey||<functions>}

``` Leave `mount` empty to use the specified mount from vault params from the inventory (see below). Same applies to the `path/in/vault` where the ref path in kapitan gets taken as default value. Parameters in the secret file are collected from the inventory of the target we gave from CLI `-t <target_name>`. If target isn't provided then kapitan will identify the variables from the environment when revealing secret. Environment variables that can be defined in kapitan inventory are `VAULTADDR`, `VAULTNAMESPACE`, `VAULTSKIPVERIFY`, `VAULTCLIENTCERT`, `VAULTCLIENTKEY`, `VAULTCAPATH` & `VAULTCACERT`. Extra parameters that can be defined in inventory are: `auth`: specify which authentication method to use like `token`,`userpass`,`ldap`,`github` & `approle` `mount`: specify the mount point of key's path. e.g if path=`alpha-secret/foo/bar` then `mount: alpha-secret` (default `secret`) `engine`: secret engine used, either `kv-v2` or `kv` (default `kv-v2`) Environment variables cannot be defined in inventory are `VAULTTOKEN`,`VAULTUSERNAME`,`VAULTPASSWORD`,`VAULTROLEID`,`VAULTSECRET_ID`. ```yaml parameters: kapitan: secrets: vaultkv: auth: userpass engine: kv-v2 mount: team-alpha-secret VAULT_ADDR: http://127.0.0.1:8200 VAULT_NAMESPACE: CICD-alpha VAULTSKIPVERIFY: false VAULTCLIENTKEY: /path/to/key VAULTCLIENTCERT: /path/to/cert ``` Considering a key-value pair like `mykey`:`mysecret` in the path `secret/foo/bar` in a transit secret engine on the vault server, to use this as a secret use: ```shell echo "any.value:whatever-youmay*like" | kapitan refs --write vaulttransit:mytarget/to/secretinsidekapitan -t <target_name> -f - ``` Parameters in the secret file are collected from the inventory of the target we gave from CLI `-t <target_name>`. If target isn't provided then kapitan will identify the variables from the environment when revealing secret. Environment variables that can be defined in kapitan inventory are `VAULTADDR`, `VAULTNAMESPACE`, `VAULTSKIPVERIFY`, `VAULTCLIENTCERT`, `VAULTCLIENTKEY`, `VAULTCAPATH` & `VAULTCACERT`. Extra parameters that can be defined in inventory are: `auth`: specify which authentication method to use like `token`,`userpass`,`ldap`,`github` & `approle` `mount`: specify the mount point of key's path. e.g if path=`my_mount` (default `transit`) `crypto_key`: Name of the `encryption key` defined in vault `alwayslatest`: Always rewrap ciphertext to latest rotated cryptokey version Environment variables cannot be defined in inventory are `VAULTTOKEN`,`VAULTUSERNAME`,`VAULTPASSWORD`,`VAULTROLEID`,`VAULTSECRET_ID`. ```yaml parameters: kapitan: vars: target: my_target namespace: my_namespace secrets: vaulttransit: VAULT_ADDR: http://vault.example.com:8200 VAULT_TOKEN: s.i53a1DL83REM61UxlJKLdQDY VAULTSKIPVERIFY: "True" auth: token mount: transit cryptokey: newkey always_latest: False parameters: target_name: secrets kapitan: secrets: vaulttransit: VAULT_ADDR: http://127.0.0.1:8200 VAULT_TOKEN: s.i53a1DL83REM61UxlJKLdQDY VAULTSKIPVERIFY: "True" auth: token mount: transit crypto_key: key always_latest: False ``` To encrypt secrets using keys stored in Azure's Key Vault, a `key_id` is required to identify an Azure key object uniquely. It should be of the form `https://{keyvault-name}.vault.azure.net/{object-type}/{object-name}/{object-version}`. This is done in the inventory under `parameters.kapitan.secrets`. ```yaml parameters: kapitan: vars: target: ${target_name} namespace: ${target_name} secrets: azkms: key: 'https://<keyvault-name>.vault.azure.net/keys/<object-name>/<object-version>' ``` The key can also be specified using the `--key` flag Secrets can be created using any of the methods described in the "creating your secret" section. For example, if the key is defined in the `prod` target file ```shell echo "myencryptedsecret" | kapitan refs --write azkms:path/to/secretinsidekapitan -t prod -f - ``` Using the `--key` flag and a `key_id` ```shell echo "myencryptedsecret" | kapitan refs --write azkms:path/to/secretinsidekapitan --key=<key_id> -f - ``` Secrets can be referenced and revealed in any of the ways described above. For example, to reveal the secret stored at `path/to/secretinsidekapitan` ```shell kapitan refs --reveal --tag "?{azkms:path/to/secretinsidekapitan}" ``` Note: Cryptographic algorithm used for encryption is rsa-oaep-256.