Package: k8s-handle
k8s-handleEasy CI/CD for Kubernetes clusters with python and jinja2k8s-handle is a command line tool that facilitates continuous delivery for Kubernetes applications.
Also k8s-handle supports environments, so you can use same deployment templates for different environments likestagingandproduction.
k8s-handle is a helm alternative, but without package managerTable of contentsFeaturesBefore you beginInstallation with pipUsage with dockerUsage with CI/CD toolsUsageExampleDocsConfiguration structureEnvironmentsCommon sectionAny other sectionsDeploy specific environmentTemplatesVariablesMerging with commonLoad variables from environmentLoad variables from yaml fileHow to use in CI/CDGitlab CINative integrationThrough variablesWorking modesSync modeStrict modeDestroyDiffOperating without config.yamlRenderApplyDeleteFeaturesEasy to use command line interfaceConfigure any variables in one configuration file (config.yaml)Templating for kubernetes resource files (jinja2) with includes, loops, if-else and so on.Loading variables from environmentIncludes for configuration (includes in config.yaml) for big deploysAsync and sync mode for deploy (wait for deployment, statefulset, daemonset ready)Strict mode, stop deploy if any warning appearEasy integration with CI pipeline (gitlab ci for example)Ability to destroy resources (deploy and destroy from git branches, gitlab environments)Before you beginSetup Kubernetes clusterhttps://kubernetes.io/docs/setup/, or use any predefinedInstallkubectlif you don't have ithttps://kubernetes.io/docs/tasks/tools/install-kubectl/Create kubeconfig(~/.kube/config) or skip if you already have one$cat>~/.kube/kubernetes.ca.crt<< EOF> <paste your cluster CA here>>EOFcat>~/.kube/config<< EOFapiVersion: v1kind: Configpreferences: {}clusters:- cluster:certificate-authority: kubernetes.ca.crtserver: < protocol://masterurl:port >name: my-clustercontexts:- context:cluster: my-clusternamespace: my-namespaceuser: my-username: my-contextcurrent-context: my-contextusers:- name: my-useruser:token: <your token>EOFInstallation with pipRequired python 3.4 or higher$ pip install k8s-handle
 -- or --
$ pip install --user k8s-handleUsage with docker$cd$WORKDIR$gitclonehttps://github.com/2gis/k8s-handle-example.git
$cdk8s-handle-example
$dockerrun--rm-v$(pwd):/tmp/-v"$HOME/.kube:/root/.kube"2gis/k8s-handlek8s-handledeploy-sstaging--use-kubeconfig
INFO:templating:File"/tmp/k8s-handle/configmap.yaml"successfullygenerated
INFO:templating:Tryingtogeneratefilefromtemplate"secret.yaml.j2"in"/tmp/k8s-handle"INFO:templating:File"/tmp/k8s-handle/secret.yaml"successfullygenerated
INFO:templating:Tryingtogeneratefilefromtemplate"deployment.yaml.j2"in"/tmp/k8s-handle"INFO:templating:File"/tmp/k8s-handle/deployment.yaml"successfullygenerated
INFO:k8s.resource:ConfigMap"k8s-starter-kit-nginx-conf"alreadyexists,replaceit
INFO:k8s.resource:Secret"k8s-starter-kit-secret"alreadyexists,replaceit
INFO:k8s.resource:Deployment"k8s-starter-kit"doesnotexist,createit_(_)_wWWWw_@@@@(_)@(_)vVVVv_@@@@(___)_(_)_@@()@@wWWWw(_)\(___)_(_)_@@()@@Y(_)@(_)@@@@(___)`|/Y(_)@(_)@@@@\|/(_)/Y\|\|//(_)\||/|\|\|/|/\|/\|/|/\|\|/\|//\|///\|//\|///\|///\|//|//\|//^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^Usage with CI/CD toolsIf you are using Gitlab CI, TeamCity or something else, you can use docker runner/agent, script will be slightly different:$k8s-handledeploy-sstagingConfigure checkout forhttps://github.com/2gis/k8s-handle-example.gitand specific branchwithout-kubeconfigAlso you need to setup next env vars:K8S_NAMESPACEK8S_MASTER_URIK8S_CA_BASE64 (optional)K8S_TOKENuse image 2gis/k8s-handle:Notice: If you use Gitlab CI, you can configureKubernetes integrationand just use--use-kubeconfigflag.Usage$k8s-handledeploy-sstaging--use-kubeconfig
INFO:templating:Tryingtogeneratefilefromtemplate"configmap.yaml.j2"in"/tmp/k8s-handle"INFO:templating:File"/tmp/k8s-handle/configmap.yaml"successfullygenerated
INFO:templating:Tryingtogeneratefilefromtemplate"secret.yaml.j2"in"/tmp/k8s-handle"INFO:templating:File"/tmp/k8s-handle/secret.yaml"successfullygenerated
INFO:templating:Tryingtogeneratefilefromtemplate"deployment.yaml.j2"in"/tmp/k8s-handle"INFO:templating:File"/tmp/k8s-handle/deployment.yaml"successfullygenerated
INFO:k8s.resource:ConfigMap"k8s-starter-kit-nginx-conf"alreadyexists,replaceit
INFO:k8s.resource:Secret"k8s-starter-kit-secret"alreadyexists,replaceit
INFO:k8s.resource:Deployment"k8s-starter-kit"doesnotexist,createit_(_)_wWWWw_@@@@(_)@(_)vVVVv_@@@@(___)_(_)_@@()@@wWWWw(_)\(___)_(_)_@@()@@Y(_)@(_)@@@@(___)`|/Y(_)@(_)@@@@\|/(_)/Y\|\|//(_)\||/|\|\|/|/\|/\|/|/\|\|/\|//\|///\|//\|///\|///\|//|//\|//^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
$kubectlgetconfigmapNAMEDATAAGE
k8s-starter-kit-nginx-conf11m
$kubectlgetsecret|grepstarter-kit
k8s-starter-kit-secretOpaque11m
$kubectlgetdeploy
NAMEDESIREDCURRENTUP-TO-DATEAVAILABLEAGE
k8s-starter-kit11113mNow set replicas_count in config.yaml to 3, and run again in sync mode$k8s-handledeploy-sstaging--use-kubeconfig--sync-mode
...
INFO:k8s.resource:Deployment"k8s-starter-kit"alreadyexists,replaceit
INFO:k8s.resource:desiredReplicas=3,updatedReplicas=3,availableReplicas=1INFO:k8s.resource:Deploymentnotcompletedon1attempt,nextattemptin5sec.
INFO:k8s.resource:desiredReplicas=3,updatedReplicas=3,availableReplicas=2INFO:k8s.resource:Deploymentnotcompletedon2attempt,nextattemptin5sec.
INFO:k8s.resource:desiredReplicas=3,updatedReplicas=3,availableReplicas=3INFO:k8s.resource:Deploymentcompletedon3attempt
...
$kubectlgetdeploy
NAMEDESIREDCURRENTUP-TO-DATEAVAILABLEAGE
k8s-starter-kit33337mExampleYou can start by examplehttps://github.com/2gis/k8s-handle-example. There are nginx with index.html and all needed kubernetes resources for deploy them.$cd$WORKDIR$gitclonehttps://github.com/2gis/k8s-handle-example.git
$cdk8s-handle-example
$k8s-handledeploy-sstaging--use-kubeconfig--sync-mode
INFO:__main__:Usingdefaultnamespacek8s-handle-test
INFO:templating:Tryingtogeneratefilefromtemplate"configmap.yaml.j2"in"/tmp/k8s-handle"INFO:templating:File"/tmp/k8s-handle/configmap.yaml"successfullygenerated
INFO:templating:Tryingtogeneratefilefromtemplate"deployment.yaml.j2"in"/tmp/k8s-handle"INFO:templating:File"/tmp/k8s-handle/deployment.yaml"successfullygenerated
INFO:templating:Tryingtogeneratefilefromtemplate"service.yaml.j2"in"/tmp/k8s-handle"INFO:templating:File"/tmp/k8s-handle/service.yaml"successfullygenerated
INFO:k8s.resource:ConfigMap"example-nginx-conf"doesnotexist,createit
INFO:k8s.resource:Deployment"example"doesnotexist,createit
INFO:k8s.resource:desiredReplicas=1,updatedReplicas=1,availableReplicas=None
INFO:k8s.resource:Deploymentnotcompletedon1attempt,nextattemptin5sec.
INFO:k8s.resource:desiredReplicas=1,updatedReplicas=1,availableReplicas=None
INFO:k8s.resource:Deploymentnotcompletedon2attempt,nextattemptin5sec.
INFO:k8s.resource:desiredReplicas=1,updatedReplicas=1,availableReplicas=1INFO:k8s.resource:Deploymentcompletedon3attempt
INFO:k8s.resource:Service"example"doesnotexist,createit
$kubectl-nk8s-handle-testgetsvcNAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGE
exampleNodePort10.100.132.168<none>80:31153/TCP52s
$curlhttp://<anynode>:31153
<h1>Helloworld!</h1>
Deployedwithk8s-handle.DocsConfiguration structurek8s-handle works with 2 components:config.yaml (or any other yaml file through -c argument) that stores all configuration for deploytemplates catalog, where your can store all required templates for kubernetes resource files (can be changed through
TEMPLATES_DIR env var)EnvironmentsIf your have testing, staging, production-zone-1, production-zone-2, etc, you can easily cover all environments with
one set of templates for your application without duplication.Common sectionIn the common section you can specify variables that you want to combine with the variables of the selected section:common:app_name:my-shiny-appapp_port:8080Both of these example variables will be added to variables of the selected section.
Common section is optional and can be omitted.Any other sectionsLet's specify testing environmenttesting:replicas:1request_cpu:100mrequest_memory:128Msome_option:disabledIn testing in most cases we don't want performance from our application so we can keep 1 replica and small
amount of resources for it. Also you can set some options to disabled state, in case when you don't want to affect
any integrated systems during testing during testing.staging:replicas:2request_cpu:200mrequest_memory:512MSome teams use staging for integration and demo, so we can increase replicas and resources for our service.production-zone-1:replicas:50request_cpu:1000mrequest_memory:1Gproduction:"true"never_give_up:"true"In production we need to process n thousand RPS, so set replicas to 50, increase resources and set all production
variables to ready for anything values.Deploy specific environmentIn your CI/CD script you can deploy any environment$k8s-handledeploy-sstaging# Or testing or production-zone-1In Gitlab CI for example you can create manual job for each environmentTemplatesTemplates in k8s-handle use jinja2 syntax and support all standard filters + some specialFilters{{ my_var | b64encode }}- encode value of my_var to base64{{ my_var | b64decode }}- decode value of my_var from base64{{ my_var | hash_sha256 }}- encode value of my_var to sha256sum{{ my_var | to_yaml(flow_style=True, width=99999) }}- Tries to render yaml representation of given variable(flow_style=True - render in one line, False multiline. width - max line width for rendered yaml lines)Warning: You can use filters only for templates and can't for config.yamlFunctions{{ include_file('my_file.txt') }}- include my_file.txt to resulting resource w/o parsing it, useful for include configs to configmap.
my_file.txt will be searched in parent directory of templates dir(most of the time - k8s-handle project dir):$ls-1
config.yaml
templates
my_file.txt
...{{ list_files('dir/or/glob*') }}- returns list of files in specified directory. Useful for including all files in folder to configmap. You specify directory path relative to parent of templates folder.Note, both fuctions support unix glob. You can import all files from directoryconf.d/*.conffor example.You can put *.j2 templates in 'templates' directory and specify it in config.yamltesting:replicas:1request_cpu:100mrequest_memory:128Msome_option:disabledtemplates:-template:my-deployment.yaml.j2the same template you can use in each section you want:staging:...templates:-template:my-deployment.yaml.j2production-zone-1:...templates:-template:my-deployment.yaml.j2You can use regular expressions (not glob) for templates selection in TEMPLATES_DIR:cluster-1:...templates:-template:dir-1/.*# All files at TEMPLATES_DIR/dir-1 will be recognised as template and renderedTemplate loader pathk8s-handle uses jinja2 template engine and initializes it with base folder specified in the TEMPLATES_DIR env variable.
Jinja environment considers template paths as specified relatively to its base init directory.Therefore, usersmustspecify paths in{% include %}(and other) blocks relatively to the base (TEMPLATES_DIR) folder, not relative to the importer template location.ExampleWe have the following templates dir content layout:templates /
     subdirectory /
         template_A.yaml
         template_B.yamlIn that scheme, if template_A contains jinja2 import of the template_B, that import statement must be{% include "subdirectory/template_B.yaml" %}despite that included template lies as the same level as the template where include is used.TagsIf you have a large deployment with many separate parts (for ex. main application and migration job), you can want to deploy them independently. In this case you have two options:Use multiple isolated sections (likeproduction_app,production_migration, etc.)Use one section and tag yours templates. For example:production:templates:-template:my-job.yaml.j2tags:migration-template:my-configmap.yaml.j2tags:['app','config']-template:my-deployment.yaml.j2tags:-app-deployment-template:my-service.yaml.j2tags:"app,service"Since you templates are tagged you can use--tags/--skip-tagskeys to partial deploy. For example, you can delete only a migration job:k8s-handle destroy --section production --tags migrationCommand line keys--tagsand--skip-tagscan be specified multiple times, for ex.:k8s-handle deploy --section production --tags=tag1 --tags=tag2 --tags=tag3GroupsYou can make groups for templates. For example:production:templates:-group:-template:my-configmap.yaml.j2-template:my-deployment.yaml.j2-template:my-service.yaml.j2tags:service-one-group:-template:my-job.yaml.j2It is useful for creating different sets of templates for other environments, or tag a bunch of templates at onceVariablesRequired parametersk8s-handle needs several parameters to be set in order to connect to k8s, such as:K8S master uriK8S CA base64K8S tokenEach of these parameters can be set in various ways in any combination and are applied with the following order
(from highest to lowest precedence):From the command line via corresponding keysFrom the config.yaml section, lowercase, underscore-delimited, e.g.k8s_master_uriFrom environment, uppercase, underscore-delimited, e.gK8S_MASTER_URIIf the --use-kubeconfig flag is used, these explicitly specified parameters are ignored.In addition, theK8S namespaceparameter also must be specified.
k8s-handle uses namespace specified inmetadata: namespaceblock of a resource.
If it is not present, the default namespace is used, which is evaluated in the following
order (from highest to lowest precedence):From the config.yamlk8s_namespacekeyFrom the kubeconfigcurrent-contextfield, if --use-kubeconfig flag is usedFrom the environmentK8S_NAMESPACEvariableIf the namespace is not specified in the resource spec, and the default namespace is also not specified,
this will lead to a provisioning error.The one of the common ways is to specify connection parameters and/or k8s_namespace in thecommonsection of your
config.yaml, but you can do it in another way if necessary.Thus, the k8s-handle provides flexible ways to set the required parameters.Merging with commonAll variables defined incommonwill be merged with deployed section and available as context dict in templates rendering,
for example:common:common_var:common_valuetesting:testing_variable:testing_valueAfter the rendering of this template some-file.txt.j2:common_var = {{ common_var }}
testing_variable = {{ testing_variable }}file some-file.txt will be generated with the following content:common_var = common_value
testing_variable = testing_valueIf the variable is declared both incommonsection and the selected one, the value from the
selected section will be chosen.If the particular variable is a dictionary in both (commonand the selected one) sections, resulting variable
will contain merge of these two dictionaries.Load variables from environmentIf you want to use environment variables in your templates(for docker image tag generated by build for example),
you can use next construction in config.yaml:common:image_version:"{{env='TAG'}}"Load variables from yaml filecommon:test:"{{file='include.yaml'}}"include.yaml:-1-2-3template:{{ test[0] }}
{{ test[1] }}
{{ test[2] }}After rendering you get:1
2
3How to use in CI/CDGitlab CINative integrationUse Gitlab CI integration with Kubernetes (https://docs.gitlab.com/ee/user/project/clusters/index.html#adding-an-existing-kubernetes-cluster)
.gitlab-ci.yaml:deploy:image:2gis/k8s-handle:latestscript:-k8s-handle deploy --section <section_name> --use-kubeconfigThrough variablesAlternatively you can setup Gitlab CI variables:K8S_TOKEN_STAGING = < serviceaccount token for staging >K8S_TOKEN_PRODUCTION = < serviceaccount token for production >Don't forget mark variables as protectedthen add next lines to config.yamlstaging:k8s_master_uri:<kubenetes staging master uri>k8s_token:"{{env='K8S_TOKEN_STAGING'}}"k8s_ca_base64:<kubernetes staging ca>production:k8s_master_uri:<kubenetes production master uri>k8s_token:"{{env='K8S_TOKEN_PRODUCTION'}}"k8s_ca_base64:<kubernetes production ca>Now just run proper gitlab job(without --use-kubeconfig option):deploy:image:2gis/k8s-handle:latestscript:-k8s-handle deploy --section <section_name>Working modesSync modeWorks only with Deployment, Job, StatefulSet and DaemonSetBy default k8s-handle just apply resources to kubernetes and exit. In sync mode k8s-handle wait for resources up and
running$k8s-handledeploy--sectionstaging--sync-mode
...
INFO:k8s.resource:Deployment"k8s-starter-kit"alreadyexists,replaceit
INFO:k8s.resource:desiredReplicas=3,updatedReplicas=3,availableReplicas=1INFO:k8s.resource:Deploymentnotcompletedon1attempt,nextattemptin5sec.
INFO:k8s.resource:desiredReplicas=3,updatedReplicas=3,availableReplicas=2INFO:k8s.resource:Deploymentnotcompletedon2attempt,nextattemptin5sec.
INFO:k8s.resource:desiredReplicas=3,updatedReplicas=3,availableReplicas=3INFO:k8s.resource:Deploymentcompletedon3attempt
...You can specify number of tries before k8s-handle exit with non zero exit code and delay before checks:--tries<tries>(360bydefault)--retry-delay<retry-delayinseconds>(5bydefault)Strict modeIn some cases k8s-handle warn you about ambiguous situations and keep working. With--strictmode k8s-handle warn and exit
with non zero code. For example when some used environment variables is empty.$k8s-handledeploy-sstaging--use-kubeconfig--strict
ERROR:__main__:RuntimeError:Environmentvariable"IMAGE_VERSION"isnotset$echo$?1DestroyIn some cases you need to destroy early created resources(demo env, deploy from git branches, testing etc.), k8s-handle
supportdestroysubcommand for you. Just usedestroyinstead ofdeploy. k8s-handle process destroy as deploy, but
call delete kubernetes api calls instead of create or replace.Sync mode is available for destroy as well.DiffYou can get diff between objects in Kubernetes API and local working copy of configuration.$k8s-handlediff-s<section>--use-kubeconfigSecrets are ignored by security reasonsOperating without config.yamlThe most common way for the most of use cases is to operate with k8s-handle viaconfig.yaml, specifying
connection parameters, targets (sections and tags) and variables in one file. The deploy command that runs after that,
at first will trigger templating process: filling your spec templates with variables, creating resource spec files.
That files become a targets for the provisioner module, which does attempts to create K8S resources.But in some cases, such as the intention to use your own templating engine or, probably, necessity to make specs
beforehand and to deploy them separately and later, there may be a need to divide the process into the separate steps:TemplatingDirect,kubectl apply-like provisioning without config.yaml context.For this reason,k8s-handle render,k8s-handle apply,k8s-handle deletecommands are implemented.Renderrendercommand is purposed for creating specs from templates without their subsequent deployment.Another purpose is to check the generation of the templates: previously, this functionality was achieved by using the--dry-runoptional flag. The support of--dry-runindeployanddestroycommands remains at this time for the
sake of backward compatibility but it'sdiscouragedfor the further usage.Just like withdeploycommand,-s/--sectionand--tags/--skip-tagstargeting options are provided to make it
handy to render several specs. Connection parameters are not needed to be specified cause no k8s cluster availability
checks are performed.Templates directory path is taken from envTEMPLATES_DIRand equal to 'templates' by default.
Resources generated by this command can be obtained in directory that set inTEMP_DIRenv variable
with default value '/tmp/k8s-handle'. Users that want to preserve generated templates might need to change this default
to avoid loss of the generated resources.TEMP_DIR="/home/custom_dir" k8s-handle render -s staging
2019-02-15 14:44:44 INFO:k8s_handle.templating:Trying to generate file from template "service.yaml.j2" in "/home/custom_dir"
2019-02-15 14:44:44 INFO:k8s_handle.templating:File "/home/custom_dir/service.yaml" successfully generatedApplyapplycommand with the-r/--resourcerequired flag starts the process of provisioning of separate resource
spec to k8s.The value of-rkey is considered as absolute path if it's started with slash. Otherwise, it's considered as
relative path from directory specified inTEMP_DIRenv variable.No config.yaml-like file is required (and not taken into account even if exists). The connection parameters can be set
via--use-kubeconfigmode which is available and the most handy, or via the CLI/env flags and variables.
Options related to output and syncing, like--sync-mode,--triesand--show-logsare available as well.$ k8s-handle apply -r /tmp/k8s-handle/service.yaml --use-kubeconfig
2019-02-15 14:22:58 INFO:k8s_handle:Default namespace "test"
2019-02-15 14:22:58 INFO:k8s_handle.k8s.resource:Using namespace "test"
2019-02-15 14:22:58 INFO:k8s_handle.k8s.resource:Service "k8s-handle-example" does not exist, create itDeletedeletecommand with the-r/--resourcerequired flag acts similarly todestroycommand and does a try to delete
the directly specified resource from k8s if any.$ k8s-handle delete -r service.yaml --use-kubeconfig

2019-02-15 14:24:06 INFO:k8s_handle:Default namespace "test"
2019-02-15 14:24:06 INFO:k8s_handle.k8s.resource:Using namespace "test"
2019-02-15 14:24:06 INFO:k8s_handle.k8s.resource:Trying to delete Service "k8s-handle-example"
2019-02-15 14:24:06 INFO:k8s_handle.k8s.resource:Service "k8s-handle-example" deletedCustom resource definitions and custom resourcesSince version 0.5.5 k8s-handle supports Custom resource definition (CRD) and custom resource (CR) kinds.
If your deployment involves use of such kinds, make sure that CRD was deployed before CR and check correctness of the CRD's scope.