Package: concoursekit
Concourse Kit - For Extreme PipeliningConcourse Kit is a highly opinionated approach to developing, generating, testing, validating, and setting, multiple, domain-specific, concourse pipelines within a single mono-repo.It's primarily aimed at platform or infrastructure teams who maintain, build, and automate against specific platforms such as Cloud Foundry, Kubernetes or manage multiple, large, and complex platforms or workflows.These automation workloads typically share a lot of common data, tasks, and deploy multiple instances of the same pipeline which target specific environments.OpinionsGenerated - Not Declared:Pipelines used to automate entire platforms typically end up in the thousands-of-lines of YAML and can get incredibly repetitive. Templating YAML can become incredibly tricky.  Concourse Kit specifies that pipelines are literally,as codeand are generated into the required pipeline YAML.Python Focused:Pipelines are written as Python 3.9 code, and all tasks are encouraged to use Python as well the reasons being are:Python is more readable than Bash scripting.Python is more testable than Bash scripting.Python is powerful enough to generate any required pipeline configuration.Python is more approachable to a wider audience vs languages such as Go.Mono Repo:Automation which operates on the same domain should be contained within the same repo.  This makes sharing resources, tasks, environment data, and commonly used scripts easily recyclable.ConventionsConcourse Kit Specifies the following Conventions to be followed.Code which generates into a pipeline config should be withinpipelinesCode which tests a pipeline config should be withinpipeline_testsEnvironment data specific to concourse environments should be withinconcourse-environmentsEnvironment data specific to automation targets should be withintarget-environmentsSetting a pipeline is done using thecc_kit.pyscript.RequirementsTo getup and running with Concourse Kit, you will need the following.Python 3.9:You will need Python 3.9+ installed as this cck makes use of the union operator for dictionaries.fly CLI:You will need the fly CLI installed as concourse kit invokes it to validate, and set pipelines for you.An Open Mind:Yes, this is a very unconventional approach to developing and managing pipelines but it might just be crazy enough to work.Installingpip install concoursekitConcoursekit is installed using pip.  This will install the module and make thecckCLI utility available.Once installed you can run the following to initialize a new working directory for concoursekit to manage.cck --initThis will place a.cck.ymlfile in the directory which provides some basic configuration defaults for cck.# The default concourse target to useconcourse_target:concourse# These options are allowed.# fly_default_options:# - non-interactive | interactive# - hide-pipeline | expose-pipeline# - unpause-pipeline | pause-pipeline# When Setting a Pipeline Fly will use these optionsfly_default_options:-interactive-hide-pipeline-unpause-pipeline# The directory to import <pipeline_name>.py files from.pipelines_dir:pipelines# The directory to look for <pipeline_name>_test.py files from.pipelines_test_dir:pipeline_tests# The directory to use for target-environmentstarget_environments_dir:target_environmentscck CLIThecckutility is the entry point into concourse kit. It has three primary commands.cck --gen-pipelineGenerates a pipeline.yml file from a pipeline.py file.cck --set-pipelineInvokes fly to set one or more pipelines.cck --test-pipelineRuns PyTest for one or more pipelines. Useful for  testing pipeline configuration results.Each of the commands above can take the following arguments.cck --set-pipeline --name <pipeline_name> --env <env>Set a specific pipeline for a specific environment.cck --set-pipeline --name <pipeline_name> --allSet a specific pipeline for all target environments it's scoped to.cck --set-pipeline --allSet all pipelines for all target environments each pipeline is scoped to.cck --gen-pipeline --name <pipeline_name> --env <env>Generate a singlepipeline.ymlfile for a specific pipeline for a specific environment.cck --test-pipeline --name <pipeline_name>Run a PyTest test for a specific pipeline.cck --test-pipeline --allRun all PyTest tests for all pipelines.The Set Command can also take a--planflag.cck --set-pipeline --name <pipeline_name> --all  --plancck --set-pipeline --name --all  --planDisplay pipeline names, their source.pyfile, the environments they will target and validates their configurations.EnvironmentsConcourse Kit organizes pipeline automation around the concept of target environments. Target environments contain all the variables and file content specific to one or more target environments.  From the perspective of automation, the automation jobs are always targeting infrastructure, APIs, or resources in a specific environment, hence the nametarget-environments.In order for thecckCLI to function and generate or set a pipeline, you must declare at least one environment directory under thetarget-environmentsdirectory.  An example of how to organize targets may look like such:target-environments
├───common
├───dev
│   ├───eu-dev
│   └───us-dev
├───prod
├───sandbox
└───stageTheconcourse-environmentsdirectory should contain data and file content related to deploying concourse itself and it's environments. This helps isolate automation data from the target data.Generating Pipeline ConfigurationsAll pipeline YAML configurations aregenerated, not declared or templated.  To configure a new pipeline, a newpipelines/<pipeline_name>.pymust be created.Hello WorldThis is the documented Hello World example on the concourse site.---jobs:-name:jobpublic:trueplan:-task:simple-taskconfig:platform:linuximage_resource:type:registry-imagesource:{repository:busybox}run:path:echoargs:["Helloworld!"]To configure this pipeline using cck, a newpipelines/hello_world.pyfile would need to be created and contain the following.  Itmusthave a top-level function defined aspipeline_configwhich returns adict.defpipeline_config():return{"jobs":[{"name":"hello-world-job","plan":[{"task":"hello-world-task","config":{"platform":"linux","image_resource":{"type":"registry-image","source":{"repository":"busybox"}},"run":{"path":"echo","args":["hello world!"]}}}]}]# end jobs}To Generate the pipeline the following is run.  Even though the environment isn't used in this example, the generate command still requires it.>cck--generate-pipeline--namehello_world--envdev
GeneratingPipelinetohello_world.ymlThis will generate the same yaml as shown above.Using EnvironmentsTypically, if you are automating infrastructure or complex platforms, you will have one pipeline which targets one or more environments. For example, if you deploy and manage a Kubernetes cluster, you would want to install it exactly the same for dev, stage, and prod with the only difference being environmental specific values for variables, endpoints, certs etc.If we think about ourpipeline.pyas a function, and thetarget-environmentdata as the input parameters, we can generate an environment specific pipeline per each environment.Given the following target environments.target-environments
  ├───dev
  │   └───vars.yml
  ├───prod
  |   └───vars.yml
  └───stage
      └───vars.ymland within each vars.yml file a environment specific endpoint.# target-environments/dev/vars.ymlapi-endpoint:dev.api.some.target.foocorp.int# target-environments/stage/vars.ymlapi-endpoint:stage.api.some.target.foocorp.int# target-environments/prod/vars.ymlapi-endpoint:prod.api.some.target.foocorp.intWithin thepipelines/example.pyfile our config looks like the below.  Note that Concourse Kit makes use of YAMLmaker to aid with YAML generation. You can read more on how to use YAMLmakerhere.The value for thetarget-environmentis always under the os environment variable:ENVIRONMENT.  When using theENVIRONMENTvariable, we can source in the variables specific for the environment we are generating a configuration for.fromyamlmakerimportSourcesfromyamlmakerimportenvdefpipeline_config():sources=Sources({"vars":"target-environments/"+env("ENVIRONMENT")+"/vars.yml"})return{"jobs":[{"name":"hello-world-job","plan":[{"task":"hello-world-task","config":{"platform":"linux","image_resource":{"type":"registry-image","source":{"repository":"busybox"}},"params":{"API_ENDPOINT":sources.grab("vars","api-endpoint")},"run":{"path":"echo","args":["hello world!"]}}}]}]# end jobs}Generate the pipeline.yml>cck--generate-pipeline--namehello_world--envdev
GeneratingPipelinetohello_world.ymlOur generated config now looks like this, as you can see the dev specific variable is used.jobs:-name:hello-world-jobplan:-task:hello-world-taskconfig:platform:linuximage_resource:type:registry-imagesource:repository:busyboxparams:API_ENDPOINT:dev.api.some.target.foocorp.intrun:path:echoargs:-hello world!The--generatecommand can only generate a config for one environment at a time.  You cannot specify the--allflag when generating a pipeline config.Setting a Single PipelineYou can set a pipeline two ways, the first being the normalflymethod and passing in a generated yaml file as the pipeline config.  The second method is to havecckinvokeflyfor you to set a pipeline for one or more environments.Setup a fly TargetTo havecckset a pipeline first set aflytarget ofconcourselike so:fly-tconcourselogin-chttp://localhost:8080-uexample-pexampleYou can target a different team if you'd like, for this example it will just use themainteam.Setting the PipelineWhen a pipeline is set, it will be generated and immediately set.  The name of the pipeline is the same as thepipeline.pyfile with underscores replaced with dashes.  In addition, thetarget-environmentis prefixed to the name.> cck --set-pipeline --name hello_world --env dev
Generating Pipeline to dev-hello-world.yml
Setting pipeline: dev-hello-world
jobs:
  job hello-world-job has been added:
+ name: hello-world-job
+ plan:
+ - config:
+     image_resource:
+       source:
+         repository: busybox
+       type: registry-image
+     params:
+       API_ENDPOINT: dev.api.some.target.foocorp.int
+     platform: linux
+     run:
+       args:
+       - hello world!
+       path: echo
+   task: hello-world-task

apply configuration? [yN]:Setting a Pipeline for Multiple Environments.Specifying the--allflag when using the--set-pipelinecommand will attempt to set the pipeline for all environments, except for defined within the.cck.ymlunderignore_environments: [].  By default, thecommonenvironment has been added for you.For this to work, everytarget-environmentmust have all the same var yaml files, content files etc. otherwise the pipeline will work for some environments, not others.> cck --set-pipeline --name hello_world --all
Generating Pipeline to dev-hello-world.yml
Setting pipeline: dev-hello-world
jobs:
  job hello-world-job has been added:
+ name: hello-world-job
+ plan:
+ - config:
...
+       - hello world!
+       path: echo
+   task: hello-world-task

apply configuration? [yN]: y
pipeline created!
you can view your pipeline here: http://localhost:8080/teams/main/pipelines/dev-hello-world

the pipeline is currently paused. to unpause, either:
  - run the unpause-pipeline command:
    fly -t concourse unpause-pipeline -p dev-hello-world
  - click play next to the pipeline in the web ui
Generating Pipeline to prod-hello-world.yml
Setting pipeline: prod-hello-world
jobs:
  job hello-world-job has been added:
+ name: hello-world-job
+ plan:
...
+       - hello world!
+       path: echo
+   task: hello-world-task

apply configuration? [yN]: y
pipeline created!
you can view your pipeline here: http://localhost:8080/teams/main/pipelines/prod-hello-world

the pipeline is currently paused. to unpause, either:
  - run the unpause-pipeline command:
    fly -t concourse unpause-pipeline -p prod-hello-world
  - click play next to the pipeline in the web ui
Generating Pipeline to stage-hello-world.yml
Setting pipeline: stage-hello-world
jobs:
  job hello-world-job has been added:
+ name: hello-world-job
+ plan:
...
+       - hello world!
+       path: echo
+   task: hello-world-task

apply configuration? [yN]: y
pipeline created!Setting for All Environments, Except One or More EnvironmentsSometimes you may wish to set all environments, except one or more of them.  To accomplish this, you can negate specific environments by appending the target-environment name with a bang!.> cck --set-pipeline --name hello_world --all  --env !sandboxThis will set for all environments, except the sandbox environment.Scoping a Pipeline to Specific Environments.It's rare that you will have a pipeline which needs to targeteveryenvironment, however, it is fairly common that pipelines will need to target multiplespecificenvironments.To declare whichtarget-environmentsa pipeline should target, you can declare the top-level variable,pipeline_environments, within thepipelines/pipeline.pyfile like so.fromyamlmakerimportSourcesfromyamlmakerimportenvpipeline_environments=["dev","stage"]defpipeline_config():sources=Sources({"vars":"target-environments/"+env("ENVIRONMENT")+"/vars.yml"})return{"jobs":[{"name":"hello-world-job","plan":[{"task":"hello-world-task","config":{"platform":"linux","image_resource":{"type":"registry-image","source":{"repository":"busybox"}},"params":{"API_ENDPOINT":sources.grab("vars","api-endpoint")},"run":{"path":"echo","args":["hello world!"]}}}]}]# end jobs}When you runcckwith the--allflag again, it will only set the pipeline for the environments declared within thepipeline_environmentslist.> cck --set-pipeline --name hello_world --all
Generating Pipeline to dev-hello-world.yml
Setting pipeline: dev-hello-world
no changes to apply
Generating Pipeline to stage-hello-world.yml
Setting pipeline: stage-hello-world
no changes to applyPipeline SuffixSometimes it's handy to label or append a suffix to the end of a pipeline name to give it some context when working with many pipelines.  For example, you may have a few pipelines which deal with operating a platform or a few pipelines which deal with just install or update specific tasks.You can provide a suffix to be applied to the pipeline name by declaring the following top level variable:pipeline_suffix.pipeline_suffix="operations"defpipeline_config():return{"jobs":[{"name":"hello-world-job","plan":[{"task":"hello-world-task","config":{"platform":"linux","image_resource":{"type":"registry-image","source":{"repository":"busybox"}},"run":{"path":"echo","args":["hello world!"]}}}]}]# end jobs}> cck --set-pipeline --name hello_world --env devGenerating Pipeline to dev-hello-world-operations.yml
Setting pipeline: dev-hello-world-operations
jobs:
...
+   task: hello-world-task

apply configuration? [yN]: y
pipeline created!
you can view your pipeline here: http://localhost:8080/teams/main/pipelines/dev-hello-world-operations

the pipeline is currently paused. to unpause, either:
  - run the unpause-pipeline command:
    fly -t concourse unpause-pipeline -p dev-hello-world-operations
  - click play next to the pipeline in the web uiFly OptionsWhen setting pipelines usingflyit's often common to run it with thenon-interactiveoption, or to immediately hide, expose, pause, or unpause a pipeline.  To make this easier and explicate, you can specify a top-level variable namedfly_options = []For example a pipeline would be configured as such, and when it's set it won't confirm with you, it will unpause it, and automatically hide it.fly_options=["non-interactive","unpause-pipeline","hide-pipeline"]defpipeline_config():...> cck --set-pipeline --name foo_mgmt --end dev
Setting pipeline: dev-foo-mgmt
Generating Pipeline to dev-foo-mgmt.yml
no changes to apply
hid 'dev-foo-mgmt'
unpaused 'dev-foo-mgmt'Specifying a Concourse TargetBy default,cckwill use the target specified within the.cck.ymlfile when running theflycommand.  If you want to set your pipeline to a different target, you can use the top-level variableconcourse_target = str.concourse_target="my-cool-team"defpipeline_config():...Setting All PipelinesIn the event you need to re-set ALL your pipelines for ALL environments (perhaps you have to re-deploy concourse, or restore them.)  It's possible to do this with one command.Take the following pipelines and environments.├───concourse-environments
├───pipelines
│   ├───bar_mgmt.py
│   └───foo_mgmt.py
├───pipeline_tests
├───resources
├───target-environments
│   ├───common
│   ├───dev
│   ├───prod
│   └───stage
└───tasksandbar_mgmtis scoped todevandstage,foo_mgmtis scoped todev,stage, andprod.Running the following command will set all pipelines for the environments they are scoped to.cckwill give you a 10 second window to abort the command if you ran it by mistake.> cck --set-pipeline --allSetting all pipelines in 10 seconds... ctl+c to cancel.
Generating Pipeline to dev-bar-mgmt-operations.yml
Setting pipeline: dev-bar-mgmt-operations
...
Generating Pipeline to stage-bar-mgmt-operations.yml
Setting pipeline: stage-bar-mgmt-operations
...
Generating Pipeline to dev-foo-mgmt-install.yml
Setting pipeline: dev-foo-mgmt-install
...
Generating Pipeline to stage-foo-mgmt-install.yml
Setting pipeline: stage-foo-mgmt-install
...
Generating Pipeline to prod-foo-mgmt-install.yml
Setting pipeline: prod-foo-mgmt-install
...> fly -t concourse pipelines
name                        paused  public  last updated
dev-foo-mgmt-install        yes     no      2021-07-04 20:41:13 -0500 CDT
stage-foo-mgmt-install      yes     no      2021-07-04 20:41:14 -0500 CDT
prod-foo-mgmt-install       yes     no      2021-07-04 20:41:16 -0500 CDT
dev-bar-mgmt-operations     yes     no      2021-07-04 20:41:10 -0500 CDT
stage-bar-mgmt-operations   yes     no      2021-07-04 20:41:12 -0500 CDTPlanning PipelinesBefore setting pipelines, it's a wise idea to preview howcckwill name, set, and toggle all the variousflyoptions before actually setting.  In addition, you may want to ensure your configuration is valid from a Concourse perspective, even though it's valid from a Python perspective.Planning a Single Pipeline and Target EnvironmentRunning the following command will display a plan for a single pipeline and environment.> cck --set-pipeline --name foo_mgmt --env dev --planPlanning a Single Pipeline and All Target Environments> cck --set-pipeline --name foo_mgmt --all --planPlanning All Pipelines and All Target EnvironmentsBelow is an example of an invalid configuration and also shows the various colors!> cck --set-pipeline --all --planPython Exceptions During PlanSometimes if you are developing a configuration and validating it, you may run into a Python Exception whichcckis unable to handle.  In this instance,cckwill provide you with just the traceback for thepipeline.pyconfig you are working on, like so, and provide you with theENVIRONMENTit was trying to generate for.Testing PipelinesBecause all of the pipelinescckmanages aregenerated, it might be a good idea to have a companion test which can ensure the pipeline will be generated correctly.Let's say you have a really complicated pipeline which has environment-specific jobs or other conditionals.  You can write a companion test to ensure that when the pipeline config is generated it has right number of jobs etc.  You can write a test as complex or as simple as you want, it's up to you.Testing a Single PipelineLet's write a test forpipelines/foo_mgmt.py.  To do this we create and write a test underpipelines_tests/foo_mgmt_test.py.#pipelines/foo_mgmt.pyfromyamlmakerimportenvfromyamlmakerimportIncludepipeline_suffix="install"pipeline_environments=["dev","stage","prod"]defpipeline_config():return{"jobs":[{"name":"foo-job-"+env("ENVIRONMENT"),"plan":[{"task":"foo-task","config":{"platform":"linux","image_resource":{"type":"registry-image","source":{"repository":"busybox"}},"run":{"path":"echo","args":["for all envs"]}}}]}]+Include.when(env("ENVIRONMENT")=="dev",[{"name":"bar-job-"+env("ENVIRONMENT"),"plan":[{"task":"bar-task","config":{"platform":"linux","image_resource":{"type":"registry-image","source":{"repository":"busybox"}},"run":{"path":"echo","args":["only in dev!"]}}}]}])# end jobs}The following test will verify that when given a specific environment, the correct number of jobs are defined. This is a simple example, but what you want to test for or verify is up to you depending on the complexity of the pipeline being generated.#pipeline_tests/foo_mgmt_test.pyimportosfrompipelines.foo_mgmtimportpipeline_configdeftest_dev_has_two_jobs():os.environ["ENVIRONMENT"]="dev"config=pipeline_config()assertlen(config["jobs"])==2deftest_prod_has_one_job():os.environ["ENVIRONMENT"]="prod"config=pipeline_config()assertlen(config["jobs"])==1Running the test.> cck --test-pipeline --name foo_mgmtTesting Pipeline: foo_mgmt
================================================================================================= test session starts =================================================================================================
platform win32 -- Python 3.9.1, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: C:\Users\Anthony\workspace\concourse-kit
collected 2 items

pipeline_tests\foo_mgmt_test.py ..

======================================================================================================= PASSES ========================================================================================================
=============================================================================================== short test summary info ===============================================================================================
PASSED pipeline_tests/foo_mgmt_test.py::test_dev_has_two_jobs
PASSED pipeline_tests/foo_mgmt_test.py::test_prod_has_one_job
================================================================================================== 2 passed in 0.02s ==================================================================================================Testing all PipelinesIf you want to verify all pipelines are passing their tests, simply run the following> cck --test-pipeline --allTesting All Pipelines
================================================================================================= test session starts =================================================================================================
platform win32 -- Python 3.9.1, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: C:\Users\Anthony\workspace\concourse-kit
collected 3 items

pipeline_tests\bar_mgmt_test.py .
pipeline_tests\foo_mgmt_test.py ..

======================================================================================================= PASSES ========================================================================================================
=============================================================================================== short test summary info ===============================================================================================
PASSED pipeline_tests/bar_mgmt_test.py::test_dev_has_two_jobs
PASSED pipeline_tests/foo_mgmt_test.py::test_dev_has_two_jobs
PASSED pipeline_tests/foo_mgmt_test.py::test_prod_has_one_job
================================================================================================== 3 passed in 0.03s ==================================================================================================