Package: cfn-stk
STKCloudFormation doesn't have to suck 😉STK - improved infrastructure management with CloudFormation.About The ProjectSTK provides an opinionated framework for better management of your AWS infrastructure with CloudFormation.Some benefits of usingstkfor CloudFormation management.Encouragescomposition of infrastructure through re-use of small, modular, sharable CloudFormation templatesover monolithic, complex templates. Think "lego" for infrastructure. Put together these Lego blocks in any number of combinations to deliver the infrastructure your business requires.Decouple CloudFormation templates (code) from configuration. It's as good for your infrastructure as it is for your applications. Templates are stored in external Git repositories separate from configuration. Compose from a variety of private and open source template repositories.Reduce change risk through improved visibilityand simpler templates. Use of Jinja2 for templates and use of explicit configuration (directly interpolated variables) over implicit configuration (AWS parameters) means you can see exactly what is going to change.Local installation (Python/Pip)STK is written in Python. Install it from GitHub using Python pip, or use the docker image for your CI/CD pipelines:Note:
Requirespython 3.10+Install STK via pippip install cfn-stkAlternativelypython3 -m pip install cfn-stkDocker images are available fromhttps://hub.docker.com/repository/docker/johnwo/stk.(back to top)Getting startedNote: Quick start: Thecfn initcommand will help you get started by creating a skeleton "configuration project":cfnconfiginitmy-project--repogit@github.com:jwoffindin/stk-templates.gitcdmy-projectNote:initwill configure an S3 bucket to store CloudFormation templates. You must select an AWS profile with currently valid credentials, otherwise the command will fail trying to list available buckets.The GitHub projecthttps://github.com/jwoffindin/stk-templateshas several starter
templates that you can use. For example, to use thevpctemplate to deploy a VPC into
your AWS account:cfnconfigaddvpcthis will create a filevpc.ymlwith starter configuration. You can deploy to AWS with:cfndeployvpcdevTerminologyEnvironment. What “level” are deploying a stack. E.g. development, test, stage, production, etc.Template. A YAML CloudFormation template (with Jinja2 templates, so yes — we're templating templates).Archetype. A common architectural building block that we deploy into each tier. I.e. same business function, but in dev, test, stage etc.Configuration File. A YAML file that defines what CloudFormation stacks are to be deployed. There are two flavours
of configuration file:Top-level configuration files. These define whatenvironmentsa given stack can be deployed - i.e. 1:1 mapping of top-level config files toarchetype.Include files. These are YAML files, mostly same structure as top-level configuration, but exist only to reduce copy
& paste between top-level configuration files. These are stored in anincludes/directory.Stack. A CloudFormation stack. For our purpose, it is an instance of an Archetype - i.e. template + configuration file + tier.CommandsA full list of commands are available by runningstk helpThe most common commands that you would use are listed below. They are invoked ascfn <command> <config-file> <environment>show-config- show configuration that will be passed to a template.show-template- displays resulting CloudFormation template by applying configuration to template.diff- show the proposed changes between locally rendered CloudFormation Template and the currently deployed template. Displays a text diff between current and proposed templatesvalidate- validates that a template is ready to be deployed. Performs validation of configuration and template, as well as submitting the template to AWS CloudFormation APIs for validation.create— deploys a new stack. Will fail if the stack exists already.update- update an existing stackdeploy- an alias for create/update commands ("upsert"). Will update if the stack exists already, otherwise it will create a new stack.Configuration fileA configuration file is simply a YAML file that declares:How we deploy a CloudFormation template into an AWS account.Any configuration required for the template.A minimal template may look like:# $CONFIG_PATH/foo.ymlaws:region:ap-southeast-2cfn_bucket:my-bucket-for-cloudformation-deploymentsenvironments:dev:test:prod:This configuration file will allow us to deploydev,testandprodinstances of a CloudFormation stack from the templatefoo.yamlinto theap-southeast-2region.Do deploy this stackfoo-devintoap-southeast-2using default credentials.stkdeployfoodevCreate and apply a change set to thefoo-teststack in ap-southeast-2 region.stkupdatefootestCloudFormation TemplatesThe CloudFormation templates are YAML (JSON may work, I've not tried it). They are processed throughJinja2(much as is done with Ansible). Any variables declared invars:section of a config file are available within the template.For example:# $CONFIG_PATH/foo.yml...vars:greeting:Hello World!...the corresponding template may look like:# $TEMPLATE_PATH/foo.yamlDescription:|I'd like to say {{ greeting }}, to you allTo avoid confusion between YAML comments and Jinja2 block level operations,stkuses a##for Jinja2. For example, if we want to allowgreetingto be optional, the template would look like:# $TEMPLATE_PATH/foo.yaml## if greeting is definedDescription:|I'd like to say {{ greeting }}, to you all## endifConfigurationConfiguration is generally stored in a different git repository from the actual templates. In the following examples,$CONFIG_DIRis used to represent where configuration files are stored, and$TEMPLATE_DIRwhere the CloudFormation templates live.StructureThe top-level keys that can be declared in a configuration file are:includes- include additional configuration filestemplate- where to find the CloudFormation templateparams- AWS CloudFormation parameter definitionsvars- Configuration variables (Jinja2)refs- Retrieve information from other CloudFormation stackstags- Default tags to apply to the CloudFormation stackaws- Defines how to connect to AWS, what region to deploy etc.environments- Environment-specific overrides.helpers- Supports defining Python helpers in templates for more complex logic.core- Adjust some corestksettings.These are explained in more detail below.includesAllows composition and sharing of common configuration to reduce unnecessary duplication.It is a list of YAML files loaded fromincludes/directory. The.yml/.yamlextension is optional.Includes are listed in highest-to-lowest precedence. Settings defined in any include file have a lower precedence than the current file.# sample.yml
includes:
  - a
  - b

# includes/a.yml
vars:
  foo: 'a'

# includes/b.yml
vars:
  foo: 'b'Two common use cases for include files are:Environment specific configuration - e.g. AWS account detailsWhere multiple config files use the sample template with common settings.templateDefines where to stk can find the CloudFormation template. Supports local files, local git repositories and remote git repositories.Templates are typically stored separately to the configuration project so they can be shared between teams.
Well-written templates are re-usable components that can be composed to build your final architecture (again, think Lego blocks).The sources for templates are:Plain filesystem (not git)Ifversionis not set, then the templates are assumed to come from local filesystem - useful
when developing templates or where templates don't need to be reused. In this case, setrootto path.E.g.environments:
  dev:
    template:
      version:
      root: ./templatesDuring development, you can use an environment variable to define the template location:environments:
  dev:
    template:
      version:
      root: {{ environ['TEMPLATE_PATH'] }}Local Git RepositorySimilar to above, but uses current commitHEADin local filesystem:environments:
  dev:
    template:
      version: HEAD
      repo: {{ environ['TEMPLATE_PATH'] }}Remote Git RepositoryRemote repositories are also supported. For example, the following will usemainbranch of a github repository:template:
  version: main
  repo: git@github.com:jwoffindin/stk-templates.gitNote:stkcreates a.template-cachedirectory when run - it's highly recommended that you add this to your.gitignorefile.paramsAWS Parameters that are passed into a stack. Minimize if possible, usevarsinstead.Good use cases forParamsinclude:passing secrets
...(that's about it?)Example:params:
  MySubnetId: sub-0123456789Params values are usually environment-specific. They can be overridden on a per environment basis, as illustrated in the following example:params:
  MySubnetId: sub-0123456789

environments:
  dev:
  test:
  stage:
    params:
      MySubnetId: sub-abcdefIn this case deployments ofdevandtestwill haveMySubnetIdofsub-0123456789(the default), andstagewill besub-abcdefParameters can be automatically JSON-encoded (for non-string values), by settingcore.encode_paramstotrue. This will simplify passing JSON-style data as parameters.For example:core:
  encode_params: true

params:
  someComplexValue: { "key": "value" }varsJinja2 variables/values primarily used to pass configuration to to the CloudFormation Template.vars are the preferred way to manage configuration (as opposed toparams`) - e.g. use template conditions rather than 'native' AWS Template Conditions - they are unwieldy and can make changes hard to reason about.As shown in examples above, vars are used in templates for interpolation, conditions etc. For example:# config filevars:queue_name:"MyQueue"# templateResources:MyQueue:Type:AWS::SQS::QueueProperties:QueueName:"{{queue_name}}"Likeparams,varscan have environment-specific overrides. Following the previous example, the following config file would allow deploying queues of different names todevandtest:# config fileenvironments:dev:vars:queue_name:"MyDevQueue"test:vars:queue_name:"MyTestQueue"Configurationvaluesare processed though Jinja2 (although the file itself isnota Jinja template). Many
patterns are stolen from Ansible, so some concepts may be familiar if you've used Ansible.For example:# $CONFIG_PATH/foo.ymlaws:cfn_bucket:my-bucket-for-{{ environment }}-cloudformation-deploymentsDeclares a different bucket per tier.Similarly:vars:hello:"hello,{{who|default('world')}}!"environments:dev:test:vars:who:meIn this example, deploying to dev,hellowill have the valuehello, world!, but the value in test will behello, me!.refs- Referencing other stack outputsUserefs:to retrieve Outputs from other stacks and feeding them as inputs into your stack.Refs is a map/dict of names and options. Refs are available to templates and configuration.For example, if you have a VPC stack deployed from which you need to retrieve subnet IDs, which are exposed asSubnetsoutput, you can do something like:refs:vpc:vars:subnets:"{{refs.vpc.Subnets}}"will query theSubnetsoutput from the vpc stack and pass in a list of subnets to the template as a var.Optional referencesSome stacks may be optional. A referenced output will return nil/blank value
if the stack doesn't exist.For example:refs:an_optional_stack:optional:truevars:some_optional_var:"{{refs.an_optional_stack.SomeOutput}}"in this case,some_optional_varwill be set iff the referenced stack exists, otherwise it is set tonil.Overriding stack namesNormally, external stack will found using standardstknaming convention ($environment-$stack_name).You can override this when needing to reference a stack that does not
follow this convernsion using thestack_nameattribute.Like allrefs:settings, you can set this on a per-environment basis if required:environments:prod:vars:foo:"{{refs.some_stack.OtherOutput}}"refs:some_stack:stack_name:"some-weirdly-named-stack"tagsTags that are applied to the stack (and thus resources within the stack)awsInformation about the AWS account being deployed into. At a minimum needs region and name of the s3 bucket for uploading deployment artifacts.environmentsIf deploying into multiple environments, any environment-specific configuration goes here.
You need to declare at least one environment in a top-level configuration file.Each sub-key ofenvironmentsis the environment name.Within a configuration file, environment-specific values take precedence over top-level configuration.For example:vars:
  a: 'foo'

environments:
  dev:
    vars:
      a: 'bar'
  test:
  prod:In this case, a dev deploymentahas valuebar, and test/prod deploymentsawill value the valuefoo.The following sections can have environment-specific overrides:varsparamsawsrefstagsFor example:...
environments:
  dev:
    tags:
      Owner: "mary.jane@acme.com"
  test:
    tags:
      Owner: "bob.smith@acme.com"
tags:
  Environment: "{{ environment }}"Valid deployment environmentsWithin an top-level config file,environmentsdefines the allowable deployment
targets. Environments defined in include files are not used for this purpose.In the following example,acan only be deployed asdevandprod, whereasbsupportsdev,testandprodas deployment environments:# a.yml
includes:
  - common
environments:
  dev:
  prod:

# b.yml
includes:
  - common
environments:
  dev:
  test:
  prod:

# includes/common.yml
environments:
  dev:
  test:
  prod:helpersCustom helper functions may be used by templates. Since we're injecting code into our runtime, the configuration file must explicitly declare any helpers here.coreConfiguration that changes behavior of the 'stk' application rather configuration/template deployment.Configuration HierarchyObviously, providing deploying the same template into dev, test, and production is not overly useful. We need to be
able to deploy environment-specific configuration to each stack.Under each environment (e.g.environments->dev), we can apply almost all the top-level configuration items to
override the defaults.For example:vars:foo:fizenvironments:dev:vars:foo:fuztest:prod:In this example,fooisfuzfor development deployments, andfizfor test and prod.Most configuration options can have environment specific overrides.For example:environments:dev:aws:region:ap-southeast-2template:version:# use version from working directorytest:aws:region:us-east-1template:version:main(back to top)Configuration Detail(back to top)Environment VariablesNamePurposeCONFIG_PATHDefault path to configuration files, override with--config-pathTEMPLATE_PATHDefault path to templates, usually overridden by config (template.*) or--template-pathcli argumentTEMPLATE_CACHEOverride the path to template cache (local copy of template project for git projects only)LoggingBy default internal logs are sent to syslog with WARN.The following environment variables can be used to change this behaviourEnvironment variablePurposeExampleCFN_LOG_LEVELChange log levelDEBUG,INFO,ERRORLOG_LEVELFallback log level. Also used by boto, so will change boto logging tooINFO,WARNLOG_FILEDirect logging to this file rather than syslog./cfn.logExit codesExit codeReason-1General failure-2Could not generate change set-9No change to applyRoadmapSee theopen issuesfor a full list of proposed features (and known issues).(back to top)Alternativesstkis rather opinionated about managing CloudFormation. If this approach doesn't work for you, check out these alternatives:Sceptre- a more mature project that takes a similar, less opinionated, approach to the problem. Supports stack sets. Only supports local (to the filesystem) templates.Sceptre is a tool to drive AWS CloudFormation. It automates the mundane, repetitive and error-prone tasks, enabling you to concentrate on building better infrastructure.ContributingContributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make aregreatly appreciated.If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement".
Don't forget to give the project a star! Thanks again!Fork the ProjectCreate your Feature Branch (git checkout -b feature/AmazingFeature)Commit your Changes (git commit -m 'Add some AmazingFeature')Push to the Branch (git push origin feature/AmazingFeature)Open a Pull Request(back to top)LicenseDistributed under the MIT License. SeeLICENSE.txtfor more information.(back to top)ContactJohn Woffindin -@john_woffindin-j.woffindin@gmail.comProject Link:https://github.com/jwoffindin/stk(back to top)AcknowledgmentsTODO(back to top)