Package: humilis
humilisHelps you deploy AWS infrastructure withCloudformation.This project is originally based on thecumulusproject. SeeCUMULUS_LICENSEfor license information.InstallationInstall theAWS CLI:pip install awscliConfigure the AWS CLI:aws configurehumiliswill use whatever credentials you introduced when
configuring your AWS CLI installation.You can now install the latest “stable” version ofhumilis:pip install humilisor the development version if you prefer that:pip install git+https://github.com/humilis/humilisAfter installation you need to configure humilis. To configure globally
for your system:humilis configureThe command above will store and read the configuration options from~/.humilis.ini. You can also store the configuration in a.humilis.inifile stored in your current working directory by using:humilis configure --localhumiliswill always read the configuration first from a.humilis.inifile under your current work directory. If it is not
found then it will read it from your system global config file~/.humilis.Development environmentAssuming you havevirtualenvinstalled:make develop

. .env/bin/activateTestingAt the moment, most tests are integration tests with the AWS SDK. This
means that you will need toset up your
systemto access AWS resources if you want to run the test suite.py.test testsQuickstartDefine your infrastructure environment following the examples in theexamples
directory.
Then to create the environment:humilis create examples/humilis-firehose.yamlTo update the environment after it has been deployed:humilis update examples/humilis-firehose.yamlAnd to delete it:humilis delete examples/humilis-firehose.yamlHumilis environmentsAhumilisenvironment is just a collection of cloudformation stacks
that are required for an application. Instead of having a monolytic CF
template for your complete application,humilisallows you to define
infrastructurelayersthat are combined into anenvironment. Eachhumilislayer translates exactly into one CF template (therefore
into one CF stack after the layer is deployed).Breaking a complex infrastructure environment into smaller layers has at
least two obvious advantages:Easier to maintain. It’s easier to maintain a simple layer that
contains just a bunch ofCF
resourcesthan serve a well-defined purpose.Easier to reuse. You should strive to define your infrastructure
layers in such a way that you can reuse them across various
environments. For instance, many projects may require a base layer
that defines a VPC, a few subnets, a gateway and some routing tables,
and maybe a (managed) NAT. You can define a humilis layer with those
resources and have a set of layer parameters (e.g. the VPC CIDR) that
will allow you to easily reuse it across environments.Environment anatomyAn environmentdefinition fileis ayamldocument that specifies
the list of layers that form your enviroment. The file should be named
as your environment. That is, for environmentmy-app-environmentthe
environment description file should be calledmy-app-environment.yaml. The contents of the environment definition
should be organized as follows:---
my-app-environment:
    description:
        A description of what this environment is for
    layers:
        # The layers that you environment requires. They will be deployed in the
        # same order as you list them. Note that you can also pass parameters
        # to a layer (more on that later).
        - {layer: name_of_first_layer, layer_param: layer_value}
        - {layer: name_of_second_layer}
        - {layer: name_of_third_layer}Layer anatomyAnything associated to a given layer must be stored in a directory with
the same name as the layer, within the same directory where the
environmentdefinition fileis located. If we consider themy-app-environmentenvironment we used above then your directory
tree should look like this:.
├── my-app-environment.yaml
├── name_of_first_layer
│   ├── meta.yaml
│   └── resources.yaml
├── name_of_second_layer
│   ├── meta.json
│   └── meta.yaml
└── name_of_third_layer
    ├── resources.json.j2
    └── resources.yaml.j2A layer must contain at least two files:meta.yaml: Meta information about the layer such as a
description, and layer parameters.resources.yaml: Basically a CF template with the resources that
the layer contains.Those two files can also be in.jsonformat (meta.jsonandresources.json). Or you can add the extension.j2if you want
the files to be pre-processed with theJinja2template compiler.Below an example of how a layermeta.yamlmay look like:---
meta:
    description:
        Creates a VPC, that's it
    parameters:
        vpc_cidr:
            description: The CIDR block of the VPC
            value: 10.0.0.0/16Above we declare only one layer parameter:vpc_cidr.humiliswill make pass that parameter to Jinja2 when compiling any template
contained in the layer. So theresources.yaml.j2for that same layer
may look like this:---
resources:
    VPC:
        Type: "AWS::EC2::VPC"
        Properties:
            CidrBlock: {{ vpc_cidr }}ReferencesYou can use references in yourmeta.yamlfiles to refer to thing
other than resources within the same layer (to refer to resources within
a layer you can simply use Cloudformation’sReforGetAttfunctions). Humilis references are used by setting the value of a layer
parameter to a dict that has arefkey. Below an ameta.yamlthat refers to a resource (with a logical nameVPC) that is
contained in another layer (calledvpc_layer):---
meta:
    description:
        Creates an EC2 instance in the vpc created by the vpc layer
    dependencies:
        - vpc
    parameters:
        vpc:
            description: Physical ID of the VPC where the instance will be created
            value:
                ref:
                    parser: layer
                    parameters:
                        layer_name: vpc_layer
                        resource_name: VPCEvery reference must have aparserkey that identifies the parser
that should be used to parse the reference. There are also two optional
keys:parameters: allows you to pass parameters to the reference
parser. You can pass either named parameters (as a dict) or
positional arguments (as a list).priority: the parsing priority. Parameters with a lower value inprioritywill be parsed before parameters with a higher value.
This allows some reference parsers to refer internally to other
parameters within the same layer. For example, thelambdaparser,
when parsing templated lambda code, it uses previously parsed layer
parameters as template parameters.More information on the reference parsers that are bundled with humilis
below.Available reference parserslayer_resourcereferenceslayer_resourcereferences allow you to refer to the physical ID of a
resource that is part of another layer.Parameters:layer_name: The name of the layer you are referring toresource_name: The logical name of the layer resourceExample:Consider the following environment definition:---
my-environment:
    description:
        Creates a VPC with a NAT in the public subnet
    layers:
        - {layer: vpc}
        - {layer: nat}Obviously thenatlayer that takes care of deploying the NAT in the
public subnet will need to know the physical ID of that subnet. You
achieve this by declaring alayer_resourcereference in themeta.yamlfor thenatlayer:---
meta:
    description:
        Creates a managed NAT in the public subnet of the NAT layer
    parameters:
        subnet_id:
            description:
                The physical ID of the subnet where the NAT will be placed
            value:
                ref:
                    parser: layer_resource
                    parameters:
                        layer_name: vpc
                        # The logical name of the subnet in the vpc layer
                        resource_name: PublicSubnetWhen parsingmeta.yamlhumilis will replace this:ref:
    parser: layer_resource
    parameters:
        layer_name: vpc
        # The logical name of the subnet in the vpc layer
        resource_name: PublicSubnetwith the physical ID you need (something likesubnet-bafa90cd). You
can then use this physical ID in theresources.yaml.j2section of
thenatlayer:{# Pseudo-content of layers/nat/resources.yaml.j2 #}
resources:
    {# An Elastic IP reservation that will be associated to the NAT #}
    NatEip:
      Type: 'AWS::EC2::EIP'
      Properties: {}
    {# Custom resource deploying the NAT #}
    NatGateway:
      Type: 'Custom::NatGateway',
      Properties:
        {# The ARN of the Lambda function backing the custom resource #}
        ServiceToken: 'arn:aws:lambda:eu-west-1:XXXX:function:CreateNatGateway'
        {# Here we use the subnet_id reference defined in meta.yaml #}
        SubnetId: {{subnet_id}}
        AllocationId:
            Ref: NatEipenvironment_resourcereferencesenvironment_outputreferences allow you to refer to resources that
belong to other humilis environments.Parameters:environment_name: The name of the environment you are referring
tolayer_name: The name of the layer you are referring toresource_name: The logical name of the layer resourcelayer_outputreferenceslayer_outputreferences allow you to refer to outputs produced by
another layer.Parameters:layer_name: The name of the layer you are referring tooutput_name: The logical name of the output parameterIn general you should prefer usinglayer_outputreferences overlayer_resourcereferences. The output parameters produced by a layer
define an informallayer interfacethat is more likely to remain
constant than the logical names of resources within a layer.boto3referencesboto3references define arbitrary calls toboto3facade. The
latter is just a simpler facade interface on top ofboto3.Parameters:service: The AWS service, e.g.ec2orcloudformation.
Note that only only AWS services that have a facade inboto3facadeare
supported.call: The corresponding facade method,
e.g.get_ami_by_name. The value of this parameter must be a
dictionary with amethodkey (the name of the facade method to
invoke) and an optionalargskey (the parameters to pass to the
facade method). Best to look at the example below to understand how
this works.output_attribute: Optional. If provided the reference parser will
return the value of this attribute from the object returned by the
facade method.Below an example of a layer that uses aboto3reference:---
meta:
    description:
        Creates an EC2 instance using a named AMI
    # More stuff omitted for brevity
    ami:
        description: The AMI to use when launching the EC2 instance
        value:
            ref:
                parser: boto3
                parameters:
                    service: ec2
                    call:
                        method: get_ami_by_name
                        args:
                            - test-ami
                    output_attribute: idhumiliswill parse the reference using this code:# Import the Ec2 facade
from boto3facade.ec2 import Ec2

# Create a facade object
ec2_facade = Ec2()

# Make the call
ami = ec2_facade.get_ami_by_name('test-ami')

# Extract the requested attribute
ref_value = ami.idfilereferencesfilereferences allow you to refer to a local file. The file will be
uploaded to S3 and the reference will evaluate to the corresponding S3
path.Parameters:path: The path to the file, relative to the layer root directory.lambdareferenceslambdareferences allow you to refer to some Python code in your
local machine. If your code follows some simple conventionshumiliswill take care of building adeployment
packagefor you, uploading it to S3, and the reference will evaluate to the S3
path of the deployment package.Parameters:path: Path to either a completely self-contained.pyfile, or
to the root directory of your lambda code. In the latter case your
code needs to follow some simple conventions for this to work. More
information below.dependencies: A list of dependencies to be included in the Lambda
deployment package. Dependencies may be either pip installable
packages, or paths to local Python packages or modules, or paths to
localrequirementsfiles.Example:ref:
    parser: lambda
    parameters:
        # Path to the root directory containing your lambda code
        path: dummy_function
        dependencies:
            # The Lambda code requires Pypi's pyyaml
            - pyyaml
            # It also requires a local package in this path
            - mycode/mypkgdir
            # And this local module
            - mycode/mymodule.pywhich will evaluate to a S3 path such as:s3://[bucket_name]/[environment_name]/[stage_name]/[func_name]-[commithash].zipCode conventions:Following the example above, the contents of the layer responsible of
deploying thedummy_functionlambda may look like this:.
├── dummy_function
│   ├── dummy_function.py
│   └── setup.py
├── meta.yaml
├── outputs.yaml.j2
└── resources.yaml.j2Basically all your code needs to be included under directorydummy_function. In this case there is only one file:dummy_function.py. External dependencies need to be specified in
yoursetup.py.secretreferencessecretreferences retrieve a secret using Python’skeyringmodule.Parameters:service: The name of the service the secret is associated to.key: The key (e.g. the username) that identifies the secret.Example:ref:
    parser: secret
    parameters: {"service": "mysqldb", "key": "adminuser"}Custom Jinja2 filtersHumilis defines the followingcustom Jinja2
filters:uuid: A random UUID. Example:{{''|uuid}}.password(length=8): A random password. Example:{{10|password}}.