Package: bossimage
bossimage=========|Build Status|Bossimage is a command line utility to convert an `Ansiblerole <http://docs.ansible.com/ansible/playbooks_roles.html>`__ into an`Amazon EC2AMI <http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html>`__.Bossimage requires just one configuration file to be added to the basedirectory of an Ansible role. Once that is done, Bossimage may be usedto build an EC2 instance, run the Ansible role on it, then "bake" itinto an AMI. After the AMI is created, Bossimage can also build a testinstance from it and run a test playbook on the instance.Bossimage is inspired by both `Packer <https://www.packer.io/>`__ and`Test Kitchen <http://kitchen.ci/>`__, but much simpler than either. Ifyou use both Ansible and AWS, you may find it useful.Bossimage has been tested on both Linux and Windows targets in EC2.Installation============Install from `PyPI <https://pypi.python.org/pypi>`__----------------------------------------------------::pip install bossimageInstall from source-------------------::git clone https://github.com/cloudboss/bossimage.gitcd bossimagepip install -r requirements.txtpip install .Quick Start===========All interaction with Bossimage is done through an executable commandcalled ``bi``, which must always be run from the base directory of anAnsible role.This introduction to Bossimage will explain how to do three things:1. Make an EC2 instance and run Ansible on it (``bi make build``).2. Make an AMI from the EC2 instance (``bi make image``).3. Make a test instance from the AMI and run a test Ansible playbook onit (``bi make test``).Later it will be explained how to do a few other things as well.First, a small amount of configuration is necessary.Note: in this guide, all commands to be run from the shell are shownpreceded by ``>`` to indicate the shell prompt.Configuration~~~~~~~~~~~~~Bossimage requires a configuration file called ``.boss.yml`` to beplaced in the root directory of the Ansible role. A minimal example ofsuch a file is as follows:::platforms:- name: amz-2015092instance_type: t2.microbuild:source_ami: amzn-ami-hvm-2015.09.2.x86_64-gp2The example contains the most minimal configuration possible, usingdefaults for all settings except those which are required: the platformname, the `instance type <https://aws.amazon.com/ec2/instance-types/>`__and the source AMI used for the ``build`` phase.Although Bossimage creates resources in AWS, it does not include any AWSauthentication code, instead preferring to pass all authenticationthrough to the underlying`SDK <http://boto3.readthedocs.io/en/latest/guide/configuration.html#guide-configuration>`__using `standard environmentvariables <https://blogs.aws.amazon.com/security/post/Tx3D6U6WSFGOK2H/A-New-and-Standardized-Way-to-Manage-Credentials-in-the-AWS-SDKs>`__.Here is an example of gaining credentials by setting ``AWS_PROFILE`` and``AWS_DEFAULT_REGION`` environment variables, assuming a credentialsfile has already been created.::> export AWS_PROFILE=uhuru> export AWS_DEFAULT_REGION=us-west-1If Bossimage is being run from an EC2 instance, an `IAM instanceprofile <http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html>`__may be used instead of environment variables, as described later.Running~~~~~~~Most ``bi`` subcommands require an *instance* argument to be passed tothem. The *instance* is derived from a *platform* together with a*profile*, i.e., ``<platform>-<profile>``. In the ``.boss.yml``configuration shown above, a single platform is defined with name``amz-2015092``. The profile is not explicitly defined, and is therefore``default``. So the instance is ``amz-2015092-default``, and that willbe the argument passed to the commands in this introduction.Platforms and profiles will be described in more detail later.bi make build^^^^^^^^^^^^^This builds an EC2 instance and runs the Ansible role on it. A uniquessh keypair is also created and assigned to the instance. This command,as with other ``bi`` commands, is idempotent and may be run multipletimes without creating a new instance each time. Subsequent runs willsimply run the Ansible role again on the existing instance.Consider ``bi make build`` the entrypoint of Bossimage: it must be runbefore ``bi make image`` or ``bi make test``.::> bi make build amz-2015092-defaultCreated keypair bossimage-oZL4NxUbAMCreated instance i-00000001Waiting for instance to be running ... okWaiting for connection to 54.xxx.xxx.xxx:22 to be available ... okPLAY ***************************************************************************TASK [setup] *******************************************************************ok: [54.xxx.xxx.xxx]TASK [test-role : add package httpd] *******************************************changed: [54.xxx.xxx.xxx]PLAY RECAP *********************************************************************54.xxx.xxx.xxx             : ok=1    changed=1    unreachable=0    failed=0bi make image^^^^^^^^^^^^^The primary goal of Bossimage is to create an AMI from an Ansible role,and that is what this command does. It may be run when ``bi make build``has completed.::> bi make image amz-2015092-defaultCreated image ami-00000001 with name test-role.default.amz-2015092.hvm.x86_64.v2Waiting for image to be available ... okImage is availablebi make test^^^^^^^^^^^^It is useful to test that ``bi make image`` generated a correct AMI, andthis is where ``bi make test`` comes into play.This command is very similar to ``bi make build``, in that it creates anEC2 instance and runs Ansible on it. However, it depends on a successfuloutcome of the ``bi make image`` command, as it uses the AMI created bythat command as the source AMI of the EC2 test instance.It also does not run the Ansible role on the instance, rather it runs atest playbook, which by default is ``tests/test.yml``, relative to theroot of the Ansible role directory. When creating an Ansible role withthe ``ansible-galaxy`` command, this test playbook is added by default.For this default test playbook to work with Bossimage, only one smallchange is needed: to changed the ``hosts`` in the playbook from``localhost`` to ``test``.::> bi make test amz-2015092-defaultCreated instance i-00000002Waiting for instance to be running ... okWaiting for connection to 52.xxx.xxx.xxx:22 to be available ... okPLAY [test] ********************************************************************TASK [setup] *******************************************************************ok: [52.xxx.xxx.xxx]TASK [check that httpd is installed] *******************************************ok: [52.xxx.xxx.xxx]TASK [check that port 80 is listening] **********************************************ok: [52.xxx.xxx.xxx]PLAY RECAP *********************************************************************52.xxx.xxx.xxx               : ok=2    changed=0    unreachable=0    failed=0Conclusion~~~~~~~~~~Having run these three commands, you will have seen the majorfunctionality of Bossimage. You will have created an AMI and then testedit.Continue reading to learn:-  How to build multiple "flavors" of AMIs for a given platform-  A shortcut for logging into build and test instances-  Clean up instances and keypairs used during the build and test phases-  Clean up AMIs that did not pass testsBossimage=========Instances, Platforms, and Profiles~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Most of the ``bi`` subcommands, such as ``make build`` or ``make test``,take an argument called the *instance*. An instance is defined by a*platform* and a *profile*, such as ``rhel6-default``, where ``rhel6``is the platform and ``default`` is the profile.Platform^^^^^^^^The platform defines the source AMI and other settings related tocreating an EC2 instance, such as security groups and block devicemappings. It also defines connection settings for Ansible to reach theinstance, such as ssh or winrm ports and default username.Profile^^^^^^^The profile defines variables that will be passed to Ansible through its``--extra-vars`` argument. By defining multiple profiles, you can buildmultiple flavors of AMIs for a given platform.For example, here is a ``.boss.yml`` with one platform and two profiles.::platforms:- name: ubuntu-16.04build:source_ami: ami-301f6f50instance_type: t2.microinventory_args:ansible_user: ubuntuansible_python_interpreter: /usr/bin/python3security_groups: [bossimage]profiles:- name: apacheextra_vars:packages:- apache2- name: nginxextra_vars:packages:- nginxRunning ``bi list`` will produce the output:::ubuntu-16.04-apache     Not createdubuntu-16.04-nginx      Not createdEach of these platform and profile combinations, or instances, can bemade into its own AMI.If ``profiles`` is not defined, every platform has an implicit profile,called ``default``. The ``default`` profile does not define anyvariables. Note that if ``profiles`` is defined, there is no longer anyimplicit ``default`` profile. In such cases you can define one that hasno ``extra_vars`` attribute, if desired..boss.yml---------The ``.boss.yml`` file is placed in the root directory of an Ansiblerole. It is the only configuration necessary for using Bossimage.To start, here is a full example for reference.::defaults:instance_type: m3.largeplatforms:- name: centos-6instance_type: t2.microconnection_timeout: 600inventory_args:ansible_user: centosbuild:source_ami: 'CentOS Linux 6 x86_64 HVM EBS 1602-74e73035-3435-48d6-88e0-89cc02ad83ee-ami-21e6d54b.3'test:instance_type: m3.mediumtags:Billing: xyzDescription: Centos 6 Build Instance- name: amz-2015092build:source_ami: amzn-ami-hvm-2015.09.2.x86_64-gp2image:ami_name: '%(role)s-%(profile)s-%(version)s-%(platform)s'block_device_mappings:- device_name: /dev/sdfebs:volume_size: 100volume_type: gp2delete_on_termination: truetags:Billing: xyzDescription: Amazon Linux 201509 Build Instance- name: win-2016build:source_ami: ami-2d360152become: falseinventory_args:ansible_connection: winrmansible_port: 5985ansible_user: Administratorimage:ami_name: '%(role)s-%(profile)s-%(version)s-%(platform)s'tags:Billing: xyzDescription: Windows Server 2016 Build Instanceprofiles:- name: default- name: nginxextra_vars:packages:- nginx- tcpdumpA ``.boss.yml`` file has three possible sections:-  ``defaults``: This section is optional, and contains default valuesto be used within ``platforms`` when not provided there.-  ``platforms``: This section is required, and defines a list ofplatforms to build instances from. There must be at least oneplatform defined in a ``.boss.yml`` configuration. Each platformdefined in the ``platforms`` section contains its own subsections foreach of the three phases ``build``, ``image``, and ``test``.-  ``profiles``: This section is optional. In here, sets of variablesmay be defined to modify each platform defined in the ``platforms``section. If this section is not given, each platform will have aprofile called ``default``, with no additional variables set.defaults~~~~~~~~The ``defaults`` section may contain the following variables.-  ``instance_type`` - type: *string*, default: ``t2.micro``The EC2 instance type.-  ``username`` - type: *string*, default: ``ec2-user``The user that Ansible will use to connect to the instance. If``inventory_args`` is defined, this value will be ignored, and``ansible_user`` should be put into ``inventory_args`` instead.-  ``connection`` - type: *string*, default: ``ssh``The type of `connection that Ansible willuse <https://docs.ansible.com/ansible/2.6/plugins/connection.html>`__.If ``inventory_args`` is defined, this value will be ignored, and``ansible_connection`` should be put into ``inventory_args`` instead.Note: Bossimage is known to work for ``ssh`` and ``winrm`` connections,but other types may need additional testing and development.-  ``connection_timeout`` - type: *integer*, default: ``300``The amount of time in seconds before Bossimage will give up trying tomake an Ansible connection.-  ``port`` - type: *integer*, default: 22The port used to connect with Ansible. If ``inventory_args`` is defined,this value will be ignored, and ``ansible_port`` should be put into``inventory_args`` instead.-  ``associate_public_ip_address`` - type: *bool*, default: ``true``Whether or not to associate a public IP address to the instance.-  ``subnet`` - type *string*The subnet in which the instance will be located.-  ``security_groups`` - type *list* of *string*, default ``[]``The security groups that are associated with the instance.-  ``iam_instance_profile`` - type *string*The name of the IAM instance profile to assign to the instance.-  ``inventory_args`` - type *map* of *string* to *string*A map of key/value pairs which will be used for building the Ansibleinventory. See `the official Ansibledocumentation <https://docs.ansible.com/ansible/2.3/intro_inventory.html#list-of-behavioral-inventory-parameters>`__for more details on available options. If this variable is defined, the``connection``, ``username``, and ``port`` variables will be ignored ifused, and should be replaced with inventory arguments``ansible_connection``, ``ansible_user``, and ``ansible_port``,respectively.Note: Bossimage normally sets ``ansible_password`` and``ansible_ssh_private_key_file`` in the inventory based on runtimegenerated values, so it is not advised to define them in``inventory_args`` unless you have good reason.-  ``tags`` - type *map* of *string* to *string*, default ``{}``A map of key/value pairs to be used for tagging the instance.-  ``user_data`` - type: *map* or *string*, default: ``''``This is the `userdata <http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html>`__that will be passed into the EC2 instance. If it is given as a map, thenit must have the key ``file``, which is the path to a file containingthe user data.If the type is a string, then it is passed verbatim as the user data forthe instance.Examples:::defaults:user_data:file: ./user-data.txt::defaults:user_data: |#!/bin/shyum update -y-  ``block_device_mappings`` - type: *list* of *map*, default: ``[]``Devices to be attached to the EC2 instance that will be part of a bakedimage.Each item in the list is a map as described in the`BlockDeviceMappings <http://boto3.readthedocs.io/en/latest/reference/services/ec2.html#EC2.ServiceResource.create_instances>`__property passed to the boto3 create\_instances operation. The onlydifference is that in. boss.yml, "CamelCase" properties should beconverted to "snake\_case".platforms~~~~~~~~~The ``platforms`` section contains a list of configurations, one foreach defined platform. Each platform configuration must have the keys:-  ``name`` - type *string*, required-  ``build`` - type *map*, requiredSee `build <#build>`__ below.-  ``image`` - type *map*, optionalSee `image <#image>`__ below.-  ``test`` - type *map*, optionalSee `test <#test>`__ below.The platform configuration may also contain any of the variables from``defaults``, and will override any of the definitions from there.build^^^^^The ``build`` section of a platform is required and may include any ofthe variables from ``defaults``. They will override any of thedefinitions given there or in the parent platform.The ``build`` section also has the following keys:-  ``source_ami`` - type: *string*, requiredThis is the source AMI to build the instance from. It may be given as anAMI ID or name, from which the ID will be found.-  ``become`` - type: *boolean*, default: ``true``This tells Ansible whether or not to "become" the superuser.image^^^^^The ``image`` section of a platform may have the following key:-  ``ami_name`` - type: *string*, default:``'%(role)s.%(profile)s.%(platform)s.%(vtype)s.%(arch)s.%(version)s'``This is a `Python formattingstring <https://docs.python.org/2/library/stdtypes.html#string-formatting>`__to use for generating the AMI name. The string may contain any of thevariables:-  ``role``: Name of Ansible role-  ``profile``: Name of profile used from ``.boss.yml``-  ``platform``: Name of platform used from ``.boss.yml``-  ``vtype``: Virtualization type, e.g. ``hvm``-  ``arch``: Architecture, e.g. ``x86_64``-  ``version``: Ansible role version, see `RoleVersions <#role-versions>`__.-  ``hv``: Hypervisor, e.g. ``xen``Of course, ``ami_name`` may also be a string used verbatim without anyinterpolated variables in it.test^^^^The ``test`` section of a platform may include any of the variables from``defaults``. They will override any of the definitions given there orin the parent platforms.In addition, the ``test`` section may have the following key:-  ``playbook`` - type: *string*, default: ``tests/test.yml``This is the playbook to run during the test phase. The default value isthe same as the test playbook that is created by running``ansible-galaxy init`` to create a new Ansible role.Commands--------The ``bi`` command must always be run from the root directory of anAnsible role, where the ``.boss.yml`` file is located.bi list^^^^^^^List instances available to be built that are configured in .boss.yml.The status of the instance is shown, which may be either ``Created`` or``Not created``.::> bi listamz-2015092-default     Createdubuntu-16.10-default    Not createdbi make build^^^^^^^^^^^^^::> bi make build <instance> [-v|--verbosity]This builds an EC2 instance and runs the Ansible role on it. A uniquessh keypair is also created and assigned to the instance. This commandis idempotent and may be run multiple times without creating a newinstance each time. Subsequent runs will simply run the Ansible roleagain on the existing instance.If your Ansible role has a ``requirements.yml`` file, then the``ansible-galaxy`` command will be used to install the dependencieslisted there.The ``-v``, or ``--verbosity`` option, gets passed through to Ansible.It may be repeated up to four times to increase Ansible's verbosity.bi make image^^^^^^^^^^^^^::> bi make image <instance> [--no-wait]This builds an AMI from the instance created by running``bi make build``. This command will not run unless ``bi make build``has run and written its state to ``.boss/<instance>-state.yml``.By default this command will complete when the image is available. Youmay pass the option ``--no-wait`` to this command so that it does notwait for the image to be available.bi make test^^^^^^^^^^^^::> bi make test <instance> [-v|--verbosity]This builds an EC2 instance from the AMI created by running``bi make image``, then runs the test playbook on it. This command willnot run unless ``bi make image`` has run and written its state to``.boss/<instance>-state.yml``.As with ``bi make build``, ``ansible-galaxy`` will be used to installany role dependencies used by the test playbook, but ``ansible-galaxy``will look for them in ``tests/requirements.yml``.The ``-v``, or ``--verbosity`` option, gets passed through to Ansible.It may be repeated up to four times to increase Ansible's verbosity.bi clean build^^^^^^^^^^^^^^::> bi clean build <instance>This deletes the instance created by ``bi make build``.bi clean image^^^^^^^^^^^^^^::> bi clean image <instance>This deletes the AMI created by ``bi make image``.bi clean test^^^^^^^^^^^^^::> bi clean test <instance>This deletes the instance created by ``bi make build``.bi login^^^^^^^^::> bi login <instance>This command works only on instances where the platform is configuredfor ssh connections, which is the default. By default this command logsinto the ``build`` phase instance, but this may be changed by passingthe ``-p|--phase`` argument, which may be either ``build`` or ``test``.::> bi login -p test <instance>bi version^^^^^^^^^^The command outputs the version of Bossimage.Role Versions-------------Ansible Galaxy does not provide a way to define a role's version in itsmetadata, it relies on git tags for versioning. So Bossimage does nothave anything it can parse to discover the version of a role.Instead, you can put a file in the root of the repository called``.role-version`` which contains the version string. Bossimage alsosupports defining the version in the environment variable``BI_ROLE_VERSION``.If neither the ``.role-version`` file or the ``BI_ROLE_VERSION``environment variable are present, then a default version ``unset`` isused.Authenticating with AWS-----------------------``bossimage`` uses standard AWS SDK environment variables forauthentication, which are described in the `boto3documentation <http://boto3.readthedocs.org/en/latest/guide/configuration.html#configuration>`__.The simplest way to authenticate if you are not running ``bossimage`` onan EC2 instance is to configure ``~/.aws/credentials`` with a`profile <http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-multiple-profiles>`__and pass its name in the environment variable ``AWS_PROFILE``.If you are running ``bossimage`` on an EC2 instance, you may assign theinstance an `IAMrole <http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html>`__upon creation, and then you do not need to pass any credentials. The IAMrole should have the policy shown below.::{"Version": "2012-10-17","Statement": [{"Effect": "Allow","Action": ["ec2:CreateImage","ec2:CreateKeyPair","ec2:CreateTags","ec2:DeleteKeyPair","ec2:DeregisterImage","ec2:DescribeImages","ec2:DescribeInstances","ec2:RunInstances","ec2:TerminateInstances"],"Resource": "*"}]}Region------You must set the AWS region you are running in. To do this, set the``AWS_DEFAULT_REGION`` environment variable.Rationale=========All I want is to spin up an EC2 instance in AWS, run an`Ansible <http://docs.ansible.com/ansible/index.html>`__ role on it,bake it into an image, and run some tests to verify the correctness ofthe image.Comparison with Packer~~~~~~~~~~~~~~~~~~~~~~Packer is a tool for creating VM and Docker images for a multitude ofcloud providers and for local use.Packer does more than I need; I only need to create EC2 AMIs. But stillit doesn't do quite enough: it doesn't provide a development phase forrapid iterative development of an Ansible role. You always have to startfrom the beginning with a new instance.Bossimage creates EC2 images and provides a development phase beforecreating an image, and a testing phase for when the image has beencreated.Comparison with Test Kitchen~~~~~~~~~~~~~~~~~~~~~~~~~~~~Test Kitchen is a tool for testing Chef cookbooks, but can be used totest Ansible and other configuration management tools using third partyplugins. It can create VM instances with Vagrant and various cloudproviders to use for developing.Test Kitchen does more than I need; I only need to test Ansible in EC2.But still it doesn't do quite enough: it doesn't provide an AMI creationphase.Bossimage creates EC2 instances and runs Ansible on them, and providesimage creation and image testing phases... |Build Status| image:: https://travis-ci.org/cloudboss/bossimage.svg?branch=master:target: https://travis-ci.org/cloudboss/bossimage