Package: ctf-cli
Containers Testing Framework (CTF) is a simple wrapper aroundBehave
testing framework. Behave is a
promising approach for testing containers, since it enables one to focus
on describing the behavior of the container from a High Level and in
simple English. For more information on how to use Behave and how to
write tests using Behave, please refer to the Behave project page.CTF tries to re-use the remote execution of steps model fromUATFrameworkso the tests
can be executed locally or on a remote machine or VM. The support is
still in progress.CTF provides:a way of running tests on Dockerfiles and imagesa way to run project-specific tests (Steps and Features)a combination of multiple tests (Steps and Features) stored in remote
repositorythe ability to run remote tests on specific Dockerfiles and/or images
without having any project-specific tests suiteHow to use the framework for your containers?If you want to implement project-specific tests, you should create the
following directory structure inside your project directorymy_project_dir:my_project_dir/
  tests/
    features/
      my_cool.feature
      my_other_cool.feature
    steps/
      my_cool_steps.py
      my_other_cool_steps.py
    environment.py
    tests.confThe best way to leverage the framework is to implement Features and
Steps that are common for a set of containers as a remote test and then
include it in the testing configuration of all containers.features/Place any features and scenarios, specific for your particular container
inside this directory. Your scenarios can use any of the steps
implemented insidetests/stepsdirectory or steps from any remote
test you listed inside thetests.conffile. CTF CLI tool will
combine all these steps together, so that Behave is able to find them
when run.steps/Place any steps that are specific for your features inside this
directory. To make sure the steps can be executed on the remote machine
or locally, always usecontext.run()for running any commands. In
the background, the command will be run locally or on remote machine,
based on the CLI configuration.The CLI tool passes some runtime arguments to Behave, when executing it.
The values are available through the context ascontext.config.userdatadictionary like object. To learn more about
it, please read theBehave
documentation.
The following values are passed:DOCKERFILEThis option contains the absolute path to the Dockerfile wich should be
tested. It is always set.IMAGEThis option contains the name of the image to test. It is passed and set
only if the name was passed to CLI tool using-ioption.tests.confThetests.conffile contains configuration telling the CTF what
extra remote tests (Steps and Features) to include when testing the
container using Behave. It is a simple INI file. Each remote test needs
to have its own section and inside the section specifyStepsandFeaturesoptions. These has to point to remote git repositories
containing the appropriate Steps and Features. Example of tests.conf:[common-tests]
Steps=https://github.com/Containers-Testing-Framework/common-steps.git
Features=https://github.com/Containers-Testing-Framework/common-features.gitThis method is not recommended and will soon be deprecated - git submodules way would be preferredenvironment.pyYou can implement any of the methods that are typically used with Behave
inside this file. They will be combined with the CTF commonenvironment.pyfile. And your methods will be invoked before the CTF
hooks.Getting StartedRequirementsgitbehave >= 1.2.5ansibleInstallationClone repositoriesgit clone https://github.com/Containers-Testing-Framework/ctf-cli.gitOptional: clone an example repositorygit clone https://github.com/Containers-Testing-Framework/example-project-postgresql.gitChange to ctf-cli directorycd ctf-cliInstall python dependencies[sudo] pip install -r requirements.txtCopy sample configuration files and edit as necessarycp ctf.conf.sample ctf.conf
cp tests.conf.sample tests.confChange to project directory and get common featurescd example-project-postgresql
../ctf-cli/ctf-cli.py remote add features https://github.com/Containers-Testing-Framework/common-features.gitRun tests../ctf-cli/ctf-cli.py runCLI toolThe key part of the framework is the CLI tool calledctf-cli. It
gathers information, reads configurations, sets up the environment for
Behave and runs it. Currentlyctf-clitool supports only ansible for
running commands on some host. To run your tests make sure you included
configuration for ansible in thectf-cliconfiguration.Configuration fileYou can pass the path to the configuration file on the command line
using the-cor--cli-configoption. If not provided the tool
tries to find the configuration in this order of preference:ctf.confin the current directoryctf.confin the user home directory~/ctf.confctf.confin thectfdirecotory inside user home directory~/ctf/ctf.confctf.confin the system configuration directory/etc/ctf.confThe configuration must include configuration for ansible. The setup on
the host you want to use needs to be done manually before running thectf-cli(e.g. setting up ssh keys, etc.) An example of the
configuration for ansible inside thectf.confcan look like this:[ansible]
Host=127.0.0.1
User=root
Method=sshUsageThectf-clishould be executed inside the directory of the project,
containing the Dockerfile. There are couple of options one can pass to
the CLI tool:-h,--help- Prints the help message and exit-v,--verbose- Makes the output (much) more verbose
(recommended)-cCLI_CONFIG_PATH,--cli-configCLI_CONFIG_PATH- Path to
CLI configuration file (default: ‘/etc/ctf-cli.conf’)-tTESTS_CONFIG_PATH,--tests-configTESTS_CONFIG_PATH- Path
to tests configuration file. By default it will be searched for in
tests/ dir-fDOCKERFILE,--dockerfileDOCKERFILE- Path to Dockerfile
to use. If not passed, will be searched for in the current directory-iIMAGE,--imageIMAGE- Image to use for testing. If not
passed, the image will be built from the DockerfileHow it works?Whenctf-cliis executed inmy_proj_dirproject directory the
following happens:CLI tool configuration is read.The directory is searched for thetests/directory.Thetests/directory is searched fortests.confconfiguration.A working directorymy_proj_dir-behave-working-diris created
inside current directory.featuresandstepsdirectories are created inside the
working directory.Project specific features are copied into thefeatures/my_proj_dir_featuresinside working directory.Project specific steps are copied into thesteps/my_proj_dir_stepsinside working directory.If environment.py is present intests/, then it is copied into
the working directory asmy_proj_dir_environment.py.Iftests.confcontained some remote test, their Features and
Steps are cloned into the working directory similarly as done for
local files in (6.) and (7.).Makes sure all subdirectories insidesteps/in working directory
contain__init__.py.Createsteps.pyinsidesteps/in working directory, which
imports everything from all steps files.Createenvironment.pyin working directory, which contains CTF
common methods and includes the project specific environment.py file
if present.Run Behave inside the working directory with all the necessary
runtime argumentsIdeas for further developmentThe steps done by CLI tool can be separated (prepare working dir,
update working dir, run tests). This would allow one to do the
partial workflow if needed.Using specific commit for remote tests. This prevents surprises when
someone breaks tests shared across multiple containers.Testing containers combinations. Idea is to tag the containers by
some roles (e.g. @webserver, @database, …) and then access these
containers just based on the tags in the steps. This way any
webserver could be tested with any database, and so on.Integrate the framework with some CIAdd support for other ways of running command remotely besides
ansibleAdd remote hosts provisioning, startng, stoppingsupport this for cloud, VMs, Vagrant boxes, etc.Decisions madewe will go with submodules instead of tests.confit is not acceptable to expect users to directly call git - needed
functionality has to be wrapped by CTFremote features/steps should be added into tests/remote/… to
prevent conflicts with project-specific features/stepswe will need export and import commands to export and import current
project tests setup for sharing between projectswe can not expect that the project is using git (maybe create git
repo if not using git only in the tests/ dir?)tests.conf will be deprecated as it is used ATMthe “generated” environment.py should be kept clean and all
containers specific code should be moved elsewhere.ReferencesBehaveUATFrameworkBehavior Driven Development