Package: thriftybuilder
[![Build Status](https://travis-ci.org/wtsi-hgi/thrifty-builder.svg?branch=master)](https://travis-ci.org/wtsi-hgi/thrifty-builder)[![codecov](https://codecov.io/gh/wtsi-hgi/thrifty-builder/branch/master/graph/badge.svg)](https://codecov.io/gh/wtsi-hgi/thrifty-builder)[![PyPI version](https://badge.fury.io/py/thriftybuilder.svg)](https://badge.fury.io/py/thriftybuilder)# Thrifty Builder_Builds Docker images, capturing information to reduce the frequency of future re-builds_## IntroductionThrifty builder stores a hash of all the ingredients that go into building a Docker image so the tool is able todetermine if an image has already been built before, even if the build cache has been emptied or if the build is takingplace on a different machine with a separate cache.In our setup, we are building a large number of Docker images in our CI. The CI job runs on a different machine eachtime (with separate caches), meaning that if `docker build` was used, all images would be rebuild every timethe CI runs. Pulling first would sped things up but it would require all images to be downloaded to the build machinefirst!The aim is to minimise our CI run time and to keep our Docker images as stable as possible (it is usuallyextremely difficult to version everything that goes into an image so each re-build will create a slightly differentimage, even if the context and Dockerfile are the same).## InstallationPrerequisites- Docker (Server API version >= 1.21)- Python 3.6+The tool can be installed from PyPi:```bashpip install thriftybuilder```Bleeding edge versions can be installed directly from GitHub:```bashpip install git+https://github.com/wtsi-hgi/thrifty-builder.git@master#egg=thriftybuilder```## Usage### ConfigurationA build configuration YAML file is required to use the tool. This file details the images that are to be built, theDocker registries to push the created images to (optional) and the location of the checksum storage.#### Storage##### stdin/stdout(Default if not specified)```yamlchecksum_storage:type: stdio```##### Local```yamlchecksum_storage:type: localpath: /root/.thrifty/checksums```##### Consul```yamlchecksum_storage:type: consulurl: https://example.com:8500           # Optional: derived from Consul environment variables if not settoken: "{{ env['CONSUL_HTTP_TOKEN'] }}"   # Optional: derived from Consul environment variables if not setkey: ci/image-checksumslock: ci/image-checksums.lock```_Note: to use Consul-backed storage, the requirements in `consul_requirements.txt` must be installed (not done so bydefault)._### CLI```usage: thrifty [-h] [-v] [--built-only] configuration-locationBuilds Docker images, capturing information to reduce the frequency of futurere-builds (v1.0.0b0)positional arguments:configuration-locationlocation of configurationoptional arguments:-h, --help            show this help message and exit-v                    increase the level of log verbosity (add multipleincrease further)--built-only          only print details about newly built images on stdout```### Example_configuration.yml_```yamldocker:images:- name: wtsi-hgi/image-1dockerfile: /images/image-1/Dockerfilecontext: /images- name: wtsi-hgi/image-2:version-1dockerfile: /images/image-2/Dockerfile# Context assumed to be /images/image-2registries:- url: docker.iousername: "{{ env['DOCKER_IO_USERNAME'] }}"password: "{{ env['DOCKER_IO_PASSWORD'] }}"checksum_storage:type: consulurl: "{{ env['CONSUL_HTTP_ADDR'] }}"token: "{{ env['CONSUL_HTTP_TOKEN'] }}"key: ci/image-checksumslock: ci/image-checksums.lock```CLI:```bashthrifty configuration.yml{"wtsi-hgi/image-1": "b2db4c1ae978201407c69573ba89a9b8", "wtsi-hgi/image-2": "f9a4d7cc9f7133756b36973cc2d888de"}```## Development### SetupInstall the tool's dependencies and the dependencies needed for testing:```bashpip install -U -r test_requirements.txtpip install -U -r requirements.txtpip install -U -r consul-requirements.txt```### TestingIn the project directory, run:```bashPYTHONPATH=. python -m unittest discover -v -s thriftybuilder/tests```The CI runs the tests and generates coverage with `run-tests.sh`. This script runs the tests as described above in,addition to testing that the tool can be installed without errors.To run the tests entirely in docker, use `run-tests-in-docker.sh` (experimental).## Alternatives- Share a build cache between all image building machines and make sure the cache is not lost.- More exotic Docker image builders might store information about the build context with the built image.