Package: dockerutils
Docker Utilities/PatternsDockerutils is a set of utilities and conventions around their use. The intent behind these utilities is to provide a
very light layer of abstraction to: simplify interaction with docker; support seamlessly running docker locally or on
ec2 instance in AWS; allow for multiple images per project; etc.Perhaps the best way to think of dockerutils is that it embodies two entities, commonly used when working
with docker, and a set of complementary commands for working with those entities. The two entities are:"dock" - the server that is hosting docker (by default, localhost)image - the standard docker imageThe commands used to operate against these entities are:CommandSet /  EntityCreationExecutionJupyterNotebookUtilityDockcreate-dockdestroy-dockstart-dockstop-docknb-docksource dockls-dockssh-dockImagebuild-imagerun-imagerun-notebookpublish-imagetransfer-imagePossible use cases include:seperating development/test dependencies out of production container, e.g. production container vs. dev/test containerseperating data science notebook container from execution containerenvironment experimentationcases in which you want to "freeze" any external dependencies in one container and use that as a base
for containers that are dependent solely on the projectConventionsCreate a docker directory tree at the root of the projectIn this directory tree there should be one sub-directory for each unique docker container type that is desired.
Each of these sub-directories would contain theDockerfilethat will be used to create the image
as well as any source specific to that image.Use versioneer for project versioning (Optional).As part of the image build, a file,_version.py.bld, will be generated and placed at the project
root. ADockerfilecan add that file to the image on creation to prevent the need for including the
.git directory tree in the container context (usually quite expensive).Create a docker/base directory to make use of built in external dependency isolation (optional)This capability supports environments where a docker build isn't able to access external dependencies (Docker Hub,
pypi, etc.), for instance a server in a "locked-down" environment. A base image can be defined to isolate any
dependencies that are required. That image can then be built andtransfer-imageused to transfer the base image
to the target environment.Subsequent images can be built based off of that image that are "self-contained" (relying only on source
from the project). The remote docker api can then be used to quickly iterate only requiring the more
cumbersome transfer-image to be used when external dependencies change.Building and running images controlled through configuration (<project_dir>/docker/dockerutils.cfg)Includes setting most docker parameters, i.e. volume mounts, ports, networks, commands, etc. with
replacement varilable support for things like user, project root, etc.Command-line InterfaceImage clibuild-imagetakes the name of one of the sub-directories in thedockerdirectory and builds the
image defined therein. The image is named <project>-<subdir>:<user>run-imagetakes the name of one of the sub-directories (or one of the synthetic images defined indockerutils.cfg),
together with any of the configuration for that image defined indockerutils.cfgand starts a docker containertransfer-imagetakes a docker image name and uses dockersaveandloadto transfer the image to a remote hostpublish-imagetakes the name of one of the sub-directories in thedockerdirectory and pushes the image built by
the docker file to the defined repository (AWS or Docker)Notebook clirun-notebookwill start a docker container using either the notebook container found in thedocker/notebookdirectory
if it exists, orresero-labs/docker-dsotherwise. The current directory will be mounted
into the container for use in the Juypter notebook environment. There are a couple of environment variable to be aware of
with this command:DOCKER_DS_DONT_PULL - if set, the version of resero-labs/docker-ds currently available will be used rather than pulling
the latest version from docker hub.RESERO_JUPYTER_DIFFS - if set, on save,.pyfiles and.htmlfiles for the notebook will be created in a.diffssubdirectory.Dock cliA "dock" is a remote system that you can connect to throughssh. You can "dock" your terminal to a remote instance and
any docker commands, including image and notebook cli above will be run against the remote docker server. Once a "dock"
is created, you can dock your terminal by issuing the commandsource dock <server IP or moniker>create-dock(register-dockif provisioning from AWS console) is used to add a remote system to the dock list with all its configuration (username, ip and a moniker)destroy-dock(unregister-dockif provisioned from AWS console) is used to remove the reference to the remote systemstop-dockwill change the instances state of a remote dock tostoppedstart-dockwill change the instance state of a remote dock torunningssh-dockopens a terminal on the remote dock with sshls-docklist (including state) any created docksnb-dockrun jupyter on the bare AMI of the dock and open a browser window to the notebook serverdockerutils.cfgFormatConfiguration indocker/dockerutils.cfgis used to configure behavior of thedockerutilsscripts.Image SectionThedockerutils.cfgfile format allows for configuration sections named corresponding to the sub-directories in the
docker directory tree. Each of these sections may contain one of the following:environment- in the form of (-e VAR=value)+ to pass environment into the containerinteractive- either -e or -itgpu- conforms to NVIDIA Docker 2.0 specification, e.g.--runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=all,run-image -gwill add this automaticallynetwork- the name of the network for the containervolumes- in the form of (-v <host-dir>:<container-dir>)+ or (--mount ...)+ or bothports- in the form of (-p <host-port>:<container-port>)+cmd- any valid DockerCMDspecificationpull_FROM_on_force- defaults to False, if True, add --pull to build command when force building image (or base image)image_repo- the repository to publish the image topublication_tag- the tag for publication (full image name + tag)pre_build_script- A shell command or script to run before a docker build is issuedpost_build_script- A shell command or script to run after a docker build has been compeleted (successfully)Synthetic ImagesAdditionally, "synthetic" images can be specified by adding arun-imagesection with asynthetic_imagesdefinition
that contains a list of "synthetic" images. Each of these may also have a named section as defined for the docker
sub-directories, but must also contain anamevalue that resolves to one of the docker sub-directories. For example:[run_image]
synthetic_images=shell

[shell]
name=dev
...Configuration-only ImagesIf there is a docker container that does what you want already, you can create a configuration-only image by
specifyingname,tagandprefix=Falsein the configuration section for the image. For example the base notebook
imageresero-labs/docker-dsis often sufficient for running a Jupyter notebook against your code, as it auto detects asetup.pyupon container start and installs the module into the notebook environment.Image TaggingThe default tag for any image created/run/etc. is the user name in the host environment when running the
utility. This can be overriden by adding atagvalue to the desired section. For example:[dev]
tag=experiment.2017.12.16
...Volume Replacement VariablesThe volume specification may contain either environment variables ($nameand${name}formats) as well as specific
variable replacement designations of the form{var}.  The supported variables include:project_root- will be replaced with the root directory name of the projectuser- will be replaced with the user name of the user running the commandproject- will be replaced with project namehome- will be replaced with the user's home directoryImage Push Replacement VariablesThepublication_tagmay contain either environment variables ($nameand${name}formats) as well as specific
variable replacement designations of the form{var}.  The supported variables include:account- AWS account designationregion- AWS regionimage- Image nametag- Image taguser- will be replaced with the user name of the user running the commandPublish to AWSIn order to publish to AWS, the repository will be the image name and the following should
be configured for images that are published to AWSimage_repo=aws
publication_tag={account}.dkr.ecr.{region}.amazonaws.com/{image}:{tag}"PatternsRunning your code in container, making live modifications outside container in your editor of choiceIf you're like me, you have a whole set of tools in your host environment that you use to work with your project.
One of the disadvantages of working with Docker can be the difficulty of transplanting those tools into the container
environment. Perhaps there is a way to have your cake and eat it too!
Thedevexample does a reasonable job of doing just this.With this pattern, you create aDockerfilethat has everything in the imageexceptfor your project source. An
emptyWORKDIRis created and thenENTRYPOINTeven does apip install -eof the contents of the emptyWORKDIR.
We get the desired results by mounting the source project directory into the container'sWORKDIR(seedevsection
ofdocker/dockertuils.cfg).With this pattern you can run tests, experiment, etc. in container, make changes to the project in your host
environment toolset and immediately observe the changes that were made.Working with a server in a locked-down environmentYou may find yourself in a situation in which you need to work with a server hosting Docker in an environment that has
limited access to the "outside world". This pattern can be used to capture all external dependencies in a base
image that is built in an environment that is open, usetransfer-imageto send this base image to the server and
then utilize a derived image dependent just on project sources and the base image to iterate without requiring open
access on the server.Adding test frameworks, code analysis tools, etc. to a container for testing/validationTensorflow for both CPU and GPU in the same containerIt is sometimes useful to try both the CPU and GPU versions of Tensorflow. The example indocker/tensorflowprovides
a pattern to do so. All tensorflow depdendencies are installed into global python site-packages. Then virtual
environments are created for both cpu and gpu versions and the appropriate version of tensorflow is install into
the respective virtual environments.Therun-imagescript makes use of NVIDIA Docker 2.0 being installed on the host os. When run with the-goption,
the NVIDIA runtime will be used to make GPUs available in container.Upon container startup, the appropriate virtual environment (cpu or gpu) will be activated dependant upon the-goption.To switch between virtual envrionments utilized the symbolic links,/cpu-envand/gpu-env, e.g.source /cpu-env.Versioneer supportIf you aren't usingversioneerto version your Python projects, you
should be. Versioneer utilizes .git to determine the project version. Having correct version information in container
is desireable in many cases, but pushing .git into container usually isn't desireable. To capture version information
for container, thegenversionutility is available and uses capabilities of versioneer to generate a _version.py.bld
file in the project root. This file is available forDockerfileto add/overwrite the version.py file within the
Docker image. An example of this is found in the cicd sub-directory..dockerignoreIn order to minimize the context sent to Docker to build images, please see examples in.dockerignorein this
repository.A Note on ImplementationWe selected to use the Docker command line from python scripts rather than theDocker
API available to Pythonas it integrates better into my development loop. As each CLI prints
the Docker command it's using, if it does something unexpected, it's each to copy and paste
the command used, modify it and be on your way without having to debug through CLI utility code,
allowing bugs/additions to the CLI to be addressed at a later point outside the context of a
project development loop.