Package: docker-playbook
docker-playbookYAML driven (nvidia) docker playbookRequirementsPython3.xNvidia-docker2 (Make sure you cannvidia-smianddocker images)PyYAML>=3.12 (both host and docker image)Template Dockerfile is provided at the bottom, which is optionalthough recommended because of itssecurity enhancementsanduseful features.Usage example./playbook.py sample.ymlFeaturesRuns withnvidia-docker2mostly in non-interactive mode (so if your script waits for input by accident, it gets signal and dies)Runs (whitelisted) steps in sequential mannerContext deduction: each step can run in different docker images or hostFull access to host network services in non-interactive modeSimple step function siginaturestep_blah(ctx)- easy to extendMinimal command line arguments./playbook.py some.ymlis all you doSingle source file & low dependency (only PyYAML is not in standard library, replacable but strongly recommend)Colorful logging for readabilityHow to add steps?Copylibrary.stepsmodule from this repository to start developing your ownlibrary.stepsmodule for this script to import.Add a functiondef step_something(ctx). Current execution context is inctxas dict. Keys are proxied to attributes to save a lot of brakets and quotes. You may useloggerto print info.defstep_something(ctx):logger.info(ctx.other_vars)Whitelist your step inlibrary.steps.whitlistwithout thestep_prefix.Add an entry to your YAML file instepswhereactionis the step function name without prefix:steps:
  - name: Some message here
    action: something
    other_vars: goes_to_ctxContext deduction rulesdocker overrides > step config > global config > parser defualts > context initializationHow to specify docker environment?You may add a default docker environment.
And usedocker_overridesto change context variables when docker is in use.docker:
  image: aleozlx/tkstack2:latest
  runtime: nvidia
  gui: False
  ports:
    - 6006:6006
  volumes:
    - /tmp:/workspace/build
    - /mnt/datasets:/workspace/datasets
  docker_overrides:
    storage: /workspace/datasets
steps:
  - name: Some message here
    action: something
    storage: /mnt/datasetsOr override the docker environment completely per stepdocker:
  # ...
steps:
  - name: Some message here
    action: something
    storage: /mnt/datasets
    docker:
      image: aleozlx/tkstack2:latest
      runtime: nvidia
      volumes:
        - /tmp:/workspace/build
    docker_overrides:
        storage: /workspace/datasetsOr use the hostdocker:
  # ...
steps:
  - name: Some message here
    action: something
    storage: /mnt/datasets
    docker: nullNote: When a docker environment is present, the playbook starts docker accordingly and resumes itself inside docker to reuse many of the playbooks' features, so that context deduction and logging have consistent behavior. (To do that, you code containing the playbook will have to be mounted read-only. This is automatic.) That's why PyYAML is also required inside your docker image. Consider you are keeping the full operationalizing mechanism with a silght dependency constraint.Security assumptions (or gotchas)Host file system: volumes specified in your playbook will be mounted RW. Although your code will be mounted RO automatically. Playbook assumes that you use a docker image that uses non-root user (like recommended) whose uid:gidhopefullymaps to you on host system.Network: network services inside docker arenot isolatedfrom host in non-interactive mode to provideconvenient accessto host databases etc. Playbook assumes whatever you are operationalizing is trusted and that your host should have a proper set of INPUT rules, and that services inside docker should be protected by an independent firewall if necessary.X11: the recommended docker image does intend to provide isolated X11 access by creating non-root user thatpresumablymaps to you on host and your X authentication files are natually mounted with proper permissions already in place. But the uid in that image is just a guess so on a host shared by many users, this is probably no good if it even works. (Docker image needs to be build in a better way)Playbook itself: the playbook itself is obviously a very capable shell program. It has execute permission by default for convenience (not necessarily good idea). It is based on a simple whitelist to allow any actions to be executed. The docker shell provided shouldn't be able to be spawned on host. But all these are best-effort, no promises.At the end of day, this is an under-2^7-line script that is supposed to just drive other things and make them work. Composition over inheritance principle makes things simpler and useful.Template Dockerfilehttps://gitlab-p01.cgi.missouri.edu/ExternalUsers/AlexYang_Fun/tkstack2/blob/master/src/DockerfileFeaturesNon-root with passwordless sudoLow maintenance: base image istensorflow/tensorflow:1.8.0-devel-gpu-py3Numpy + Scipy linked against libblas + liblapackIsolated X11 GUI supportDetails athttps://gitlab-p01.cgi.missouri.edu/ExternalUsers/AlexYang_Fun/tkstack2LicenseLicensed under either ofApache License, Version 2.0, (LICENSE-APACHEorhttp://www.apache.org/licenses/LICENSE-2.0)MIT license (LICENSE-MITorhttp://opensource.org/licenses/MIT)at your option.