Package: copili
copili - container pipelineRun a series of containers, in a coordinated mannerMaintainer:tim.bleimehl@dzd-ev.deLicence: MITissue tracker:https://git.connect.dzd-ev.de/dzdtools/pythonmodules/-/issues?label_name%5B%5D=copiliHINT: This Readme is WIP. Expect changes and additions![[TOC]]What?copiliis a python tool to run a series of scripts that are wrapped into a docker container/image.You can create pipelines based on containers with central definitions. The pipeline definition supports yaml,json, python-dict.copiliwill manage the runs of docker containers;manage dependencieshandle failed runsmanage periodic runsmanage log(-files)Example Scenario & Backgroundcopiliwas created for developing a dataloading pipeline for theCovid*Graph, a Covid19 knowledge graph around a Neo4j database.In Covid*Graph we have contributions, from many developers in diverse programming languages, to load data into the database; So calleddataloaders.To reproducable bootstrap the graph and create the needed environment for each dataloader we put all the dataloader scripts into docker images.At the beginning we started the containers sequentially, but with a growing count of dataloaders and more complex dependencies among those dataloaders, a manual execution was not feasible anymore.Here comescopiliinto the game:Withcopiliwe can define a sequence of containers and the dependencies among them.copiliis the base library formotherlodeWithmotherlodewe now can rebuild the graph from scratch. we just need to startcopili/motherlodewith our pipeline definition, which lives in a yaml file.Now everybody can easily get an overview how the graph is created or create a local copy of the graph. This is important for an open source community project to make lives of the the developers easier.Also we can now add new dataloaders with no effort.On top we can create "service" definitions which automatically update our knowledge graph. More on that in the docs...UsageInstallStablepip3 install copiliDevpip3 install git+https://git.connect.dzd-ev.de/dzdpythonmodules/copili.gitGet startedQuick exampleSee this short example to get an idea how copili works. After that we will go into more detail.importdockerimportschedulefromcopiliimportPipelined=docker.DockerClient(base_url="unix://var/run/docker.sock")# pipelindata - this could be also a path to a yaml-,json-file or just a python dictpipeline_description_yaml="""ExmaplePipeline:- name: dataloader_02image_repo: stakater/exit-containerdependencies:- dataloader_01env_vars:EXIT_CODE: 0- name: dataloader_01image_repo: stakater/exit-container- name: dataloader_03image_repo: stakater/exit-containerdependencies:- dataloader_02- dataloader_01- name: servicecontainer01image_repo: hello-worldis_service_container: truedependencies:- dataloader_02"""p=Pipeline(description=pipeline_description_yaml,docker_client=d)# run all containers oncep.run()# Optional define custom service schedule (https://schedule.readthedocs.io)# default is once a day at 00:00p.service_schedule=schedule.every(10).minutes.do(p.run_service_containers)# Step into service modep.start_service_mode()# now servicecontainer01 will run every 10 minutesPipeline description formatA pipeline defintion consist of a name and an array of container descriptions. These container descriptions can have dependencies among each other.
Container descriptions can be provided as python dict or as a json/yaml string or file.A pipeline description will be overhanded to copili via thecopili.Pipeline-descriptionparametere.g.importcopilip=Pipeline(description="path/to/my/pipelinefile.json")Container description propertiesOne container description can have following propertiesnameName of the container description. Serves as identifier within copili.MandatoryType(python/json/yaml)DefaultExample Value(s)TruestringNoneMY_FIRST_PIPELINE_CONTAINERinfo_linkLink to the code repository or some other info about the pipeline memberMandatoryType(python/json/yaml)DefaultExample Value(s)TruestringNonehttps://github.com/me/myrepodescShort description of the pipeline memberMandatoryType(python/json/yaml)DefaultExample Value(s)TruestringNoneLoads stuff into the databaseimage_repoName of the repo where copili can download the image from. Usually a dockerhub repo. Custom repo urls are supportedMandatoryType(python/json/yaml)DefaultExample Value(s)TruestringNonemy-docker-namespace/my-container,my-own-registry.com:443/my-own-namespace/my-containerimage_reg_usernameIf we need to authorize to download the image from a certain registry, we can pass a username here (SECURITY HINT: Environment variables are supported as well and should be used here)MandatoryType(python/json/yaml)DefaultExample Value(s)FalsestringNonemy-username,${USERNAME-FROM-DOT-ENV_FILE}image_reg_passwordIf we need to authorize to download the image from a certain registry, we can pass a password here (SECURITY HINT: Environment variables are supported as well and should be used here)MandatoryType(python/json/yaml)DefaultExample Value(s)FalsestringNonemy-password,$PASSWORD-FROM-SYSTEM-ENV-VARtagThe tag of the imageMandatoryType(python/json/yaml)DefaultExample Value(s)Falsestringlateststable,beta01,yetanothertagis_service_containerDoes the container run once per pipeline run or should it run periodically (if the pipeline enters service mode). Ssetypedfor more detailsMandatoryType(python/json/yaml)DefaultExample Value(s)FalseboolFalseTrueenv_varsProvide customenvironment variablesper containerMandatoryType(python/json/yaml)DefaultExample Value(s)Falsedict/json-object/record{}{'MY_ENV_VAR':'value01',MY_OTHER_ENV_VAR:'val02'}labelsAttach docker labels to the container.MandatoryType(python/json/yaml)DefaultExample Value(s)Falsedict/json-object/record{}{'my-super-label':'my-super-value','stuff.company.com/enabled':"true"}dependenciesProvide a list of copili container description **name*s which need to run successfull before this container is allowd to runMandatoryType(python/json/yaml)DefaultExample Value(s)Falselist of strings[]['NAME_OF_OTHER_CONTAINER','NAME_OF_ANOTHER_CONTAINER']exlude_in_envSkip this container if we run in a certain environment. Set environment variableENVto set the environmentMandatoryType(python/json/yaml)DefaultExample Value(s)Falselist of strings[]['PROD','QA']volumesA volumes desc. The format is given by thepython-docker-sdk. Seevolumes-parameterMandatoryType(python/json/yaml)DefaultExample Value(s)Falsedict/json-object/record{}{"/tmp/data": {"bind": "/data/", "mode": "rw"},{'/home/user1/': {'bind': '/mnt/vol2', 'mode': 'rw'},'/var/www': {'bind': '/mnt/vol1', 'mode': 'ro'}}commandDockercommandlist. Similar todocker composecommandMandatoryType(python/json/yaml)DefaultExample Value(s)Falselist of strings[]['-p' ,'3000']sidecarsStart helper containers with your container. E.g. if your container needs a redis database for cachingMandatoryType(python/json/yaml)DefaultExample Value(s)Falselist of container descriptions[][{"name": "redis01", "image_repo": "redis"}]force_rerunSkip all checks if container can be skipped.MandatoryType(python/json/yaml)DefaultExample Value(s)FalseboolFalsetruejson-Pipeline DescriptionTo provide a pipeline description via json, provide a json object starting with a name and the list of container descriptions{"my-pipeline-name":[{"name":"my-first-container","repo":"hello-world"}]}This will run the containerhello-worldonce, when the pipeline is started.Now, lets add another dependecy that is only allowed to run, if our hello world container ran successfully:{"my-pipeline-name":[{"name":"my-first-container","repo":"hello-world"},{"name":"my-second-container","repo":"chentex/random-logger","dependency":["my-first-container"]}]}This again will run ourhello-worldcontainer and after that thechentex/random-loggercontainer.It should be noted, the order of the container desciptions in the list does not matter for the dependencies. copili figures our the needed sequence itself.Now, lets add a sidecar container to our second container{"my-pipeline-name":[{"name":"my-first-container","repo":"hello-world"},{"name":"my-second-container","repo":"chentex/random-logger","dependency":["my-first-container"],"sidecars":[{"name":"redis01","repo":"redis"}]}]}This again will run ourhello-worldcontainer and after that thechentex/random-loggercontainer.
But additionally with the second container arediscontainer will be started. This can be helpful for containers that need this as a caching database for example.yaml-Pipeline DescriptionSame rules apply for yaml pipeline descriptions as forjson.Json follows the same structure as yaml and is just another way of formating the same informations. seehttps://www.json2yaml.com/Also have a look at thequick start example, which is provided in yaml formatContainer description typesvia the propertyis_service_containerwe can define if a container is static or service container.staticA static container will run only once when pipeline is started.
If you want to run the container only once on first pipeline run you have to setcopili.Pipeline.container_did_run_check_override_callbackand provide the information if a container already ran (e.g. from a database)serviceContainer will run periodicallyEnvironment Variable SupportYou can use (environment variables)[https://en.wikipedia.org/wiki/Environment_variable] in the pipeline description.Either just by setting system env vars (e.g.EXPORT MYPASSWORD=hello123) or by passing a .env file viaPipeline classsee codeDesc still missing... todoContainerManager classsee codeAttributesImage
Instance ofdocker.models.images.Image. The image the container will run onContainer
Instance ofdocker.models.containers.Container. The actual python representation of the docker containerexit_codeNoneas long the container did exited.0if the container run successfull. > 0 if the container failed to run..ToBeCompletedCallback / Function overridesYou can override these functions to modifiy the behaviour of yourcopiliinstancecopili.Pipeline.container_pre_pull_callback(copili.ContainerManager)Will be called before the image for the container is pulledcopili.Pipeline.container_pre_run_callback(copili.ContainerManager)Will be called before the containers is started. Runs only if container is not skippedcopili.Pipeline.container_post_run_callback(copili.ContainerManager)Will be called after the containers exited. Runs only if container is not skippedcopili.Pipeline.container_pre_processing_callback(copili.ContainerManager)Will be called before the containers is startedcopili.Pipeline.container_post_processing_callback(copili.ContainerManager)Will be called after the containers exitedcopili.Pipeline.container_did_run_check_override_callback(copili.ContainerRegistryItem) -> BoolWill be called before the container is started. if functions returns 'False' container run will be skippedcopili.Pipeline.container_dependency_check_override_callback(copili.ContainerManager, List[copili.ContainerManager]) -> BoolWill be called before the container is started. if functions returns 'False' the current dependency branch will be stopped. Can be used for checking if all previously runned containers accomplish all dependencies.

If set to None `copili` checks the dependencies by recognizing that all containers which are in `copili.ContainerRegistryItem.dependencies` ran with exit code `0`. 

If you need a more sophisticated dependency check, use this function. (e.g. a check which takes the state of previous pipeline runs in account and these state informations are stored in an external database)..ToBeCompletedDevelopementgit clone ssh://git@git.connect.dzd-ev.de:22022/dzdpythonmodules/copili.gitpip install -e .ToDo:Custom schedules per service containerAlternative to an docker image a git repo with Dockerfile can be provided which will be build and runreplace service-containers concept with amax_ageattribute per container desc. when a container did not run a certain time its allowed to rerun. much more simple...