Package: hubmap-pipeline-release-mgmt
HuBMAP pipeline release managementOverviewThis package provides a convenience script which automates some aspects of
tagging and releasing production-ready versions of HuBMAP computational
analysis pipelines.InstallationRunpython3-mpip installhubmap-pipeline-release-mgmt.UsageOnce the package is installed, navigate to a repository containing a
pipeline, and ensure that the main branch is what you would like
to release as a new tagged version, including thecommittedstate of any
submodules. (SeeConfigurationfor setting persistent configuration
parameters globally or for each pipeline, including the name of the “main”
branch.)Choose a new version number (preferably starting withv), likev1.0,
and run:tag_release_pipeline v1.0Most of the script is automated, but Git will ask you for a tag message (by
opening a text editor) unless a tag message is given as an argument to this
script via the--tag-messageargument.To sign Git tags with GPG, append--sign(and if you want to sign with
a non-default key, add--sign=preferred@email.address).(Your local main branch can be behind or ahead of its remote version
– if behind, it will be updated withgit pull, and if ahead
the remote branch will be updated withgit push. Your main branch
and its remote version cannot havediverged, however;tag_release_pipelinewill abort if this is the case.)Thetag_release_pipelinescript makes several assumptions about the state
of your repository, and if these assumptions are violated, the script will
probably fail loudly and leave your local copy in an arbitrary state. Make sure
you have no local modifications for best results (though you shouldn’t anyway,
if preparing a release version of a pipeline).At a high level,tag_release_pipelinedoes:Checkout the main branch, pull/push so it and its remote version matchCheckout or create a release branch,git pull--ff-onlyif checking out
a local branch that already existsSync the main branch to the release branch – note that this isnota
merge; the previous contents of the release branch are overwritten entirelyUpdate the content of all submodules to match the versions committed in the
main branchBuild all Docker images indocker_images.txt, using themulti-docker-buildpackageTag all images aslatestand with the new tag namePush all Docker images/tags to Docker HubUpdate all CWL files to use tagged versions of any images built from the
pipeline repository (i.e.those listed indocker_images.txt)Commit the updated CWL files (on the release branch)Tag the new commit, signed or notPush the main and release branches, and the new tagOptions:--pretendDon’t run anything that would make any modifications to any Git
repositories or Docker images. This will still rungit branch-ato obtain the list of Git branches, however.
This will print all commands which would be run.--tag-message(alias:-m) Use this string as the tag message. This is
given to Git as the-margument togit tag, which stops
Git from asking for a tag message interactively.--signSign the new tag with GPG using your default identity.--sign=identitySign the new tag with GPG, using the specified
identity (email address).--no-pushDon’t push anything to Docker Hub or the Git remote repository.
Everything will be committed, tagged, and built locally.--main-branchName of the main branch. Overrides the default (master)
and anything found in configuration files.--release-branchName of the release branch. Overrides the default (release)
and anything found in configuration files.--remote-repositoryName of the remote repository. Overrides the default
(origin) and anything found in configuration files.ConfigurationThis package uses theconfuselibrary to read user and pipeline configuration. The default configuration
specifies branch names, the remote repository name, and whether to sign each
release version of a pipeline, via the following contents ofconfig_default.yaml:main_branch: master
release_branch: release
remote_repository: origin
sign: falseThis configuration can be overridden globally (affecting all usage of this
package) and separately for each repository. These configuration parameters
are read in this order, with each source overriding earlier ones:Package default configuration shown aboveGlobal (user) configuration from~/.config/hubmap_pipeline_release_mgmt/config.yaml(on Linux)Pipeline configuration options, frompipeline_release_mgmt.yamlin the
base directory of the pipeline repositoryCommand-line arguments passed to thetag_release_pipelinescriptFor example, to sign all Git tags by default with a specific GPG key, you could
create the user configuration file noted above, containing:sign: mruffalo@cs.cmu.eduThe defaultmain_branchofmasteris likely to change in the near future.RequirementsPython 3.6 or newer.The following package dependencies should be automatically installed when
installing viapiporpython setup.py install:Version 0.7.1 or newer of themulti-docker-buildPyPI packageconfuse, (recent) version
unimportant