Package: reproman
ReproManReproMan aims to simplify creation and management of computing environments
in Neuroimaging.  While concentrating on Neuroimaging use-cases, it is
by no means is limited to this field of science and tools will find
utility in other fields as well.StatusReproMan is under rapid development. While
the code base is still growing the focus is increasingly shifting towards
robust and safe operation with a sensible API. There has been no major public
release yet, as organization and configuration are still subject of
considerable reorganization and standardization.SeeCONTRIBUTING.mdif you are interested in
internals and/or contributing to the project.InstallationReproMan requires Python 3 (>= 3.6).Linux'es and OSX (Windows yet TODO) - via pipBy default, installation via pip (pip install reproman) installs core functionality of reproman
allowing for managing datasets etc.  Additional installation schemes
are available, so you could provide enhanced installation viapip install 'reproman[SCHEME]'whereSCHEMEcould betests
to also install dependencies used by unit-tests battery of the repromanfull
to install all of possible dependencies, e.g.DataLadFor installation throughpipyou would need some external dependencies
not shipped from it (e.g.docker,singularity, etc.) for which please refer to
the next section.Debian-based systemsOn Debian-based systems we recommend to enableNeuroDebianfrom which we will soon provide recent releases of ReproMan.  We will also provide backports of
all necessary packages from that repository.DependenciesOursetup.pyand corresponding packaging describes all necessary dependencies.
On Debian-based systems we recommend to enableNeuroDebiansince we use it to provide backports of recent fixed external modules we
depend upon.  Additionally, if you would
like to develop and run our tests battery seeCONTRIBUTING.mdregarding additional dependencies.A typical workflow forreproman runThis example is heavily based on the"Typical workflow"example created for///repronim/containerswhich we refer you to discover more about YODA principles etc.  In this reproman example we will
follow exactly the same goal -- running MRIQC on a sample dataset -- but this time utilizing
ReproMan's ability to run computation remotely. DataLad and///repronim/containerswill
still be used for data and containers logistics, while reproman will establish a littleHTCondorcluster in the AWS cloud, run the analysis, and fetch the results.Step 1: Create the HTCondor AWS EC2 clusterIf it is the first time you are using ReproMan to interact with AWS cloud services, you should first provide
ReproMan with secret credentials to interact with AWS. For that edit its configuration file
(~/.config/reproman/reproman.cfgon Linux,~/Library/Application Support/reproman/reproman.cfgon OSX)[aws]
access_key_id = ...
secret_access_key = ...Disclaimer/Warning: Never share or post those secrets publicly.filling out the...s.  Ifrepromanfails to find this information, error messageUnable to locate credentialswill appear.Run (need to be done once, makes resource available forreproman loginorreproman run):repromancreateaws-hpc2-taws-condor-bsize=2-binstance_type=t2.mediumto create a new ReproMan resource: 2 AWS EC2 instances, with HTCondor installed (we useNITRC-CEinstances).Disclaimer/Warning: It is important to monitor your cloud resources in the cloud provider dashboard(s)
to ensure absent run away instances etc. to help avoid incuring heavy cost for used cloud services.Step 2: Create analysis DataLad dataset and run computation on aws-hpc2Following script is an exact replica from///repronim/containerswhere only thedatalad containers-runcommand, which fetches data locally and runs computation locally and serially, is replaced withreproman runwhich publishes dataset (without data) to the remote resource, fetches the data, runs computation
via HTCondor in parallel across 2 nodes, and then fetches results back:#!/bin/sh(# so it could be just copy pasted or used as a scriptPS4='> ';set-xeu# to see what we are doing and exit upon error# Work in some temporary directorycd$(mktemp-d${TMPDIR:-/tmp}/repro-XXXXXXX)# Create a dataset to contain mriqc outputdataladcreate-dds000003-qc-ctext2gitcdds000003-qc# Install our containers collection:dataladinstall-d.///repronim/containers# (optionally) Freeze container of interest to the specific version desired# to facilitate reproducibility of some older resultsdataladrun-m"Downgrade/Freeze mriqc container version"\containers/scripts/freeze_versionsbids-mriqc=0.16.0# Install input data:dataladinstall-d.-shttps://github.com/ReproNim/ds000003-demosourcedata# Setup git to ignore workdir to be used by pipelinesecho"workdir/">.gitignore&&dataladsave-m"Ignore workdir".gitignore# Execute desired preprocessing in parallel across two subjects# on remote AWS EC2 cluster, creating a provenance record# in git history containing all condor submission scripts and logs, and# fetching them locallyrepromanrun-raws-hpc2\--subcondor--orcdatalad-pair\--jp"container=containers/bids-mriqc"--bpsubj=02,13--follow\--input'sourcedata/sub-{p[subj]}'\--output.\'{inputs}'.participantgroup-wworkdir--participant_label'{p[subj]}')ReproMan: Executedocumentation section
provides more information on the underlying principles behindreproman runcommand.Step 3: Remove resourceWhenever everything is computed and fetched, and you are satisfied with the results, usereproman delete aws-hpc2to terminate
remote cluster in AWS, to not cause unnecessary charges.LicenseMIT/ExpatDisclaimerIt is in a beta stage -- majority of the functionality is usable but
Documentation and API enhancements is WiP to make it better.  Please do not be
shy of filing an issue or a pull request. SeeCONTRIBUTING.mdfor the guidance.