Package: hera-opm
hera_opmhera_opmprovides a convenient and flexible framework for developing data
analysis pipelines for operating on HERA data. It facilitates "offline
processing", and is portable enough to operate on computer clusters with
batch submission systems or on local machines.How It WorksThehera_opmpackage uses themakeflowsystem, which is a part of theCooperative Computing Tools
packagedeveloped by theCooperative Computing Lab. Thehera_opmpackage
essentially converts a pipeline defined in a configuration file into a format
that can be parsed bymakeflow. This process is also aware of aspects specific
to HERA data, such as the polarization features of the data, in order to build
an appropriate software pipeline. Once themakeflowinstructions file has been
generated, themakeflowprogram itself is used to execute the steps in the
pipeline.There are generally 5 steps required to "build a pipeline":Writetask scriptsthat will be executed bymakeflowfor a given stage in
the pipeline. These scripts should generally be as atomic as possible, and
perform only a single logical component of a pipeline (though it may in turn
call several supporting scripts or commands).Write aconfiguration filewhich defines the order of tasks to be
completed. This configuration file defines the logical flow of the pipeline, as
well as prerequisites for each task. It also allows for defining compute and
memory requirements, for systems that support resource management.Use the providedbuild_makeflow_from_config.pyscript to build amakeflowinstruction file that specifies the pipeline tasks applied to the data files.Use the providedmakeflow_nrao.shormakeflow_local.shto execute the
pipeline in either the NRAO batch scheduler environment, or on a local machine,
respectively.(Optional) Use the providedclean_up_makeflow.pyto clean up the work
directory for makeflow. This will remove the wrapper scripts and output files,
and generate a single log file for all jobs in the makeflow.InstallationTo install thehera_opmpackage, simply:pip install .As mentioned above,hera_opmusesmakeflowas the backing pipeline management
software. As such,makeflowmust be installed. To installmakeflowin your
home directory:gitclonehttps://github.com/cooperative-computing-lab/cctools.gitcdcctools
./configure--prefix=${HOME}/cctools
makeclean
makeinstallexportPATH=${PATH}:${HOME}/cctools/binFor convenience, it is helpful to add theexportstatement to your.bashrcfile, so that themakeflowcommands are always on yourPATH.DependenciesWhen installing the package, setuptools will attempt to download and install any
missing dependencies. If you prefer to manage your own python environment
(through conda or pip or some other manager), you can install them yourself.Requiredtoml >= 0.9.4Optionalhera_calGenerating anlstbinpipeline (instead ofanalysis) requires that hera_cal
be installed. The main package and tests can be run without this requirement.Task Scripts and Config FilesFor documentation on building task scripts, seethe task scipts docs
page. For documentation on config files, seethe config
file docs page.Testinghera_opmusespytestas its testing framework. To run the test suite, do:pytestfrom the root repo directory. This may require runningpip install .[test]to
install testing dependencies.