Package: asimov
AsimovAsimov is a workflow management and automation platform for scientific analyses.Documentation·Releases·Issue TrackerAsimov was developed to manage and automate the parameter estimation analyses used by the LIGO, Virgo, and KAGRA collaborations to analyse gravitational wave signals, but it aims to provide tools which can be used for other workflows.Asimov has been used to organise and run the major catalogue analyses from the third observing run, O3, but it's designed to be flexible enough to allow new pipelines and analyses to be added to the framework.Branch notesThese notes relate to in-development features on this branch, and what's described here is only expected to be relevant during development.
More generally useful documentation will move to the main documentation before moving to production.Starting the logging serverRun inasimovdirectory:export FLASK_APP=server
flask runFeaturesJob monitoring and managementAsimov is able to interact with high throughput job management tools, and can submit jobs to clusters, monitor them for problems, and initiate post-processing tasks.Uniform pipeline interfaceAsimov provides an API layer which allows a single configuration to be deployed to numerous different analysis pipelines.
Current gravitational wave pipelines which are supported arelalinference,bayeswave,RIFT, andbilby.Centralised configurationAsimov records all ongoing, completed, and scheduled analyses, allowing jobs, configurations, and results to be found easily.Reporting overviewAsimov can provide both machine-readible and human-friendly reports of all jobs it is monitoring, while collating relevant log files and outputs.Results managementYour results are important, and Asimov provides special tools to help manage the outputs of analyses as well as ensuring their veracity.Do I need Asimov?Asimov makes setting-up and running parameter estimation jobs easier.
It can generate configuration files for several parameter estimation pipelines, and handle submitting these to a cluster.
Whether you're setting-up a preliminary analysis for a single gravitational wave event, or analysing hundreds of events for a catalog, Asimov can help.Installing AsimovAsimov is written in Python, and is available onpypi.
It can be installed by running$ pip install asimovIt is also available on conda, and can be installed by running$ conda install -c conda-forge ligo-asimovAsimov also requires that you havegitinstalled on your machine, and that you've set it up by running:$ git config --global user.email "you@example.com"
$ git config --global user.name "Your Name"Get startedAsimov supports a variety of different ways of running, but the simplest way, running a workflow on a local machine, can be set up with a single command.We start by setting up a project, which is a directory which keeps all of the analyses and the required metadata together.
A project can include just a single event or a whole selection of events, for example if you're producing a catalogue.
First create a directory to store your project:$ mkdir my-new-project
$ cd my-new-projectand then get asimov to set things up$ asimov init "Test project"where you can replace"Test project"with the name you want to give your project.
A project will be set-up in your current working directory.In order to start setting up analyses we next need to download some default settings.An analysis is a pipeline run, and asimov supportsbayeswave,bilbyin the default installation.We'll download the default configurations for jobs which are going to be run on the LIGO data grid.
We do this using theasimov applycommand, which pulls-in data from a file either locally or online.$ asimov apply -f https://git.ligo.org/asimov/data/-/raw/main/defaults/production-pe.yamland the load default priors the same way:$ asimov apply -f https://git.ligo.org/asimov/data/-/raw/main/defaults/production-pe-priors.yamlNow you can add an existing event, by downloading the event data settings using theasimov applyfunction, for example, to add GW150914 to the project you can run$ asimov apply -f https://git.ligo.org/asimov/data/-/raw/main/events/gwtc-2-1/GW150914_095045.yamlThis downloads the settings which were used for analysing GW150914 for the GWTC-2.1 catalogue paper, and stores them in the ledger file in the project (.asimov/ledger.yml).Many analyses can be run on a single event (these are called "productions" in asimov parlence).
We can add some pre-configured analyses by downloading some analysis configuration settings.$ asimov apply -f https://git.ligo.org/asimov/data/-/raw/main/analyses/production-default.yaml -e GW150914_095045Note that if you omit the-eargument asimov will ask which event the analyses should be applied to.You can now build and submit your jobs to the cluster.
First useasimov manage buildto create the configuration files for each analysis:$ asimov manage buildThese will be added to the git repositories in thecheckoutsdirectory inside the project directory.
You then submit the jobs to the cluster by running$ asimov manage submitIt will normally take a long time for a parameter estimation job to finish, but you can get asimov to check up on it by running$ asimov monitorIf the job is finishedasimovwill start post-processing usingPESummary, and if it's fallen over it will attempt to fix the problem and resubmit it to the cluster.
If the post-processing has finished it will move the results to a read-only directory.You can also set the asimov monitor to automatically check the status of the job every 15 minutes by running$ asimov startwhich will automatically start any post-processing, and start any jobs once their dependencies are met.
You can stop this automatic monitoring process any time by running$ asimov stopFor a full description of the workflow management process see the documentation.I want to help develop new features, or add a new pipelineGreat! We're always looking for help with developing asimov!
Please take a look at ourcontributors' guideto get started!RoadmapGravitic pipelinesWhile Asimov already supports a large number of pre-existing pipelines, and provides a straightforward interface for adding new pipelines, we also intend to support pipelines constructed usinggravitic, allowing experimental tools to be used without constructing an entire new pipeline, while also allowing asimov to manage the training of machine learning algorithms.Workflow replication, extension and duplicationAsimov will allow an existing workflow to be duplicated, in a similar way to agit clone, and then extended, with new jobs gaining access to the completed jobs in the workflow.
It will also allow entire workflows to be re-run, providing a straightforward way to replicate results, or make minor modifications.AuthorsAsimov is made by the LIGO, Virgo, and KAGRA collaborations.
The primary maintainer of the project is Daniel Williams.
Its development is supported by the Science and Technology Facilities Council.