Package: scicommander
SciCommanderThis is a small tool that executes single shell commands in a scientifically
more reproducible and robust way, by doing the following things:Auditing: Creating an audit log of most output filesCaching: Skipping executions where output files already exist(Coming soon): Atomic writes - Writes files to a temporary location until
command is finishedRequirementsA unix like operating system such as Linux or Mac OS (On Windows you can useWSLorMSYS2)Python 3.6 or higherA bash shellFor graph plotting for the HTML report, you needGraphVizand itsdotcommand.InstallationpipinstallscicommanderThis will install thescicmdcommand into yourPATHvariable, so that it
should be executable from your shell.UsageTo view the options of thescicmdcommand, execute:scicmd-hTo get the benefits from SciCommander, do the following:Prepend all your shell commands with thescicmd -ccommand.Wrap the command itself in quotes, either""or''. This is not strictly
required always, but will be required for example if using redirection using>or piping with|(Alternatively one can just add quotes around those).Wrap definitions of input fields in{i:INPATH}and output files in{o:OUTPATH}for output paths.You can also just prepend input paths withi:and output paths witho:,
but this is a slightly less robust method, that might fail to wrap the
correct number of characters in some situations.Then run your script as usual.Now you will notice that if you run your script again, it will skip all
commands that have already finished and produced output files.You will also have files with the extension.au.jsonfor every output that
you decorated with the syntax above.To convert such an audit report into a nice HTML-report, run the following:scicmd--to-html<audit-file>ExampleTo demonstrate how you can use SciCommander, imagine that you want to write the
following little bioinformatics pipeline, that writes some DNA and converts its
reverse complement, as a shell script,my_pipeline.sh:#!/bin/bash# Create a fasta file with some DNAechoAAAGCCCGTGGGGGACCTGTTC>o:dna.fa# Compute the complement sequencecati:dna.fa|trACGTTGCA>o:dna.compl.fa# Reverse the DNA stringcati:dna.compl.fa|rev>o:dna.compl.rev.faNow, to make the commands run through SciCommander, change the syntax in the
script like this:#!/bin/bash# Create a fasta file with some DNAscicmd-cechoAAAGCCCGTGGGGGACCTGTTC'>'o:dna.fa# Compute the complement sequencescicmd-ccati:dna.fa'|'trACGTTGCA'>'o:dna.compl.fa# Reverse the DNA stringscicmd-ccati:dna.compl.fa'|'rev'>'o:dna.compl.rev.faNotice how all input paths are prepended withi:and output paths witho:,
and also that we had to wrap all pipe characters (|) and redirection
characters (>) in quotes. This is so that they are not grabbed by bash
immediately, but instead passed with the command to SciCommander, and executed
as part of its execution.Now you can run the script as usual, e.g. with:bashmy_pipeline.shNow, the files in your folder will look like this, if you list them withls -tr:my_pipeline.sh
dna.fa.au.json
dna.fa
dna.compl.fa.au.json
dna.compl.fa
dna.compl.rev.fa.au.json
dna.compl.rev.faNow, you see that the last.au.jsonfile isdna.compl.rev.fa.au.json.To convert this file to HTML and view it in a browser, you can do:scicmd--to-htmldna.compl.rev.fa.au.jsonThen you will seean HTML page like thisExperimental: Bash integrationThere is very early and experimental support for running SciCommander commands
in bash, without needing to run them via thescicmd -ccommand.To do this, start the SciCommander shell with the following command:scishellAnd then, you can run the example commands above as follows:# Create a fasta file with some DNAechoAAAGCCCGTGGGGGACCTGTTC>o:dna.fa# Compute the complement sequencecati:dna.fa|trACGTTGCA>o:dna.compl.fa# Reverse the DNA stringcati:dna.compl.fa|rev>o:dna.compl.rev.faIn other words, only thei:ando:markers are now needed, and no extra
syntax.Notes[1] Although Nextflow and Snakemake already take care of some of the benefits,
such as atomic writes, SciCommander adds additional features such as detailed
per-output audit logs.