Package: batchx-dev
Welcome toBatchX' python toolbox, or"the toolbox"for short.
The toolbox makes the life of BatchX (python) developersmore pleasant.
To do so, it provides tools that help devs to work more effectively.Table of ContentsInstallationLocalInside Docker imageThe manifest as single source of truthbx-readmeCommandBuilderConnecting CLI parameters to BatchX parametersUnderstanding parameter typesAdd actionSpecify constraintsInstallationThe toolbox is available onPyPi.LocalSimple: do apip installpython-mpipinstallbatchx-devFor reproducibility, it often makes sense to 'lock' yourself into a specific version.
This can be done as follows:python-mpipinstallbatchx-dev==1.0.63Inside Docker imageInstallation of the toolbox inside the docker image of a BatchX tool isalmostidentical to a local install, except for the fact that thepip installcommand now has to be part of the Dockerfile.
For the sake of argument, we assume that you are working on bringing a bioinformatics tool calledbiotoolinto BatchX, and that your directory structure looks like this,biotool
├── Dockerfile
├── run_biotool.py
└── manifest
    ├── manifest.json
    ├── picture.png
    └── readme.mdSuppose that, initially,biotool/Dockerfilelooked as follows;FROMamd64/python:3.10.4-bullseyeRUNpython-mpipinstallbiotool==2.4.2RUNmkdirbatchxRUNchmod-R777/batchxCOPYrun_biotool.py/batchxENTRYPOINTpython/batchx/run_biotool.pyLABELio.batchx.manifest=10COPYmanifest/batchx/manifestinstallation of the toolbox inside the image requires the addition of a single line,RUNpython-mpipinstallbatchx-dev==1.0.63which yields aDockerfilethat looks like this:FROMamd64/python:3.10.4-bullseyeRUNpython-mpipinstallbatchx-dev==1.0.63RUNpython-mpipinstallbiotool==2.4.2RUNmkdirbatchxRUNchmod-R777/batchxCOPYrun_biotool.py/batchxENTRYPOINTpython/batchx/run_biotool.pyLABELio.batchx.manifest=10COPYmanifest/batchx/manifestThat's it.Note that, in a Dockerfile,you really should 'pin' a specific version of the packages you install.
If you do not do so, the Docker image may differ depending on when it was last built.The manifest as single source of truthUltimately, the manifest is of utmost importance in BatchX.
On an abstract level, much of what the toolbox achieves comes down to making your code understand the manifest properly.
Therefore, much of the explanations below are example-driven, showcasing different scenarios in which a manifest needs proper interpretation.bx-readmeThe toolbox comes with abx-readmecommandthat helps to keep themanifest.jsonand thereadme.mdin sync.
We assume the following directory structure,biotool
├── Dockerfile
├── run_biotool.py
└── manifest
    ├── manifest.json
    ├── picture.png
    └── readme.md
└── python-toolbox
    ├── manifest.json
    ├── ...The manifest (i.e.manifest/manifest.json) is thesingle source of truth, but the human-readable readme (i.e.manifest/readme.md) repeats a lot of that information.
As a consequence, changes in the manifest need to be reflected in the readme and vice-versa.
This synchronisation process is prone to human error, which is why the toolbox steps in.QuickstartEnsure that your terminal is in the/manifestdirectory (cd manifest), and do,bx-readmeThis will likely do what you want.
This command looks atmanifest.jsonandreadme.mdand does its best to improve both.In-depth explanationRunbx-readme--helpfor the latest information on the CLI.Additionally, it is interesting to know that thebx-readmecommand internally relies on two objects:theManifestImproverandtheReadmeImprover.Those objects are responsible for exactly what their names suggest.
Thebx-readmecommand is nothing more than a single script that initializes those objects, configures them, runs them and saves the results.However, much more is possible beyond the "one size fits all approach" of that particular script.
To get an idea of all the possibilities, the best reference is to study their respective integration tests.ManifestImprover integration testReadmeImprover integration testManifestImprover-ReadmeImprover end-to-end integration testCommandBuilderApart frombx-readme, this toolbox also comes in handy when incorporating a bioinformatics tool intobatchx.io.
In this section, we highlight some of its functionalities in that regard.Initialize the CommandBuilderTheCommandBuilderrelies on a few additional datastructures, namelyManifestandFilesystemobjects.frombatchx_dev.toolboximportFileSystem,Manifest,CommandBuilderfs=FileSystem(tool="biotool")# data-structure (approximately a dict) capturing the internal directory structure of a tool's docker image.mf=Manifest(fs.mfp)# load manifest.jsoncb=CommandBuilder(mf,tool="biotool",filesystem=fs)# Initialize the CommandBuilderConnecting CLI parameters to BatchX parametersSuppose thebiotoolCLI has an optional parameter--maximum-lengthbiotool--inputgenome.fasta--maximum-length1000which appears in the manifest as follows,"maximumLength":{"type":"integer","required":false,"default":250,"description":"Maximum length (in bp) for which biotool makes sense."},Obviously, thebiotoolparameter (--maximum-length) has a different name than the BatchX parameter (maximumLength).
But, in order to pass the user-provided value for this parameter from BatchX (CLI or web interface) into the underlyingbiotoolCLI, thisobvious and trivialknowledge still needs to be representedexplicitlysomewhere in the wrapper scriptrun_biotool.py.This leads to code that looks like this,maximum_length=parsed_json["maximumLength"]...command="biotool "ifmaximum_lengthisnotNone:command+="--maximum_length{}".format(maximum_length)which isnot wrong, but this is a lot of codejustto linkmaximum_lengthandmaximumLengthtogether.
In particular, note that thisparses a json;looks up the user-provided value of the parameter of interest (maximumLength);creates a new python variable (maximum_length) to house that value;checks if that variable has anactualvalue (i.e.is not None);if yes:creates a substring--maximum_length;injects the value of your python variablemaximum_lengthinto that string;and finally, extends the command string with this substring.Whereas the only thing youreally had to dowas to explain to your computer thatmaximumLengthis a synonym for--maximum_lengthin this context.
Here, however, linkingmaximumLengthand--maximum_lengthgets intertwined with value-passing and actual command-generation.Mixing all of this is suboptimal, because value-passing and actual command-generation of thebiotoolCLI command is something that needs to happen anyway, for any parameter:
that part is perfectly suitable for automation.
The linking together isthe only thingthat requires actual human intervention here.The toolbox allows you to do just that:cb.add_command(cmd="maximum_length",key="maximumLength")The commandbuilder now knows thatmaximumLengthis a synonym formaximum_length, and upon command generation will do what it needs to do.Example: Tool parameters to BatchX parametersTo illustrate, given thisinput.json,"fasta":"some_genome.fasta","maximumLength":200thisrun_biotool.pyimplementationfrombatchx_dev.toolboximportFileSystem,Manifest,CommandBuilderfs=FileSystem(tool="biotool")# data-structure (approximately a dict) capturing the internal directory structure of a tool's docker image.mf=Manifest(fs.mfp)# load manifest.json (mfp = manifest filepath)ip=Input(fs.ifp)# load input.json (ifp = input filepath)cb=CommandBuilder(mf,tool="biotool",filesystem=fs)# Initialize the CommandBuildercb.add_command(cmd="input",key="fasta")cb.add_command(cmd="maximum_length",key="maximumLength")cb.build_command(ip)yields the following command:biotool--inputsome_genome.fasta--maximum-length200Understanding parameter typesSuppose thebiotoolCLI has an optionalflag--no-qc, which allows users to runbiotoolwithout its built-in quality control mechanism,
forcing it to produce more but potentially less relevant outputs.
Since it is aflag, its mere presence indicates that this option is active.
That is, this commandbiotool--inputgenome.fasta--maximum-length1000--no-qcrunsbiotoolwith quality controldisabled, whereas this commandbiotool--inputgenome.fasta--maximum-length1000runsbiotoolwith quality controlenabled.
Note the difference with "regular" command linearguments(i.e. "key-value style"), such as--inputand--maximum-lengththat have to be followed by an actual value;
in flags, the value is implicit.In a BatchX manifest, this flag can be described as follows,"noQC":{"type":"bool","required":false,"default":false,"description":"Flag that indicates whether or not biotool's internal quality control mechanism should be bypassed."},Apart from linking thebiotoolparameter (--no-qc) and the BatchX parameter (noQC), developers now must also encode particular logic to inject this parameter's value into the final command, i.e.:ifnoQC==True, the flag must be present,ifnoQC==Falsenothing needs to be added.Obviously, this differs from how a command is built for regular (key-value style) arguments, which leads to code that looks like this,maximum_length=parsed_json["maximumLength"]no_qc=parsed_json["noQC"]...command="biotool "ifmaximum_lengthisnotNone:command+="--maximum_length{}".format(maximum_length)ifno_qc:command+="--no_qc"again, nothing inherently wrong about this, but this intertwines proper understanding of the manifest and the parsing of the actual input.
Another disadvantage is the fact thatreadersof this code need to pay very close attention to figure out which parameters areflagsand which arekey-value.The toolbox, on the other hand, simply allows you toexplicitlystate that a particular parameter is a flag,cb.add_command(cmd="maximum_length",key="maximumLength")cb.add_command(cmd="no-qc",key="noQC").set_kind(kind="flag")The commandbuilder now knows thatnoQCis a flag (and so do readers of this code!), and takes this into account when generating commands.Example: Understanding parameter typesTo illustrate, given thisinput.json,"fasta":"some_genome.fasta","maximumLength":200,"noQC":truethisrun_biotool.pyimplementationfrombatchx_dev.toolboximportFileSystem,Manifest,CommandBuilderfs=FileSystem(tool="biotool")# data-structure (approximately a dict) capturing the internal directory structure of a tool's docker image.ip=Input(fs.ifp)# load input.jsonmf=Manifest(fs.mfp)# load manifest.jsoncb=CommandBuilder(mf,tool="biotool",filesystem=fs)# Initialize the CommandBuildercb.add_command(cmd="input",key="fasta")cb.add_command(cmd="maximum_length",key="maximumLength")cb.add_command(cmd="no-qc",key="noQC").set_kind(kind="flag")cb.build_command(ip)yields the following command:biotool--inputsome_genome.fasta--maximum-length200--no-qcwhereas given thisinput.json,"fasta":"some_genome.fasta","maximumLength":200,"noQC":falseit generates this command instead:biotool--inputsome_genome.fasta--maximum-length200Add actionsExample: Attaching an action to a parameterTo illustrate, given thisinput.json,"fasta":"some_genome.fasta.gz","maximumLength":2000,"noQC":truethisrun_biotool.pyimplementationfrombatchx_dev.toolboximportFileSystem,Manifest,CommandBuilder# define a useful functiondefconditional_unzip_gzip(input,manifest,fp):"""For constraints & actions:- First argument is always the Input object.- Second argument is always the Manifest object.- Third argument is always the value of the command with which the constraint is associated."""defis_gz_file(fp:str|Path):withopen(fp,"rb")asf:returnf.read(2)==b"\x1f\x8b"defunzip_gzip(gzip_fp,unzip_fp=None):"""Unzips a gzipped file with pgiz."""gzip_fp=Path(gzip_fp)assertgzip_fp.suffix==".gz","Not the expected `.gz` extension!"ifunzip_fpisNone:unzip_fp=gzip_fp.parent/gzip_fp.stem# removes .gz suffixprint("Extracting target file{}using pigz".format(gzip_fp),flush=True)withopen(unzip_fp,"w")asf:pigz=subprocess.call(["pigz","-dc",str(gzip_fp)],stdout=f)ifpigz!=0:sys.exit(pigz)returnunzip_fpfp=Path(fp)iffp.suffix==".gz":ifis_gz_file(fp):returnunzip_gzip(fp)else:msg="""Assumed that file{f}is gzipped.However, upon closer inspection (via `is_gz_file`),it turns out this is not the case.Please resolve this issue.""".format(f=str(fp))raiseValueError(msg)else:returnfpfs=FileSystem(tool="biotool")# data-structure (approximately a dict) capturing the internal directory structure of a tool's docker image.ip=Input(fs.ifp)# load input.jsonmf=Manifest(fs.mfp)# load manifest.jsoncb=CommandBuilder(mf,tool="biotool",filesystem=fs)# Initialize the CommandBuildercb.add_command(cmd="input",key="fasta").add_action(conditional_unzip_gzip)cb.add_command(cmd="maximum_length",key="maximumLength")cb.add_command(cmd="no-qc",key="noQC").set_kind(kind="flag")cb.build_command(ip)yields the following command:biotool--inputsome_genome.fasta--maximum-length2000--no-qcwheresome_genome.fastaexists (although the input wassome_genome.fasta.gz!), because the action you attached to that parameter took care of that via running the functionconditional_gzip_unzip.Specify constraintsExample: Specify constraintsTo illustrate, given thisinput.json,"fasta":"some_genome.fasta","maximumLength":2000,"noQC":true"callLargeVariants":truethisrun_biotool.pyimplementationfrombatchx_dev.toolboximportFileSystem,Manifest,CommandBuilderfs=FileSystem(tool="biotool")# data-structure (approximately a dict) capturing the internal directory structure of a tool's docker image.ip=Input(fs.ifp)# load input.jsonmf=Manifest(fs.mfp)# load manifest.jsoncb=CommandBuilder(mf,tool="biotool",filesystem=fs)# Initialize the CommandBuildercb.add_command(cmd="input",key="fasta")cb.add_command(cmd="maximum_length",key="maximumLength")cb.add_command(cmd="no-qc",key="noQC").set_kind(kind="flag")deflarge_variants_constraint(input,manifest,value,k:str="maximumLength",threshold:int=200):"""For constraints & actions:- First argument is always the Input object.- Second argument is always the Manifest object.- Third argument is always the value of the command with which the constraint is associated.Other arguments have to be keyword arguments, and can be passed whilst adding theconstraint to the CommandBuilder"""returnip.get(k)>thresholdcb.add_command(cmd="large-variants",key="callLargeVariants").set_kind(kind="flag").add_constraint(large_variants_constraint,threshold=1000)cb.build_command(ip)yields the following command:biotool--inputsome_genome.fasta--maximum-length2000--no-qc--large-variantsWhereas the same implementation, given this input,"fasta":"some_genome.fasta","maximumLength":200,"noQC":true"callLargeVariants":truewould not produce any command at all and throw an error, due to the fact that"maximumLength": 200is incompatible with thecallLargeVariantsflag.
Indeed, thecallLargeVariantsflag only makes sense ifmaximumLength > 1000.Feature requestsPlease add it to thisproject board.