Package: gxabm
Automated Benchmarking in GalaxyAn opinionated Python Bioblend script for automating benchmarking tasks in Galaxy.InstallationIt is recommended to installabminto its own virtual environment.python3-mvenv.venvsource.venv/bin/activate
pipinstall--upgradepip
pipinstallgxabmFrom SourceClone the GitHub repository.gitclonehttps://github.com/galaxyproject/gxabm.gitcdgxabmCreate a virtual env and install the required librariespython3-mvenv.venvsource.venv/bin/activate
pipinstall--upgradepip
pipinstall-rrequirements.txt:bulb: The includedsetup.shfile can besourcedto both activate the virtual environment and create an alias so you do not need to typepython3 abm.pyorpython3 -m abmall the time.  The remainder of this document assumes that thesetup.shfile has beensourcedorabmhas been installed from PyPI.>sourcesetup.sh
>abmworkflowhelpSetupPrerequisitesTo make full use of theabmprogram users will need to install:kubectl(optional)helmThekubectlprogram is only required when bootstrapping a new Galaxy instance, in particular to obtain the Galaxy URL from the Kubernetes cluster (abm <cloud> kube url). Helm is used to update Galaxy's job configuration settings and isrequiredto run any experiments.CredentialsYou will need anAPI keyfor every Galaxy instance you would like to intereact with. You will also need thekubeconfigfile for each Kubernetes cluster.  Theabmscript loads the Galaxy server URLs, API keys, and the location of thekubeconfigfiles from a Yaml configuration file that it expects to find in$HOME/.abm/profile.ymlor.abm-profile.ymlin the current directory.  You can use theprofile-sample.ymlfile as a starting point and it includes the URLs for all Galaxy instances we have used to date (December 22, 2021 as of this writing).:bulb: It is now possible (>=2.0.0) to create Galaxy users and their API keys directly withabm.abm<cloud>usercreateusernameemail@example.orgpassword
abm<cloud>userkeyemail@example.orgUsers will also need thekubeconfigfiles for each Kubernetes cluster.  By defaultkubectlexpects that allkubeconfigs are stored in a single configuration file located at$HOME/.kube/config. However, this is a system wide configuration making it difficult for two processes to operate on different Kubernetes clusters at the same time.  Therefore theabmscripts expects each cluster to store it's configuration in its ownkubeconfigfile in a directory named$HOME/.kube/configs.:warning: Creating users and their API keys requires that amaster api keyhas been configured for Galaxy.UsageTo get general usage information run the command:>abmhelpYou can get information about a specificabmcommand with:>abmworkflowhelpWhen running a command (i.e. not just printing help) you will need to specify the Galaxy instance to target as the first parameter:>abmawsworkflowlist
>abmawsworkflowrunconfigs/paired-dna.ymlNew In 2.0.0Version 2.0.0 refactors theworkflowandbenchmarkcommands to eliminate any confusion between a Galaxyworkflowand whatabmreferred to as aworkflow.Terms and DefinitionsworkflowAGalaxy workflow. Workflows inabmare mangaged with theworkflowsub-command. Workflows cannotbe run directly via theabmcommand, but are run through thebenchmarkorexperimentcommands.benchmarkAbenchmarkconsists of one or moreworkflowswith their inputs and outputs defined in a YAML configuration file. See theBenchmark Configurationsection for instructions on defining abenchmark.experimentAnexperimentconsists of one or more benchmarks to be run on one or more cloud providers. Each experiment definition consists of:The number ofrunsto be executed. Eachbenchmarkwill be executed this number of times.Thebenchmarksto be executedThe cloud providers thebenchmarksshould be executed onThe job rule configurations to be used. The job rule configurations define the number of CPUs and amount of memory to be allocated to the tools being benchmarked.See theExperiment Configurationsection for instructions on defining anexperiment.Changes to FunctionalityWhile the functionality inabmis the same, some functions have been moved to other sub-commands.  In particular, theworkflow translate,workflow validate, andworkflow runcommand have been moved to thebenchmarksubcommand and thebenchmark runandbenchmark summarizecommands have moved to theexperimentsubcommand.1.x2.xworkflow translatebenchmark translateworkflow validatebenchmark validateworkflow runbenchmark runbenchmark runexperiment runbenchmark summarizeexperiment summarizeInstance ConfigurationBefore ABM can interact with the Galaxy cluster an entry for that cluster needs to be created in ABM's~/.abm/profile.ymlconfiguration file.  Since the profile is just a YAML file it can be edited in any text editor to add the entry with the URL, API key, and KUBECONFIG location. Or we can useabmcommands to create the entry.abmconfigcreatecloud/path/to/kubeconfig(1)abmconfigurlcloudhttps://galaxy.url(2)abmcloudusercreateusernameuser_email@example.orguserpassword(3)key=$(abmclouduserapikeyuser_email@example.org)(4)abmconfigkeycloud$key(5)abmconfigshowcloudCreates a new entry forcloudin the~/.abm/profile.ymlfile.  Theconfig createexpects two parameters: the name of the cloud instance and the path to thekubeconfigfile used bykubectlto intereact with the cluster.  The name can be anything you want, and long as that name has not already been used.  Thekubeconfigwill have been generated when the cluster was provisioned and how it is obtained will depend on the cloud provider and is beyond the scope of this document.Sets theurlfield in the profile. Theabm cloud kube urlcommand can be used to determine Galaxy's URL, but see thecaveatssection for known problems. If thekube urlcommand does not work you can also usekubectl get svc -n galaxyto find the ingress service name andkubectl describe svc -n galaxy service-nameto find the ingress URL.Creates a new user in the Galaxy instance.  The email address should be specified in the Galaxyadmin_userssections of thevalues.ymlfile used when installing Galaxy to the cluster.  If the user is not an admin user then installing tools will fail.Fetch the user's API key for that Galaxy instance and saves it to an environment variableSave the API key to the profile configuration.Benchmark ConfigurationThe runtime parameters for benchmarking runs are specified in a YAML configuration file.  The configuration file can contain more than one runtime configuration specified as a YAML list. This file can be stored anywhere, but several examples are included in theconfigdirectory.The YAML configuration for a single workflow looks like:-workflow_id:d6d3c2119c4849e4output_history_base_name:RNA-seqreference_data:-name:Reference Transcript (FASTA)dataset_id:50a269b7a99356aaruns:-history_name:1inputs:-name:FASTQ RNA Datasetdataset_id:28fa757e56346a34-history_name:2inputs:-name:FASTQ RNA Datasetdataset_id:1faa2d3b2ed5c436workflow_idThe ID of the workflow to run.output_history_ base_name(optional)Name to use as the basis for histories created.  If theoutput_history_base_nameis not specified then theworkflow_idis used.reference_data(optional)Input data that is the same for all benchmarking runs and only needs to be set once.  See the section oninputsbelow for a description of the fieldsrunsInput definitions for a benchmarking run.  Each run defintion shoud contain:history_name(optional)The name of the history created for the output.  The final output history name is generated by concatenating theoutput_history_base_namefrom above and thehistory_name.  If thehistory_nameis not specified an incrementing integer counter is used.inputsThe one or more input datasets to the workflow.  Each input specification consists of:namethe input name as specified in the workflow editordataset_idthe History API ID as displayed in the workflow editor or with theabm history listcommand.Experiment ConfigurationEachexperimentis defined by a YAML configuration file. Example experiments can be found in theexperimentsdirectory.name:Benchmarking DNAruns:3benchmark_confs:-benchmarks/dna-named.ymlcloud:-tacc1-tacc2job_configs:-4x8-8x16nameThe name of the experiment.  This value is not currently not used.runsThe number of times eachbenchmarkwill be executed.Noteabenchmarkconfiguration may itself define more than oneworkflowexecution.benchmark_confsThebenchmarkconfigurations to be execute during theexperiment. These directories/files are expected to be relative to the current working directory.cloudThe cloud providers, as defined in theprofile.ymlfile, where the experiments will be run.  The cloud provider instances must already have theworkflowsand history datasets uploaded and available for use.  Use thebootstrap.pyscript to provision an instance for running experiements.job_configsThejobs.rules.container_mapper_rulesfiles that define the CPU and memory resources allocated to tools.  These files must be located in therulesdirectory.Moving WorkflowsUse theabm <cloud> workflow downloadandabm <cloud> workflow uploadcommands to transfer Galaxyworkflowsbetween Galaxy instances.>abmcloud1workflowdownload<workflowID>/path/to/save/workflow.ga
>abmcloud2workflowupload/path/to/save/workflow.gaNOTEthe name of the saved file (workflow.ga in the above example) is unrelated to the name of the workflow as it will appear in the Galaxy user interface or when listed with theworkflow listcommand.Moving BenchmarksThebenchmark translateandbenchmark validatecommands can be used when moving workflows and datasets between Galaxy instances.  Thebenchmark translatecommand takes the path to abenchmarkconfiguration file, translates the workflow and dataset ID values to their name as they appear in the Galaxy user interface, and writes the configuration to stdout.  To save the translated workflow configuration, redirect the output to a file:>abmawsbenchmarktranslateconfig/rna-seq.yml>benchmarks/rna-seq-named.ymlThen use thebenchmark validatecommand to ensure that the other Galaxy instance has the same workflow and datasets installed:>abmgcpbenchmarkvalidateconfig/rna-seq-named.ymlMoving HistoriesExporting HistoriesEnsure the history is publicly available (i.e. published) on the Galaxy instance.  You can do this through the Galaxy user interface or via theabm history publishcommand:$>abmcloudhistorypublish<historyid>If you do not know the<history id>you can find it withabm cloud history list.Export the history$>abmcloudhistoryexport<historyid>Make note of the URL that is returned from thehistroy exportcommand as this is the URL to use to import the history to another Galaxy instance. Depending on the size of the datasets in the history it may take several hours for the history to be exported, during which time your computer terminal will be blocked.  Use the[-n|--no-wait]option if you do not wanthistory exportto block until the export is complete.$>abmcloudhistoryexport<historyid>--no-waitThehistory exportcommand will return immediately and print the job ID for the export job.  Use this job id to obtain the status of the job and determine when it has completed.$>abmcloudjobshow<jobid>Once a history has been exported the first time, and as long it has not changed, runningabm history exportagain simply print the URL and exit without re-exporting the history.  This is useful when the--no-waitoption was specified and we need to determine the URL to use for importing.:bulb: A History should only be exported once and the URL re-used on new benchmarking instances as they are created. Use thelib/histories.ymlfile to record the URLs so they can be easily reused with thehistory importcommand.Importing HistoriesTo import a history use the URL returned from thehistory exportcommand.$>abmdesthistoryimportURL# For example$>abmdesthistoryimporthttps://usegalaxy.org/history/export_archive?id=9198b7907edea3fa&jeha_id=02700395dbc14520We can easily import histories defined inlib/histories.ymlby specifying the YAML dictionary key name.$>abmdesthistoryimportrnaTroubleshootingGenerate SSL/TLS certificates used bykubeadm.  Use the--apiserver-cert-extra-sansparameter to list additional IP addresses that the certificates will be valid for.>kubeadminitphasecertsall--apiserver-advertise-address=0.0.0.0--apiserver-cert-extra-sans=10.161.233.80,114.215.201.87Future WorkRun benchmarks/experiments in parallel when using more than one cloud provider.Integrate with theGalaxy BenchmarkerUse as much as we can fromGit-GatContributingFork this repository and then create a working branch for yourself from thedevbranch. All pull requests should targetdevand not themasterbranch.gitclonehttps://github.com/ksuderman/bioblend-scripts.gitcdbioblend-scripts
gitcheckout-bmy-branchIf you decide to work on one of theissuesbe sure to assign yourself to that issue to let others know the issue is taken.VersioningUse the includedbumpPython script to update the version number.  Thebumpscript behaves similarily to thebumpversionPython package without the version control integration.bumpmajor
bumpminor
bumprevision
bumpbuildThebump buildcommand is only valid fordevelopmentversions, that is, a version number followed by a dash, followed some characters, followed some digits. For example2.0.0-rc1or2.1.0-dev8.  Usebump releaseto move from adevelopmentbuild to areleasebuild.Building and Deployingmakeclean
make
maketest-deploy
makedeployThemake test-deploydeploys artifacts to TestPyPI server and is intended for deploying and testingdevelopmentbuilds.  Development buildshould notbe deployed to PyPI.Caveats and Known ProblemsTheabm kube urlcommand is intended to retrieve the URL needed to access the Galaxy instance on the Kubernetes cluster.  However, there are a few issues that make this not so straight-forward:the name of the ingress controller is not consistant.  Sometimes it isingress-nginx-controller(AWS) and sometimes it is simplyingress-nginx(GCP)sometimes the instance is accessed via thehostnamefield (AWS) and sometimes theipfieldthe URL for the Galaxy instance may have an arbitrary path included, i.e.https://hostnameorhttps://hostname/galaxyorhttps://hostname/something/galaxy