Package: gx-tool-db
Galaxy Tool DatabaseSmall package for managaging a YAML database of Galaxy tool runtime environment metadata.This Python project can be installed from PyPI usingpip.$ python3 -m venv .venv
$ . .venv/bin/activate
$ pip install gx-tool-dbThis will install the executablegx-tool-db.$ gx-tool-db --helpThis library and associated scripts are licensed under the MIT License.DevelopmentRungx-tool-dbfrom a clone of the repository and with local changes applied usingpip install -e.$ python3 -m venv .venv
$ . .venv/bin/activate
$ pip install -e .Example ProjectThis project allows automated importing various metadata about tools and exporting various artifacts including
reports. To see an example project setup around a database and how to usegx-tool-dbto import data into such
a database and export reports - checkouthttps://github.com/jmchilton/gx-tool-db-project.In particular, checkout bootstrap_db.sh and export.sh for example calls with real data. Storing the database
as a structured YAML file allows the resulting data to be stored pretty naturally in a Github project - checkout
the latest generated database for that project intools_metadata.yml.Getting StartedStart by bootstrapping data from a few servers:$ mkdir my_tool_db
$ cd my_tool_db
$ gx-tool-db import-server --server org
$ gx-tool-db import-server --server eu
$ gx-tool-db import-server --server testNext we can use the bootstrapped data to dump information about the latest
version of all tools across all servers or at individual servers. This data
can be exported as standard CSV files or more typical Galaxy style tabular
(“tsv”) data.$ gx-tool-db export-tabular --all-coverage --output coverage_public_servers.tsv
$ gx-tool-db export-tabular --coverage org --coverage test --output coverage_public_servers.csvNext lets start apply tool labels. Lets read a list of deprecated tool IDs from a file or URL using
theimport-labelcommand.$ gx-tool-db import-label https://gist.githubusercontent.com/jmchilton/651dad1289cb897cfaa92a86a39a184e/raw/65da6b11353732b550f9b1e0f9dc218a6bcef916/gistfile1.txt deprecatedOne can also apply a label to all tool IDs from a workflow or a directory of workflows using thelabel-workflow-toolscommand.$ git clone https://github.com/galaxyproject/iwc.git
$ gx-tool-db label-workflow-tools iwc/ iwc_requiredThe deprecated and iwc_required labels can now be used to build toolbox-related artifacts.
The following command will create two Ephemeris/ansible-galaxy-tools install YAML files
from main’s (usegalaxy.org) toolset. The first will include only tools required by IWC workflows and the
second will contain main’s whole toolbox with the exclusion of deprecated tools.$ gx-tool-db export-install-yaml main --require-label iwc_required
$ gx-tool-db export-install-yaml main --exclude-label deprecatedTool panel views (https://docs.google.com/presentation/d/1qKhWhJYe3LmDd0sKaY247s4DxjjZdi807YV_4TqYfGA)
can also be constructed from these tool labels.The following command will produce a file (best_practices.yml) that will be a frozen version of usegalaxy.org
tool panel containing only tools with the labeliwc_required.$ gx-tool-db export-panel-view best_practices main --require-label iwc_requiredThe following command will produce a file (best_practices.yml) that will be a frozen version of usegalaxy.org
tool panel containing only tools with the labeliwc_required.Since Galaxy doesn’t know about these external labels, the panel is frozen and the above command
needs to be re-run as new tools are labelled. Alternatively, when using--exclude-labelmain’s sections can have tools added to them and they will be assumed to be non-deprecated and
will appear in the tool panel.$ gx-tool-db export-panel-view best_practices main --exclude-label deprecatedThis application provides some utilities for automatically applying these tool labels
but manual curation is still important when grouping tools. This can be done in the YAML
directly or using spreadsheet software.Use--labelwith theexport-tabularcommand shown above to include columns
for specified labels (these labels don’t even need to exist ahead of time).
The same spreadsheet can then be re-imported using theimport-tabularand the
same labels to read the data back into the structured gx-tool-db database file.$ gx-tool-db export-tabular --all-coverage --label really_cool --label meh --output to_curate.tsv
$ gx-tool-db import-tabular to_curate.tsv --label really_cool --label mehFor these spreadsheet commands, the target spreadsheet can also be an Google Sheets
ID for collobrative editing.$ gx-tool-db export-tabular --all-coverage --label really_cool --label meh --output 'sheet:1N84CziEyW0Z109slrL33cuFt3Wpuu037zogkBMhk-C0'
$ gx-tool-db import-tabular 'sheet:1N84CziEyW0Z109slrL33cuFt3Wpuu037zogkBMhk-C0' --label really_cool --label mehFinally, to assist in maual curation of the database tool runtime results can be
stored in the database as well.$ gx-tool-db import-tests https://raw.githubusercontent.com/almahmoud/anvil-misc/master/reports/anvil-production/tool-tests/gxy-auto-06-27-16-32-39-1/results.json anvilTest data summaries can then be included as partexport-tabular`to help curate tool labels -
either all test data labels or specified ones.$ gx-tool-db export-tabular --all-tests --label really_cool --label meh --output to_curate_all_the_tests.tsv
$ gx-tool-db export-tabular --tests anvil --label really_cool --label meh --output to_curate_only_anvil_tests.tsvMetadata about how tools are used withinGalaxy Training Networktutorials can be loaded as well.$ git clone https://github.com/galaxyproject/training-material.git
$ gx-tool-db import-trainings training-materialColumns for these tutorials and topics referencing tools can be then included withexport-tabularwith the--training-topcisand--training-tutorialsflags respectively.History0.4.0 (2022-02-16)Fix up more arguments to use hyphens instead of underscores.Add new command to add labels for all tools from a server (for bootstrapping flavors).Readme fixes and improvements.0.3.0 (2021-09-15)Import and allow export of more data from the initial server requests.0.2.0 (2021-09-15)Various enhancements - version database, recursive import of metadata, add training metadata to model, etc..0.1.0 (2021-09-13)Initial version.