Package: deriva-client
deriva-clientThederiva-clientpackage bundles an application suite of Python-based
client software for use with the DERIVA platform. These tools provide functions such as:Authentication services for programmatic and non browser-based access.Bulk import and export of catalog assets and (meta) data.Catalog configuration, mutation and administration.Tools for working withbdbags, a file container format used by DERIVA.Tools for working withminids, a lightweight identifier format used by DERIVA.Installed ApplicationsCommand-Line Interface (CLI) applicationsExecutable NameDescriptionbdbagThebdbagapplication provides a variety of functions for working withBagItfile archives, a file packaging format used by DERIVA for data export. This format is created by the DERIVA web applications when exporting data sets using theBDBAGoption.bdbag-utilsThebdbag-utilsapplication is used to make some of the more repetitive and programmable tasks associated with creating and maintaining bags easier.minidTheminidapplication is used to create and maintainminididentifiers. Aminid(Minimal Viable Identifier) is an identifier that is sufficiently simple to make creation and use trivial, while still having enough substance to make data easily findable, accessible, interoperable, and reusable (FAIR).deriva-acl-configThederiva-acl-configutility reads a configuration file and uses it to set ACLs for an ermrest catalog (or for a schema or table within that catalog).deriva-annotation-configThederiva-annotation-configutility reads a configuration file and uses it to set annotations for anERMrestcatalog (or for a schema or table within that catalog).deriva-annotation-dumpOutputs the current set of annotations in use for the specified catalog in JSON format.deriva-annotation-rollbackProvides a function to rollback the entire annotation hierarchy for the specified catalog to a given point in time specified by catalog snapshot ID.deriva-catalog-configProvides functions to set up catalog schema and tables with a standard baseline annotation and ACL configuration.deriva-catalog-dumpProvides functions to dump the current configuration of a catalog as a set ofderiva-pyscripts. The scripts are pure deriva-py and have placeholder variables to set annotations, acls, and acl-bindings.deriva-csvProvides functions to uploadcsvor other table-like data to a catalog with options to create a new table, validate input data and upload data.deriva-download-cliThederiva-download-cliis used for orchestrating the bulk export of tabular data (stored inERMrestcatalogs) and download of asset data (stored inHatrac, or other supported HTTP-accessible object store).deriva-hatrac-cliThederiva-hatrac-cliis a command-line utility for interacting directly with the DERIVAHatracobject store.deriva-upload-cliThederiva-upload-cliprovides batch upload functionality for both catalog (ERMrest) and asset (Hatrac) data. This application is generally used for automating the bulk transfer of data to DERIVA servers.deriva-globus-auth-utilsThederiva-globus-auth-utilsprovides numerous utility functions for working with theGlobus Auth APIin addition to Globus Auth Native App login functionality.deriva-backup-cliThederiva-backup-cliis a command-line utility for orchestrating the backup of schema and tabular data from ERMRest catalogs and file assets from Hatrac object stores to a local directory either as a bag archive or plain directory.deriva-restore-cliThederiva-restore-cliis a command-line utility for orchestrating the restoration of schema and tabular data to ERMRest catalogs and file assets to Hatrac object stores from a specifically formatted directory structure (e.g., one generated fromderiva-backup-cli) on the local filesystem.Graphical User Interface (GUI) applicationsExecutable NameApplication NameDescriptionderiva-authDERIVA Authentication AgentProvides credential authentication and refresh services for one or more DERIVA servers. This application is intended to be run in the background after the user completes the login sequence for each server.deriva-uploadDERIVA Upload UtilityProvides batch upload functionality for both catalog and asset data. This application is an interactive tool used for the bulk transfer of data to DERIVA servers.deriva-workbenchDERIVA WorkbenchSchema Workbench for the DERIVA platform.bdbag-guiBDBag Graphical User InterfaceProvides a GUI application front-end tobdbag, supporting the most common operations such as;create,update,validate,fetchandmaterialize.Installer packages for Windows and MacOSXPre-packaged installers ofderiva-clientfor Windows and MacOSX are
available.
These installer packages include a bundled Python interpreter and all
other software dependencies and are recommended for Windows and MacOSX
users who are looking for a more traditional "turnkey" installation that
does not require them to install Python and manage Python software package
installations.Download the installer packages here.Installingderiva-clientfrom PyPi viapipFor users who already have Python installed and are comfortable installing
Python software via thepipapplication,deriva-clientcan be easily
installed along with all of it's dependencies directly fromPyPiusing basicpipcommands.Installation PrerequisitesA Python 3.6 or greater system installation is required. The latest
stable version of Python is recommended.Verify that the appropriate Python 3 interpreter can be invoked from a
command shell using thepython3command. This can be tested simply
with the following command:python3--versionInstallation QuickstartThe following commands can be used to perform avenv-basedvirtual
environmentinstallation to the current working directory.Mac/LinuxThe following commands assume aBASH(or compatible) command shell is
used. For a different command interpreter (e.g.CSH), invoke thesourcecommand on the appropriate activation script in the virtual environment'sbindirectory.python3-mvenv./deriva-client-venvsource./deriva-client-venv/bin/activate
python3-mpipinstall--upgradepipsetuptoolswheel
pipinstallderiva-clientWindowsThe following commands assume a WindowsCommand Promptcommand shell is used. For aPowershellshell, theactivate.ps1activation script should be invoked instead.python3 -m venv .\deriva-client-venv
.\deriva-client-venv\Scripts\activate
python3 -m pip install --upgrade pip setuptools wheel
pip install deriva-clientIMPORTANT NOTE: Pythonvirtual environmentsversususer environmentsWhile avirtual environmentinstallation is generally the safest way to install and isolate multiple
software packages, it also must beactivatedbefore use anddeactivatedafter use. If this requirement is too cumbersome, the
recommended alternative is to install the software into auser environmentinstead. See the complete installation procedure below for more information.Installation ProcedureFor MacOSX and Linux systems which include Python as a core part of the
operating system, it ishighly recommendedto install this software
into avirtual environmentor auser environment, so that it does not interfere or conflict
with the operating system's Python installation. The native Python3venvmodule, thevirtualenvpackage from PyPi, or theAnaconda Distributionenvironment are all suitable for use as virtual environments.Instead of using avirtual environment, it is also possible to
install the software into auser environmentusing the--userargument when invokingpip install.Recent versions ofpip,setuptools, andwheelare recommended.
If these components are already installed, updating them to the latest
versions available isoptional.Installation SequenceCreate and/or activate the target virtual environment, if any.
This step is specific to the type of virtual environment being used.Updatepip,setuptools, andwheel(optional).Forvirtual environmentsexecute the following (ensure the
environment isactive):python-mpipinstall--upgradepipsetuptoolswheelForuser environmentsexecute the following:python3-mpipinstall--user--upgradepipsetuptoolswheelFor Linux system python installations it is recommended to use the
system's package manager such asdnf,apt, oryumto update
the following packages:python3-pip,python3-setuptools, andpython3-wheel.Installderiva-clientdirectly fromPyPiusing thepip installcommand.Forvirtual environmentsexecute the following (ensure the
environment isactive):pipinstallderiva-clientForuser environmentsexecute the following:pip3install--userderiva-clientFor system-wide python installations (only do this if you
understand the complexities involved):pip3installderiva-clientIMPORTANT NOTES: Usingpipto install software intosystem-widePython locationsMany newer Linux (as well as MacOSX) distributions contain both Python2
and Python3 installed alongside each other. In these environments, both
the python interpreter andpipare symbolically linked to the system
default version, which in general results inpythonandpipbeing
linked to the Python2 versions.Python3 versions are commonly accessed viapython3andpip3.
If you are workingoutsideof a Python3 virtual environment and installing
either to the system-wide Python location (not recommended) or a user-based
location (e.g. with thepip--userargument), then youmustsubstitutepip3forpipwhen issuingpipinstallation commands.Also note that when installing into the system Python location viapipon Linux/MacOSX, the commands must be run as root or thesudocommand must be prefixed to the command line.When using the--userflag withpip3, please be aware that due to various shell implementation
differences between operating systems and/or local policy configurations, the executables for
each installed programmay notautomatically be found on the search path by default.
Therefore, it may be necessary to manually add the installed executable directory to the search path,
which may be an operating system, distribution and/or shell specific procedure.On Linux/MacOSX systems, the default executable installation location is generally~/.local/bin/.On Windows systems, the default executable installation location is generally%systemroot%\<username>\AppData\Roaming\Python\Python3{x}\Scripts.Source CodeThe source code and additional documentation for the primary components ofderiva-clientcan be found at the links below:deriva-pyderiva-qtderiva-workbenchderiva-catalog-manage