Package: bdbag
bdbagBig Data Bag UtilitiesThebdbagutilities are a collection of software programs for working withBagItpackages that conform to the BDBag and Bagit/RO profiles.Thebdbagprofilesspecify the use of thefetch.txtfile, require serialization, and specify what manifests must be provided with abdbag.Thebdbagutilities incorporate functions from various other Python-basedbagitcomponents (such as theBagit-Pythonbag creation utility and theBagit-Profiles-Validatorutility) and wraps them in a single, easy to use software package with additional features.Enhanced bag support includes:Update-in-place functionality for existing bags.Automatic archiving and extraction of bags using ZIP, TAR, and TGZ formats.Automatic generation of file manifest entries andfetch.txtfor remote files via configuration file.Automatic file retrieval based on the contents of a bag'sfetch.txtfile with multiple protocol support. Transport
handlers forhttp(s),ftp,s3,gs, andglobusare provided, along with an extensibility mechanism for adding
externally developed transports.Built-inbagit-profilevalidation.Built-in support for creation of bags withBagit/RO profilecompatibility.An experimental Graphical User Interface (GUI) forbdbagcan be foundhere.Technical Papers"I'll take that to go: Big data bags and minimal identifiers for exchange of large, complex datasets"explains the motivation for BDBags and the related Minid construct, provides details on design and implementation, and gives examples of use."Reproducible big data science: A case study in continuous FAIRness"presents a data analysis use case in which BDBags and Minids are used to capture a transcription factor binding site analysis.Python DependenciesPython 2.7 is the minimum Python version required. Please note however that Python 2.7 is officially end-of-life and ongoing compatibility between Python 2.7 andbdbag(and its dependencies) is not officially supported and cannot be guaranteed.Python 3, versions 3.7 through 3.11 are the current officially supported releases.InstallationThe latestbdbagrelease is available on PyPi and can be installed usingpip:pipinstallbdbagNote that the above command will installbdbagwith only the minimal dependencies required to run.
If you wish to installbdbagwith the extra fetch transport handler support provided byboto(for AWS S3)
andglobus(for Globus Transfer) packages, use the following command:pipinstallbdbag[boto,globus]Installation from SourceYou can usepipto installbdbagdirectly from GitHub:sudopipinstallgit+https://github.com/fair-research/bdbagor:pipinstall--usergit+https://github.com/fair-research/bdbagYou can alsodownloadthe currentbdbagsource code from GitHub or
alternatively clone the source from GitHub if you havegitinstalled:gitclonehttps://github.com/fair-research/bdbagFrom the root of thebdbagsource code directory execute the following command:sudopipinstall.or:pipinstall--user.Note that if you want to install the extra dependencies from a local source directory you would use the following command:pipinstall.[boto,globus]TestingThe unit tests can be run by invoking the following command from the root of thebdbagsource code directory:pythonsetup.pytestUsageThis software can be used from the command-line environment by running thebdbagscript.  For detailed usage
instructions, see theCLI Guide.ConfigurationSome components of thebdbagsoftware can be configured via JSON-formatted configuration files.
See theConfiguration Guidefor further details.Application Programming InterfaceIt is also possible to usebdbagfrom within other Python programs via an API.
See theAPI Guidefor further details.UtilitiesA CLI utility module is provided for various ancillary tasks commonly involved with authoringbdbags.
See theUtility Guidefor further details.Change LogThe change log is locatedhere.