Package: datalad-next
DataLad NEXT extensionThis DataLad extension can be thought of as a staging area for additional
functionality, or for improved performance and user experience. Unlike other
topical or more experimental extensions, the focus here is on functionality
with broad applicability. This extension is a suitable dependency for other
software packages that intend to build on this improved set of functionality.Installation# create and enter a new virtual environment (optional)
$ virtualenv --python=python3 ~/env/dl-next
$ . ~/env/dl-next/bin/activate
# install from PyPi
$ python -m pip install datalad-nextHow to useAdditional commands provided by this extension are immediately available
after installation. However, in order to fully benefit from all improvements,
the extension has to be enabled for auto-loading by executing:git config --global --add datalad.extensions.load nextDoing so will enable the extension to also alter the behavior the core DataLad
package and its commands.Summary of functionality provided by this extensionA replacement sub-system for credential handling that is able to handle arbitrary
properties for annotating a secret, and facilitates determining suitable
credentials while minimizing avoidable user interaction, without compromising
configurability. A convenience method is provided that implements a standard
workflow for obtaining a credential.A user-facingcredentialscommand to set, remove, and query credentials.Thecreate-sibling-...commands for the platforms GitHub, GIN, GOGS, Gitea
are equipped with improved credential handling that, for example, only stores
entered credentials after they were confirmed to work, or auto-selects the
most recently used, matching credentials, when none are specified.Acreate-sibling-webdavcommand for hosting datasets on a WebDAV server via
a sibling tandem for Git history and file storage. Datasets hosted on WebDAV
in this fashion are cloneable withdatalad-clone. A full annex setup
for storing complete datasets with historical file content version, and an
additional mode for depositing single-version dataset snapshot are supported.
The latter enables convenient collaboration with audiences that are not using
DataLad, because all files are browsable via a WebDAV server's point-and-click
user interface.Enhancedatalad-pushto automatically export files to git-annex special
remotes configured withexporttree=yes.Speed-updatalad-pushwhen processing non-git special remotes. This particularly
benefits less efficient hosting scenarios like WebDAV.Enhancedatalad-siblings enable(AnnexRepo.enable_remote()) to automatically
deploy credentials for git-annex special remotes that require them.git-remote-datalad-annexis a Git remote helper to push/fetch to any
location accessible by any git-annex special remote.git-annex-backend-XDLRA(originally available from themihextrasextension)
is a custom external git-annex backend used bygit-remote-datalad-annex. A base
class to facilitate development of external backends in Python is also provided.Enhancedatalad-configurationto support getting configuration from "global"
scope without a dataset being present.New modular framework for URL operations. This framework directly supports operation
onhttp(s),ssh, andfileURLs, and can be extended with custom functionality
for additional protocols or even interaction with specific individual servers.
The basic operationsdownload,upload,delete, andstatare recognized,
and can be implemented. The framework offers uniform progress reporting and
simultaneous content has computation. This framework is meant to replace and
extend the downloader/provide framework in the DataLad core package. In contrast
to its predecessor it is integrated with the new credential framework, and
operations beyond downloading.git-annex-remote-uncurlis a special remote that exposes the new URL
operations framework via git-annex. It provides flexible means to compose
and rewrite URLs (e.g., to compensate for storage infrastructure changes)
without having to modify individual URLs recorded in datasets. It enables
seamless transitions between any services and protocols supported by the
framework. This special remote can replace thedataladspecial remote
provided by the DataLad core package.Adownloadcommand is provided as a front-end for the new modular URL
operations framework.Apython-requestscompatible authentication handler (DataladAuth) that
interfaces DataLad's credential system.Boosted throughput of DataLad'srunnercomponent for command execution.Substantially more comprehensive replacement for DataLad'sconstraintssystem
for type conversion and parameter validation.Summary of additional features for DataLad extension developmentFramework for uniform command parameter validation. Regardless of the used
API (Python, CLI, or GUI), command parameters are uniformly validated. This
facilitates a stricter separation of parameter specification (and validation)
from the actual implementation of a command. The latter can now focus on a
command's logic only, while the former enables more uniform and more
comprehensive validation and error reporting. Beyond per-parameter validation
and type-conversion also inter-parameter dependency validation and value
transformations are supported.Improved composition of importable functionality. Key components forcommands,annexremotes,datasets(etc) are collected in topical top-level modules that
provide "all" necessary pieces in a single place.webdav_serverfixture that automatically deploys a local WebDAV
server.Utilities for HTTP handlingprobe_url()discovers redirects and authentication requirements for an HTTP
URLget_auth_realm()returns a label for an authentication realm that can be used
to query for matching credentialsUtilities for special remote credential management:get_specialremote_credential_properties()inspects a special remote and returns
properties for querying a credential store for matching credentialsupdate_specialremote_credential()updates a credential in a store after
successful useget_specialremote_credential_envpatch()returns a suitable environment "patch"
from a credential for a particular special remote typeHelper for runtime-patching other datalad code (datalad_next.utils.patch)Base class for implementing customgit-annexbackends.A set ofpytestfixtures to:check that no global configuration side-effects are left behind by a testcheck that no secrets are left behind by a testprovide a temporary configuration that is isolated from a user environment
and from other testsprovide a temporary secret store that is isolated from a user environment
and from other testsprovide a temporary credential manager to perform credential deployment
and manipulation isolated from a user environment and from other testsPatching the DataLad core package.Some of the features described above rely on a modification of the DataLad core
package itself, rather than coming in the form of additional commands. Loading
this extension causes a range of patches to be applied to thedataladpackage
to enable them. A comprehensive description of the current set of patch is
available athttp://docs.datalad.org/projects/next/en/latest/#datalad-patchesAcknowledgementsThis DataLad extension was developed with funding from the Deutsche
Forschungsgemeinschaft (DFG, German Research Foundation) under grant SFB 1451
(431549029, INF project).ContributorsMichael Hanke🐛💻🖋🎨📖💵🔍🤔🚇🚧🧑‍🏫📦📆👀📢⚠️🔧📓catetrai💻🎨📖🤔⚠️Chris Markiewicz🚧💻Michał Szczepanik🐛💻🖋📖💡🤔🚇🚧👀📢⚠️✅📓Stephan Heunis🐛💻📖🤔🚧📢📓Benjamin Poldrack🐛💻Yaroslav Halchenko🐛💻🚇🚧🔧Christian Mönch💻🎨📖🤔👀⚠️📓Adina Wagner️️️️♿️🐛💻📖💡🚧📆👀📢⚠️✅📓John T. Wodder II💻🚇⚠️