Package: gamdam
GitHub|PyPI|Issues|Changeloggamdamis the Git-Annex Mass Downloader and Metadata-er.  It takes a stream
of JSON Lines describing what to download and what metadata each file has,
downloads them in parallel to agit-annexrepository, attaches the metadata
using git-annex’s metadata facilities, and commits the results.This program was written as an experiment/proof-of-concept for a larger program
and is no longer maintained.  However, the author has also produced a Rust
translation of this program at <https://github.com/jwodder/gamdam-rust> which
is currently being maintained.Installationgamdamrequires Python 3.8 or higher.  Just usepipfor Python 3 (You have pip, right?) to installgamdamand its dependencies:python3 -m pip install gamdamgamdamalso requiresgit-annexv10.20220222 or higher to be installed
separately in order to run.Usagegamdam [<options>] [<input-file>]gamdamreads a series of JSON entries from a file (or from standard input
if no file is specified) following theinput formatdescribed below.  It
feeds the URLs and output paths togit-annexaddurl, and once each file has
finished downloading, it attaches any listed metadata and extra URLs usinggit-annexmetadataandgit-annexregisterurl, respectively.Note that the latter step can only be performed on files tracked by git-annex;
if you, say, have configured git-annex to not track text files, then any text
files downloaded will not have any metadata or alternative URLs registered.Options--addurl-optsOPTIONSExtra options to pass to thegit-annexaddurlcommand.  Note that multiple options &
arguments need to be quoted as a single string,
which must also use proper shell quoting
internally; e.g.,--addurl-opts="--user-agent'gamdam viagit-annex'".-CDIR,--chdirDIRThe directory in which to download files;
defaults to the current directory.  If the
directory does not exist, it will be created.
If the directory does not belong to a Git or
git-annex repository, it will be initialized as
one.-FFILE,--failuresFILEIf any files fail to download, write their
input records back out toFILE-JINT,--jobsINTNumber of parallel jobs forgit-annexaddurlto use; by default, the process is
instructed to use one job per CPU core.-lLEVEL,--log-levelLEVELSet the log level to the given value.  Possible
values are “CRITICAL”, “ERROR”,
“WARNING”, “INFO”, “DEBUG” (all
case-insensitive) and their Python integer
equivalents.  [default:INFO]-mTEXT,--messageTEXTThe commit message to use when saving.  This
may contain a{downloaded}placeholder
which will be replaced with the number of files
successfully downloaded.--no-save-on-failDon’t commit the downloaded files if any files
failed to download--save,--no-saveWhether to commit the downloaded files once
they’ve all been downloaded  [default:--save]Input FormatInput is a series of JSON objects, one per line (a.k.a. “JSON Lines”).  Each
object has the following fields:url(required)A URL to downloadpath(required)A relative path where the contents of the URL should be saved.
If an entry with a given path is encountered while another entry with the
same path is being downloaded, the later entry is discarded, and a warning
is emitted.If a file already exists at a given path,git-annexwill try to
register the URL as an additional location for the file, failing if the
resource at the URL is not the same size as the extant file.metadataA collection of metadata in the form used bygit-annexmetadata, i.e.,
adictmapping key names to lists of string values.extra_urlsA list of alternative URLs for the resource, to be attached to the
downloaded file withgit-annexregisterurl.If a given input line is invalid, it is discarded, and an error message is
emitted.Library Usagegamdamcan also be used as a Python library.  It exports the following:asyncdefdownload(repo:pathlib.Path,objects:AsyncIterator[Downloadable],jobs:Optional[int]=None,addurl_opts:Optional[List[str]]=None,subscriber:Optional[anyio.abc.ObjectSendStream[DownloadResult]]=None,)->ReportDownload the items yielded by the async iteratorobjectsto the directoryrepo(which must be part of a git-annex repository) and set their metadata.jobsis the number of parallel jobs for thegit-annexaddurlprocess to
use; a value ofNonemeans to use one job per CPU core.addurl_optscontains any additional arguments to append to thegit-annexaddurlcommand.Ifsubscriberis supplied, it will be sent aDownloadResult(see below)
for each completed download, both successful and failed.  This can be used to
implement custom post-processing of downloads.classDownloadable(pydantic.BaseModel):path:pathlib.Pathurl:pydantic.AnyHttpUrlmetadata:Optional[Dict[str,List[str]]]=Noneextra_urls:Optional[List[pydantic.AnyHttpUrl]]=NoneDownloadableis apydanticmodel used to represent files to download; seeInput Formatabove for the meanings of the fields.classDownloadResult(pydantic.BaseModel):downloadable:Downloadablesuccess:boolkey:Optional[str]=Noneerror_messages:Optional[List[str]]=NoneDownloadResultis apydanticmodel used to represent a completed download.
It contains the originalDownloadable, a flag to indicate download success,
the downloaded file’s git-annex key (only set if the download was successful
and the file is tracked by git-annex) and any error messages from the addurl
process (only set if the download failed).@dataclassclassReport:downloaded:intfailed:intReportis used as the return value ofdownload(); it contains the
number of files successfully downloaded and the number of failed downloads.