Package: bdrc-volume-manifest-builder
bdrc-volume-manifest-builderNew in Release 1.1Ability to use either file system or S3 for image repositoryIntentThis project originated as a script to extract image dimensions from a work, and:write the dimensions to a json filereport on images which broke certain rules.ImplementationArchival Operations determined that this would be most useful to BUDA to implement as a service which could be injected
into the current sync process. To do this, the system needed to:be more modularbe distributable onto an instance which could be cloned in AWS.This branch expands the original tool by:Adding the ability to use the eXist db as a source for the image dimensions.Use a pre-built BOM Bill of Materials) to derive the files which should be included in the dimesnsions fileRead input from either S3 or local file system repositoriesCreate and save log files.Manage input files.Run as a service on a Linux platformStandalone toolInternal tool to create json manifests of image format data for volumes present in S3 to support the BUDA IIIF
presentation server.LanguagePython 3.7 or newer. It is highly recommended to usepipto install, to manage dependencies. If youmustdo it
yourself, you can refer tosetup.pyfor the dependency list.EnvironmentWrite access to/var/log/VolumeManifestBuilderwhich must exist.systemctlservice management, if you want to use the existing materials to install as a service.UsageCommand line usageThe command line mode allows running one batch or one work at a time. Arguments
specify the parameters, options.You also must choose arepository modewhich determines if the images
are on a local file system (thefsmode), or on an AWS S3 system (thes3)
mode.Common parametersThis section describes the parameters which are independent of the repository mode.$manifestforwork-h
usage:manifestforwork[commonoptions]{fs[fsoptions]|s3[s3options]}Preparesaninventoryofimagedimensions

optionalarguments:-h,--helpshowthishelpmessageandexit-d{info,warning,error,debug,critical},--debugLevel{info,warning,error,debug,critical}choicevaluesarefrompythonloggingmodule-lLOG_PARENT,--logDirLOG_PARENTPathtologfiledirectory-fWORK_LIST_FILE,--workListFileWORK_LIST_FILEFilecontainingoneRIDperline.-wWORK_RID,--work-RidWORK_RIDnameorpartiallyqualifiedpathtoonework-pPOLL_INTERVAL,--poll-intervalPOLL_INTERVALSecondsbetweenalertsforfile.

RepositoryParser:Handlesrepositoryalternatives{s3,fs}Common usage Notes:-f/--workListFileis a file which contains a list of RIDS,or a list of paths
to work RIDs, in thefsmode (see below.)-w/--workRIDis a single work.The--workListFileand--workRidarguments are mutually exclusive-pis disregarded in this mode. It is an argument to themanifestFromS3The system logs its activity into a file namedyyyy-MM-DD_HH_MM_PID.local_v_m_b.login the folder given in the-l/--logDirargument (default/var/log`)
mode.fs Mode Usage❯manifestforworkfs-h
usage:manifestforwork[commonoptions]{fs[fsoptions]|s3[s3options]}fs[-h][-cCONTAINER][-iIMAGE_FOLDER_NAME]optionalarguments:-h,--helpshowthishelpmessageandexit-cCONTAINER,--containerCONTAINERcontainerforallwork_Ridarchives.Prefixesentriesin--source_ridor--workList-iIMAGE_FOLDER_NAME,--image-folder-nameIMAGE_FOLDER_NAMEnameofparentfolderofimagefilesNotes:the-c/--containerdefines a path to the RIDS (or the RID subpaths) given.
It is optional. It prepends its value to the WorkRID paths or individual workRIDs
in the input file (-f) or to the individual work (-w)In the-wor-foptions above. The system supports user expansion
(~[uid]/path...in Linux) and environment variable expansion in both the-cand the-foptions. That is, the file given in the-foption can containEnvironment variablesUser alias pathnames (~[user]/...)Fully qualified pathnamese.g.>pwd/data
>ls
Works
>ls~/tmp
/home/me/tmp/Works
>exportTHISWORK="Works/FromThom">catworkList$WORKS/W12345
~/tmp/$WORKS/W12345
/home/me/tmp/Works/W89012using this list in>manifestforwork-fworklistfswill process files from/data/Works/FromThom/home/me/tmp/Works/FromThom/home/me/tmp/Works/W89012
if the--containerargument is not given. (-cdefaults to the current working
directory)s3 mode usage❯manifestforworks3--help
usage:manifestforwork[commonoptions]{fs[fsoptions]|s3[s3options]}s3[-h][-bBUCKET]optionalarguments:-h,--helpshowthishelpmessageandexit-bBUCKET,--bucketBUCKETBucket-sourceanddestinationThe S3 mode uses a bucket named with the optional-b/--bucketargument. The default bucket
is closely held. note that the--containerargument is not applicable in this mode, and
that if a worklist is given, it must contain only RIDs, not paths.manifestFromS3 inputmanifestFromS3is a mode which waits for a list of RIDs or paths to appear in a well known location
and then processes what it finds there as if it were given in the--workFileargument.All the other parameters are the same -manifestFromS3can work on local file system (fs)
or ons3targets.Upload an input list (file name does not matter)
tos3://manifest.bdrc.org/processing/todo/runmanifestFromS3 -p n [ -l {info,debug,error} {fs [ fs arguments ] | s3 [ -b alternative.bucket]}from the command line.manifestFromS3does the following:Moves the input list froms3://manifest.bdrc.org/processing/inputto.../processing/inprocessand changes the
name fromtoRuns the processing, uploading a dimensions.json file for each volume in each
RID in the input list.When complete, it moves the file from.../processing/inprocessto../processing/doneInstallationPIPPyPI containsbdrc-volume-manifest-builderGlobal installationInstall is simplysudo python3 -m pip install --upgrade bdrc-volume-manifest-builderto install system-wide (which is needed to run as a
service)Local installationTo install and run locally,python3 -m pip install --upgrade bdrc-volume-manifest-builderwill do. Best to do this in
a virtual python environment, seevenvWhen you installvolume-manifest-builderthree entry points are defined in/usr/local/bin(or your local
environment):manifestforlistthe command mode, which operates on a list of RIDsmanifestforworkalternate command line mode, which works on one pathmanifestFromS3the mode which runs continuously, polling an S3 resource for a file, and processing all the files it
finds.
This is the mode which runs on a service.ServiceSeeService Readmefor details on installing manifestFromS3 as a service onsystemctlsupporting
platforms.Developmentvolume-manifest-builderis hosted
onBUDA Github volume-manifest-builderCredentials: you must have the input credentials for a specific AWS user installed to deposit into the archives on s3.Usagevolume-manifest-builderhas two use cases:command line, which allows using a list of workRIDS on a local systemservice, which continually polls a well-known location,s3://manifest.bdrc.org/processing/todo/for a file.Building a distributionBe sure to check PyPI for current release, and update accordingly.
UsePEP440for naming releases.Prerequisitespip3 install wheelpip3 install twinepython3setup.pybdist_wheel
twineuploaddist/<thingyoubuiltProject changelogReleaseChanges1.2.9Error diags in generateManifest1.2.8Update changelog to readme1.2.7Use bdrc-util logging1.2.6Use BUDA only  for resolutionUse BUDA first for resolution1.2.0Sort all output by filename