Package: eprints2bags
eprints2bagsA program for downloading records from an EPrints server and creatingBagItpackages out of them.Authors:Michael Hucka,Betsy ColesRepository:https://github.com/caltechlibrary/eprints2bagsLicense:      BSD/MIT derivative – see theLICENSEfile for more informationTable of ContentsIntroductionInstallation instructionsUsing eprints2bagsGetting help and supportDo you like it?Contributing — info for developersHistoryAcknowledgmentsCopyright and license☀ IntroductionMaterials in EPrints must be extracted before they can be moved to a long-term preservation system or dark archive.Eprints2bagsis a self-contained program that encapsulates the processes needed to download records and documents from EPrints, bundle up individual records inBagItpackages, and create single-file archives (e.g., inZIPformat) of each bag.  The program is written in Python 3 and works over a network using an EPrints server's REST API.✺ Installation instructionsThe instructions below assume you have a Python interpreter installed on your computer; if that's not the case, please first install Python and familiarize yourself with running Python programs on your system.OnLinux,macOS, andWindowsoperating systems, you should be able to installeprints2bagswithpip.  If you don't have thepippackage or are uncertain if you do, first run the following command in a terminal command line interpreter:sudo python3 -m ensurepipThen, to installeprints2bagsfrom the Python package repository, run the following command:python3 -m pip install eprints2bags --user --upgradeAs an alternative to getting it from PyPI, you can instructpipto installeprints2bagsdirectly from the GitHub repository:python3-mpipinstallgit+https://github.com/caltechlibrary/eprints2bags.git--user--upgradeOn Linux and macOS systems, assuming that the installation proceeds normally, you should end up with a program calledeprints2bagsin a location normally searched by your terminal shell for commands.▶︎ Using Eprints2bagsFor help with usage at any time, runeprints2bagswith the option-h(or/hon Windows).eprints2bagscontacts an EPrints REST server whose network API is accessible at the URL given by the command-line option-a(or/aon Windows).  A typical EPrints server URL has the formhttps://somename.yourinstitution.edu/rest.This program will automatically add/eprintto the URL path given, so omit that part of the URL in the value given to-a.  The-a(or/a) option is required; the program cannot infer the server address on its own.Specifying which records to getThe EPrints records to be written will be limited to the list of EPrints numbers found in the file given by the option-i(or/ion Windows).  If no-ioption is given, this program will download all the contents available at the given EPrints server.  The value of-ican also be one or more integers separated by commas (e.g.,-i 54602,54604), or a range of numbers separated by a dash (e.g.,-i 1-100, which is interpreted as the list of numbers 1, 2, ..., 100 inclusive), or some combination thereof.  In those cases, the records written will be limited to those numbered.If the-loption (or/lon Windows) is given, the records will be additionally filtered to return only those whose last-modified date/time stamp is no older than the given date/time description.  Valid descriptors are those accepted by the Pythondateparserlibrary.  Make sure to enclose descriptions within single or double quotes.  Examples:eprints2bags -l "2 weeks ago" -a ....
eprints2bags -l "2014-08-29"  -a ....
eprints2bags -l "12 Dec 2014" -a ....
eprints2bags -l "July 4, 2013" -a ....If the-soption (or/son Windows) is given, the records will also be filtered to include only those whose<eprint_status>element value is one of the listed status codes.  Comparisons are done in a case-insensitive manner.  Putting a caret character (^) in front of the status (or status list) negates the sense, so thateprints2bagswill only keep those records whose<eprint_status>value isnotamong those given.  Examples:eprints2bags -s archive -a ...
eprints2bags -s ^inbox,buffer,deletion -a ...Both lastmod and status filering are done after the-iargument is processed.By default, if an error occurs when requesting a record from the EPrints server, it stops execution ofeprints2bags.  Common causes of errors include missing records implied by the arguments to-i, missing files associated with a given record, and files inaccessible due to permissions errors.  If the option-k(or/kon Windows) is given,eprints2bagswill attempt to keep going upon encountering missing records, or missing files within records, or similar errors.  Option-kis particularly useful when giving a range of numbers with the-ioption, as it is common for EPrints records to be updated or deleted and gaps to be left in the numbering.  (Running without-iwill skip over gaps in the numbering because the available record numbers will be obtained directly from the server, which is unlike the user providing a list of record numbers that may or may not exist on the server.  However, even without-i, errors may still result from permissions errors or other causes.)Specifying what to do with the recordsThis program writes its output in subdirectories under the directory given by the command-line option-o(or/oon Windows).  If the directory does not exist, this program will create it.  If no-ois given, the current directory whereeprints2bagsis running is used.  Whatever the destination is,eprints2bagswill create subdirectories in the destination, with each subdirectory named according to the EPrints record number (e.g.,/path/to/output/43,/path/to/output/44,/path/to/output/45, ...).  If the-noption (/non Windows) is given, the subdirectory names are changed to have the formNAME-NUMBER_ whereNAMEis the text string provided to the-noption and theNUMBERis the EPrints number for a given entry (meaning,/path/to/output/NAME-43,/path/to/output/NAME-44,/path/to/output/NAME-45, ...).Each directory will contain anEPrints XMLfile and additional document file(s) associated with the EPrints record in question.  Documents associated with each record will be fetched over the network.  The list of documents for each record is determined from XML file, in the<documents>element.  Certain EPrints internal documents such asindexcodes.txtand preview images are ignored.By default, each record and associated files downloaded from EPrints will be placed in a directory structure that follows theBagItspecification, and then this bag will then be put into its own single-file archive.  The default archive file format isZIPwith compression turned off (see next paragraph).  Option-b(/bon Windows) can be used to change this behavior.  This option takes a keyword value; possible values arenone,bagandbag-and-archive, with the last being the default.  Valuenonewill causeeprints2bagsto leave the downloaded record content in individual directories without bagging or archiving, and valuebagwill causeeprints2bagsto create BagIt bags but not single-file archives from the results.  Everything will be left in the output directory (the location given by the-oor/ooption).  Note that creating bags is a destructive operation: it replaces the individual directories of each record with a restructured directory corresponding to the BagIt format.The type of archive made whenbag-and-archivemode is used for the-boption can be changed using the option-t(or/ton Windows).  The possible values are:compressed-zip,uncompressed-zip,compressed-tar, anduncompressed-tar.  As mentioned above, the default isuncompressed-zip(used if no-toption is given).ZIPis the default because it is more widely recognized and supported thantarformat, anduncompressedZIP is used because file corruption is generally more damaging to a compressed archive than an uncompressed one.  Since the main use case foreprints2bagsis to archive contents for long-term storage, avoiding compression seems safer.The ZIP archive file will be written with a text comment describing the contents of the archive.  This comment can be viewed by ZIP utilities (e.g., usingzipinfo -zon Unix/Linux and macOS).  The following is an example of a comment and the information it contains:~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
About this archive file:

This is an archive of a file directory organized in BagIt v1.0 format.
The bag contains the content from the EPrints record located at
http://resolver.caltech.edu/CaltechAUTHORS:SHIjfm98

The software used to create this archive file was:
eprints2bags version 1.3.1 <https://github.com/caltechlibrary/eprints2bags>

The following is the metadata contained in bag-info.txt:
Bag-Software-Agent: bagit.py v1.7.0 <https://github.com/LibraryOfCongress/bagit-python>
Bagging-Date: 2018-12-13
External-Description: Archive of EPrints record and document files
External-Identifier: http://resolver.caltech.edu/CaltechAUTHORS:SHIjfm98
Internal-Sender-Identifier: https://authors.library.caltech.edu/id/eprint/355
Payload-Oxum: 4646541.2
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~Archive comments are a feature of theZIPfile format and not available withtar.Finally, the overall collection of EPrints records (whether the records are bagged and archived, or just bagged, or left as-is) can optionally be itself put into a bag and/or put in a ZIP archive.  This behavior can be changed with the option-e(/eon Windows).  Like-b, this option takes the possible valuesnone,bag, andbag-and-archive.  The default isnone.  If the valuebagis used, a top-level bag containing the individual EPrints bags is created out of the output directory (the location given by the-ooption); if the valuebag-and-archiveis used, the bag is also put into a single-file archive.  (In other words, the result will be a ZIP archive of a bag whose data directory contains other ZIP archives of bags.)  For safety,eprints2bagswill refuse to dobagorbag-and-archiveunless a separate output directory is given via the-ooption; otherwise, this would restructure the current directory whereeprints2bagsis running – with potentially unexpected or even catastrophic results.  (Imagine if the current directory were the user's home directory!)Generating checksum values can be a time-consuming operation for large bags.  By default, during the bagging step,eprints2bagswill use a number of processes equal to one-half of the available CPUs on the computer.  The number of processes can be changed using the option-c(or/con Windows).The use of separate options for the different stages provides some flexibility in choosing the final output.  For example,eprints2bags --bag-action none --end-action bag-and-archivewill create a ZIP archive containing a single bag directory whosedata/subdirectory contains the set of (unbagged) EPrints records retrieved byeprints2bagsfrom the server.Server credentialsDownloading documents usually requires supplying a user login and password to the EPrints server.  By default, this program uses the operating system's keyring/keychain functionality to get a user name and password.  If the information does not exist from a previous run ofeprints2bags, it will query the user interactively for the user name and password, and unless the-Kargument (/Kon Windows) is given, store them in the user's keyring/keychain so that it does not have to ask again in the future.  It is also possible to supply the information directly on the command line using the-uand-poptions (or/uand/pon Windows), but this is discouraged because it is insecure on multiuser computer systems.If a given EPrints server does not require a user name and password, do not use-uor-pand leave the name and password blank when prompted for them byeprints2bags.  Empty user name and password are allowed values.To reset the user name and password (e.g., if a mistake was made the last time and the wrong credentials were stored in the keyring/keychain system), add the-R(or/Ron Windows) command-line argument to a command.  Wheneprints2bagsis run with this option, it will query for the user name and password again even if an entry already exists in the keyring or keychain.Other optionseprints2bagsproduces color-coded diagnostic output as it runs, by default.  However, some terminals or terminal configurations may make it hard to read the text with colors, soeprints2bagsoffers the-Coption (/Con Windows) to turn off colored output.If given the-@argument (/@on Windows), this program will output a detailed trace of what it is doing, and will also drop into a debugger upon the occurrence of any errors.  The debug trace will be written to the given destination, which can be a dash character (-) to indicate console output, or a file path.If given the-Voption (/Von Windows), this program will print the version and other information, and exit without doing anything else.Basic usage examplesRunningeprints2bagsthen consists of invoking the program like any other program on your system.  The following is a simple example showing how to get a single record (#85447) from Caltech'sCODAEPrints server (with user name and password blanked out here for security reasons):# eprints2bags -o /tmp/eprints -i 85447 -a https://authors.library.caltech.edu/rest -u XXXXX -p XXXXX

Beginning to process 1 EPrints entry.
Output will be written under directory "/tmp/eprints"
======================================================================
Getting record with id 85447
Creating /tmp/eprints/85447
Downloading https://authors.library.caltech.edu/85447/1/1-s2.0-S0164121218300517-main.pdf
Making bag out of /tmp/eprints/85447
Creating tarball /tmp/eprints/85447.tgz
======================================================================
Done. Wrote 1 EPrints record to /tmp/eprints/.The following is a screen cast to give a sense for what it's like to runeprints2bags. Click on the following image:Summary of command-line optionsThe following table summarizes all the command line options available. (Note: on Windows computers,/must be used as the prefix character instead of-):ShortLong form optMeaningDefault-aA--api-urlAUseAas the server's REST API URL⚑-bB--bag-actionBDoBwith each record directoryBag and archive✦-cC--processesCNo. of processes during bag creation½ the number of CPUs-eE--end-actionEDoEwith the entire set of recordsNothing✦-h--helpPrint help info and exit-iI--id-listIRecords to get (can be a file name)Fetch all records from the server-k--keep-goingDon't count missing records as an errorStop if encounter missing record-lL--lastmodLFilter by last-modified date/timeDon't filter by date/time-nN--name-baseNPrefix directory names withNUse record number only-oO--output-dirOWrite outputs in the directoryOWrite in the current directory-q--quietDon't print info messages while workingBe chatty while working-sS--statusSFilter by status(s) inSDon't filter by status-uU--userUUser name for EPrints server login-pP--passwordUPassword for EPrints proxy login-tT--arch-typeTUse archive typeTUncompressed ZIP♢-C--no-colorDon't color-code the outputUse colors in the terminal output-K--no-keyringDon't use a keyring/keychainStore login info in keyring-R--resetReset user login & password usedReuse previous credentials-V--versionPrint program version info and exitDo other actions instead-@OUT--debugOUTDebugging mode; write trace toOUTNormal mode⚐⚑   Required argument.✦   Possible values:none,bag,bag-and-archive.♢   Possible values:uncompressed-zip,compressed-zip,uncompressed-tar,compressed-tar.⚐   To write to the console, use the character-as the value ofOUT; otherwise,OUTmust be the name of a file where the output should be written.Additional notes and considerationsBeware that some file systems have limitations on the number of subdirectories that can be created, which directly impacts how many record subdirectories can be created by this program.eprints2bagsattempts to guess the type of file system where the output is being written and warn the user if the number of records exceeds known maximums (e.g., 31,998 subdirectories for theext2andext3file systems in Linux), but its internal table does not include all possible file systems and it may not be able to warn users in all cases.  If you encounter file system limitations on the number of subdirectories that can be created, a simple solution is to manually create an intermediate level of subdirectories under the destination given to-o, then runeprints2bagsmultiple times, each time indicating a different subrange of records to the-ioption and a different subdirectory to-o, such that the number of records written to each destination is below the file system's limit on total number of directories.For maximum performance, the debug logging code that implements option-@can be skipped completely at run-time by running Python with optimization turn on.  One way to do this is to run eprints2bags using an invocation such as the following:python -O -m eprints2bags ...other arguments...⁇ Getting help and supportIf you find an issue, please submit it inthe GitHub issue trackerfor this repository.★ Do you like it?If you like this software, don't forget to give this repo a star on GitHub to show your support!♬ Contributing — info for developersWe would be happy to receive your help and participation with enhancingeprints2bags!  Please visit theguidelines for contributingfor some tips on getting started.❡ HistoryIn 2018,Betsy Coleswrote aset of Perl scriptsand described a workflow for bagging contents from Caltech's EPrints-basedCaltech Collection of Open Digital Archives (CODA)server.  The original code is still available in this repository in thehistoricalsubdirectory.  In late 2018, Mike Hucka sought to expand the functionality of the original tools and generalize them in anticipation of having to stop using DPN because on 2018-12-04, DPN announced they were shutting down. Thus was bornEprints2bags.☺︎ AcknowledgmentsThevector artworkof a bag used as a logo for this repository was created byStoneHubfrom the Noun Project.  It is licensed under the Creative CommonsCC-BY 3.0license.We thank the following people for suggestions and ideas that led to improvements ineprints2bags: Robert Doiel, Tom Morrell, Tommy Keswick.eprints2bagsmakes use of numerous open-source packages, without which it would have been effectively impossible to developeprints2bagswith the resources we had.  We want to acknowledge this debt.  In alphabetical order, the packages are:bagit– Python library for working withBagItstyle packagescolorama– makes ANSI escape character sequences work under MS Windows terminalsdateparser– parse dates in almost any string formathumanize– helps write large numbers in a more human-readable formipdb– the IPython debuggerkeyring– access the system keyring service from Pythonlxml– an XML parsing library for Pythonplac– a command line argument parserpsutil– process and system utilitiesrequests– an HTTP library for Pythonsetuptools– library forsetup.pytermcolor– ANSI color formatting for output in terminaltwine– Twine is a utility for publishing Python packages on PyPIurllib3– HTTP client library for Pythonvalidators– data validation package for Python☮︎ Copyright and licenseCopyright (C) 2019–2023, Caltech.  This software is freely distributed under a BSD/MIT type license.  Please see theLICENSEfile for more information.