Package: microarchiver
MicroarchiverA program to create archives of articles frommicroPublication.orgfor sending toPortico.Authors:Michael Hucka,Tom MorrellRepository:https://github.com/caltechlibrary/microarchiverLicense:      BSD/MIT derivative – see theLICENSEfile for more informationTable of ContentsIntroductionInstallationUsageKnown issues and limitationsGetting help and supportContributingLicenseAuthors and historyAcknowledgments☀ IntroductionThe Caltech Library is the publisher of the online journalmicroPublicationand provides services to the journal that include archiving the journal in a dark archive (specifically,Portico).  The archiving process involves pulling down articles frommicroPublicationand packaging them up in a format suitable for sending to Portico.microarchiveris a program to automate this process.✺ InstallationOnLinux,macOS, andWindowsoperating systems, you should be able to install Microarchiver directly from the GitHub repository usingpip.  If you don't have thepippackage or are uncertain if you do, first run the following command in a terminal command line interpreter:sudo python3 -m ensurepipThen, install this software by running the following command on your computer:python3-mpipinstallgit+https://github.com/caltechlibrary/microarchiver.gitAlternatively, you can clone this GitHub repository and then runsetup.py:gitclonehttps://github.com/caltechlibrary/microarchiver.gitcdmicroarchiver
python3-mpipinstall.▶︎ UsageMicroarchiver is a command-line program.  The installation process should put a program namedmicroarchiverin a location normally searched by your shell interpreter.  For help with usage at any time, runmicroarchiverwith the option-h(or/hon Windows).microarchiver-hBasic usageThe simplest use ofmicroarchiverinvolves running it without any options.  By default, it will contactmicroPublication.orgto get a list of current articles, and create an archive of all the articles in a subdirectory of the current directory.microarchiverIf given the option-o(or/oon Windows), the output will be written to the directory named after the-o.  (If no-ois given, the output will be written to the current directory instead.)  For example:microarchiver-o/tmp/micropublication-archiveIf the option-dis given,microarchiverwill download only articles whose publication dates areafterthe given date.  Valid date descriptors are those accepted by the Pythondateparserlibrary.  Make sure to enclose descriptions within single or double quotes.  Examples:microarchiver -d "2014-08-29"   ....
  microarchiver -d "12 Dec 2014"  ....
  microarchiver -d "July 4, 2013"  ....
  microarchiver -d "2 weeks ago"  ....As it works,microarchiverwrites information to the terminal about the archives it puts into the archive, including whether any problems are encountered. To save this info to a file, use the option-r(or/ron Windows), which will makemicroarchiverwrite a report file.  By default, the format isCSV; to change the format of the report to HTML, use the option-f html(or/f htmlon Windows).  The title of the report will be named after the current date, unless the option-t(or/ton Windows) is used to supply a different title.The following is a screen recording of an actual run ofmicroarchiver:Previewing the list of articlesIf given the option-p(or/pon Windows),microarchiverwillonlyprint a list of articles it will archive and stop short of creating the archive. This is useful to see what would be produced without actually doing it.  However, note that because it does not attempt to download the articles and associated files, it will not be able to report on errors that might occur when not operating in preview mode.  Consequently, do not use the output of-pas a prediction of eventual success or failure.If given the option-g(or/gon Windows),microarchiverwillonlywrite a file namedarticle-list.xmlcontaining the complete current article list from the micropublication.org server, and exit without doing anything else.  This is useful as a starting point for creating the file used by option-a.  It's probably a good idea to redirect the output to a file; e.g.,microarchiver -g > article-list.xmlOutputUnless given the option-gor-p, microarchiver will download articles from micropublication.org and create archive files out of them.The value supplied after the option-s(or/son Windows) determines the structure of the archive generated by this program.  Currently, two output structures are supported: PMC, and a structure suitable for Portico.  (The PMC structure follows the "naming and delivery" specifications defined athttps://www.ncbi.nlm.nih.gov/pmc/pub/filespec-delivery/.) If the output will be sent to PMC, use-s pmc; else, use-s porticoor leave off the option altogether (because Portico is the default).The output will be written to the directory indicated by the value of the option-o(or/oon Windows).  If no-ois given, the output will be written to the directory in whichmicroarchiverwas started. Each article will be written to a subdirectory named after the DOI of the article. The output for each article will consist of an XML metadata file describing the article, the article itself in PDF format, and a subdirectory namedjatscontaining the article in JATS XML format along with any image that may appear in the article.  The image is always converted to uncompressed TIFF format (because it is considered a good preservation format).Unless the option-Z(or/Zon Windows) is given, the output will be archived inZIPformat. If the output structure (as determine by the-soption) is being generated for PMC, each article will be put into its own individual ZIP archive; else, the default action is to put the collected output of all articles into a single ZIP archive file.  The option-Zmakesmicroarchiverleave the output unarchived in the output directory determined by the-ooption.Additional command-line optionsIf given the option-a(or/aon Windows) followed by a file name, the given file will be read for the list of articles instead of getting the list from the server. The contents of the file can be either a list of DOIs, or article data in the same XML format as the list obtained from micropublication.org.  (See option-gabove for a way to get an article list in XML from the server.)Microarchiver always downloads the JATS XML version of articles from micropublication.org (in addition to downloading the PDF version), and by default, microarchiver validates the XML content against the JATS DTD.  To skip the XML validation step, use the option-X(/Xon Windows).microarchiverwill print informational messages as it works. To reduce messages to only warnings and errors, use the option-q(or/qon Windows). Also, output is color-coded by default unless the-Coption (or/Con Windows) is given; this option can be helpful if the color control sequences create problems for your terminal emulator.If given the-@option (/@on Windows), this program will output a detailed real-time trace of what it is doing.  The output will be written to the given destination, which can be a dash character (-) to indicate console output, or a file path.If given the-Voption (/Von Windows), this program will print version
information and exit without doing anything else.Return valuesThis program exits with a return code of0if no problems are encountered while fetching data from the server.  It returns a nonzero value otherwise, following conventions for use in shells such as bash which only understand return code values of0to255.  If no network is detected, it returns a value of 1; if it is interrupted (e.g., usingctrl-c) it returns a value of2; if it encounters a fatal error, it returns a value of3.  If it encounters any non-fatal problems (such as a missing PDF file or JATS validation error), it returns a nonzero value equal to 100 + the number of articles that had failures.   Summarizing the possible return codes:Return valueMeaning0No errors were encountered – success1No network detected – cannot proceed2The user interrupted program execution3An exception or fatal error occurred100+nEncountered non-fatal problems on a total ofnarticlesSummary of command-line optionsThe following table summarizes all the command line options available. (Note: on Windows computers,/must be used as the prefix character instead of-):ShortLong form optMeaningDefault-aA--articlesAGet list of articles from fileAGet list from server-C--no-colorDon't color-code the outputColor the terminal output-dD--after-dateDOnly get articles published after dateDGet all articles⬥-fF--rep-formatFReport format, eitherhtmlorcsvcsv-g--get-xmlPrint the server's article list & exitDo other actions instead-oO--output-dirOWrite output in directoryOWrite in current dir-p--previewPreview what would be obtainedObtain the articles-q--quietOnly print important messagesBe chatty while working-rR--rep-fileRWrite list of article & results in fileRDon't write a report-sS--structureSStructure output for Portico or PMCPortico-tT--rep-titleTUseTas the report titleUse the current date-V--versionPrint program version info and exitDo other actions instead-X--no-checkDon't validate JATS XML filesValidate JATS XML-Z--no-zipDon't put output into one ZIP archiveZIP up the output-@OUT--debugOUTDebugging mode; write trace toOUTNormal mode⚑⬥   Enclose the date in quotes if it contains space characters; e.g.,"12 Dec 2014".⚑   To write to the console, use the character-(a single dash) as the value ofOUT; otherwise,OUTmust be the name of a file where the output should be written.⚑ Known issues and limitationsCurrently, the only way to indicate that a subset of articles should be obtained from microPublication.org is to use the option-ain combination with a file that contains the list of desired articles, or the-doption to indicate a cut-off for the article publication date.⁇ Getting help and supportIf you find an issue, please submit it inthe GitHub issue trackerfor this repository.♬ ContributingWe would be happy to receive your help and participation with enhancingmicroarchiver!  Please visit theguidelines for contributingfor some tips on getting started.☥ LicenseCopyright © 2019-2020, Caltech.  This software is freely distributed under a BSD/MIT type license.  Please see theLICENSEfile for more information.❡ Authors and historyTom Morrelldeveloped the original algorithm for extracting metadata from DataCite and creating XML files for use with Portico submissions of microPublication.org articles.Mike Huckacreated the much-expanded second version of the software, now known as Microarchiver.♥︎ AcknowledgmentsThevector artworkused as a starting point for the logo for this repository was created byThomas Helbigfor theNoun Project.  It is licensed under the Creative CommonsAttribution 3.0 Unportedlicense.  The vector graphics was modified by Mike Hucka to change the color.Nick Stifflerfrom themicroPublication.org teamhelped figure out the requirements for PMC output (introduced in version 1.9), helped guide development of microarchiver, and engaged in many discussions about microPublication.org's needs.Microarchiver makes use of numerous open-source packages, without which it would have been effectively impossible to develop Microarchiver with the resources we had.  We want to acknowledge this debt.  In alphabetical order, the packages are:colorful– terminal/text string stylingcommonpy– a collection of commonly-useful Python functionsdateparser– parser for human-readable datesdateutil– extensions to the Pythondatetimemodulehumanize– make numbers more easily readable by humanslxml– an XML parsing library for PythonPillow– a fork of the Python Imaging Libraryplac– a command line argument parserpypubsub– a publish-and-subscribe message-passing library for Pythonrecordclass– a mutable version of Python named tuplesrequests– an HTTP library for Pythonsetuptools– library forsetup.pyslack-cli– a command-line interface to Slack written inBashurllib3– a powerful HTTP library for Pythonxmltodict– a module to make working with XML feel like working with JSONwxPython– a cross-platform GUI toolkit for the Python languageFinally, we are grateful for computing & institutional resources made available by the California Institute of Technology.