Package: pubarchiver
PubArchiverA program that creates archives of articles from specific journal sites (currentlymicroPublicationandPrompt) for sending toPorticoandPMC.Authors:Michael Hucka,Tom MorrellRepository:https://github.com/caltechlibrary/pubarchiverLicense:      BSD/MIT derivative – see theLICENSEfile for more informationTable of ContentsIntroductionInstallationUsageGetting help and supportContributingLicenseAuthors and historyAcknowledgments☀ IntroductionThe Caltech Library is the publisher of a few academic journals and provides services for them. The services include archiving in adark archive(specifically,Portico) as well as submitting articles toPMC.  The archiving process involves pulling down articles from the journals and packaging them up in a format suitable for sending to the archives.  PubArchiver is a program to help automate this process.✺ InstallationThere are multiple ways of installing PubArchiver.  Please choose the alternative that suits you.Alternative 1: installing PubArchiver usingpipxYou can usepipxto install PubArchiver. Pipx will install it into a separate Python environment that isolates the dependencies needed by PubArchiver from other Python programs on your system, and yet the resultingpubarchivercommand wil be executable from any shell – like any normal program on your computer. If you do not already havepipxon your system, it can be installed in a variety of easy ways and it is best to consultPipx's installation guidefor instructions. Once you have pipx on your system, you can install PubArchiver with the following command:pipxinstallpubarchiverPipx can also let you run PubArchiver directly usingpipx run pubarchiver, although in that case, you must always prefix everypubarchivercommand withpipx run.  Consult thedocumentation forpipx runfor more information.Alternative 2: installing PubArchiver usingpipThe instructions below assume you have a Python 3 interpreter installed on your computer.  Note that the default on macOS at least through 10.14 (Mojave) is Python2– please first install Python version 3 and familiarize yourself with running Python programs on your system before proceeding further.OnLinux,macOS, andWindowsoperating systems, you should be able to installpubarchiverwithpipfor Python 3.  To installpubarchiverfrom thePython package repository (PyPI), run the following command:python3-mpipinstallpubarchiverAs an alternative to getting it fromPyPI, you can usepipto installpubarchiverdirectly from GitHub:python3-mpipinstallgit+https://github.com/calitechlibrary/pubarchiver.gitIf you already installed PubArchiver once before, and want to update to the latest version, add--upgradeto the end of either command line above.Alternative 3: installing PubArchiver from sourcesIf  you prefer to install PubArchiver directly from the source code, you can do that too. To get a copy of the files, you can clone the GitHub repository:gitclonehttps://github.com/caltechlibrary/pubarchiverAlternatively, you can download the files as a ZIP archive using this link directly from your browser using this link:https://github.com/caltechlibrary/pubarchiver/archive/refs/heads/main.zipNext, after getting a copy of the files,  runsetup.pyinside the code directory:cdpubarchiver
python3setup.pyinstall▶︎ UsagePubArchiver is a command-line program.  The installation process should put a program namedpubarchiverin a location normally searched by your shell interpreter.  For help with usage at any time, runpubarchiverwith the option--help(or-hfor short).pubarchiver-hBasic usageOptions topubarchiveruse a dash (-) as the prefix character on macOS and Linux, and forward slash (/) on Windows.The journal whose articles are to be archived must be indicated using the required option--journal(or-jfor short). To see a list of supported journals, you can use--journal listlike this:pubarchiver--journallistIf not given any additional options besides a--journaloption to select the journal,pubarchiverwill proceed to contact the journal website as well as eitherDataCiteorCrossref, and create an archive containing articles and their metadata for all articles published to date by the journal.  The options below can be used to select articles and influence otherpubarchiverbehaviors.Printing information without doing anythingThe option--list-dois(or-lfor short) can be used to obtain a list of all DOIs for all articles published by the selected journal. When--list-doisis used,pubarchiverprints the list to the terminal and exits without doing further work. This can be useful if you intend to use the--doi-fileoption discussed below.If given the option--preview(or-pfor short),pubarchiverwillonlyprint a list of articles it will archive and stop short of creating the archive. This is useful to see what would be produced without actually doing it.  Note, however, that because it does not attempt to download the articles and associated files, it cannot report errors thatmightoccur when actually creating an archive.  Consequently, do not use the output of--previewas a prediction of eventual success or failure.Selecting the archive format and archive output locationThe value supplied after the option--dest(or-dfor short) can be used to tellpubarchiverthe intended destination where the archive will be sent.  The option changes the structure and content of the archive created bypubarchiver. The possible alternatives areporticoandpmc. Portico is assumed to be the default destination if no--destoption is given.By default,pubarchiverwill write its output to a new subdirectory it creates in the directory from whichpubarchiveris being run. The option--output-dir(or/oon Windows) can be used to select a different location. For example,pubarchiver--journalmicropublication--output-dir/tmp/micropubThe materials for each article will be written to an individual subdirectory named after the DOI of the article.  The output for each article will consist of an XML metadata file describing the article, the article itself in PDF format, and (if the journal providesJATS) a subdirectory namedjatscontaining the article in JATS XML format along with any image that may appear in the article.  The image is always converted to uncompressed TIFF format, because it is considered a good preservation format. The PMC structure follows thenaming and deliveryspecifications defined athttps://www.ncbi.nlm.nih.gov/pmc/pub/filespec-delivery/.Unless the option--no-zip(or-Zfor short) is given, the output will be archived inZIPformat.  If the output structure (as determine by the--destoption) is being generated for PMC, each article will be put into its own individual ZIP archive; else, the default action is to put the collected output of all articles into a single ZIP archive file.  The option--no-zipmakespubarchiverleave the output unarchived in the directory determined by the--output-diroption.Selecting a subset of articlesIf the option--after-dateis given,pubarchiverwill download only articles whose publication dates areafterthe given date.  Valid date descriptors are those accepted by the Pythondateparserlibrary.  Make sure to enclose descriptions within single or double quotes.  Examples:pubarchiver --after-date "2014-08-29"   ....
  pubarchiver --after-date "12 Dec 2014"  ....
  pubarchiver --after-date "July 4, 2013"  ....
  pubarchiver --after-date "2 weeks ago"  ....The option--doi-file(or-ffor short) can be used to tellpubarchiverto read a file containing DOIs and only fetch those particular articles instead of asking the journal for all articles.  The format of the file indicated after the--doi-fileoption must be a simple text file containing one DOI per line.The selection by date performed by the--after-dateoption is performed after reading the list of articles using the--doi-fileoption if present, and thus can be used to filter by date the articles whose DOIs are provided.Writing a reportAs it works,pubarchiverwrites information to the terminal about the articles it puts into the archive, including whether any problems are encountered.  To save this information to a file, use the option--rep-file(or-rfor short), which will makepubarchiverwrite a report file.  By default, the format of the report file isCSV; the option--rep-fmt(or-sfor short) can be used to selectcsvorhtml(or both) as the format.  The title of the report will be based on the current date, unless the option--rep-title(or-tfor short) is used to supply a different title.Additional command-line optionsWhenpubarchiverdownloads the JATS XML version of articles from the journal site, it will by default validate the XML content against the JATS DTD.  To skip the XML validation step, use the option--no-check(or-Xfor short).pubarchiverwill print informational messages as it works. To reduce messages to only warnings and errors, use the option--quiet(or-qfor short). Also, output is color-coded by default unless the--no-coloroption (or-Cfor short) is given; this option can be helpful if the color control sequences create problems for your terminal emulator.If given the--debugoption (or-@for short), this program will output a detailed real-time trace of what it is doing.  The output will be written to the given destination, which can be a dash character (-) to indicate console output, or a file path.If given the--versionoption (or-Vfor short), this program will print version information and exit without doing anything else.Return valuesThis program exits with a return code of0if no problems are encountered while fetching data from the server.  It returns a nonzero value otherwise, following conventions for use in shells such as bash which only understand return code values of0to255.  If no network is detected, it returns a value of 1; if it is interrupted (e.g., usingctrl-c) it returns a value of2; if it encounters a fatal error, it returns a value of3.  If it encounters any non-fatal problems (such as a missing PDF file or JATS validation error), it returns a nonzero value equal to 100 + the number of articles that had failures.   Summarizing the possible return codes:Return valueMeaning0No errors were encountered – success1No network detected – cannot proceed2The user interrupted program execution3An exception or fatal error occurred100+nEncountered non-fatal problems on a total ofnarticlesSummary of command-line optionsThe following table summarizes all the command line options available. (Note: on Windows computers,/must be used as the prefix character instead of-):ShortLong form optMeaningDefault-aA--after-dateAOnly get articles published after dateAGet all articles⬥-C--no-colorDon't color-code info messagesColor-code terminal output-dD--destDDestination for archive: Portico or PMCPortico-fF--doi-fileFOnly get articles whose DOIs are in fileFGet all articles-jJ--journalJWork with journalJ★-l--list-doisPrint a list of all known DOIs & exitDo other actions instead-oO--output-dirOWrite output in directoryOWrite in current dir-p--previewPreview what would be archived & exitObtain the articles-q--quietOnly print important messagesBe chatty while working-rR--rep-fileRWrite list of article & results in fileRDon't write a report-sS--rep-fmtSWith-r, write eitherhtmlorcsvcsv-tT--rep-titleTWith-r, useTas the report titleUse the current date-V--versionPrint program version info & exitDo other actions instead-X--no-checkDon't validate JATS XML filesValidate JATS XML-Z--no-zipDon't put output into one ZIP archiveZIP up the output-@OUT--debugOUTDebugging mode; write trace toOUTNormal mode⚑⬥   Enclose the date in quotes if it contains space characters; e.g.,"12 Dec 2014".★   Required argument.⚑   To write to the console, use the character-(a single dash) as the value ofOUT; otherwise,OUTmust be the name of a file where the output should be written.⁇ Getting help and supportIf you find an issue, please submit it inthe GitHub issue trackerfor this repository.♬ ContributingWe would be happy to receive your help and participation with enhancingpubarchiver!  Please visit theguidelines for contributingfor some tips on getting started.☥ LicenseCopyright © 2019-2022, Caltech.  This software is freely distributed under a BSD 3-clause license.  Please see theLICENSEfile for more information.❡ Authors and historyTom Morrelldeveloped the original algorithm for extracting metadata from DataCite and creating XML files for use with Portico submissions of microPublication.org articles.  Starting with that original script,Mike Huckacreated the much-expanded Microarchiver program (later renamed to PubArchiver).♥︎ AcknowledgmentsThevector artworkused as a starting point for the logo for this repository was created byCuby Designfor theNoun Project.  It is licensed under the Creative CommonsAttribution 3.0 Unportedlicense.  The vector graphics was modified by Mike Hucka to change the color.Nick Stifflerfrom themicroPublication.org teamhelped figure out the requirements for PMC output (introduced in Microarchiver version 1.9), helped guide development of Microarchiver/PubArchiver, and engaged in many discussions about microPublication.org's needs.PubArchiver makes use of numerous open-source packages, without which it would have been effectively impossible to develop PubArchiver with the resources we had.  We want to acknowledge this debt.  In alphabetical order, the packages are:Beautiful Soup– an HTML parsing librarybun– a set of basic user interface classes and functionscommonpy– a collection of commonly-useful Python functionscrossrefapi– a python library that implements the Crossref APIdateparser– parser for human-readable dateshumanize– make numbers more easily readable by humanslxml– an XML parsing library for PythonPillow– a fork of the Python Imaging Libraryplac– a command line argument parserrecordclass– a mutable version of Python named tuplessetuptools– library forsetup.pysidetrack– simple debug logging/tracing packageslack-cli– a command-line interface to Slack written inBashurllib3– a powerful HTTP library for Pythonxmltodict– a module to make working with XML feel like working with JSONFinally, we are grateful for computing & institutional resources made available by the California Institute of Technology.