Package: iga
IGAIGA is theInvenioRDM GitHub Archiver, a standalone program as well as aGitHub Actionsworkflow that lets you automatically archive GitHub software releases in anInvenioRDMrepository.Table of contentsIntroductionInstallationQuick startUsageKnown issues and limitationsGetting helpContributingLicenseAcknowledgmentsIntroductionInvenioRDMis the basis for many institutional repositories such asCaltechDATAthat enable users to preserve software and data sets in long-term archive. Though such repositories are critical resources, creating detailed records and uploading assets can be a tedious and error-prone process if done manually. This is where theInvenioRDM GitHub Archiver(IGA) comes in.IGA creates metadata records and sends releases from GitHub to an InvenioRDM-based repository server. IGA can be invoked from the command line; it also can be set up as aGitHub Actionsworkflow to archive GitHub releases automatically for a repository each time they are made.➜IGA offers many notable features:Automatic metadata extraction from GitHub pluscodemeta.jsonandCITATION.cfffilesThorough coverage ofInvenioRDM record metadatausing painstaking proceduresRecognition of identifiers in CodeMeta & CFF files:ORCID,DOI,PMCID, and moreAutomatic lookup of publication data inDOI.org,PubMed, Google, and other sourcesAutomatic lookup of organization names inROR(assuming ROR id's are provided)Automatic lookup of human names inORCID.org(assuming ORCID id's are provided)Automatic splitting of human names into family & given names usingMLmethodsSupport for InvenioRDMcommunitiesSupport for overriding the record that IGA creates, for complete control if you need itSupport for using the GitHub API without aGitHub access tokenin simple casesExtensive use of logging so you can see what's going on under the hoodInstallationIGA can be installed as either (or both) a command-line program on your computer or aGitHub Actionin a GitHub repository.IGA as a standalone programPlease choose an approach that suits your situation and preferences.Alternative 1: usingpipxPipxlets you install Python programs in a way that isolates Python dependencies from other Python programs on your system, and yet the resultingigacommand can be run from any shell and directory – like any normal program on your computer. If you usepipxon your system, you can install IGA with the following command:pipxinstalligaAfter installation, a program namedigashould end up in a location where other command-line programs are installed on your computer.  Test it by running the following command in a shell:iga--helpAlternative 2: usingpipIGA is available from thePython package repository PyPIand can be installed usingpip:python3-mpipinstalligaAs an alternative to getting it fromPyPI, you can installigadirectly from GitHub:python3-mpipinstallgit+https://github.com/caltechlibrary/iga.gitIf you already installed IGA once before, and want to update to the latest version, add--upgradeto the end of either command line above.After installation, a program namedigashould end up in a location where other command-line programs are installed on your computer.  Test it by running the following command in a shell:iga--helpAlternative 3: from sourcesIf  you prefer to install IGA directly from the source code, first obtain a copy by either downloading the source archive from theIGA releases page on GitHub, or by usinggitto clone the repository to a location on your computer. For example,gitclonehttps://github.com/caltechlibrary/igaNext, after getting a copy of the files,  runsetup.pyinside the code directory:cdiga
python3setup.pyinstallOnce you have installediga, the next steps are (1)get an InvenioRDM tokenand (2)configureigafor running locally.IGA as a GitHub Actions workflowAGitHub Actionsworkflow is an automated process that runs on GitHub's servers under control of a file in your repository. Follow these steps to create the IGA workflow file:In the main branch of your GitHub repository, create a.github/workflowsdirectoryIn the.github/workflowsdirectory, create a file named (e.g.)iga.ymland copy thefollowing contentsinto it:# ╭────────────────────────────────────────────╮# │ Configure this section                     │# ╰────────────────────────────────────────────╯env:# 👋🏻 Set the next variable to your InvenioRDM server address 👋🏻INVENIO_SERVER:https://your-invenio-server.org# Set to an InvenioRDM record ID to mark release as a new version.parent_record:none# The variables below are other IGA options. Please see the docs.community:nonedraft:falseall_assets:falseall_metadata:falsedebug:false# ╭────────────────────────────────────────────╮# │ The rest of this file should be left as-is │# ╰────────────────────────────────────────────╯name:InvenioRDM GitHub Archiveron:release:types:[published]workflow_dispatch:inputs:release_tag:description:"Thereleasetag(empty=latest):"parent_record:description:"IDofparentrecord(forversioning):"community:description:"NameofInvenioRDMcommunity(ifany):"draft:description:"Marktherecordasadraft"type:booleanall_assets:description:"AttachallGitHubassets"type:booleanall_metadata:description:"IncludeadditionalGitHubmetadata"type:booleandebug:description:"PrintdebuginfointheGitHublog"type:booleanrun-name:Archive ${{inputs.release_tag || 'latest release'}} in InvenioRDMjobs:run_iga:name:"Sendto${{needs.get_repository.outputs.server}}"runs-on:ubuntu-latestneeds:get_repositorysteps:-uses:caltechlibrary/iga@mainwith:INVENIO_SERVER:${{env.INVENIO_SERVER}}INVENIO_TOKEN:${{secrets.INVENIO_TOKEN}}all_assets:${{github.event.inputs.all_assets || env.all_assets}}all_metadata:${{github.event.inputs.all_metadata || env.all_metadata}}debug:${{github.event.inputs.debug || env.debug}}draft:${{github.event.inputs.draft || env.draft}}community:${{github.event.inputs.community || env.community}}parent_record:${{github.event.inputs.parent_record || env.parent_record}}release_tag:${{github.event.inputs.release_tag || 'latest'}}get_repository:name:"Getrepositoryname"runs-on:ubuntu-latestoutputs:server:${{steps.parse.outputs.host}}steps:-name:Extract name from INVENIO_SERVERid:parserun:echo "host=$(cut -d'/' -f3 <<< ${{env.INVENIO_SERVER}} | cut -d':' -f1)" >> $GITHUB_OUTPUTEdit the value of theINVENIO_SERVERvariable (line 7 above)↑Optionally, change the values of other options (parent_record,community, etc.)Save the file, commit the changes to git, and push your changes to GitHubOnce you have installed the GitHub Action workflow for IGA, the next steps are (1)get an InvenioRDM tokenand (2)configure the GitHub Action workflow.Quick startNo matter whether IGA is run locally on your computer or as a GitHub Actions workflow, in both cases it must be provided with a personal access token (PAT) for your InvenioRDM server. Getting one
is the first step.Getting an InvenioRDM tokenLog in to your InvenioRDM accountGo to theApplicationspage in your account profileClick theNew tokenbutton next to "Personal access tokens"On the page that is shown after you click that button, name your token (the name does not matter) and click theCreatebuttonAfter InvenioRDM creates and shows you the token,copy it to a safe locationbecause InvenioRDM will not show it againConfiguring and running IGA locallyTo send a GitHub release to your InvenioRDM server, IGA needs this information:(Required) The identity of the GitHub release to be archived(Required) The address of the destination InvenioRDM server(Required) A personal access token for InvenioRDM (fromabove)(Optional) Apersonal access token for GitHubThe identity of the GitHub release is always given as an argument to IGA on the command line; the remaining values can be provided either via command-line options or environment variables. One approach is to set environment variables in shell scripts or your interactive shell. Here is an example using Bash shell syntax, with fake token values:exportINVENIO_SERVER=https://data.caltech.eduexportINVENIO_TOKEN=qKLoOH0KYf4D98PGYQGnC09hiuqw3Y1SZllYnonRVzGJbWz2exportGITHUB_TOKEN=ghp_wQXp6sy3AsKyyEo4l9esHNxOdo6T34ZsthzOnce these are set, use of IGA can be as simple as providing a URL for a release in GitHub. For example, the following command creates a draft record (the-doption is short for--draft) for another project in GitHub and tells IGA to open (the-ooption is short for--open) the newly-created InvenioRDM entry in a web browser:iga-d-ohttps://github.com/mhucka/taupe/releases/tag/v1.2.0More options are described in the section ondetailed usage informationbelow.Configuring and running IGA as a GitHub Actions workflowAfter doing theGitHub Actions installationsteps andobtaining an InvenioRDM token, one more step is needed: the token must be stored as a "secret" in your GitHub repository.Go to theSettingspage of your GitHub repositoryIn the left-hand sidebar, findSecrets and variablesin the Security section, click on it to revealActionsunderneath, then click onActionsIn the next page, click the greenNew repository secretbuttonName the variableINVENIO_TOKENand paste in your InvenioRDM tokenFinish by clicking the greenAdd secretbuttonTesting the workflowAfter setting up the workflow and storing the InvenioRDM token in your repository on GitHub, it's a good idea to run the workflow manually to test that it works as expected.Go to theActionstab in your repository and click on the name of the workflow in the sidebar on the leftClick theRun workflowbutton in the right-hand side of the blue stripIn the pull-down, change the value of "Mark the record as a draft" totrueClick the greenRun workflowbutton near the bottomRefresh the web page and a new line will be shown named after your workflow fileClick the title of the workflow to see the IGA workflow progress and resultsRunning the workflow when releasing softwareOnce the personal access token from InvenioRDM is stored as a GitHub secret, the workflow should run automatically every time a new release is made on GitHub – no further action should be needed. You can check the results (and look for errors if something went wrong) by going to theActionstab in your GitHub repository.UsageThis section provides detailed information about IGA's operation and options to control it.Identifying the InvenioRDM serverThe server address must be provided either as the value of the option--invenio-serveror in an environment variable namedINVENIO_SERVER.  If the server address does not begin withhttps://, IGA will prepend it automatically.Providing an InvenioRDM access tokenA personal access token (PAT) for making API calls to the InvenioRDM server must be also supplied when invoking IGA. The preferred method is to set the value of the environment variableINVENIO_TOKEN. Alternatively, you can use the option--invenio-tokento pass the token on the command line, butyou are strongly advised to avoid this practice because it is insecure.To obtain a PAT from an InvenioRDM server, first log in to the server, then visit the page at/account/settings/applicationsand use the interface there to create a token. The token will be a long string of alphanumeric characters such asOH0KYf4PGYQGnCM4b53ejSGicOC4s4YnonRVzGJbWxY; set the value of the variableINVENIO_TOKENto this string.Providing a GitHub access tokenItmaybe possible to run IGA without providing a GitHub access token. GitHub allows up to 60 API calls per minute when running without credentials, and though IGA makes several API calls to GitHub each time it runs, for some public repositories IGA will not hit the limit. However, if you are archiving a private repository, run IGA multiple times in a row, or the repository has many contributors, then you will need to supply a GitHub access token. The preferred way of doing that is to set the value of the environment variableGITHUB_TOKEN. Alternatively, you can use the option--github-tokento pass the token on the command line, butyou are strongly advised to avoid this practice because it is insecure.  To obtain a PAT from GitHub, visithttps://docs.github.com/en/authenticationand follow the instructions for creating a "classic" personal access token.Note that when you run IGA as a GitHub Actions workflow, you do not need to create or set a GitHub token because it is obtained automatically by the GitHub Actions workflow.Specifying a GitHub releaseA GitHub release can be specified to IGA in one of two mutually-exclusive ways:The full URL of the web page on GitHub of a tagged release. In this case,
the URL must be the final argument on the command line invocation of IGA
and the options--accountand--repomust be omitted.A combination ofaccount name,repository name, andtag. In this
case, the final argument on the command line must be thetag, and in
addition, values for the options--accountand--repomust be provided.Here's an example using approach #1 (assuming environment variablesINVENIO_SERVER,INVENIO_TOKEN, andGITHUB_TOKENhave all been set):igahttps://github.com/mhucka/taupe/releases/tag/v1.2.0and here's the equivalent using approach #2:iga--github-accountmhucka--github-repotaupev1.2.0Note that when using this form of the command, the release tag (v1.2.0above) must be the last item given on the command line.Gathering metadata for an InvenioRDM recordThe record created in InvenioRDM is constructed using information obtained using GitHub's API as well as several other APIs as needed. The information includes the following:(if one exists) acodemeta.jsonfile in the GitHub repository(if one exists) aCITATION.cfffile in the GitHub repositorydata available from GitHub for the releasedata available from GitHub for the repositorydata available from GitHub for the account of the ownerdata available from GitHub for the accounts of repository contributorsfile assets associated with the GitHub releasedata available from ORCID.org for ORCID identifiersdata available from ROR.org for Research Organization Registry identifiersdata available from DOI.org, NCBI, Google Books, & others for publicationsdata available from spdx.org for software licensesIGA tries to useCodeMeta.jsonfirst andCITATION.cffsecond to fill out the fields of the InvenioRDM record. If neither of those files are present, IGA uses values from the GitHub repository instead. You can make it always use all sources of info with the option--all-metadata. Depending on how complete and up-to-date yourCodeMeta.jsonandCITATION.cffare, this may or may not make the record more comprehensive and may or may not introduce redundancies or unwanted values.To override the auto-created metadata, use the option--read-metadatafollowed by the path to a JSON file structured according to the InvenioRDM schema used by the destination server. When--read-metadatais provided, IGA doesnotextract the data above, but still obtains the file assets from GitHub.Specifying GitHub file assetsBy default, IGA attaches to the InvenioRDM recordonlythe ZIP file asset created by GitHub for the release. To make IGA attach all assets associated with the GitHub release, use the option--all-assets.To upload specific file assets and override the default selections made by IGA, you can use the option--filefollowed by a path to a file to be uploaded.  You can repeat the option--fileto upload multiple file assets. Note that if--fileis provided, then IGAdoes not use any file assets from GitHub; it is the user's responsibility to supply all the files that should be uploaded.If both--read-metadataand--fileare used, then IGA does not actually contact GitHub for any information.Handling communitiesTo submit your record to a community, use the--communityoption together with a community name. The option--list-communitiescan be used to get a list of communities supported by the InvenioRDM server. Note that submitting a record to a community means that the record will not be finalized and will not be publicly visible when IGA finishes; instead, the record URL that you receive will be for a draft version, pending review by the community moderators.Indicating draft versus published recordsIf the--communityoption is not used, then by default, IGA will finalize and publish the record. To make it stop short and leave the record as a draft instead, use the option--draft. The draft option also takes precedence over the community option: if you use both--draftand--community, IGA will stop after creating the draft record and willnotsubmit it to the community.  (You can nevertheless submit the record to a community manually once the draft is created, by visiting the record's web page and using the InvenioRDM interface there.)Versioning recordsThe option--parent-recordcan be used to indicate that the record being constructed is a new version of an existing record. This will make IGA use the InvenioRDM API forrecord versioning. The newly-created record will be linked to a parent record identified by the value passed to--parent-record. The value must be either an InvenioRDM record identifier (which is a sequence of alphanumeric characters of the formXXXXX-XXXXX, such asbknz4-bch35, generated by the InvenioRDM server), or a URL to the landing page of the record in the InvenioRDM server. (Note that such URLs end in the record identifier.) Here is an example of using this option:iga --parent-record xbcd4-efgh5 https://github.com/mhucka/taupe/releases/tag/v1.2.0Other options recognized by IGARunning IGA with the option--save-metadatawill make it create a metadata record, but instead of uploading the record (and any assets) to the InvenioRDM server, IGA will write the result to the given destination. This can be useful not only for debugging but also for creating a starting point for a custom metadata record: first run IGA with--save-metadatato save a record to a file, edit the result, then finally run IGA with the--read-metadataoption to use the modified record to create a release in the InvenioRDM server.The--modeoption can be used to change the run mode. Four run modes are available:quiet,normal,verbose, anddebug. The default mode isnormal, in which IGA prints a few messages while it's working. The modequietwill make it avoid printing anything unless an error occurs, the modeverbosewill make it print a detailed trace of what it is doing, and the modedebugwill make IGA even more verbose. In addition, indebugmode, IGA will drop into thepdbdebugger if it encounters an exception during execution. On Linux and macOS, debug mode also installs a signal handler on signal USR1 that causes IGA to drop into thepdbdebugger if the signal USR1 is received. (Usekill -USR1 NNN, where NNN is the IGA process id.)By default, informational output is sent to the standard output (normally the terminal console). The option--log-destcan be used to send the output to the given destination instead. The value can be-(i.e., a dash) to indicate console output, or it can be a file path to send the output to the file. A special exception is that even if a log destination is given, IGA will still print the final record URL to stdout.  This makes it possible to invoke IGA from scripts that capture the record URL while still saving diagnostic output in case debugging is needed.By default, IGA prints only the record URL when done. The option--print-doiwill make it also print the DOI of the record. (Note that this only works when publishing records; if options--draftor--communityare used, then there will be no DOI. In those case, only the URL will be printed.)Reading and writing large files may take a long time; on the other hand, IGA should not wait forever on network operations before reporting an error if a server or network becomes unresponsive. To balance these conflicting needs, IGA automatically scales its network timeout based on file sizes. To override its adaptive algorithm and set an explicit timeout value, use the option--timeoutwith a value in seconds.If given the--versionoption, this program will print its version and other information, and exit without doing anything else.Running IGA with the option--helpwill make it print help text and exit without doing anything else.Summary of command-line optionsAs explain above, IGA takes one required argument on the command line: either (1) the full URL of a web page on GitHub of a tagged release, or (2) a release tag name which is to be used in combination with options--github-accountand--github-repo. The following table summarizes all the command line options available.Long form optionShortMeaningDefault--all-assets-AAttach all GitHub assetsAttach only the release source ZIP--all-metadata-MInclude additional metadata from GitHubFavor CodeMeta & CFF--communityC-cCSubmit record to RDM communityCDon't submit record to any community--draft-dMark the RDM record as a draftPublish record when done--fileF-fFUpload local fileFinstead of GitHub assetsUpload only GitHub assets⚑--github-accountA-aALook in GitHub accountAGet account name from release URL✯--github-repoR-rRLook in GitHub repositoryRof accountAGet repo name from release URL✯--github-tokenT-tTUse GitHub access tokenTUse value in env. var.GITHUB_TOKEN--help-hPrint help info and exit--invenio-serverS-sSSend record to InvenioRDM server at addressSUse value in env. var.INVENIO_SERVER--invenio-tokenK-kKUse InvenioRDM access tokenKUse value in env. var.INVENIO_TOKEN--list-communities-LList communities available for use with--community--log-destL-lLWrite log output to destinationLWrite to terminal⚐--modeM-mMRun in modequiet,normal,verbose, ordebugnormal--open-oOpen record's web page in a browser when doneDo nothing when done--parent-recordN-pNMake this a new version of existing recordNNew record is unrelated to other records❖--print-doi-iPrint both the DOI & record URL when donePrint only the record URL--read-metadataR-RRRead metadata record from fileR; don't build oneBuild metadata record--save-metadataD-SDSave metadata record to fileD; don't upload itUpload to InvenioRDM server--timeoutX-TXWait on network operations a max ofXsecondsAuto-adjusted based on file size--version-VPrint program version info and exit⚑   Can repeat the option to specify multiple files.⚐   To write to the console, use the character-as the value ofOUT; otherwise,OUTmust be the name of a file where the output should be written.✯   When using--github-accountand--github-repo, the last argument on the command line must be a release tag name.❖   The record identifier must be given either as a sequence of alphanumeric characters of the formXXXXX-XXXXX(e.g.,bknz4-bch35), or as a URL to the landing page of an existing record in the InvenioRDM server.Return valuesThis program exits with a return status code of 0 if no problem is encountered.  Otherwise, it returns a nonzero status code. The following table lists the possible values:CodeMeaning0success – program completed normally1interrupted2encountered a bad or missing value for an option3encountered a problem with a file or directory4encountered a problem interacting with GitHub5encountered a problem interacting with InvenioRDM6the personal access token was rejected7an exception or fatal error occurredKnown issues and limitationsThe following are known issues and limitations.As of mid-2023, InvenioRDM requires names of record creators and other contributors to be split into given (first) and family (surname). This is problematic for multiple reasons. The first is that mononyms are common in many countries: a person's name may legitimately be only a single word which is not conceptually a "given" or "family" name.  To compound the difficulty for IGA, names are stored as single fields in GitHub account metadata, so unless a repository has acodemeta.jsonorCITATION.cfffile (which allow authors more control over how they want their names represented), IGA is forced to try to split the single GitHub name string into two parts.A foolproof algorithm for doing this does not exist, so IGA will sometimes get it wrong. (That said, IGA goes to extraordinary lengths to try to do a good job.)InvenioRDM requires that identities (creators, contributors, etc.) to be labeled as personal or organizational. The nature of identities is usually made clear incodemeta.jsonandCITATION.cfffiles. GitHub also provides a flag that is meant to be used to label organizational accounts, but sometimes people don't set the GitHub account information correctly. Consequently, if IGA has to use GitHub data to get (e.g.) the list of contributors on a project, it may mislabel identities in the InvenioRDM record it produces.Some accounts on GitHub are software automation or "bot" accounts, but are not labeled as such. These accounts are generally indistinguishable from human accounts on GitHub, so if they're not labeled as bot or organizational accounts in GitHub, IGA can't recognize that they're humans. If such an account is the creator of a release in GitHub, and IGA tries to use its name-splitting algorithm on the name of the account, it may produce a nonsensical result. For example, it might turn "Travis CI" into an entry with a first name of "Travis" and last name of "CI".Funder and funding information can only be specified incodemeta.jsonfiles; neither GitHub norCITATION.cffhave provisions to store this kind of metadata. The CodeMeta specification defines two fields for this purpose:funderandfunding. Unfortunately, these map imperfectly to the requirements of InvenioRDM's metadata format. In addition, people don't always follow the CodeMeta guidelines, and sometimes they write funding information as text strings (instead of structured objects), the interpretation of which would require software that can recognize grant and funding agency information from free-text descriptions. This combination of factors means IGA often can't fill in the funding metadata in InvenioRDM records even if there is some funding information in thecodemeta.jsonfile.Getting helpIf you find an issue, please submit it inthe GitHub issue trackerfor this repository.ContributingYour help and participation in enhancing IGA is welcome!  Please visit theguidelines for contributingfor some tips on getting started.LicenseSoftware produced by the Caltech Library is Copyright © 2023 California Institute of Technology.  This software is freely distributed under a BSD-style license.  Please see theLICENSEfile for more information.AcknowledgmentsThis work was funded by the California Institute of Technology Library.IGA uses multiple other open-source packages, without which it would have taken much longer to write the software. I want to acknowledge this debt. In alphabetical order, the packages are:Aenum– package for advanced enumerationsArrow– a library for creating & manipulating datesBoltons– package of miscellaneous Python utilitiescaltechdata_api– package for using the CaltechDATA APICommonPy– a collection of commonly-useful Python functionsdemoji– find or remove emojis from textdirtyjson– JSON decoder that copes with problematic JSON files and reports useful error messagesflake8– Python code linter and style analyzerhttpx– HTTP client library that supports HTTP/2humanize– make numbers more easily readable by humansidutils– package for validating and normalizing various kinds of persistent identifiersipdb– the IPython debuggeriptools– utilities for dealing with IP addressesisbnlib– utilities for dealing with ISBNsjson5– extended JSON format parserlatexcodec– lexer and codec to work with LaTeX code in Pythonlinkify-it-py– a link recognition library with full unicode supportlxml– an XML parsing libraryMarkdown– Python package for working with Markdownmarkdown-checklist– GitHub-style checklist extension for Python Markdown packagemdx-breakless-lists– GitHub-style Markdown lists that don't require a line break above themmdx_linkify– extension for Python Markdown will convert text that look like links to HTML anchorsMyST-parser– A Sphinx and Docutils extension to parse an extended version of Markdownnameparser– package for parsing human names into their individual componentsprobablepeople– package for parsing names into components using ML-based techniquespybtex– BibTeX parser and formatterpybtex-apa7-style– plugin forpybtexthat provides APA7 style formattingpymdown-extensions– extensions for Python Markdownpytest– testing frameworkpytest-cov– coverage reports for use withpytestpytest-mock– wrapper around themockpackage for use withpytestPyYAML– YAML parserRich– library for writing styled text to the terminalrich-click– CLI interface built on top ofRichsetuptools– library forsetup.pySidetrack– simple debug logging/tracing packagespaCy– Natural Language Processing packagespacy-alignments– alternate alignments forspaCyspacy-legacy–spaCylegacy functions and architectures for backwards compatibilityspacy-loggers– loggers forspaCyspacy-pkuseg– Chinese word segmentation toolkit forspaCyspacy-transformers– pretrained Transformers forspaCySphinx– documentation generator for Pythonsphinx-autobuild– rebuild Sphinx docs automaticallysphinx-material– a responsive Material Design theme for Sphinxsphinxcontrib-mermaid– support Mermaid diagrams in Sphinx docsStringDist– library for calculating string distancesTwine– utilities for publishing Python packages onPyPIurl-normalize– URI/URL normalization utilitiesvalidators– data validation package for Pythonwheel– setuptools extension for building wheels