DOI: 10.5281/zenodo.8327611
title: Catalogue of Life Repackaged and Sorted hash://sha256/e7130fb557d9aee033ac7147f4d5c4c75f12223dd43e53c7cbb141372f9579cd hash://md5/882b8744b3ebd5fae371fa659ee52a2b

description: Taxonomic name alignment is a necessary, and often time consuming, task when integration biodiversity datasets for re-use. This publication aims to facilitate fast (e.g., &gt; 1k names/s), offline-enabled, and reproducible, name alignment workflows through the Catalogue of Life by repackaging optimized versions of the wealth of names contained in the Catalogue of Life. <strong>Introduction</strong> The Catalogue of Life (Bánki 2023) is “[…] is an assembly of expert-based global species checklists with the aim to build a comprehensive catalogue of all known species of organisms on Earth. […]” This data publication contains a verifiable copy of the Catalogue of Life as well as a reverse sorted version of the NameUsage.tsv table. The aims of this publication are to: provide a signed citation (Elliott, Poelen, and Fortes 2023) for a copy of Catalogue of Life prepare Catalogue of Life to be included in the Nomer Corpus of Taxonomic Resources (J. H. (ed. ). Poelen 2023). pre-process the Catalogue of Life to facilitate optimized indexing and offline taxonomic name alignments using tools like Nomer (J. Poelen and Salim 2023). Overall, the publications aims to facilitate taxonomic name alignment using the wealth of information provided by the Catalogue of Life to help enable fast, reproduceable, offline-enabled alignment of namelists with taxonomic resources of known provenance (or origin). An example of an application facilitated by this publication is the Taxonomic Name Alignment tool as provided through https://github.com/globalbioticinteractions/name-alignment-template. This template repository implements an automated workflow using GitHub Action to align scientific names in csv/tsv files and darwin core archive with common taxonomic name lists like Catalogue of Life, NCBI Taxonomy, Integrated Taxonomic Information System (ITIS), and GBIF Backbone taxonomy. <strong>Methods</strong> To capture and process the Catalogue of Life, the following steps were taken: track and archive a copy of Catalogue of Life reverse sort NameUsage.tsv assign an alias to the processed resources Steps 1-3 are captured and documented using Preston, a biodiversity data tracker. Preston not only helps to documents the steps, but also includes the digital resources that were used and produced. Track and Archive To track and archive a copy of Catalogue of Life, the following command was issued: <pre><code>preston track https://download.catalogueoflife.org/col/latest_coldp.zip</code></pre> With this, a copy of https://download.catalogueoflife.org/col/latest_coldp.zip is downloaded and their sha256 checksum (or hash) is calculated. Also, the download process is captured machine-readable rdf/nquads statement. The content id (or sha256 hash) of the copy included in this publication can be found using: <pre><code>preston alias\ --anchor hash://sha256/e7130fb557d9aee033ac7147f4d5c4c75f12223dd43e53c7cbb141372f9579cd\ --remote https://zenodo.org/deposit/8327611/files\ https://download.catalogueoflife.org/col/latest_coldp.zip\ head -n1</code></pre> and is <pre><code>&lt;https://download.catalogueoflife.org/col/latest_coldp.zip&gt; &lt;http://purl.org/pav/hasVersion&gt; &lt;hash://sha256/d512d769a3e68b6f3be523b97f9f3c05b10d317607f95cb24ddcda27bded03e1&gt; &lt;urn:uuid:f00d82a2-d965-4b1d-9030-0a2f8833e004&gt; .</code></pre> meaning that <code>hash://sha256/d512d769a3e68b6f3be523b97f9f3c05b10d317607f95cb24ddcda27bded03e1</code> is the content id (or sha256 hash) of the content produced by <code>https://download.catalogueoflife.org/col/latest_coldp.zip</code> at the time this publication was compiled. Reverse Sort NameUsage.tsv The second step was to pre-process (or transform) part of the tracked Catalogue of Life data package and reverse sort their content. We use the following bash script to do this processing: <pre><code>(sed -u 1q; LC_ALL=C sort -r) | gzip</code></pre> The script first prints the header (i.e., <code>sed -u 1q</code>), then reverse sorts the following content (i.e., <code>LC_ALL=C sort -r</code>). Finally, the output is compressed using gzip. This script is part of this data publication, and can be retrieved via: <pre><code>preston cat\ --remote https://zenodo.org/record/8327611/files/\ hash://sha256/03292e3e40a04c565c83debd26ed521516f158a29c011b57a96d1c720c40b6cd</code></pre> To help make it easier to reference the script, an alias was created using <pre><code>preston alias\ urn:example:reverse-sort.sh\ hash://sha256/03292e3e40a04c565c83debd26ed521516f158a29c011b57a96d1c720c40b6cd ~~ Following the authoring of the reverse sort script, and documenting their alias (i.e. 'urn:example:reverse-sort.sh`), we applied the script to the acquired copy of the Catalogue of Life using: </code></pre> preston cat<br> –remote https://zenodo.org/record/8327611/files/<br> ‘zip:hash://sha256/d512d769a3e68b6f3be523b97f9f3c05b10d317607f95cb24ddcda27bded03e1!/NameUsage.tsv’<br> | preston bash<br> –anchor hash://sha256/e7130fb557d9aee033ac7147f4d5c4c75f12223dd43e53c7cbb141372f9579cd<br> –remote https://zenodo.org/record/8327611/files/<br> -c urn:example:reverse-sort.sh ~~~ The result of this process was the content identified by sha256 hash <pre><code>hash://sha256/1008433bfa5fe7fb059a720eddfe995e18b3e0e8f25ac0c990f1477c177d18cc</code></pre> as documented in line 22 of the associated preston processing log in <pre><code>preston cat\ --remote https://zenodo.org/record/8327611/files/\ 'line:hash://sha256/8ba35deafc847f0d5d69d357241a431b7fd9b6f2735189575b2b7168d523caa9!/L22' </code></pre> Assign An Alias to Processed Resources The alias ‘col:NameUsage.tsv.gz’ was defined to help make it easier to point to the result using: <pre><code>preston alias\ col:NameUsage.tsv.gz\ hash://sha256/1008433bfa5fe7fb059a720eddfe995e18b3e0e8f25ac0c990f1477c177d18cc</code></pre> With this, the following command was executed to list the first three lines of the produced resource: <pre><code>preston cat\ --anchor hash://sha256/e7130fb557d9aee033ac7147f4d5c4c75f12223dd43e53c7cbb141372f9579cd\ --remote https://zenodo.org/record/8327611/files/\ col:NameUsage.tsv.gz\ | gunzip\ | head -n3</code></pre> Where <code>preston cat ...</code> prints the produced resource, <code>gunzip</code> uncompresses the result, and the first three lines are selected using <code>head -n3</code>. The result of the operation is shown below: col:ID col:alternativeID col:nameAlternativeID col:sourceID col:parentID col:basionymID col:status col:scientificName col:authorship col:rank col:notho col:uninomial col:genericName col:infragenericEpithet col:specificEpithet col:infraspecificEpithet col:cultivarEpithet col:namePhrase col:nameReferenceID col:publishedInYear col:publishedInPage col:publishedInPageLink col:code col:nameStatus col:accordingToID col:accordingToPage col:accordingToPageLink col:referenceID col:scrutinizer col:scrutinizerID col:scrutinizerDate col:extinct col:temporalRangeStart col:temporalRangeEnd col:environment col:species col:section col:subgenus col:genus col:subtribe col:tribe col:subfamily col:family col:superfamily col:suborder col:order col:subclass col:class col:subphylum col:phylum col:kingdom col:sequenceIndex col:branchLength col:link col:nameRemarks col:remarks ffc77d7d-2ede-49ff-ab12-03410a1c25db 55434 93MTR 4WGYN provisionally accepted [Semiothisa] lapidata Warren, 1906 species zoological acceptable false ff82a38f-348b-4fd7-891e-a2a68d8edce4 55434 93MTR 78YFQ provisionally accepted [Sabulodes] arnissa Druce, 1891 species zoological acceptable false <strong>Results</strong> As described in our methods, this publication derived the resource with alias col:NameUsage.tsv.gz and content id <code>hash://sha256/1008433bfa5fe7fb059a720eddfe995e18b3e0e8f25ac0c990f1477c177d18cc</code> . This resource contains a reverse-sorted copy of the NameUsage.tsv provided in the Catalogue of Life data package retrieved from https://download.catalogueoflife.org/col/latest_coldp.zip with content identifier <code>hash://sha256/d512d769a3e68b6f3be523b97f9f3c05b10d317607f95cb24ddcda27bded03e1</code>. The following tools were used to process the Catalogue of Life resource: Tools used in this data publication tool name preston bash gzip sed head sort <strong>Discussion</strong> This publication is intended to facilitate re-use of the Catalogue of Life data package in taxonomic name alignment workflows. While the primary goal was to generate a resource for use in Nomer v0.4.5 (J. Poelen and Salim 2023), other usage can be imagined such as: Lots of Copies Keeps Stuff Safe (LOCKSS (Maniatis et al. 2005)): keep an identical copy of Catalogue of Life data package outside of the Catalogue of Life infrastructure. demonstrating how data transformation processes can be documented using Preston making a streamable copy of a reverse-sorted copy of Catalogue of Life available via https://zenodo.org/record/8327611/files/1008433bfa5fe7fb059a720eddfe995e18b3e0e8f25ac0c990f1477c177d18cc for use in workflows like looking up the first record that contain <em>Enhydra lutris</em> (Sea otter): <pre><code> curl -L 'https://zenodo.org/record/8327611/files/1008433bfa5fe7fb059a720eddfe995e18b3e0e8f25ac0c990f1477c177d18cc'\ | gunzip\ | grep "Enhydra lutris"\ | head -n1</code></pre> <strong>Acknowledgements</strong> This work stands on the shoulders of contributors to open source software and openly accessible datasets. Thank you! <strong>References</strong> Bánki, O et al. 2023. “Catalogue of Life Checklist (Version 2023-08-17).” <em>Catalogue of Life</em>. Catalogue of Life. https://doi.org/10.48580/dft7. Elliott, Michael J., Jorrit H. Poelen, and José A. B. Fortes. 2023. “Signing Data Citations Enables Data Verification and Citation Persistence.” <em>Scientific Data</em> 10 (1). https://doi.org/10.1038/s41597-023-02230-y. Maniatis, Petros, Mema Roussopoulos, Thomas J Giuli, David SH Rosenthal, and Mary Baker. 2005. “The LOCKSS Peer-to-Peer Digital Preservation System.” <em>ACM Transactions on Computer Systems (TOCS)</em> 23 (1): 2–50. Poelen, Jorrit H. (ed.). 2023. “Nomer Corpus of Taxonomic Resources hash://sha256 /12051b8aa59930d6561a3ed46b7cf3f67a31a98445a457d78 894f6b8a8e81641 hash://md5/1ff6b3628d7afc15b882cc0c9b1c3815.” Zenodo. https://doi.org/10.5281/zenodo.8326175. Poelen, Jorrit, and José Augusto Salim. 2023. “Globalbioticinteractions/Nomer: 0.5.4.” Zenodo. https://doi.org/10.5281/zenodo.8329422.