DOI: 10.5281/zenodo.8329453
title: Catalogue of Life Repackaged and Sorted hash://sha256/e7130fb557d9aee033ac7147f4d5c4c75f12223dd43e53c7cbb141372f9579cd hash://md5/882b8744b3ebd5fae371fa659ee52a2b

description: Taxonomic name alignment is a necessary, and often time consuming, task when integration biodiversity datasets for re-use. This publication aims to facilitate fast (e.g., > 1k names/s), offline-enabled, and reproducible, name alignment workflows through the Catalogue of Life by repackaging optimized versions of the wealth of names contained in the Catalogue of Life.

Introduction

The Catalogue of Life (Bánki 2023) is “[…] an assembly of expert-based global species checklists with the aim to build a comprehensive catalogue of all known species of organisms on Earth. […]”

This data publication contains a verifiable copy of the Catalogue of Life as well as a reverse sorted version of the NameUsage.tsv table. The aims of this publication are to:



provide a signed citation (Elliott, Poelen, and Fortes 2023) for a copy of Catalogue of Life

prepare Catalogue of Life to be included in the Nomer Corpus of Taxonomic Resources (J. H. (ed. ). Poelen 2023).

pre-process the Catalogue of Life to facilitate optimized indexing and offline taxonomic name alignments using tools like Nomer (J. Poelen and Salim 2023).


Overall, the publications aims to facilitate taxonomic name alignment using the wealth of information provided by the Catalogue of Life to help enable fast, reproduceable, offline-enabled alignment of namelists with taxonomic resources of known provenance (or origin).

An example of an application facilitated by this publication is the Taxonomic Name Alignment tool as provided through https://github.com/globalbioticinteractions/name-alignment-template. This template repository implements an automated workflow using GitHub Action to align scientific names in csv/tsv files and darwin core archive with common taxonomic name lists like Catalogue of Life, NCBI Taxonomy, Integrated Taxonomic Information System (ITIS), and GBIF Backbone taxonomy.

Methods

To capture and process the Catalogue of Life, the following steps were taken:



track and archive a copy of Catalogue of Life

reverse sort NameUsage.tsv

assign an alias to the processed resources


Steps 1-3 are captured and documented using Preston, a biodiversity data tracker. Preston not only helps to documents the steps, but also includes the digital resources that were used and produced.

Track and Archive

To track and archive a copy of Catalogue of Life, the following command was issued:

preston track https://download.catalogueoflife.org/col/latest_coldp.zip

With this, a copy of https://download.catalogueoflife.org/col/latest_coldp.zip is downloaded and their sha256 checksum (or hash) is calculated. Also, the download process is captured machine-readable rdf/nquads statement.

The content id (or sha256 hash) of the copy included in this publication can be found using:

preston alias\
  --anchor hash://sha256/e7130fb557d9aee033ac7147f4d5c4c75f12223dd43e53c7cbb141372f9579cd \
  --remote https://zenodo.org/deposit/8327611/files \
   https://download.catalogueoflife.org/col/latest_coldp.zip \
  head -n1

and is

<https://download.catalogueoflife.org/col/latest_coldp.zip> <http://purl.org/pav/hasVersion> <hash://sha256/d512d769a3e68b6f3be523b97f9f3c05b10d317607f95cb24ddcda27bded03e1> <urn:uuid:f00d82a2-d965-4b1d-9030-0a2f8833e004> .

meaning that hash://sha256/d512d769a3e68b6f3be523b97f9f3c05b10d317607f95cb24ddcda27bded03e1 is the content id (or sha256 hash) of the content produced by https://download.catalogueoflife.org/col/latest_coldp.zip at the time this publication was compiled.

Reverse Sort NameUsage.tsv

The second step was to pre-process (or transform) part of the tracked Catalogue of Life data package and reverse sort their content. We use the following bash script to do this processing:

(sed -u 1q; LC_ALL=C sort -r) | gzip

The script first prints the header (i.e., sed -u 1q), then reverse sorts the following content (i.e., LC_ALL=C sort -r). Finally, the output is compressed using gzip.

This script is part of this data publication, and can be retrieved via:

preston cat \
 --remote https://zenodo.org/record/8327611/files/ \
 hash://sha256/03292e3e40a04c565c83debd26ed521516f158a29c011b57a96d1c720c40b6cd

To help make it easier to reference the script, an alias was created using

preston alias \
 urn:example:reverse-sort.sh \
 hash://sha256/03292e3e40a04c565c83debd26ed521516f158a29c011b57a96d1c720c40b6cd

Following the authoring of the reverse sort script, and documenting their alias (i.e. 'urn:example:reverse-sort.sh`), we applied the script to the acquired copy of the Catalogue of Life using:

preston cat \
 --remote https://zenodo.org/record/8327611/files/ \
'zip:hash://sha256/d512d769a3e68b6f3be523b97f9f3c05b10d317607f95cb24ddcda27bded03e1!/NameUsage.tsv' \
| preston bash \
 --anchor hash://sha256/e7130fb557d9aee033ac7147f4d5c4c75f12223dd43e53c7cbb141372f9579cd \
 --remote https://zenodo.org/record/8327611/files/ \
-c urn:example:reverse-sort.sh 

 

The result of this process was the content identified by sha256 hash

hash://sha256/1008433bfa5fe7fb059a720eddfe995e18b3e0e8f25ac0c990f1477c177d18cc

as documented in line 22 of the associated preston processing log in

preston cat \
  --remote https://zenodo.org/record/8327611/files/ \
 'line:hash://sha256/8ba35deafc847f0d5d69d357241a431b7fd9b6f2735189575b2b7168d523caa9!/L22' 

Assign An Alias to Processed Resources

The alias ‘col:NameUsage.tsv.gz’ was defined to help make it easier to point to the result using:

preston alias \
  col:NameUsage.tsv.gz \
  hash://sha256/1008433bfa5fe7fb059a720eddfe995e18b3e0e8f25ac0c990f1477c177d18cc

With this, the following command was executed to list the first three lines of the produced resource:

preston cat \
 --anchor hash://sha256/e7130fb557d9aee033ac7147f4d5c4c75f12223dd43e53c7cbb141372f9579cd \
 --remote https://zenodo.org/record/8327611/files/ \
 col:NameUsage.tsv.gz \
 | gunzip \
 | head -n3

Where preston cat ... prints the produced resource, gunzip uncompresses the result, and the first three lines are selected using head -n3.

The result of the operation is shown below:




col:ID
col:alternativeID
col:nameAlternativeID
col:sourceID
col:parentID
col:basionymID
col:status
col:scientificName
col:authorship
col:rank
col:notho
col:uninomial
col:genericName
col:infragenericEpithet
col:specificEpithet
col:infraspecificEpithet
col:cultivarEpithet
col:namePhrase
col:nameReferenceID
col:publishedInYear
col:publishedInPage
col:publishedInPageLink
col:code
col:nameStatus
col:accordingToID
col:accordingToPage
col:accordingToPageLink
col:referenceID
col:scrutinizer
col:scrutinizerID
col:scrutinizerDate
col:extinct
col:temporalRangeStart
col:temporalRangeEnd
col:environment
col:species
col:section
col:subgenus
col:genus
col:subtribe
col:tribe
col:subfamily
col:family
col:superfamily
col:suborder
col:order
col:subclass
col:class
col:subphylum
col:phylum
col:kingdom
col:sequenceIndex
col:branchLength
col:link
col:nameRemarks
col:remarks




ffc77d7d-2ede-49ff-ab12-03410a1c25db
 
 
55434
93MTR
4WGYN
provisionally accepted
[Semiothisa] lapidata
Warren, 1906
species
 
 
 
 
 
 
 
 
 
 
 
 
zoological
acceptable
 
 
 
 
 
 
 
false
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 


ff82a38f-348b-4fd7-891e-a2a68d8edce4
 
 
55434
93MTR
78YFQ
provisionally accepted
[Sabulodes] arnissa
Druce, 1891
species
 
 
 
 
 
 
 
 
 
 
 
 
zoological
acceptable
 
 
 
 
 
 
 
false
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 




Results

As described in our methods, this publication derived the resource with alias col:NameUsage.tsv.gz and content id hash://sha256/1008433bfa5fe7fb059a720eddfe995e18b3e0e8f25ac0c990f1477c177d18cc . This resource contains a reverse-sorted copy of the NameUsage.tsv provided in the Catalogue of Life data package retrieved from https://download.catalogueoflife.org/col/latest_coldp.zip with content identifier hash://sha256/d512d769a3e68b6f3be523b97f9f3c05b10d317607f95cb24ddcda27bded03e1.

The following tools were used to process the Catalogue of Life resource:

Tools used in this data publication




tool name




preston


bash


gzip


sed


head


sort




Discussion

This publication is intended to facilitate re-use of the Catalogue of Life data package in taxonomic name alignment workflows. While the primary goal was to generate a resource for use in Nomer v0.4.5 (J. Poelen and Salim 2023), other usage can be imagined such as:



Lots of Copies Keeps Stuff Safe (LOCKSS (Maniatis et al. 2005)): keep an identical copy of Catalogue of Life data package outside of the Catalogue of Life infrastructure.

demonstrating how data transformation processes can be documented using Preston

making a streamable copy of a reverse-sorted copy of Catalogue of Life available via https://zenodo.org/record/8327611/files/1008433bfa5fe7fb059a720eddfe995e18b3e0e8f25ac0c990f1477c177d18cc for use in workflows like looking up the first record that contain Enhydra lutris (Sea otter):


 curl -L 'https://zenodo.org/record/8327611/files/1008433bfa5fe7fb059a720eddfe995e18b3e0e8f25ac0c990f1477c177d18cc' \
   | gunzip \
   | grep "Enhydra lutris" \
   | head -n1

Acknowledgements

This work stands on the shoulders of contributors to open source software and openly accessible datasets. Thank you!

References

Bánki, O et al. 2023. “Catalogue of Life Checklist (Version 2023-08-17).” Catalogue of Life. Catalogue of Life. https://doi.org/10.48580/dft7.

Elliott, Michael J., Jorrit H. Poelen, and José A. B. Fortes. 2023. “Signing Data Citations Enables Data Verification and Citation Persistence.” Scientific Data 10 (1). https://doi.org/10.1038/s41597-023-02230-y.

Maniatis, Petros, Mema Roussopoulos, Thomas J Giuli, David SH Rosenthal, and Mary Baker. 2005. “The LOCKSS Peer-to-Peer Digital Preservation System.” ACM Transactions on Computer Systems (TOCS) 23 (1): 2–50. https://doi.org/10.1145/1047915.1047917 .

Poelen, Jorrit H. (ed.). 2023. “Nomer Corpus of Taxonomic Resources hash://sha256 /12051b8aa59930d6561a3ed46b7cf3f67a31a98445a457d78 894f6b8a8e81641 hash://md5/1ff6b3628d7afc15b882cc0c9b1c3815.” Zenodo. https://doi.org/10.5281/zenodo.8326175.

Poelen, Jorrit, and José Augusto Salim. 2023. “Globalbioticinteractions/Nomer: 0.5.4.” Zenodo. https://doi.org/10.5281/zenodo.8329422.