DOI: 10.5281/zenodo.16616872
title: A Versioned Literature Corpus derived from Biodiversity Heritage Library hash://md5/b3cd9de0685deeebf57a5d225e59c10f

description: THIS DESCRIPTION IS A WORK IN PROGRESS, THE CONTENT OF THIS VERSIONED CORPUS IS FINAL.

Biodiversity Heritage Library (BHL, https://biodiversitylibrary.org) contains hundreds of thousands of digital works related to biodiversity. This publications contains a versioned snapshot of work metadata and the digital signatures of their associated pdfs as seen in period 2025-03-26/2025-06-25 . 
description: Files included in this publication are:

HEAD - described the signature of this literature corpus as a MD5 hash.

part.txt - list of publication parts considered in this corpus with digital signature hash://md5/f235166f8cb094f56063680af4d1328a as seen at https://biodiversitylibrary.org/data/part.txt .

bhlpart.ris.zip - publication part metadata expressed in RIS bibliographic records.  https://www.biodiversitylibrary.org/data/RIS/bhlpart.ris.zip

license-map.nq - describes the licenses associated with bhl item parts. Licenses are identified by their SPDX license identifiers (see https://spdx.org/licenses/) .

as well as the following files named after the content their contain using the md5 hash algorithm [1]:

016c9f11f52e6ce67c7399a4d588c2eb01ae1d4fc5dae07417d07ea15b1ebcbb0502e4ea67c725e22f71aef16b16430317428dc255c56dadd82f80a0a006b1c917589c09f48219cc210bb17233f905871beac84c691ca2c581d7c7126a6c9fa31fb70a2c683eb7a5545900bc6668950f27f552c25bc733d05a5cc67e9ba638502c1fc95f367642f134adf283e4a76c4634e11bf61f64d9a9c841337c7091c31b352fa9f7e46576462cf7840b2992de0d36c22c990b2ce8c187796aaac3247cdc3e437559b980d2e88daf3c05ec2f4d7743357f8a0019ca92d2c06c27495cdb0545c0336103993c893d1ed56b9ddbcbd94c9cceb77db62729b04e0743a929cb344f56e60040ce1a8519b6b16c18f56c8053e144641ffded6800dea502a8bb47ed54c2199d04c207ac0a72ba70cbc033c554dcecd04cb55ac693b0dd700a83539d581141d3f4d1847c833d9e2aa64656f25f7f443f990afb27b7b2894aa5ccf62363af62d86d0f3d4d26bfd534bf7282bb6dcbfbcf1ad373c6749027c866d31f3170f6ac67e5fffa937106e6814a3364ad7290c3e847ed1d10e47e4fd12188b752750d2b81977beb6e354479fd0247fc1b77fbd090bbd649aa4f5f45a77543627c9379bb3c6b6f91b7da72231a2881dbb598fbf06892fcf919bc2107446147cc4a99cbba7ff89fed781368d2b557900bc3a250d215282393ad43494dd9de11da4aa28a615337c2c187a5d3286923642ad6a445e6eb1138710be4b93db45ab7b4a9a6dbf5d82868e0fece05f47c7e3a42e7ac44c9ac6746a9fe2babaac45ec58860ad6e37caf05deee6e0260b55564ecb87ad70be22c8150717f3edab08ff4727a8b061254939327dde119942947203f6bcb3cd9de0685deeebf57a5d225e59c10fb5f366d040d34d85ee1ea2dcf4e2de28b855da1ec4dd0600c8fd736a0f413ec2b9014b0b403516b1e852748d72633c7dbb263189781c35bc2ecf714e6f2e3bb9be94e9bd64eb83df76146e3091f176aec06de184c76c2a8a586c903a5e7ca582c157994807032e7b22b652eb44dbd500c4b9251c4f6db873a941b42544c1fc96c71376e752134950cf96e1e5467ce8d5c97bb72c0b15f3877c4e24e03ed423cacceaf490770b1e3468c4e271f8fdca3dce5c0bafd951a9a805d3de7c2c6f6f3ace804fb3370465a27a89d5f731490da6d268444ddf44545c53e6bf4635b3ae57d86a121f36f8e6f081a20803444bc1aadb008838f260749cf34ae9fb6775e9aae19ad69093efab2b86c3ca04de8f95baf206133d0b42fc9126e36a30654748ecf235166f8cb094f56063680af4d1328afab4cb4d5766302a59936ebe6312ecc2feb6470b553aa9e104a6d7114a424a05
description: This part is a technical description that defines this corpus and their associated contents.

History

The (version) history of this corpus can be queried by:

preston history\
 --algo md5\
 --anchor hash://md5/b3cd9de0685deeebf57a5d225e59c10f\ --remote https://linker.bio,https://zenodo.org

to produce:

<hash://md5/b3cd9de0685deeebf57a5d225e59c10f> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/2c1fc95f367642f134adf283e4a76c46> .<hash://md5/2c1fc95f367642f134adf283e4a76c46> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/36c22c990b2ce8c187796aaac3247cdc> .<hash://md5/36c22c990b2ce8c187796aaac3247cdc> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/fab4cb4d5766302a59936ebe6312ecc2> .<hash://md5/fab4cb4d5766302a59936ebe6312ecc2> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/43357f8a0019ca92d2c06c27495cdb05> .<hash://md5/43357f8a0019ca92d2c06c27495cdb05> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/581141d3f4d1847c833d9e2aa64656f2> .<hash://md5/581141d3f4d1847c833d9e2aa64656f2> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/17589c09f48219cc210bb17233f90587> .<hash://md5/17589c09f48219cc210bb17233f90587> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/c4b9251c4f6db873a941b42544c1fc96> .<hash://md5/c4b9251c4f6db873a941b42544c1fc96> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/be94e9bd64eb83df76146e3091f176ae> .<hash://md5/be94e9bd64eb83df76146e3091f176ae> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/53e144641ffded6800dea502a8bb47ed> .<hash://md5/53e144641ffded6800dea502a8bb47ed> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/b855da1ec4dd0600c8fd736a0f413ec2> .<hash://md5/b855da1ec4dd0600c8fd736a0f413ec2> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/7290c3e847ed1d10e47e4fd12188b752> .<hash://md5/7290c3e847ed1d10e47e4fd12188b752> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/3e437559b980d2e88daf3c05ec2f4d77> .<hash://md5/3e437559b980d2e88daf3c05ec2f4d77> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/352fa9f7e46576462cf7840b2992de0d> .<hash://md5/352fa9f7e46576462cf7840b2992de0d> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/ac44c9ac6746a9fe2babaac45ec58860> .<hash://md5/ac44c9ac6746a9fe2babaac45ec58860> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/d86a121f36f8e6f081a20803444bc1aa> .<hash://md5/d86a121f36f8e6f081a20803444bc1aa> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/1fb70a2c683eb7a5545900bc6668950f> .<hash://md5/1fb70a2c683eb7a5545900bc6668950f> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/63af62d86d0f3d4d26bfd534bf7282bb> .<hash://md5/63af62d86d0f3d4d26bfd534bf7282bb> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/b9014b0b403516b1e852748d72633c7d> .<hash://md5/b9014b0b403516b1e852748d72633c7d> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/f206133d0b42fc9126e36a30654748ec> .<hash://md5/f206133d0b42fc9126e36a30654748ec> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/98fbf06892fcf919bc2107446147cc4a> .<hash://md5/98fbf06892fcf919bc2107446147cc4a> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/a28a615337c2c187a5d3286923642ad6> .<hash://md5/a28a615337c2c187a5d3286923642ad6> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/e19ad69093efab2b86c3ca04de8f95ba> .<hash://md5/e19ad69093efab2b86c3ca04de8f95ba> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/feb6470b553aa9e104a6d7114a424a05> .<hash://md5/feb6470b553aa9e104a6d7114a424a05> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/a445e6eb1138710be4b93db45ab7b4a9> .<hash://md5/a445e6eb1138710be4b93db45ab7b4a9> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/016c9f11f52e6ce67c7399a4d588c2eb> .<hash://md5/016c9f11f52e6ce67c7399a4d588c2eb> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/a250d215282393ad43494dd9de11da4a> .<hash://md5/a250d215282393ad43494dd9de11da4a> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/0502e4ea67c725e22f71aef16b164303> .<hash://md5/0502e4ea67c725e22f71aef16b164303> <http://www.w3.org/ns/prov#wasDerivedFrom> <hash://md5/d268444ddf44545c53e6bf4635b3ae57> .<urn:uuid:0659a54f-b713-4f86-a917-5be166a14110> <http://purl.org/pav/hasVersion> <hash://md5/d268444ddf44545c53e6bf4635b3ae57> .

 

Volume

The total volume of this corpus can be estimated using

preston ls\
 --algo md5\
 --anchor hash://md5/b3cd9de0685deeebf57a5d225e59c10f\
 | preston cat\
 | pv\
 > /dev/null

to produce:

Also, the number of unique pdfs associated with content referenced in the corpus can be estimated using

preston ls\
 --algo md5\
 --anchor hash://md5/b3cd9de0685deeebf57a5d225e59c10f\ --remote https://linker.bio,https://zenodo.org\
| grep hasVersion\
| grep partpdf\
| grep -oE hash://md5/[a-f0-9]{32}\| sort\| uniq\
| wc -l

to produce 375,763 items, or over 375k tracked pdfs. Note also that out of a total of  375,763  items, pdf content related to 38,826 part items, or about 10%, failed to be retrieved, even after multiple retries. This number can be estimated using:

diff\ <(preston ls\
 --algo md5\
 --anchor hash://md5/b3cd9de0685deeebf57a5d225e59c10f\ --remote https://linker.bio,https://zenodo.org\
| grep hasVersion\
| grep partpdf\
| grep well-known\| grep -Eo partpdf/[0-9]+\| sort\| uniq)\ <(preston ls\
 --algo md5\
 --anchor hash://md5/b3cd9de0685deeebf57a5d225e59c10f\ --remote https://linker.bio,https://zenodo.org\
| grep hasVersion\
| grep partpdf\
| grep -v well-known\| grep -Eo partpdf/[0-9]+\| sort\| uniq)\| grep "^<"\| wc -l