# Bioschemas data harvesting project report

Alasdair J. G. Gray, Petros Papadopoulos, Alban Gaignard, Thomas Rosnet, Ivan Mičetić, Sébastien Moretti

<figure><img src="image_1.png" /></figure>

To cite this version:

Alasdair J. G. Gray, Petros Papadopoulos, Alban Gaignard, Thomas Rosnet, Ivan Mičetić, et al.. Bioschemas data harvesting project report. 2025. hal-05130616

HAL Id: hal-05130616

https://cnrs.hal.science/hal-05130616v1

Preprint submitted on 26 Jun 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Bioschemas data harvesting project report

Alasdair Gray¹, Petros Papadopoulos¹, Alban Gaignard², Thomas Rosnet³, Ivan Mičetić⁴, and Sébastien Moretti⁵

1 Heriot-Watt University, Edinburgh, UK 2 Nantes Université, CNRS, INSERM, l'institut du thorax, F-44000 Nantes, France 3 Institut Français de Bioinformatique, CNRS UAR 3601, France 4 University of Padua, Padova, Italy 5 Université de Lausanne, Lausanne, Switzerland

## Introduction

The promise of Bioschemas is that it makes consuming data from multiple resources more straightforward. However, this hypothesis has not been tested by conducting a large scale harvest of deployed markup and making this available for others to reuse. Therefore, the goal of this hackathon project is to harvest a collection of Bioschemas markup from a number of different sites listed on the Bioschemas live deploys page using the Bioschemas Markup Scraper and Extractor (BMUSE). The harvested data will be made available for others and loaded into a triplestore to allow for further exploration.

## Data Harvesting

Prior to the BioHackathon, we set about harvesting data from as many of the Bioschemas live deploy sites as possible. At the time of the BioHackathon, there were 70 sites listed, and 137 profile deployments (a site can deploy multiple profiles, e.g. Dataset and DataCatalog). Not all deployments could be harvested since they do not provide sitemaps listing the pages within the site. At the time of the BioHackathon there were 25 sites with sitemaps. Several of these do not list the pages containing data, limiting the amount that could be harvested.

The list of sites to be harvested were gathered in a GitHub project board so that progress could be tracked. The cards in this board were annotated to state whether the source was known to use a static site deployment (i.e. the markup is embedded in the page source by the server) or dynamic single page application (i.e. the page content is generated client side using Javascript), and also whether they were known to have data content or limited content of Dataset and DataCatalog.

The Bioschemas Markup Scraper and Extractor (BMUSE) was used for the harvesting of the data. During the harvesting we found two key issues with BMUSE which arose due to the scale of the data harvest. The first was that errors in the JSON-LD were not correctly identified and logged. The second was a memory limit relating to JSoup which meant that only about 24,104 pages were scraped out of the 50,000 in the sitemap file (BMUSE #82). Fixes to these issues were applied resulting in BMUSE v0.5.2 being used for most of the harvesting.

The data harvesting workflow consisted of the following steps:

1. Pick one of the sites to be harvested: priority was given to static sites with data content since these could be harvested more quickly and went beyond Dataset/DataCatalog markup.

2. For each sitemap in the sitemap index, harvest the content from the source pages.

3. Merge the individual nquad files for each page in the (sub)sitemap into a single nquad file.

4. Load the merged nquad file into the triplestore.

5. Make the merged nquad file available on the web.

6. Update the project README with details of what had been harvested.
Where issues were found with the source site, these were fefback to the data provider to
allow them to revise their markup. For example, it was found that MassBank were including
characters in their string values such as " that need to be escaped to generate valid JSON-LD
(MassBank-Web #316).

In total, six sites were found to be unscrapable. These were

* InterPro: a dynamic site providing a sitemap. However, the sitemap did not conform with the sitemap standard.

* Scholia COVID-19 URL list: a site generated via SPARQL queries over the Wikidata endpoint. Unable to scrape due to timeout being reached before the data being available.

* SwissModel: the provided sitemap did not conform with the sitemap stanard.

* WikiPathways: sitemap was empty.

* IPPIDB: a dynamic site with data content corresponding to MolecularEntities. However, the pages exhibit inconsistent rendering when tested in the browser and could not be harvested with BMUSE.

* OrphaNet: a static site with disease markup. The sitemap conforms with the older Google proposal for sitemaps rather than the widely used 0.9 version expected by BMUSE

## Data Analysis

We reused the notebook originally developed at BioHackathon 2020 (Gray et al., 2021) and since evolved for the Intrinsically Disordered Protein Knowledge Graph (IDP-KG) (Gray et al., 2022). We include the HCLS Dataset Description profile statistics queries<fnref n="1" /> (Dumontier et al., 2016), read in from an existing repository. We also include queries developed specifically for the analysis of the Bioschemas harvested data.

To use the notebook (MyBinder launcher), you simply need to run all cells and then select the query you would like to execute from the resulting dropdown menu.

We now present the results of the queries obtained during the hackathon, i.e. the data values are as they were on 11 November 2021. Running the notebook in March 2022 obtains different results due to more harvested data having been added.

## HCLS Dataset Statistics

We include here a selection of results from some of the HCLS statistics queries. We focus on those providing the most interesting statistics for the available data. For the full set of queries and results, please run the notebook.

### Number of triples

This is the raw count of the number of triples contained in the triplestore repository.

triples
---
10,610,743

### Number of named graphs

The result presented here is equivalent to number of pages harvested since BMUSE generates a named graph for each page harvested.

<footnote n="1">¹Dataset Descriptions: HCLS Community Profile §6 accessed March 2022</footnote>
graphs
---
413,748

Number of instance per class

There are many different types included in the markup. BMUSE extracts all markup, not just
Bioschemas profiles.

The results are ordered by the Class IRI; in the notebook you can edit the query and change
the ordering of results.

(57 results)

<table><thead><tr><td>Class</td><td>distinctInstances</td></tr></thead><tbody><tr><td>http://rdfs.org/sioc/ns#Item</td><td>57</td></tr><tr><td>http://xmlns.com/foaf/0.1/Document</td><td>89</td></tr><tr><td>http://xmlns.com/foaf/0.1/Image</td><td>219</td></tr><tr><td>https://bioschemas.org/Gene</td><td>238,079</td></tr><tr><td>https://bioschemas.org/Protein</td><td>1,262</td></tr><tr><td>https://bioschemas.org/Taxon</td><td>55,884</td></tr><tr><td>https://schema.org/AboutPage</td><td>1</td></tr><tr><td>https://schema.org/Action</td><td>3</td></tr><tr><td>https://schema.org/Answer</td><td>8</td></tr><tr><td>https://schema.org/BioChemEntity</td><td>49,823</td></tr><tr><td>https://schema.org/BreadcrumbList</td><td>14,037</td></tr><tr><td>https://schema.org/ChemicalSubstance</td><td>29</td></tr><tr><td>https://schema.org/CollectionPage</td><td>187</td></tr><tr><td>https://schema.org/CollegeOrUniversity</td><td>2</td></tr><tr><td>https://schema.org/ContactPoint</td><td>148</td></tr><tr><td>https://schema.org/CreativeWork</td><td>14,299</td></tr><tr><td>https://schema.org/DataCatalog</td><td>7,439</td></tr><tr><td>https://schema.org/DataDownload</td><td>1,497</td></tr><tr><td>https://schema.org/Dataset</td><td>201,302</td></tr><tr><td>https://schema.org/DefinedTerm</td><td>4,261</td></tr><tr><td>https://schema.org/DefinedTermSet</td><td>4,112</td></tr><tr><td>https://schema.org/DigitalDocument</td><td>1</td></tr><tr><td>https://schema.org/EducationalOrganization</td><td>3</td></tr><tr><td>https://schema.org/Event</td><td>12,818</td></tr><tr><td>https://schema.org/FAQPage</td><td>1</td></tr><tr><td>https://schema.org/Gene</td><td>39</td></tr><tr><td>https://schema.org/GeoShape</td><td>19,398</td></tr><tr><td>https://schema.org/GovernmentOrganization</td><td>1</td></tr><tr><td>https://schema.org/ItemList</td><td>187</td></tr><tr><td>https://schema.org/ListItem</td><td>28,137</td></tr><tr><td>https://schema.org/MolecularEntity</td><td>199,350</td></tr><tr><td>https://schema.org/NGO</td><td>11,717</td></tr><tr><td>https://schema.org/Offer</td><td>5</td></tr><tr><td>https://schema.org/Organization</td><td>206,715</td></tr><tr><td>https://schema.org/PeopleAudience</td><td>2,475</td></tr><tr><td>https://schema.org/Person</td><td>326,935</td></tr><tr><td>https://schema.org/Place</td><td>19,438</td></tr><tr><td>https://schema.org/PostalAddress</td><td>307,406</td></tr><tr><td>https://schema.org/ Property Value</td><td>144,002</td></tr><tr><td>https://schema.org/Protein</td><td>4,462</td></tr></tbody></table>
<table><thead><tr><td>Class</td><td>distinctInstances</td></tr></thead><tbody><tr><td>https://schema.org/QAPage</td><td>1</td></tr><tr><td>https://schema.org/Question</td><td>8</td></tr><tr><td>https://schema.org/ScholarlyArticle</td><td>9,350</td></tr><tr><td>https://schema.org/SearchAction</td><td>5</td></tr><tr><td>https://schema.org/SequenceAnnotation</td><td>15,786</td></tr><tr><td>https://schema.org/SequenceRange</td><td>15,786</td></tr><tr><td>https://schema.org/SoftwareApplication</td><td>4</td></tr><tr><td>https://schema.org/SoftwareSourceCode</td><td>4</td></tr><tr><td>https://schema.org/Study</td><td>4,328</td></tr><tr><td>https://schema.org/Thing</td><td>27,872</td></tr><tr><td>https://schema.org/URL</td><td>1</td></tr><tr><td>https://schema.org/WebApplication</td><td>3</td></tr><tr><td>https://schema.org/WebPage</td><td>55,114</td></tr><tr><td>https://schema.org/WebSite</td><td>5</td></tr><tr><td>https://schema.org/contact</td><td>40</td></tr><tr><td>https://schema.org/hostInstitution</td><td>40</td></tr><tr><td>https://schema.org/url</td><td>10,360</td></tr></tbody></table>

## Bioschemas Queries

The following queries focus on features of interest to the Bioschemas community.

### Instances per Bioschemas Class

Note that due to the data content we need to include some properties with both a Bioschemas namespace and a Schema.org namespace.

The results are ordered by the count of the number of instances; in the notebook you can edit the query and change the ordering of results.

(18 results)

<table><thead><tr><td>Class</td><td>instances</td></tr></thead><tbody><tr><td>https://schema.org/Person</td><td>326,935</td></tr><tr><td>https://bioschemas.org/Gene</td><td>238,079</td></tr><tr><td>https://schema.org/Organization</td><td>206,715</td></tr><tr><td>https://schema.org/Dataset</td><td>201,302</td></tr><tr><td>https://schema.org/MolecularEntity</td><td>199,350</td></tr><tr><td>https://bioschemas.org/Taxon</td><td>55,884</td></tr><tr><td>https://schema.org/BioChemEntity</td><td>49,823</td></tr><tr><td>https://schema.org/SequenceAnnotation</td><td>15,786</td></tr><tr><td>https://schema.org/SequenceRange</td><td>15,786</td></tr><tr><td>https://schema.org/Event</td><td>12,818</td></tr><tr><td>https://schema.org/ScholarlyArticle</td><td>9,350</td></tr><tr><td>https://schema.org/DataCatalog</td><td>7,439</td></tr><tr><td>https://schema.org/Protein</td><td>4,462</td></tr><tr><td>https://schema.org/Study</td><td>4,328</td></tr><tr><td>https://bioschemas.org/Protein</td><td>1,262</td></tr><tr><td>https://schema.org/Gene</td><td>39</td></tr><tr><td>https://schema.org/ChemicalSubstance</td><td>29</td></tr><tr><td>https://schema.org/SoftwareApplication</td><td>4</td></tr></tbody></table>
## Number of Domains

This result informs us how many web domains were harvested. This is approximately equal to the number of datasets, but some sites may host more than one dataset so not necessarily an exact correspondence.

count

25

## Number of Pages per Domain

We now report the number of pages that have been harvested from each domain. Note that we do not understand the empty domain as all markup was extracted from a web domain.

(25 results)

<table><thead><tr><td>domain</td><td>count</td></tr></thead><tbody><tr><td>massbank.eu</td><td>76,253</td></tr><tr><td>scholia.toolforge.org</td><td>74,319</td></tr><tr><td>www.gbif.org</td><td>68,167</td></tr><tr><td>test.intermine.org</td><td>49,959</td></tr><tr><td>bgee.org</td><td>49,022</td></tr><tr><td>www.metanetx.org</td><td>49,012</td></tr><tr><td>tess.elixir-europe.org</td><td>13,939</td></tr><tr><td>ega-archive.org</td><td>11,833</td></tr><tr><td>fairsharing.org</td><td>6,351</td></tr><tr><td>prosite.expasy.org</td><td>5,858</td></tr><tr><td>ippidb.pasteur.fr</td><td>2,433</td></tr><tr><td>mobidb.org</td><td>2,082</td></tr><tr><td>disprot.org</td><td>2,043</td></tr><tr><td>pcddb.cryst.bbk.ac.uk</td><td>1,402</td></tr><tr><td>www.ebi.ac.uk</td><td>672</td></tr><tr><td>proteinensemble.org</td><td>187</td></tr><tr><td>www.france-bioinformatique.fr</td><td>86</td></tr><tr><td>pairedomicsdata.bioinformatics.nl</td><td>78</td></tr><tr><td>www.covid19dataportal.org</td><td>19</td></tr><tr><td></td><td>12</td></tr><tr><td>www.alliancegenome.org</td><td>11</td></tr><tr><td>biopragmatics.github.io</td><td>3</td></tr><tr><td>nanocommons.github.io</td><td>3</td></tr><tr><td>bridgedb.github.io</td><td>2</td></tr><tr><td>www.uniprot.org</td><td>2</td></tr></tbody></table>

## Count of Types per Domain

We now report the number of instances of each type on each domain. What is interesting here is the fact that Bgee has many proteins listed on their pages.

The results are ordered by the count of the number of instances; in the notebook you can edit the query and change the ordering of results.

(146 results)
<table><thead><tr><td>domain</td><td>type</td><td>count</td></tr></thead><tbody><tr><td>www.gbif.org</td><td>https://schema.org/PostalAddress</td><td>297,090</td></tr><tr><td>www.gbif.org</td><td>https://schema.org/Person</td><td>291,260</td></tr><tr><td>bgee.org</td><td>https://bioschemas.org/Gene</td><td>263,793</td></tr><tr><td>www.gbif.org</td><td>https://schema.org/Organization</td><td>186,688</td></tr><tr><td>www.gbif.org</td><td>https://schema.org/PropertyValue</td><td>126,268</td></tr><tr><td>massbank.eu</td><td>https://schema.org/Dataset</td><td>76,249</td></tr><tr><td>massbank.eu</td><td>https://schema.org/MolecularEntity</td><td>76,249</td></tr><tr><td>scholia.toolforge.org</td><td>https://schema.org/CreativeWork</td><td>74,310</td></tr><tr><td>scholia.toolforge.org</td><td>https://schema.org/MolecularEntity</td><td>74,310</td></tr><tr><td>www.gbif.org</td><td>https://schema.org/Dataset</td><td>63,134</td></tr><tr><td>test.intermine.org</td><td>https://schema.org/Dataset</td><td>49,959</td></tr><tr><td>test.intermine.org</td><td>https://schema.org/BioChemEntity</td><td>49,823</td></tr><tr><td>bgee.org</td><td>https://bioschemas.org/Taxon</td><td>49,059</td></tr><tr><td>bgee.org</td><td>https://schema.org/WebPage</td><td>49,009</td></tr><tr><td>www.metanetx.org</td><td>https://schema.org/CreativeWork</td><td>49,002</td></tr><tr><td>www.metanetx.org</td><td>https://schema.org/MolecularEntity</td><td>49,001</td></tr><tr><td>prosite.expasy.org</td><td>https://schema.org/Person</td><td>31,364</td></tr><tr><td>tess.elixir-europe.org</td><td>https://schema.org/ListItem</td><td>27,872</td></tr><tr><td>tess.elixir-europe.org</td><td>https://schema.org/Thing</td><td>27,872</td></tr><tr><td>www.gbif.org</td><td>https://schema.org/GeoShape</td><td>19,398</td></tr><tr><td>www.gbif.org</td><td>https://schema.org/Place</td><td>19,398</td></tr><tr><td>tess.elixir-europe.org</td><td>https://schema.org/BreadcrumbList</td><td>13,938</td></tr><tr><td>tess.elixir-europe.org</td><td>https://schema.org/Event</td><td>12,778</td></tr><tr><td>prosite.expasy.org</td><td>https://schema.org/Organization</td><td>11,715</td></tr><tr><td>prosite.expasy.org</td><td>https://schema.org/NGO</td><td>11,714</td></tr><tr><td>disprot.org</td><td>https://schema.org/PropertyValue</td><td>110,406</td></tr><tr><td>disprot.org</td><td>https://schema.org/SequenceAnnotation</td><td>11,046</td></tr><tr><td>disprot.org</td><td>https://schema.org/SequenceRange</td><td>11,046</td></tr><tr><td>prosite.expasy.org</td><td>https://schema.org/url</td><td>10,360</td></tr><tr><td>tess.elixir-europe.org</td><td>https://schema.org/PostalAddress</td><td>10,316</td></tr><tr><td>ega-archive.org</td><td>https://schema.org/DataCatalog</td><td>7,431</td></tr><tr><td>ega-archive.org</td><td>https://schema.org/Dataset</td><td>7,431</td></tr><tr><td>tess.elixir-europe.org</td><td>https://schema.org/Organization</td><td>7,110</td></tr><tr><td>prosite.expasy.org</td><td>https://bioschemas.org/Taxon</td><td>6,796</td></tr><tr><td>prosite.expasy.org</td><td>https://schema.org/ScholarlyArticle</td><td>6,681</td></tr><tr><td>disprot.org</td><td>https://schema.org/DefinedTerm</td><td>6,599</td></tr><tr><td>fairsharing.org</td><td>https://schema.org/Dataset</td><td>6,328</td></tr><tr><td>prosite.expasy.org</td><td>https://schema.org/WebPage</td><td>6,093</td></tr><tr><td>prosite.expasy.org</td><td>https://schema.org/CreativeWork</td><td>5,857</td></tr><tr><td>fairsharing.org</td><td>https://schema.org/CreativeWork</td><td>5,542</td></tr><tr><td>mobidb.org</td><td>https://schema.org/PropertyValue</td><td>4,486</td></tr><tr><td>mobidb.org</td><td>https://schema.org/SequenceAnnotation</td><td>4,486</td></tr><tr><td>mobidb.org</td><td>https://schema.org/SequenceRange</td><td>4,486</td></tr><tr><td>ega-archive.org</td><td>https://schema.org/Study</td><td>4,328</td></tr><tr><td>disprot.org</td><td>https://schema.org/DefinedTermSet</td><td>4,076</td></tr><tr><td>mobidb.org</td><td>https://schema.org/DefinedTerm</td><td>3,400</td></tr><tr><td>mobidb.org</td><td>https://schema.org/DefinedTermSet</td><td>3,400</td></tr><tr><td>tess.elixir-europe.org</td><td>https://schema.org/Person</td><td>3,298</td></tr><tr><td>tess.elixir-europe.org</td><td>https://schema.org/CreativeWork</td><td>2,876</td></tr><tr><td>disprot.org</td><td>https://schema.org/ScholarlyArticle</td><td>2,857</td></tr><tr><td>tess.elixir-europe.org</td><td>https://schema.org/PeopleAudience</td><td>2,475</td></tr><tr><td>proteinensemble.org</td><td>https://schema.org/PropertyValue</td><td>2,202</td></tr><tr><td>mobidb.org</td><td>https://schema.org/CreativeWork</td><td>2,073</td></tr></tbody></table>
<table><thead><tr><td>domain</td><td>type</td><td>count</td></tr></thead><tbody><tr><td>mobidb.org</td><td>https://schema.org/Protein</td><td>2,073</td></tr><tr><td>disprot.org</td><td>https://schema.org/CreativeWork</td><td>2,038</td></tr><tr><td>disprot.org</td><td>https://schema.org/Protein</td><td>2,038</td></tr><tr><td>proteinensemble.org</td><td>https://schema.org/DefinedTerm</td><td>1,626</td></tr><tr><td>pcddb.cryst.bbk.ac.uk</td><td>https://schema.org/Organization</td><td>1,402</td></tr><tr><td>pcddb.cryst.bbk.ac.uk</td><td>https://schema.org/DataDownload</td><td>1,394</td></tr><tr><td>prosite.expasy.org</td><td>https://bioschemas.org/Protein</td><td>1,376</td></tr><tr><td>pcddb.cryst.bbk.ac.uk</td><td>https://schema.org/Dataset</td><td>697</td></tr><tr><td>pcddb.cryst.bbk.ac.uk</td><td>https://schema.org/Person</td><td>697</td></tr><tr><td>biopragmatics.github.io</td><td>https://schema.org/Dataset</td><td>287</td></tr><tr><td>www.france-bioinformatique.fr</td><td>https://schema.org/ListItem</td><td>265</td></tr><tr><td>proteinensemble.org</td><td>https://schema.org/Protein</td><td>254</td></tr><tr><td>proteinensemble.org</td><td>https://schema.org/SequenceAnnotation</td><td>254</td></tr><tr><td>proteinensemble.org</td><td>https://schema.org/SequenceRange</td><td>254</td></tr><tr><td>pairedomicsdata.bioinformatics.nl</td><td>https://schema.org/Person</td><td>222</td></tr><tr><td>www.ebi.ac.uk</td><td>http://xmlns.com/foaf/0.1/Image</td><td>222</td></tr><tr><td>proteinensemble.org</td><td>https://schema.org/CollectionPage</td><td>187</td></tr><tr><td>proteinensemble.org</td><td>https://schema.org/CreativeWork</td><td>187</td></tr><tr><td>proteinensemble.org</td><td>https://schema.org/DefinedTermSet</td><td>187</td></tr><tr><td>proteinensemble.org</td><td>https://schema.org/ItemList</td><td>187</td></tr><tr><td>proteinensemble.org</td><td>https://schema.org/ScholarlyArticle</td><td>181</td></tr><tr><td>pairedomicsdata.bioinformatics.nl</td><td>https://schema.org/ContactPoint</td><td>148</td></tr><tr><td>www.covid19dataportal.org</td><td>https://schema.org/Organization</td><td>148</td></tr><tr><td>www.covid19dataportal.org</td><td>https://schema.org/Dataset</td><td>110</td></tr><tr><td>www.france-bioinformatique.fr</td><td>https://schema.org/BreadcrumbList</td><td>99</td></tr><tr><td>test.intermine.org</td><td>https://schema.org/Dataset</td><td>97</td></tr><tr><td>www.ebi.ac.uk</td><td>https://schema.org/Protein</td><td>97</td></tr><tr><td>bgee.org</td><td>http://xmlns.com/foaf/0.1/Document</td><td>91</td></tr><tr><td>pairedomicsdata.bioinformatics.nl</td><td>https://schema.org/Person</td><td>90</td></tr><tr><td>pairedomicsdata.bioinformatics.nl</td><td>https://schema.org/Dataset</td><td>87</td></tr><tr><td>pairedomicsdata.bioinformatics.nl</td><td>https://schema.org/CreativeWork</td><td>74</td></tr><tr><td>pairedomicsdata.bioinformatics.nl</td><td>https://schema.org/DataCatalog</td><td>74</td></tr><tr><td>pairedomicsdata.bioinformatics.nl</td><td>https://schema.org/DataDownload</td><td>74</td></tr><tr><td>pairedomicsdata.bioinformatics.nl</td><td>https://schema.org/Dataset</td><td>74</td></tr><tr><td>pairedomicsdata.bioinformatics.nl</td><td>https://schema.org/Organization</td><td>74</td></tr><tr><td>www.ebi.ac.uk</td><td>http://rdfs.org/sioc/ns#Item</td><td>59</td></tr><tr><td>www.france-bioinformatique.fr</td><td>https://schema.org/Event</td><td>40</td></tr><tr><td>www.france-bioinformatique.fr</td><td>https://schema.org/Place</td><td>40</td></tr><tr><td>www.france-bioinformatique.fr</td><td>https://schema.org/contact</td><td>40</td></tr><tr><td>www.france-bioinformatique.fr</td><td>https://schema.org/hostInstitution</td><td>40</td></tr><tr><td>test.intermine.org</td><td>https://schema.org/Gene</td><td>39</td></tr><tr><td>nanocommons.github.io</td><td>https://bioschemas.org/Taxon</td><td>29</td></tr><tr><td>bridgedb.github.io</td><td>https://schema.org/ChemicalSubstance</td><td>29</td></tr><tr><td>bridgedb.github.io</td><td>https://schema.org/Organization</td><td>27</td></tr><tr><td>www.covid19dataportal.org</td><td>https://schema.org/DataDownload</td><td>23</td></tr><tr><td>www.uniprot.org</td><td>https://schema.org/Dataset</td><td>23</td></tr><tr><td>nanocommons.github.io</td><td>https://schema.org/DataCatalog</td><td>19</td></tr><tr><td>bgee.org</td><td>https://schema.org/Organization</td><td>14</td></tr><tr><td>bgee.org</td><td>https://schema.org/CreativeWork</td><td>9</td></tr><tr><td>bgee.org</td><td>https://schema.org/Answer</td><td>8</td></tr><tr><td>bgee.org</td><td>https://schema.org/CreativeWork</td><td>8</td></tr><tr><td>bgee.org</td><td>https://schema.org/Question</td><td>8</td></tr></tbody></table>
<table><thead><tr><th>domain</th><th>type</th><th>count</th></tr></thead><tbody><tr><td></td><td>https://schema.org/WebPage</td><td>7</td></tr><tr><td>nanocommons.github.io</td><td>https://schema.org/Dataset</td><td>7</td></tr><tr><td>nanocommons.github.io</td><td>https://schema.org/DataDownload</td><td>6</td></tr><tr><td></td><td>https://schema.org/DataCatalog</td><td>5</td></tr><tr><td>bgee.org</td><td>https://schema.org/Offer</td><td>5</td></tr><tr><td></td><td>https://schema.org/SearchAction</td><td>4</td></tr><tr><td>bgee.org</td><td>https://schema.org/SoftwareSourceCode</td><td>4</td></tr><tr><td>nanocommons.github.io</td><td>https://schema.org/CreativeWork</td><td>4</td></tr><tr><td></td><td>https://schema.org/EducationalOrganization</td><td>3</td></tr><tr><td></td><td>https://schema.org/NGO</td><td>3</td></tr><tr><td></td><td>https://schema.org/ScholarlyArticle</td><td>3</td></tr><tr><td>bgee.org</td><td>https://schema.org/WebApplication</td><td>3</td></tr><tr><td>biopragmatics.github.io</td><td>https://schema.org/Person</td><td>3</td></tr><tr><td>prosite.expasy.org</td><td>https://schema.org>Action</td><td>3</td></tr><tr><td></td><td>https://schema.org/CollegeOrUniversity</td><td>2</td></tr><tr><td></td><td>https://schema.org/WebSite</td><td>2</td></tr><tr><td>bgee.org</td><td>https://schema.org/SoftwareApplication</td><td>2</td></tr><tr><td>biopragmatics.github.io</td><td>https://schema.org/WebPage</td><td>2</td></tr><tr><td>bridgedb.github.io</td><td>https://schema.org/CreativeWork</td><td>2</td></tr><tr><td>www.uniprot.org</td><td>https://schema.org/GovernmentOrganization</td><td>2</td></tr><tr><td>www.uniprot.org</td><td>https://schema.org/NGO</td><td>2</td></tr><tr><td>www.uniprot.org</td><td>https://schema.org/WebPage</td><td>2</td></tr><tr><td></td><td>https://schema.org/GovernmentOrganization</td><td>1</td></tr><tr><td>bgee.org</td><td>https://schema.org/AboutPage</td><td>1</td></tr><tr><td>bgee.org</td><td>https://schema.org/FAQPage</td><td>1</td></tr><tr><td>biopragmatics.github.io</td><td>https://schema.org/WebSite</td><td>1</td></tr><tr><td>bridgedb.github.io</td><td>https://schema.org/SoftwareApplication</td><td>1</td></tr><tr><td>bridgedb.github.io</td><td>https://schema.org/WebPage</td><td>1</td></tr><tr><td>bridgedb.github.io</td><td>https://schema.org/WebSite</td><td>1</td></tr><tr><td>massbank.eu</td><td>https://schema.org/DataCatalog</td><td>1</td></tr><tr><td>massbank.eu</td><td>https://schema.org/ScholarlyArticle</td><td>1</td></tr><tr><td>nanocommons.github.io</td><td>https://schema.org/DataCatalog</td><td>1</td></tr><tr><td>nanocommons.github.io</td><td>https://schema.org/Person</td><td>1</td></tr><tr><td>nanocommons.github.io</td><td>https://schema.org/URL</td><td>1</td></tr><tr><td>nanocommons.github.io</td><td>https://schema.org/WebSite</td><td>1</td></tr><tr><td>prosite.expasy.org</td><td>https://schema.org/DigitalDocument</td><td>1</td></tr><tr><td>prosite.expasy.org</td><td>https://schema.org/SearchAction</td><td>1</td></tr><tr><td>www.metanetx.org</td><td>https://schema.org/SoftwareApplication</td><td>1</td></tr><tr><td>www.uniprot.org</td><td>https://schema.org/Dataset</td><td>1</td></tr><tr><td>www.uniprot.org</td><td>https://schema.org/QAPage</td><td>1</td></tr></tbody></table>

## Connectivity of the Data

We were interested to gain some insight as to how connected the data was both internally, and how many points where it would link up with other knowledge graphs. The queries in this section focus on the connectedness of the data.

We first investigated the number of nodes that only contained incoming edges. We report the total number of object nodes there are (excluding literals), and the number of edge IRIs, i.e. those that only have incoming properties. Only 4.65% of the nodes only contain incoming edges.
<table><thead><tr><th>Object IRIs</th><th>Edge IRIs</th></tr></thead><tbody><tr><td>2,057,094</td><td>95,610</td></tr></tbody></table>

We then investigated the number of outgoing links per class. We report here the top 20 results.

<table><thead><tr><th>s</th><th>class</th><th>nb_out_edges</th></tr></thead><tbody><tr><td>https://bioschemas.org/crawl/v1/<br>bgee/?page=gene&amp;gene_id=ENSBTAG00000027937/20211110/90020/<br>bgee.org/?page=gene&amp;gene_id=ENSBTAG00000027937/1779564251<br>https://www.metanetx.org/chem_info/<br>MNXM1944<br>https://doi.org/10.15468/hb9rjv</td><td>https://bioschemas.org/<br>org/Gene</td><td>856</td></tr><tr><td>https://bioschemas.org/crawl/v1/<br>bgee/?page=gene&amp;gene_id=ENSBTAG00000043564/20211110/94715/<br>bgee.org/?page=gene&amp;gene_id=ENSBTAG00000043564/1772156424<br>https://bioschemas.org/crawl/v1/<br>bgee/?page=gene&amp;gene_id=ENSBTAG00000043584/20211110/94734/<br>bgee.org/?page=gene&amp;gene_id=ENSBTAG00000043584/2022662406<br>https://doi.org/10.15468/m5vrza</td><td>https://schema.org/<br>MolecularEntity<br>https://schema.org/<br>Dataset<br>https://bioschemas.org/<br>org/Gene</td><td>654<br>594<br>519</td></tr><tr><td>http://www.ebi.ac.uk/pdbe/about/<br>past-events<br>http://www.ebi.ac.uk/pdbe/about/<br>past-events<br>https://doi.org/10.15468/vmf5ye</td><td>https://bioschemas.org/<br>org/Gene</td><td>474</td></tr><tr><td>https://bioschemas.org/crawl/v1/<br>bgee/?page=gene&amp;gene_id=ENSBTAG00000043559/20211110/94710/<br>bgee.org/?page=gene&amp;gene_id=ENSBTAG00000043559/1377128066<br>https://doi.org/10.5281/zenodo.291971</td><td>https://schema.org/<br>Dataset<br>http://rdfs.org/sioc/<br>ns#Item<br>http://xmlns.com/<br>foaf/0.1/Document<br>https://schema.org/<br>Dataset<br>https://bioschemas.org/<br>org/Gene</td><td>406<br>346<br>346<br>296<br>292</td></tr><tr><td>https://bioschemas.org/crawl/v1/<br>bgee/?page=gene&amp;gene_id=ENSBTAG00000043546/20211110/94697/<br>bgee.org/?page=gene&amp;gene_id=ENSBTAG00000043546/1804549344<br>https://doi.org/10.15472/hy9nif</td><td>https://schema.org/<br>Dataset<br>https://bioschemas.org/<br>org/Gene</td><td>289<br>284</td></tr><tr><td>https://bioschemas.org/crawl/v1/<br>bgee/?page=gene&amp;gene_id=ENSBTAG00000043550/20211110/94701/<br>bgee.org/?page=gene&amp;gene_id=ENSBTAG00000043550/1476242225<br>https://www.metanetx.org/chem_info/<br>MNXM383</td><td>https://schema.org/<br>Dataset<br>https://bioschemas.org/<br>org/Gene</td><td>282<br>269<br>264</td></tr></tbody></table>
<table><tr><td>https://bioschemas.org/crawl/v1/bgee/?page=gene&amp;gene_id=ENSBTAG00000043577/20211110/94727/bgee.org/?page=gene&amp;gene_id=ENSBTAG00000043577/1610681495</td><td>https://bioschemas.org/Gene</td><td>261</td></tr><tr><td>https://bioschemas.org/crawl/v1/bgee/?page=gene&amp;gene_id=ENSBTAG00000043556/20211110/94707/bgee.org/?page=gene&amp;gene_id=ENSBTAG00000043556/1277162978</td><td>https://bioschemas.org/Gene</td><td>240</td></tr><tr><td>https://bioschemas.org/crawl/v1/bgee/?page=gene&amp;gene_id=ENSBTAG00000043568/20211110/94719/bgee.org/?page=gene&amp;gene_id=ENSBTAG00000043568/2065005542</td><td>https://bioschemas.org/Gene</td><td>235</td></tr><tr><td>https://bioschemas.org/crawl/v1/bgee/?page=gene&amp;gene_id=ENSBTAG00000043560/20211110/94711/bgee.org/?page=gene&amp;gene_id=ENSBTAG00000043560/1818154049</td><td>https://bioschemas.org/Gene</td><td>229</td></tr></table>

## Discussion

Through this Hackathon project we have demonstrated that it is possible to harvest Bioschemas markup from a number of sites and load them into a triplestore. However, this has revealed a number of challenges that need to be resolved.

Harvesting content from a whole site is very time consuming, particularly for dynamic sites. Harvesting requires visiting each page with markup in turn and extracting the markup. In the case of dynamic sites the content needs to be rendered before it can be extracted. Most of the sites that could be completed harvested did not contain data content beyond Dataset and DataCatalog.

The quality of deployed markup is very problematic. As reported above, a number of sites could not be harvested due to issues with their sitemap. Of those that could be harvested, a number of pages were not harvested due to issues with the markup contained within them, e.g. inclusion of non-escaped characters within strings was a common error. Even the markup that could be extracted contained errors, e.g. the use of different namespaces for the declaration of the Gene type as identified in the Instances per Bioschemas Class query. This highlights the need for a Bioschemas validator capable of both syntactic and semantic checking (see the data validation outputs of Project 29).

## Future work

The next steps for this work would be to improve the robustness of the data harvesting pipeline, including automating the manual steps of iterating over an index of sitemap files; merging individual files for each harvested page into a single file per subsitemap; and loading the harvested data into the triplestore.

The use of data dumps for sites should be considered to eliminate the need for data harvesting, which is a costly process for both data producers – who have to have sufficient bandwidth and compute resources to serve the content – and data consumers – who have to have sufficient compute resources to retrieve, render, and extract the content from each page.
## Jupyter notebooks, GitHub repositories and data repositories

* GitHub repository: https://github.com/BioSchemas/bioschemas-data-harvesting

* Jupyter Notebook: https://github.com/BioSchemas/bioschemas-data-harvesting/blob/main/AnalysisQueries.ipynb

- MyBinder launch: https://mybinder.org/v2/gh/BioSchemas/bioschemas-data-harvesting/HEAD?labpath=AnalysisQueries.ipynb

* SPARQL Endpoint: https://swel.macs.hw.ac.uk/data/repositories/bioschemas

- Snorql Extended Interface: https://swel.macs.hw.ac.uk/bioschemas/

* Data download director: https://swel.macs.hw.ac.uk/bioschemas-data/

## Acknowledgements

This work was done during the BioHackathon Europe 2021 organised by ELIXIR and run in November 2021. We wish to thank the organizers and supporters of the BioHackathon Europe 2021 for offering the venue for improving Bioschemas community efforts.

## References

Dumontier, M., Gray, A. J., Marshall, M. S., Alexiev, V., Ansell, P., Bader, G., Baran, J., Bolleman, J. T., Callahan, A., Cruz-Toledo, J. (2016). The health care and life sciences community profile for dataset descriptions. *PeerJ*, 4, e2331. https://doi.org/10.7717/peerj.2331

Gray, A. J. G., Papadopoulos, P., Asif, I., Mičetić, I., & Hatos, A. (2022, January 11). Creating and exploiting the intrinsically disordered protein knowledge graph (IDP-KG). *13th International Semantic Web Applications and Tools for Health Care and Life Sciences Conference (Swat4hcls 2022)*. http://www.swat4ls.org/

Gray, A. J. G., Papadopoulos, P., Mičetić, I., & Hatos, A. (2021). Exploiting bioschemas markup to populate IDPcentral. BioHackrXiv. https://doi.org/10.37044/osf.io/v3jct