fils/oihBook huggingface dataset:
df4pd = df4.to_pandas()  # convert cudf dataframe to pandas dataframe
df4pd['sdate'] = pd.to_datetime(df4pd['sdate'], format='%Y-%m-%d')  # convert date to datetime
courseByYear = df4pd.groupby(pd.Grouper(key='sdate', freq='Y')).size() # group by year
ax = courseByYear.plot.bar(rot=80, stacked=True, figsize=(15, 5)) # plot
More on Query#
Here we have gone through an introdcution to the SPARQL query language and how to use it to query our graph.
We can look at the SPARQL we use on the larger Ocean Infohub graph.
References#
This is a just a quick overview of the SPARQL language.  You can find a large amount of information online.
https://derwen.ai/docs/kgl/ex4_0/
https://www.stardog.com/tutorials/getting-started-1/
https://www.dataversity.net/introduction-to-sparql/
https://www.slideshare.net/olafhartig/an-introduction-to-sparql
OIH SPARQL#
About#
This page will hold some information about the SPARQL queries we use and
how they connect with some of the profile guidance in this document.    We will
show how this relates to and depends on the Gleaner prov as well as the
Authoritative Reference elements of the patterns.  It is expected that the Gleaner
prov will be present, though this can be made optional in case other
indexing systems are used that do not provide this prov shape.    The SPARQL will
be looking for both Gleaner prov and the Authroitative Reference elements.
This will be different for different patterns.  For example, it might
relate to the publisher provider elements for Creativeworks, but to
the identity element for People and Organizations.
 1prefix prov: <http://www.w3.org/ns/prov#>
 2PREFIX con: <http://www.ontotext.com/connectors/lucene#>
 3PREFIX luc: <http://www.ontotext.com/owlim/lucene#>
 4PREFIX con-inst: <http://www.ontotext.com/connectors/lucene/instance#>
 5PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
 6PREFIX schema: <https://schema.org/>
6PREFIX schema: <https://schema.org/>
 7PREFIX schemaold: <http://schema.org/>
 8PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
 9
10SELECT DISTINCT ?g  ?s  ?wat ?orgname ?domain ?type ?score ?name ?url ?lit ?description ?headline
11WHERE {
12   ?lit bds:search "coral" .
13   ?lit bds:matchAllTerms "false" .
14   ?lit bds:relevance ?score .
15   graph ?g {
16    ?s ?p ?lit .
17    ?s rdf:type ?type . 
18    OPTIONAL { ?s schema:name ?name .   }
19    OPTIONAL { ?s schema:headline ?headline .   }
20    OPTIONAL { ?s schema:url ?url .   }
21    OPTIONAL { ?s schema:description ?description .    }
22  }
23   ?sp prov:generated ?g  .
24   ?sp prov:used ?used .
25   ?used prov:hadMember ?hm .
26   ?hm prov:wasAttributedTo ?wat .
27   ?wat rdf:name ?orgname .
28   ?wat rdfs:seeAlso ?domai
29}
28   ?wat rdfs:seeAlso ?domai
29}
30ORDER BY DESC(?score)
31LIMIT 30
32OFFSET 0
Lines 12-14#
It should be noted that the above SPARQL is not standards compliant.  It leverages some
vender specific syntax that is not part of the SPARQL standard.  This is not uncommon
as groups will often add their own syntax to offer additional functionality.
A common one is what is seen here where a full text index is leveraged to allow for more complex
and faster searches than can be done with FILTER regex.  These three lines will
only work in the current OIH triplestore (Blazegraph).   Other triplestores like Jena
and other do similar built in function extensions.
Lines 18-21#
These line demonstrate the use of the OPTIONAL keyword.  These triples are not required
to be present in a resource.  If they are, they will be returned.
Lines 23-28#
These lines are standard SPARQL but are searching across triples not from the provider
graphs.  Rather, they are looking at triples generated by the OIH indexing program
used, Gleaner.
Note, that Gleaner is not a dependency of this project and other
indexing approaches and software could be used.  As pointed out in the documentation,
indexing approaches and software could be used.  As pointed out in the documentation,
this approach is based on structured data on the web and web architecture approaches.
So, any indexing system following those approaches can be used.
These triples are used to track the indexing event and the sources indexed.  It provides
some additional provenance to the information collected, but does not change or even
extend what the providers are publishing.
As such, these statements could be removed and all that would be lost of indexing
activity information.
Lines 30-32#
These lines represent three specific SPARQL parameters.
First is the ORDER BY directive.  This is used to order the results by one of the
returned variables.  In this case we are using the ?score variable which comes from
the vendor specific syntax noted in lines 11-13.  This score is the ranking score
for a resource search against the full text index.  However, this could be any
variable coming from standards compliant SPARQL calls too.   Sorting can be done
on alphanumeric values in ascending (ASC) or descending (DESC) order.
The  LIMIT is used to limit the number of results returned.  We follow this with,
OFFSET which is used to skip the first n results.  These two are useful for pagination when
combined with the ORDER BY directive.
Users#
About#
The reference client is available at
search.oceaninfohub.org.
Using OIH search in Chrome#
It is possible to set Ocean InfoHub as a search shortcut in Chrome.
To do this go to the Mange Search … section of your settings.
There you will see a button “Add” in the Site Search section.
You can make an entry like: https://search.oceaninfohub.org/results/?search_text=%s&page=0
The shortcut, here oih can be used in the address bar to quickly invoke this search.
Simple type: oih or whatever you set the shortcut to and then hit the tab key.  The address bar will convert to a search mode where you can type your search and hit return.
APIs#
About#
The Ocean InfoHub graph is accessible through several approaches.
The graph is implemented in RDF and expressed through a standards compliant triplestore.
This triplestore exposes a SPARQL endpoint that can be queried using the SPARQL 1.1 Query Language.
To do this you can visit a web based query interface as discussed in other sections of this document.
It is also possible to access this service following RESTful principles.
SPARQL HTTP Protocol#
One approach to this RESTful approach is to use the
SPARQL 1.1 Graph Store HTTP Protocol.
This is a simple HTTP protocol that allows you to query the graph using SPARQL and obtain the results in JSON.
The OIH triplestore exposes the graph following this pattern for queries.
Examples#
curl -X POST https://graph.collaborium.io/blazegraph/namespace/aquadocs/sparql --data-urlencode 'query=SELECT * { ?s ?p ?o } LIMIT 1' -H 'Accept:application/sparql-results+json'
If run this from the command line we will get something like the following.
If run this from the command line we will get something like the following.
✗ curl -X POST https://graph.collaborium.io/blazegraph/namespace/aquadocs/sparql --data-urlencode 'query=SELECT * { ?s ?p ?o } LIMIT 1' -H 'Accept:application/sparql-results+json'
{
  "head" : {
    "vars" : [ "s", "p", "o" ]
  },
  "results" : {
    "bindings" : [ {
      "s" : {
        "type" : "uri",
        "value" : "https://hdl.handle.net/1834/10030"
      },
      "p" : {
        "type" : "uri",
        "value" : "https://schema.org/propertyID"
      },
      "o" : {
        "type" : "literal",
        "value" : "https://hdl.handle.net/"
      }
    } ]
  }
}                                   
While this is unlikely how you will want to interact with the graph,
it desmonstrates the HTTP based access API that can be used in tools, notebooks
or other applications.
This is the same basic approach the used in the web client.  There the
This is the same basic approach the used in the web client.  There the
axios library (https://axios-http.com/) is used with a code snippet like:
 axios.get(url.toString())
      .then(function (response) {
        // handle success
        console.log(response);
        const el = document.querySelector('#container2');
        render(showresults(response), el);
      })
      .catch(function (error) {
        // handle error
        console.log(error);
      })
      .then(function () {
        // always executed
      });
References#
A broad collection of references.
General#
Science on Schema
BioSchemas
Ocean Best Practices on Schema
PID policy for European Open Science Cloud
DCAT Schema.org mappings
DCAT US Data.gov reference
FAIR Semantics
Developer References#
Schema.org releases
Schema.org RDF graph (turtle format)
JSON-LD Playground
Google Developers Search Gruid
Google Developers Fact Check
Structured Data Testing Tool
Rich Results Testing Tool
JSON-LD
Ruby JSON-LD
schema.org Java
Perl classes for schema.org markup
Content Management and Web Server support#
Drupal Support
Wordpress Claim Review
Organizations#
Google Open Source
DataCommons  & DataCommons REST
Indexers#
Gleaner
BMUSE
Graph tools#
Wikipedia SPARQL implementation list
RDFlib
Any23
rdfjs
shemaram
validatingrdf
Structured Data Linter
Blogs and Press Releases#
Yandex: What is Scheme.org
Bing: Fact Check Label
Bing: Contextual Awareness
Facebook: Marketing API
Facebook: fact checking
Amazon: Alex skills
Google Developers mail invoice
Not Categorized Yet#
Lighthouse Plugins
Lighthouse
Science on Schema
BioSchemas
CodeMeta
Linter Structured Data
JSON-LD Playground
JSON-LD.org
SHACL playground
Google Structured Data testing tool (
JSON-LD.org
SHACL playground
Google Structured Data testing tool (
Google Dataset for  developers
Press  article
Rich results
SchemaApp.com
Yandex
Schema dev
Chromeextension
Google Rich Results
Datashapes
ACL Web Alexa Meaning Representation Language
hash.aio Volcano schema
Yoast Structured Data Guide
Schema App
Springer: Scigraph
iPhylo Biodiversity KG
ozymandias
RDA group meeting notes
RDA Plenary meeting
Registries#
Documents and Datasets#
DOI
A not-for-profit membership organization that is the governance and management body for the federation of Registration Agencies providing Digital Object Identifier (DOI) services and registration, and is the registration authority for the ISO standard (ISO 26324) for the DOI system. The DOI system provides a technical and social infrastructure for the registration and use of persistent interoperable identifiers, called DOIs, for use on digital networks.
Datacite
Locate, identify, and cite research data with the leading global provider of DOIs for research data.
Archival Resource Key or ARK:  and
N2T ARKs and Names to Thinkgs
People#
Orcid
ORCID’s mission is to enable transparent and trustworthy connections between researchers, their contributions, and their affiliations by providing a unique, persistent identifier for individuals to use as they engage in research, scholarship, and innovation activities.
Organizations#
re3data
A registry of research data repositories
ROR
ROR is a community-led project to develop an open, sustainable, usable, and unique identifier for every research organization in the world.
Grid
GRID is a free and openly available global database of research-related organisations, cataloging research-related organisations and providing each with a unique and persistent identifier. With GRID you have over 99,609 carefully curated records at hand, enabling you to identify and distinguish research-related institutions worldwide.
Physical Samples#
IGSN
The objective of the IGSN e.V. is to implement and promote standard methods for identifying, citing, and locating physical samples with confidence by operating an international IGSN registration service.
Known Issues#
About#
This document will collect some of the various issues we have encounter in publishing
the JSON-LD documents.
control characters in URL string for sitemap or in the JSON-LD documents#
Make sure there are no control characters such as new line, carriage returns,
tabs or others in the document.  These can be problematic both for processing and
display.
context is a map (changed from 1.0)#
Be sure to use a context style like:
    "@context": {
        "@vocab": "https://schema.org/"
    },
The context section must be a map starting in JSON-LD 1.1
data graphs need @id#
Be sure to include an @id in your graph that points to the identifier or the
web address of the resource providing the metadata.  This is not the material the
metadata is about, but rather the metadata record itself.
string literals must be valid#
The string literals must be sure to not have quotation marks or other invalid
characters without escaping or encoding them.
Appendix#
About#
A collection of items related to the OIH development.  Mostly related
to examples around representing concepts in the graph such as
date and time, language etc.
As these develop they may be moved into other sections of the book.
Controlled Vocabularies#
About#
See also:  ODIS-ARCH Vocabularies
A list of possible controlled vocabularies to use in the schema.org documents.
Many such resources can be found by searching at BARTOC.org
or the UNESCO Thesaurus.
Note, at present this is an exploration and there is not yet a recommendation
for use in OIH.
List#
Essential Climate Variables
Vocabulary based on DFG’S Classification of Subject Area, Review Board, Research Area and Scientific Discipline (2016 - 2019)
Nature Subjects Ontology
Indexing with Gleaner#
Gleaner (app)#
The Gleaner applications performs the retrieval and loading of JSON-LD documents
from the web following structured data on the web patterns.  Gleaner is available for Linux, Mac OS X and Windows.
While Gleaner is a stand alone app, it needs to interact with
an object store to support data storage and other operations.  These dependencies are met within the
Gleaner Indexing Services or Data Service Docker compose files.
Warning
This documentation is in development.  The primary testing environments are Linux and other UNIX based platforms
such as Mac OS X.   If you are on Windows, there may be some issues.  If you can use a Linux subsystem on Windows,
you may experience better results.  We will test with Windows eventually and update documentation as needed.
Quick Start steps#
This quick start guide is focused on setting up and testing Gleaner in a local environnement.  It is similar to
how you might run Gleaner in a production environment but lacks the routing and other features likely desired for
such a situation.
Note
This documentation assumes a basic understanding of Docker and experience with basic Docker activities like
starting and stopping containers.  It also assumes an understanding of using a command line interface and
editing configuration files in the YAML format.
Command
From this point down, the documentation will attempt to put all commands
Command
From this point down, the documentation will attempt to put all commands
you should issue in this admonition style box.
In the end, this is the table of applications and config files you will need.  In this guide we will go through
downloading, setting them up and running Gleaner to index documents from the web.
Table 1 Required Applications and Their Config Files#
Gleaner
Docker
Minio Client
config.yaml
setenv.sh 
load2blaze.sh
schemaorg-current-https.jsonld
gleaner-DS-NoRouter.yml
Grab Gleaner and the support files we need#
We will need to get the Gleaner binary for your platform and also the Gleaner configuration file
template.  To do this, visit the Gleaner Releases page 
and pick the release Ocean InfoHubdev rc1.  Under the Assets drop down you should see the files we need.  Get:
Gleaner for your platform
Gleaner config template: template_v2.0.yaml
Gleaner indexing service compose file: gleaner-IS.yml
Helper environment setup script: setenvIS.sh
For this demonstration, we will be running on linux, so this would look something like:
Command
curl -L -O https://github.com/earthcubearchitecture-project418/gleaner/releases/download/2.0.25/gleaner
curl -L -O https://github.com/earthcubearchitecture-project418/gleaner/releases/download/2.0.25/gleaner-IS.yml
curl -L -O https://github.com/earthcubearchitecture-project418/gleaner/releases/download/2.0.25/setenvIS.sh
curl -L -O https://github.com/earthcubearchitecture-project418/gleaner/releases/download/2.0.25/template_v2.0.yaml
Note
You can download these with any tool you wish or through the browser.  Above we downloaded used the command
line curl tool.  For GitHub, be sure to add the -L to inform curl to follow redirects to the object to download.
Command
You may need to change the permission on your gleaner file to ensure it can be run.   On Linux this would
look something like the following.
chmod 755 gleaner
We then need to visit Schema.org for Developers to pull down the
appropriate JSON-LD context.  For this work we will want to pull down the schemaorg-current-https in JSON-LD format.
It also should work to do something similar to the following:
Command
curl -O https://schema.org/version/latest/schemaorg-current-https.jsonld
About the compose file(s)#
The above steps have collected the resources for the indexer.   We now want to set up the services that
Gleaner will use to perform the indexing.  To do that we use Docker or an appropriate run time alternative like
Podman or others.   For this example, we will assume you are using the Docker client.
As noted, a basic understanding of Docker and the ability to issue Docker cli commands to start and stop
containers is required. If you are new do Docker, we recommend you visit and read:
Get Started with Docker.
We need to select the type of services we wish to run.  The various versions of these Docker compose
file can be found in the Gleaner-compose deployment directory.
Why pick one over the other?
Choose Gleaner IS if you simply wish to retrieve the JSON-LD into a data warehouse to use in your own workflows
Choose Gleaner DS if you wish to build out a graph and want to use the default contains used by Gleaner.
Note
We wont look at this file in detail here since there will hopefully be no
required edits.  You can see the file in detail in the Index Services
section.
Edit environment variables setup script#
We have Docker and the appropriate compose file.  The compose files require a set of environment variables
to be populated to provide the local hosts information needed to run.  You can set these yourself or
use or reference the setenv.sh file in the Gleaner-compose repository in the
Gleaner-compose deployment directory.
Gleaner-compose deployment directory.
You may also need to visit information about permissions at
Post-installation steps for Linux if you are
having permission issues.
Let’s take a look at the script.
 1#!/bin/bash
 2
 3# Object store keys
 4export MINIO_ACCESS_KEY=worldsbestaccesskey
 5export MINIO_SECRET_KEY=worldsbestsecretkey
 6
 7# local data volumes
 8export GLEANER_BASE=/tmp/gleaner/
 9mkdir -p ${GLEANER_BASE}
10export GLEANER_OBJECTS=${GLEANER_BASE}/datavol/s3
11export GLEANER_GRAPH=${GLEANER_BASE}/datavol/graph
You may wish to edit file to work better with your environment.  By default it will attempt to
use localhost to resolve with and host local runtime data in a /tmp/gleaner directory.
Spin up the containers#
Load our environment variables to the shell:
Command
source setenv.sh
Then start the containers:
Command
docker-compose -f gleaner-IS.yml up -d
If all has gone well, you should be able to see your running containers with
Command
docker ps
and see results similar to:
CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS              PORTS                    NAMES
CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS              PORTS                    NAMES
c4b7097f5e06        nawer/blazegraph                 "docker-entrypoint.s…"   8 seconds ago       Up 7 seconds        0.0.0.0:9999->9999/tcp   test_triplestore_1
ca08c24963a0        minio/minio:latest               "/usr/bin/docker-ent…"   8 seconds ago       Up 7 seconds        0.0.0.0:9000->9000/tcp   test_s3system_1
24274eba0d34        chromedp/headless-shell:latest   "/headless-shell/hea…"   8 seconds ago       Up 7 seconds        0.0.0.0:9222->9222/tcp   test_headless_1
Edit Gleaner config file#
We have all the files we need and we have our support services running.  The next and
final step is to edit our Gleaner configuration file.  This will let Gleaner know
the location of the support services, the JSON-LD context file and the locations
of the resources we wish to index.
Let’s take a look at the full configuration file first and then break down each section.
 1---
 2minio:
1---
 2minio:
 3  address: 0.0.0.0
 4  port: 9000
 5  accessKey: worldsbestaccesskey      
 6  secretKey: worldsbestsecretkey  
 7  ssl: false
 8  bucket: gleaner
 9gleaner:
10  runid: oih # this will be the bucket the output is placed in...
11  summon: true # do we want to visit the web sites and pull down the files
12  mill: true
13context:
14  cache: true
15contextmaps:
16- prefix: "https://schema.org/"
17  file: "./jsonldcontext.json"  # wget http://schema.org/docs/jsonldcontext.jsonld
18- prefix: "http://schema.org/"
19  file: "./jsonldcontext.json"  # wget http://schema.org/docs/jsonldcontext.jsonld
20summoner:
21  after: ""      # "21 May 20 10:00 UTC"   
22  mode: full  # full || diff:  If diff compare what we have currently in gleaner to sitemap, get only new, delete missing
23  threads: 1
23  threads: 1
24  delay: 0  # milliseconds (1000 = 1 second) to delay between calls (will FORCE threads to 1) 
25  headless: http://0.0.0.0:9222  # URL for headless see docs/headless
26millers:
27  graph: true
28  #geojson: false
29sitegraphs:
30- name: aquadocs
31  url: https://oih.aquadocs.org/aquadocs.json 
32  headless: false
33  pid: https://www.re3data.org/repository/aquadocs
34  properName: AquaDocs
35  domain: https://aquadocs.org 
36sources:
37- name: samplesearth
38  url: https://samples.earth/sitemap.xml
39  headless: false
40  pid: https://www.re3data.org/repository/samplesearth
41  properName: Samples Earth (DEMO Site)
42  domain: https://samples.earth  
43- name: marinetraining
44  url: https://www.marinetraining.eu/sitemap.xml
45  headless: false
46  pid: https://www.re3data.org/repository/marinetraining
47  properName: Marine Training EU
47  properName: Marine Training EU
48  domain: https://marinetraining.eu/
49- name: marineie
50  url: http://data.marine.ie/geonetwork/srv/eng/portal.sitemap
51  headless: true
52  pid: https://www.re3data.org/repository/marineie
53  properName: Marine Institute Data Catalogue
54  domain: http://data.marine.ie
55- name: oceanexperts
56  url: https://oceanexpert.org/assets/sitemaps/sitemapTraining.xml
57  headless: false
58  pid: https://www.re3data.org/repository/oceanexpert
59  properName: OceanExpert UNESCO/IOC Project Office for IODE 
60  domain: https://oceanexpert.org/
61# - name: obis
62#   url: https://obis.org/sitemap/sitemap_datasets.xml
63#   headless: false
64#   pid: https://www.re3data.org/repository/obis
65#   properName: Ocean Biodiversity Information System
66#   domain: https://obis.org  
Object store#
1minio:
2  address: 0.0.0.0
3  port: 9000
4  accessKey: worldsbestaccesskey
3  port: 9000
4  accessKey: worldsbestaccesskey      
5  secretKey: worldsbestsecretkey  
6  ssl: false
7  bucket: gleaner
The minio section defines the IP and port of the object store.  For this case, we are
using minio and these are the IP and port from our docker compose steps above.  Note,
if you were to use Ceph or AWS S3, this section is still labeled minio.  You simply
need to update the property values.
Gleaner#
1gleaner:
2  runid: oih # this will be the bucket the output is placed in...
3  summon: true # do we want to visit the web sites and pull down the files
4  mill: true
This passes a few high level concpets.
runid:
summon
mill
Context sections#
1context:
2  cache: true
3contextmaps:
4- prefix: "https://schema.org/"
5  file: "./jsonldcontext.json"  # wget http://schema.org/docs/jsonldcontext.jsonld
6- prefix: "http://schema.org/"
7  file: "./jsonldcontext.json"  # wget http://schema.org/docs/jsonldcontext.jsonld
Comments for the context sections
Summoner section#
1summoner: