fils/oihBook huggingface dataset:
Comments for the context sections
Summoner section#
1summoner:
2  after: ""      # "21 May 20 10:00 UTC"   
3  mode: full  # full || diff:  If diff compare what we have currently in gleaner to sitemap, get only new, delete missing
4  threads: 1
5  delay: 0  # milliseconds (1000 = 1 second) to delay between calls (will FORCE threads to 1) 
6  headless: http://0.0.0.0:9222  # URL for headless see docs/headless
Comments for the summoner sections
Millers section#
1millers:
2  graph: true
3  #geojson: false
Comments for the miller sections
Site graphs section#
1sitegraphs:
2- name: aquadocs
3  url: https://oih.aquadocs.org/aquadocs.json 
4  headless: false
5  pid: https://www.re3data.org/repository/aquadocs
6  properName: AquaDocs
7  domain: https://aquadocs.org 
Comments for the sitegrpah sections
Sources section#
 1sources:
 2- name: samplesearth
 3  url: https://samples.earth/sitemap.xml
3  url: https://samples.earth/sitemap.xml
 4  headless: false
 5  pid: https://www.re3data.org/repository/samplesearth
 6  properName: Samples Earth (DEMO Site)
 7  domain: https://samples.earth  
 8- name: marinetraining
 9  url: https://www.marinetraining.eu/sitemap.xml
10  headless: false
11  pid: https://www.re3data.org/repository/marinetraining
12  properName: Marine Training EU
13  domain: https://marinetraining.eu/
14- name: marineie
15  url: http://data.marine.ie/geonetwork/srv/eng/portal.sitemap
16  headless: true
17  pid: https://www.re3data.org/repository/marineie
18  properName: Marine Institute Data Catalogue
19  domain: http://data.marine.ie
20- name: oceanexperts
21  url: https://oceanexpert.org/assets/sitemaps/sitemapTraining.xml
22  headless: false
23  pid: https://www.re3data.org/repository/oceanexpert
24  properName: OceanExpert UNESCO/IOC Project Office for IODE 
25  domain: https://oceanexpert.org/
26# - name: obis
25  domain: https://oceanexpert.org/
26# - name: obis
27#   url: https://obis.org/sitemap/sitemap_datasets.xml
28#   headless: false
29#   pid: https://www.re3data.org/repository/obis
30#   properName: Ocean Biodiversity Information System
31#   domain: https://obis.org  
Comments for the sources sections
Run gleaner#
For this example we are going to run Gleaner directly.  In a deployed instance you may
run Gleaner via a script or cron style service.  We will document that elsewhere.
We can do a quick test of the setup.
Command
 ./gleaner -cfg template_v2.0 -setup
For now, we are ready to run Gleaner.  Try:
Command
 ./gleaner -cfg template_v2.0
Note
Leave the suffix like .yaml off the name of the config file.  The config system can also read
json and other formats.  So simply leave the suffix off and let the config code inspect the
contents.
Load results to a graph and test#
You have set up the server environment and Gleaner and done your run.  Things look good
but you don’t have a graph you can work with yet.    You need to load the JSON-LD into
the triplestore in order to start playing.
Minio Object store#
To view the object store you could use your browser and point it on the default minio
port at 9000.  This typically something like localhost:9000.
If you wish to continue to use the command line you can use the Minio client at
Minio Client Quickstart guide.
Once you have it installed and working, you can write an entry for our object store with:
Command
 ./mc alias set minio http://0.0.0.0:9000 worldsbestaccesskey worldsbestsecretkey
Load Triplestore#
We now want to load these objects, which are JSON-LD files holding RDF based graph
data, into a graph database.  We use the term, triplestore, to define a graph database
designed to work with the RDF data model and provide SPARQL query support over that
graph data.
Simple script loading
Nabu
Try out a simple SPARQL query
References#
The following are some reference which may provide more information on the various
technologies used in this approach.
Google: Understanding how structured data works
Google Dataset Search By the Numbers
Google Dataset Search: Building a search engine for datasets in an open Web ecosystem
W3C SPARQL
SHACL
Triplestores
Interfaces#
About#
In the end the goal is to provide use of the generated index.  There are several
possible used for an index.
Web UI such as the reference client at oceans.collaborium.io
A variation on this is the development of web components that can be easily included in
domain sites to perform operations on the OIH index
graph access via SPARQL
access to the graph and objects via workflows like Jupyter notebooks
Gleaner Web UI (WUI)#
The user of the index may take several forms.  A user may be a software developer creating a web based
interface to the generated index.  It may also be an end user accessing the index (indexes) through notebooks
or special clients.
Those wishing to run a web site can augemnt the compose files to run their
preferred web server, object server (to serve files from the object store) or
software such as node or others to support their deployment pattern.
Fig. 6 Gleaner Optional Web UI#
Alternatives#
Options#
While Gleaner will be used during initial OIH development it is not
the only or required approach.   The web architecture foundation means there are many other tools that
can be used and might be leveraged in a production environment including:
Extrunct
BioSchemas Tools
LDSpider
StormCrawler
Squirrel
Nutch (Apache)
Laundromat
DataArchiver
OD Archiver
These different tools may better fit into the workflow and available
skill sets for a group.
Graph First Approach#
About#
During the early adopters meetings and in discussion with others an alternative publication pattern came up.  This is the pattern where it is not possible to update the web resources with the metadata content.  This may be due to access or technical issues.  Regardless, what was possible was to generate the metadata in bulk locally and make the resulting document available.
This approach is not ideal since it is a non-standard pattern and makes the data and information more obscure to other users.  However, it is one the OIH architecture can adapt to and is preferable to the option of excluding those partners in this activity.
As such, we are making some changes to allow for this pattern.  This means documenting the published graph structure based on the existing thematic patterns and some updates in the indexing workflow to obtain and integrate these graphs into the OIH graph.
Warning
Anti-pattern:
Using the approach here is not in alignment with
Google guidance nor with W3C patterns for structured
data on the web.
It is documented here for edge cases
where this is the minimum viable approach.  The hope is it
could act as a gateway to a more standards aligned implementation later.
Graph Only#
There are cases where it is only possible to generate the graph
based on the metadata.  Access to the HTML pages is either difficult or the process of inserting the
data into the pages is not supportable.
data into the pages is not supportable.
For this case the goal is to create a simple graph in JSON-LD.  To do this we need a collection
approach that is valid for a range of Things.
For this it is proposed to use ItemList which can be used on a list of type Thing, ie anything
type in the Schema.org vocabulary.
This would define a ListItem with item of any type.  Below is an example for a CreativeWork (map)
and a Course.  Once you are in a “item” any of the details from the other thematic type descriptions
can be used.
{
  "@context": "https://schema.org/",
  "@type": ["ItemList", "CreativeWork"],
  "@id": "https://example.org/id/graph/X",
  "name": "Resource collection for site X",
  "author": "Creator of the list",
  "itemListOrder": "https://schema.org/ItemListUnordered",
  "numberOfItems": 2,
  "itemListElement": [
    {
      "@type": "ListItem",
      "item": {
           "@id": "ID_for_this_metadata_record1",
           "@type": "Map",
            "@id": "https://example.org/id/XYZ",
            "name": "Name or title of the document",
"name": "Name or title of the document",
            "description": "Description of the map to aid in searching",
            "url":  "https://www.sample-data-repository.org/creativework/map.pdf"
      }
    },
    {
      "@type": "ListItem",
      "item": {
           "@id": "ID_for_this_metadata_record2",
            "@type": "Course",
            "courseCode": "F300",
            "name": "Physics",
            "provider": {
                "@type": "CollegeOrUniversity",
                "name": "University of Bristol",
                "url": {
                    "@id": "/provider/324/university-of-bristol"
                }
            }
        }
    }
  ]
}
In the case of schema:Dataset one might use schema:DataCatlogue for the following approach.  However,
since OIH is addressing a wide range of types a more generic collection of Things or CreativeWorks
approach is needed.
Item Catalogue Page#
It’s not hard to generate a simple HTMl page based on the structured metadata file.  This doesn’t
alter the content of the graph, just builds an automated HTMl page around it.
Publishing and referencing#
Testing#
Publishing and referencing#
Testing#
Since we are now dealing with a graph that is pulled as a complete entity there are a few thoughts.
How do ensure a connection between a record in the list and a resolvable URL?  Do we need
to:
ensure each record has a IRI it is subject of
in the case where IRI is or can be URL, do a validation of at least a 200 on it
How do we publish this?
entry in robots.txt (might be able due to reasons above?)
published and provided to OIH
Need guidance on format and structure
Data Services#
The typical functional goal of this work is the development and use of a Graph that can be accessed via a triplestore (Graph Database).  To do that we need a set of additional containers to support this and expose these services on the web through a single domain with https support.
Object Store 
An S3 compliant object store supporting S3 APIs including S3Select.  For open source this is best satisfied with the Minio Object Store.  For commercial cloud AWS S3 or hosted Ceph services will work.
Graph Database
Web Router (technically optional)
Gleaner Data Services (DS)#
If you wish to work with a triplestore and wish to use the default app used by OIH
you can use the compose file that sets up the Gleaner Data Services environment.
This adds the Blazegraph triplestore to the configuration along with the object store.
The details of the OIH data services are found in the
Data Services section.
Fig. 3 Gleaner Data Service Activity Workflow#
Typically, a user would wish to run the full Gleaner DS stack which supports
both the indexing process and the serving of the resulting data warehouse and graph
database capacity.
Combined, these would then look like the following where the indexing and
data services shared a common object store.
Fig. 4 Gleaner Indexing and Data Service Combined#
Object store pattern#
Fig. 4 Gleaner Indexing and Data Service Combined#
Object store pattern#
Within in the object store the following digital object pattern is used.
This is based on the work of the RDA Digital Fabric working group.
Fig. 5 Gleaner Digital Object Pattern#
At this point the graph and data warehouse (object store) can be exposed to the net for use by clients such as jupyter notebooks or direct client calls to the S3 object APIs and SPARQL endpoint.
Gleaner Data Services (DS) Environment Variables
The Docker Compose file used to launch the Gleaner DS has a set of configurable elements that can be set and passed to the orchestration system with environment variables.
These can be set manually or through the command line.  A simple script to set the variables could look like:
– Environment Var settings script
The actual services can be deployed via a Docker Compose file (also works with Podman).  An example of that file and details about it follow.
Let’s take a look at this.
 1#!/bin/bash
 2
 3# domains 
 4export GLEANER_ADMIN_DOMAIN=admin.local.dev
 5export GLEANER_OSS_DOMAIN=oss.local.dev
 6export GLEANER_GRAPH_DOMAIN=graph.local.dev
 7export GLEANER_WEB_DOMAIN=web.local.dev
7export GLEANER_WEB_DOMAIN=web.local.dev
 8export GLEANER_WEB2_DOMAIN=web2.local.dev
 9
10# Object store keys
11export MINIO_ACCESS_KEY=worldsbestaccesskey
12export MINIO_SECRET_KEY=worldsbestsecretkey
13
14# local data volumes
15export GLEANER_BASE=/tmp/gleaner/
16export GLEANER_TRAEFIK=${GLEANER_BASE}/config
17export GLEANER_OBJECTS=${GLEANER_BASE}/datavol/s3
18export GLEANER_GRAPH=${GLEANER_BASE}/datavol/graph
19
– Break down the compose file here
  1version: '3'
  2
  3# ${GLEANER_ADMIN_DOMAIN}
  4# ${GLEANER_OSS_DOMAIN}
  5# ${GLEANER_GRAPH_DOMAIN}
  6# ${GLEANER_WEB_DOMAIN}  
  7# ${GLEANER_WEB2_DOMAIN}
  8# ${MINIO_ACCESS_KEY}  
  9# ${MINIO_SECRET_KEY}
 10#
 11# ${GLEANER_TRAEFIK}
 12# ${GLEANER_OBJECTS}
 13# ${GLEANER_GRAPH}
 14
 15services:
 16  triplestore:
14
 15services:
 16  triplestore:
 17    image: nawer/blazegraph 
 18    environment:
 19      JAVA_XMS: 2g
 20      JAVA_XMX: 8g
 21      JAVA_OPTS: -Xmx6g -Xms2g --XX:+UseG1GC
 22    ports:
 23      - 9999:9999
 24    labels:
 25      - "traefik.enable=true"
 26      - "traefik.http.routers.triplestore.entrypoints=http"
 27      - "traefik.http.routers.triplestore.rule=Host(`${GLEANER_GRAPH_DOMAIN}`)"
 28      - "traefik.http.middlewares.triplestore-https-redirect.redirectscheme.scheme=https"
 29      - "traefik.http.routers.triplestore.middlewares=triplestore-https-redirect"
 30      - "traefik.http.routers.triplestore-secure.entrypoints=https"
 31      - "traefik.http.routers.triplestore-secure.rule=Host(`${GLEANER_GRAPH_DOMAIN}`)"
 32      - "traefik.http.routers.triplestore-secure.tls=true"
33      - "traefik.http.routers.triplestore-secure.tls.certresolver=http"
 34      - "traefik.http.routers.triplestore-secure.service=triplestore"
 35      - "traefik.http.middlewares.triplestore-secure.headers.accesscontrolallowmethods=GET,OPTIONS,PUT,POST"
 36      - "traefik.http.middlewares.triplestore-secure.headers.accesscontrolalloworigin=*"
 37      - "traefik.http.middlewares.triplestore-secure.headers.accesscontrolmaxage=200"
 38      - "traefik.http.middlewares.triplestore-secure.headers.addvaryheader=true"
 39      - "traefik.http.middlewares.triplestore-secure.headers.accesscontrolallowcredentials=true"
 40      - "traefik.http.middlewares.triplestore-secure.headers.accesscontrolallowheaders=Authorization,Origin,Content-Type,Accept"
 41      - "traefik.http.middlewares.triplestore-secure.headers.customresponseheaders.Access-Control-Allow-Headers=Authorization,Origin,Content-Type,Accept"
 42      - "traefik.http.routers.triplestore-secure.middlewares=triplestore-secure@docker" 
 43      - "traefik.http.services.triplestore.loadbalancer.server.port=9999"
44      - "traefik.docker.network=traefik_default"
 45    volumes:
 46      - ${GLEANER_GRAPH}:/var/lib/blazegraph
 47    networks:
 48      - traefik_default
 49
 50  s3system:
 51    image: minio/minio:latest
 52    ports:
 53      - 9000:9000
 54    labels:
 55      - "traefik.enable=true"
 56      - "traefik.http.routers.s3system.entrypoints=http"
 57      - "traefik.http.routers.s3system.rule=Host(`${GLEANER_OSS_DOMAIN}`)"
 58      - "traefik.http.middlewares.s3system-https-redirect.redirectscheme.scheme=https"
 59      - "traefik.http.routers.s3system.middlewares=s3system-https-redirect"
 60      - "traefik.http.routers.s3system-secure.entrypoints=https"
 61      - "traefik.http.routers.s3system-secure.rule=Host(`${GLEANER_OSS_DOMAIN}`)"
 62      - "traefik.http.routers.s3system-secure.tls=true"
63      - "traefik.http.routers.s3system-secure.tls.certresolver=http"
 64      - "traefik.http.routers.s3system-secure.service=s3system"
 65      - "traefik.http.services.s3system.loadbalancer.server.port=9000"
 66      - "traefik.docker.network=traefik_default"
 67    volumes:
 68      - ${GLEANER_OBJECTS}:/data
 69    environment:
 70      - MINIO_ACCESS_KEY=${MINIO_ACCESS_KEY}
 71      - MINIO_SECRET_KEY=${MINIO_SECRET_KEY}
 72    networks:
 73      - traefik_default
 74    command: ["server", "/data"]
 75
 76  features:
 77    image: fils/grow-general:latest
 78    ports:
 79      - 8080:8080
 80    environment:
 81      - S3ADDRESS=s3system:9000
 82      - S3BUCKET=sites
 83      - S3PREFIX=domain
 84      - DOMAIN=https://${GLEANER_WEB_DOMAIN}/
 85      - S3KEY=${MINIO_ACCESS_KEY}
 86      - S3SECRET=${MINIO_SECRET_KEY}
 87    labels:
86      - S3SECRET=${MINIO_SECRET_KEY}
 87    labels:
 88      - "traefik.enable=true"
 89      - "traefik.http.routers.features.entrypoints=http"
 90      - "traefik.http.routers.features.rule=Host(`${GLEANER_WEB_DOMAIN}`, `${GLEANER_WEB2_DOMAIN}`)"
 91      - "traefik.http.middlewares.features-https-redirect.redirectscheme.scheme=https"
 92      - "traefik.http.routers.features.middlewares=features-https-redirect"
 93      - "traefik.http.routers.features-secure.entrypoints=https"
 94      - "traefik.http.routers.features-secure.rule=Host(`${GLEANER_WEB_DOMAIN}`,`${GLEANER_WEB2_DOMAIN}`)"
 95      - "traefik.http.routers.features-secure.tls=true"
 96      - "traefik.http.routers.features-secure.tls.certresolver=http"
 97      - "traefik.http.routers.features-secure.service=features"
 98      - "traefik.http.services.features.loadbalancer.server.port=8080"
 99      - "traefik.docker.network=traefik_default"
99      - "traefik.docker.network=traefik_default"
100      - "traefik.http.middlewares.features.headers.accesscontrolallowmethods=GET,OPTIONS,PUT,POST"
101      - "traefik.http.middlewares.features.headers.accesscontrolalloworigin=*"
102      - "traefik.http.middlewares.features.headers.accesscontrolmaxage=100"
103      - "traefik.http.middlewares.features.headers.addvaryheader=true"
104      - "traefik.http.middlewares.features-secure.headers.accesscontrolallowheaders=*"
105      - "traefik.http.middlewares.features-secure.headers.customresponseheaders.Access-Control-Allow-Headers=*"
106    networks:
107      - traefik_default
108
109networks:
110  traefik_default:
111
NOTE:  DS also needs the object -> graph sync (via Nabu)
NOTE:  Should also add in (here or to the side) the ELT local Data Lake to Data Warehouse path (ala CSDCO VaultWalker)
Indexing Services#
Gleaner can not run alone and relies on a couple of Open Container Initiative (OCI) containers to support it.  For this document, we will assume you are using Docker but this will work with Podman or other OCI compliant orchestration environments.   These Gleaner Indexing Services are necessary to use Gleaner.   The exception to this would be if you are using a 3rd party objects store like AWS S3 or Wasabi.
Object Store 
An S3 compliant object store supporting S3 APIs including S3Select.  For open source this is best satisfied with the Minio Object Store.  For commercial cloud AWS S3 or hosted Ceph services will work.
Headless Chrome (technically optional) 
This is only needed where you expect the sources you index to use Javascript to include the JSON-LD in the pages.  If you know your sources do not use this publishing pattern and rather include the JSON-LD in the static page, then you don’t need this container running.
IS represent the minimum required services to support Gleaner.  With IS you have an object
store in the form of Minio and a headless chrome server in the form of
chromedp/headless-shell.
As shown in the figure below, and support the basic harvesting of resources with Gleaner
and loading the JSON-LD objects into Minio.
and loading the JSON-LD objects into Minio.
It does not result in these objects ending up in a graph / triplestore.   You would use
this option if you intend to work on the JSON-LD objects yourself.  Perhaps loading
them into a alternative graphdb like Janus or working on them with python tooling.
Fig. 2 Basic Gleaner Indexing Service Activity Workflow#
Gleaner Indexing Services (IS) Environment Variables
The Docker Compose file used to launch the Gleaner IS has a set of configurable elements that can be set and passed to the orchestration system with environment variables.
These can be set manually or through the command line.  A simple script to set the variables could look like:
#!/bin/bash
# domains 
export GLEANER_ADMIN_DOMAIN=admin.local.dev
export GLEANER_OSS_DOMAIN=oss.local.dev
export GLEANER_GRAPH_DOMAIN=graph.local.dev
export GLEANER_WEB_DOMAIN=web.local.dev
export GLEANER_WEB2_DOMAIN=web2.local.dev
# Object store keys
export MINIO_ACCESS_KEY=worldsbestaccesskey
export MINIO_SECRET_KEY=worldsbestsecretkey
# local data volumes
export GLEANER_BASE=/tmp/gleaner/
export GLEANER_TRAEFIK=${GLEANER_BASE}/config
export GLEANER_TRAEFIK=${GLEANER_BASE}/config
export GLEANER_OBJECTS=${GLEANER_BASE}/datavol/s3
export GLEANER_GRAPH=${GLEANER_BASE}/datavol/graph
The actual services can be deployed via a Docker Compose file (also works with Podman).  An example of that file and details about it follow.
– Break down the compose files here  link to them
Aggregator#
Intoduction#
This section introduces the the OIH approach to indexing. Currently,  OIH is
using the Gleaner software to do the indexing and leverages the Gleaner IO
gleaner-compose Docker
Compose files for the server side architecture.  For more information on Docker Compose files visit the
Overiew of Docker Compose.  The gleaner-compose repository holds Docker compose files that can set up
various environments that Gleaner needs.
The figure below gives a quick overview of the various compose options for setting up
the supporting architecture for Gleaner. A fully configured system where all the indexing
and data services are running and exposing services to the net, a total of five containers are run.  In many case
you may run fewer than this.
Fig. 1 The various compose options for Gleaner#
Container overview#
S3 (Minio / AWS): This is the only container that is required in all cases to run.  Gleaner needs an S3 compatible object store.  By default
we use the Minio object store
Chrome Headless: In cases where providers place the JSON-LD documents into the pages with Javascript, we need to render the page before
reading and accessing the DOM.  This is done using Chrome Headless
reading and accessing the DOM.  This is done using Chrome Headless
Graph data base:  Gleaner extracts JSON-LD documents from resources.  These JSON-LD documents are representations of the RDF data mode.  To
queries on them at scale, it easiest to load the triples into a compatible graph database.  Sometimes we call this a triplestore.  For OIH we
use the Blazgraph triplestore.
Router: If we wish to deploy this setup onto the net, we will route to route all the services through a single domain.  To do this network routing
we use Traefik.  This router is not required for local use and alternative routers like Caffdy
or nginx are also valid options.
Web Server: If you wish to serve a web UI for the index, then you can also leverage this setup to serve that.  Again, this is optional and your web site
may be hosted elswhere and simply call to the index in compliance with CORS settings.  There is an example web server that leverages the object store
available in this setup.
Gleaner#
As mentioned Gleaner is a single binary app (ie, one file).  It can be run on Linux, Mac OS X or Windows.  It does
not need to be run on the same machine as the supporting services as it can connect to them