fils/oihBook huggingface dataset:
not need to be run on the same machine as the supporting services as it can connect to them
over the network.  So, for example, they could be hosted in commercial cloud services or
on remote servers.
You can download and compile the code from the previously mentioned github repository or
the releases page.
A single configuration file provides the settings Gleaner needs.  Additionally, a local
copy of the current schema.org context file should be downloaded and available to the app.  This file is needed
for many operations and access it over the net is slow and often rate limited depending on the
source.   You can download the file at the Schema.org for Developers page.
This setup show in the above figure is the typical setup for Gleaner and is
detailed in the Quick Start section.
Indexing Services#
This is the basic indexing service requirements.  At a minimum we need the object store and the Chrome headless containers
scoped in the Indexing Sevices box above. More details on this set can be found in Indexing Servives.
Data Services#
A more expanded set of services is defined in the Data Services section.  This section discussion a setup more
designed to address a server setup tht will support indexing and also present the resulting indexes to the broader internet.
Web UI#
Web UI#
As mentioned, if you wish to serve a web UI for the index, then you can leverage this setup to serve that.  Again, this is optional and your web site
can be hosted elsewhere and simply call to the index in compliance with CORS settings. Some tailes on this can be found in the Interfaces section.
Alternatives#
Note, the Gleaner ecosystem is not a requirement.  OIH follows the structured data on the web and data on the web best practices patterns.  Being web architecture based, there are many open source tools and scripting solutions you might use.  You may wish to explore the Alternative Approaches section for more on this.
What follows is a bit more detail on the setup used by Gleaner.  Experienced users will
see where they can swap out elements for their own preference.  Like a different
triplestore, or wish to leverage a commercial object store?  Simply modify the architecture
to do so.
ODIS Catalog as Index Source#
Before we discuss indexing source a key question is what source will be indexed.
OIH is not a web crawl in that it doesn’t move from source to source based on
the content of those sources.
Rather, the OIH index is based on a list of sources selected ahead of time.  At
this time that set of sources if based on those partners engaged in the
this time that set of sources if based on those partners engaged in the
development phase of OIH.  As the work moves to a more routine operation
the sources will come from the ODIS Catalog.
The ODIS Catalog will then act as a curated source of domains for inclusion
in the Ocean InfoHub.   This will provide a level of curation and vetted of sources
and ensure sources are aware of the technical requirements for inclusion in the
OIH index.
Gleaner CLI Docker#
About#
This is a new approach for quick starts with Gleaner.  It is a script that exposes
a containerized version of Gleaner as a CLI interface.
You can use the -init flag to pull down all the support files you need including
the Docker Compose file for setting up the object store, a triplestore and
the support for headless indexing.
Prerequisites#
You need Docker installed.  Later, to work with the results and load them into a
triplestore, you will also need an S3 compatible client.  We will use the Minio
client, mc, for this.
Steps#
Download the script gleanerDocker.sh from https://github.com/earthcubearchitecture-project418/gleaner/tree/master/docs/cliDocker You may need to make it run-able with
curl -O https://raw.githubusercontent.com/earthcubearchitecture-project418/gleaner/master/docs/cliDocker/gleanerDocker.sh
chmod 755 gleanerDocker.sh
Next you can run the script with the -init flag to pull down all the support files you need.
./gleanerDocker.sh -init
This will also download the needed docker image and the support files.
Your directory should look like this now:
fils@ubuntu:~/clidocker# ls -lt
total 1356
fils@ubuntu:~/clidocker# ls -lt
total 1356
-rw-r--r-- 1 fils fils    1281 Aug 15 14:07 gleaner-IS.yml
-rw-r--r-- 1 fils fils     290 Aug 15 14:07 setenvIS.sh
-rw-r--r-- 1 fils fils    1266 Aug 15 14:07 demo.yaml
-rw-r--r-- 1 fils fils 1371350 Aug 15 14:07 schemaorg-current-https.jsonld
-rwxr-xr-x 1 fils fils    1852 Aug 15 14:06 gleanerDocker.sh
Let’s see if we can setup our support infrastructure for Gleaner.  The
file gleaner-IS.yml is a docker compose file that will set up the object store,
and a triplestore.
To do this we need to set up a few environment variables.  To do this we can
leverage the setenvIS.sh script.  This script will set up the environment we need.
Note you can also use a .env file or other approaches.  You can references
the Environment variables in Compose documentation.
root@ubuntu:~/clidocker# source setenvIS.sh 
root@ubuntu:~/clidocker# docker-compose -f gleaner-IS.yml up -d
root@ubuntu:~/clidocker# docker-compose -f gleaner-IS.yml up -d
Creating network "clidocker_traefik_default" with the default driver
Creating clidocker_triplestore_1 ... done
Creating clidocker_s3system_1    ... done
Creating clidocker_headless_1    ... done
Note:  In a fresh run all the images will be pulled down.  This may take a while.
In the end, you should be able to see these images running:
root@ubuntu:~/clidocker# docker ps
CONTAINER ID        IMAGE                            COMMAND                  CREATED              STATUS              PORTS                                              NAMES
a26f7c945479        nawer/blazegraph                 "docker-entrypoint.s…"   About a minute ago   Up About a minute   0.0.0.0:9999->9999/tcp                             clidocker_triplestore_1
f3a4197c42be        minio/minio:latest               "/usr/bin/docker-ent…"   About a minute ago   Up About a minute   0.0.0.0:9000->9000/tcp, 0.0.0.0:54321->54321/tcp   clidocker_s3system_1
062f029462b1        chromedp/headless-shell:latest   "/headless-shell/hea…"   About a minute ago   Up About a minute   0.0.0.0:9222->9222/tcp  
At this point we should be able to do a run.  During the init process a
working config file was downloaded.
Note:  This config file will change…  it’s pointing to an OIH partner
and I will not do that for the release.  I have a demo site I will use.
Next we need to setup our object for Gleaner.  Gleaner itself can do this
task so we will use
root@ubuntu:~/clidocker# ./gleanerDocker.sh -setup -cfg demo
main.go:35: EarthCube Gleaner
main.go:110: Setting up buckets
check.go:58: Gleaner Bucket gleaner not found, generating
main.go:117: Buckets generated.  Object store should be ready for runs
Note:  Here is where we go off the rails.  The config file uses 0.0.0.0 as the
location and this is not working.   You need to edit the config file with the
“real” IP of the host machine.  In may case is this 192.168.122.77.  This is
obviously still a local network IP but it does work.  I am still investigating
this issue.
We can now do a run with the example template file.
Note:  Best to delete the “sitegraph” node, I will do that soon.  It should
work, but is currently slow and gives little feedback
If everything goes well, you should see something like the following:
root@ubuntu:~/clidocker# ./gleanerDocker.sh -cfg demo
main.go:35: EarthCube Gleaner
main.go:122: Validating access to object store
check.go:39: Validated access to object store: gleaner.
org.go:156: Building organization graph (nq)
org.go:163: {samplesearth  https://samples.earth/sitemap.xml false https://www.re3data.org/repository/samplesearth Samples Earth (DEMO Site) https://samples.earth}
main.go:154: Sitegraph(s) processed
summoner.go:16: Summoner start time: 2021-08-15 14:34:08.907152656 +0000 UTC m=+0.067250623 
resources.go:74: samplesearth : 202
resources.go:74: samplesearth : 202
 100% |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| (202/202, 20 it/s)        
summoner.go:34: Summoner end time: 2021-08-15 14:34:20.36804137 +0000 UTC m=+11.528139340 
summoner.go:35: Summoner run time: 0.191015 
webfeed.go:37: 1758
millers.go:26: Miller start time: 2021-08-15 14:34:20.368063453 +0000 UTC m=+11.528161421 
millers.go:40: Adding bucket to milling list: summoned/samplesearth
millers.go:51: Adding bucket to prov building list: prov/samplesearth
 100% |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| (202/202, 236 it/s)
graphng.go:77: Assembling result graph for prefix: summoned/samplesearth to: milled/samplesearth
graphng.go:78: Result graph will be at: results/runX/samplesearth_graph.nq
pipecopy.go:16: Start pipe reader / writer sequence
graphng.go:84: Pipe copy for graph done
millers.go:80: Miller end time: 2021-08-15 14:34:21.84702814 +0000 UTC m=+13.007126109 
millers.go:81: Miller run time: 0.024649 
Working with results#
If all has gone well, at this point you have downloaded the JSON-LD documents into Minio or
some other object store.Next we will install a client that we can use to work with these objects.
Note, there is a web interface exposed on the port mapped in the Docker compose file.
In the case of these demo that is 9000.  You can access it at
http://localhost:9000/ with the credentials set in the environment variable file.
However, to work with these objects it would be better to use a command line tool, like mc.
The Minio Client, can be installed
following their Minio Client Quickstate Guide.
Be sure to place it somewhere where it can be seen from the command line, ie, make sure it
is in your PATH variable.
If you are on Linux this might look something like:
wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
./mc --help
There is also a Minio Client Docker image
that you can use as well but it will be more difficult to use with the following scripts due
to container isolation.
To man an entry in the mc config use:
mc alias set oih  http://localhost:9000 worldsbestaccesskey worldsbestsecretkey
We should now be able to list our object store.  We have set it up using the alias oih.
user@ubuntu:~/clidocker# mc ls oih
[2021-08-15 14:31:20 UTC]     0B gleaner/
user@ubuntu:~/clidocker# mc ls oih/gleaner
[2021-08-19 13:36:04 UTC]     0B milled/
[2021-08-19 13:36:04 UTC]     0B orgs/
[2021-08-19 13:36:04 UTC]     0B prov/
[2021-08-19 13:36:04 UTC]     0B results/
[2021-08-19 13:36:04 UTC]     0B summoned/
You can explore mc and see how to copy and work with the object store.
Loading to the triplestore#
As part of our Docker compose file we also spun up a triplestore.  Let’s use that now.
Now Download the minio2blaze.sh script.
curl -O https://raw.githubusercontent.com/earthcubearchitecture-project418/gleaner/master/scripts/minio2blaze.sh
chmod 755 minio2blaze.sh 
The content we need to load into the triplestore needs to be in RDF for Blazegraph.  We also need
to tell the triplestore how we have encoded that RDF.   If look in the object store at
mc ls oih/gleaner/milled
[2021-08-19 13:26:52 UTC]     0B samplesearth/
We should see a bucket that is holding the RDF data converted from the JSON-LD.  Let’s use this
in our test.   We can pass this path to the minio2blaze.sh script.  This script will go looking
for the mc command we installed above, so be sure it is in a PATH location that script can see.
./minio2blaze.sh oih/gleaner/milled/samplesearth
./minio2blaze.sh oih/gleaner/milled/samplesearth
...   lots of results removed 
If all has gone well, we should have RDF in the triplestore.  We started our triplestore as part of
the docker-compose.yml file.  You can visit the triplestore at http://localhost:9999/blazegraph/#splash
Note, you may have to try other addresses other than localhost if your networking is a bit different with
Docker.  For me, I had to use a real local IP address for my network, you might also try 0.0.0.0.
Hopefully you will see something like the following.
We loaded into the default kb namespace, so we should be good there.  We can see that is listed
as the active namespace at the Current Namespace: kb report.
Let’s try a simple SPARQL query.  Click on the Query tab to get to the query user interfaced. We
can use something like:
select * 
where 
{ 
  ?s ?p ?o
}
LIMIT 10
A very simple SPARQL to give us the first 10 results from the triplestore.  If all has gone well,
we should see something like:
You can explore more about SPARQL and the wide range of queries you can do with it
You can explore more about SPARQL and the wide range of queries you can do with it
at the W3C SPARQL 1.1 Query Language reference.
Conclusion#
We have attempted here to give a quick introduction to the use of Gleaner in a Docker
environment.  This is a very simple example, but it should give you an idea of the approach
used.  This approach can then be combined with other approaches documented to establish a
more production oriented implementation.  Most of this documentation will be located
at the Gleaner.io GitHub repository and Gleaner repository.
Note: The plan is to merge the Gleaner.io GitHub repository into the first.
Prov#
About#
This is the start of some discussion on issues around prov tracking in OIH.
This may take two paths.  One would be the prov tracking indexers might do
and the other prov that providers would encode to provide specific prov
the community requests.
Gleaner Prov#
The Gleaner application generates a prov graph of the activity of accessing
and indexing provider resources.  The main goal of this prov is to connect
an indexed URL to the digital object stored in the object store.  This
digital object should be the JSON-LD data graph presented by the provider.
By contrast, the authoritative reference in the various profiles will connect
the the data graph ID, or in the absence of that the data graph URL or the
referenced resources URL by gleaner, to another reference.  This may be
an organization ID or a PID of the connected resource.
 1{
 2    "@context": {
 3        "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
 4        "prov": "http://www.w3.org/ns/prov#",
 5        "rdfs": "http://www.w3.org/2000/01/rdf-schema#"
 6    },
 7    "@graph": [
 8        {
6    },
 7    "@graph": [
 8        {
 9            "@id": "https://www.re3data.org/repository/obis",
10            "@type": "prov:Organization",
11            "rdf:name": "Ocean Biodiversity Information System",
12            "rdfs:seeAlso": "https://obis.org"
13        },
14        {
15            "@id": "https://obis.org/dataset/9381239f-3d64-48b4-80c9-b9ebb674edc2",
16            "@type": "prov:Entity",
17            "prov:wasAttributedTo": {
18                "@id": "https://www.re3data.org/repository/obis"
19            },
20            "prov:value": "https://obis.org/dataset/9381239f-3d64-48b4-80c9-b9ebb674edc2"
21        },
22        {
23            "@id": "https://gleaner.io/id/collection/7c1eaa1aaed95861330109026c42e57a31ecae55",
24            "@type": "prov:Collection",
25            "prov:hadMember": {
25            "prov:hadMember": {
26                "@id": "https://obis.org/dataset/9381239f-3d64-48b4-80c9-b9ebb674edc2"
27            }
28        },
29        {
30            "@id": "urn:gleaner:milled:obis:7c1eaa1aaed95861330109026c42e57a31ecae55",
31            "@type": "prov:Entity",
32            "prov:value": "7c1eaa1aaed95861330109026c42e57a31ecae55.jsonld"
33        },
34        {
35            "@id": "https://gleaner.io/id/run/7c1eaa1aaed95861330109026c42e57a31ecae55",
36            "@type": "prov:Activity",
37            "prov:endedAtTime": {
38                "@value": "2021-04-20",
39                "@type": "http://www.w3.org/2001/XMLSchema#dateTime"
40            },
41            "prov:generated": {
40            },
41            "prov:generated": {
42                "@id": "urn:gleaner:milled:obis:7c1eaa1aaed95861330109026c42e57a31ecae55"
43            },
44            "prov:used": {
45                "@id": "https://gleaner.io/id/collection/7c1eaa1aaed95861330109026c42e57a31ecae55"
46            }
47        }
48    ]
49}
import json
from rdflib.extras.external_graph_libs import rdflib_to_networkx_multidigraph
from rdflib.extras.external_graph_libs import rdflib_to_networkx_graph
from pyld import jsonld
import graphviz
import os, sys
currentdir = os.path.dirname(os.path.abspath(''))
parentdir = os.path.dirname(currentdir)
sys.path.insert(0, parentdir)
from lib import jbutils
with open("../../../odis-in/dataGraphs/indexing/prov/graphs/gleaner.json") as dgraph:
    doc = json.load(dgraph)
frame = {
  "@context": {"@vocab": "https://schema.org/",
  "prov": "http://www.w3.org/ns/prov#"},
  "@explicit": "false",
"@explicit": "false",
  "@type":     "prov:Activity",
   "prov:generated": {},
   "prov:endedAtTime": {},
   "prov:used": {}
}
context = {
  "@vocab": "https://schema.org/",
  "prov": "http://www.w3.org/ns/prov#"
}
compacted = jsonld.compact(doc, context)
framed = jsonld.frame(compacted, frame)
jd = json.dumps(framed, indent=4)
print(jd)
{
    "@context": {
        "@vocab": "https://schema.org/",
        "prov": "http://www.w3.org/ns/prov#"
    },
    "@id": "https://gleaner.io/id/run/7c1eaa1aaed95861330109026c42e57a31ecae55",
    "@type": "prov:Activity",
    "prov:endedAtTime": {
        "@type": "http://www.w3.org/2001/XMLSchema#dateTime",
        "@value": "2021-04-20"
    },
    "prov:generated": {
        "@id": "urn:gleaner:milled:obis:7c1eaa1aaed95861330109026c42e57a31ecae55",
"@type": "prov:Entity",
        "prov:value": "7c1eaa1aaed95861330109026c42e57a31ecae55.jsonld"
    },
    "prov:used": {
        "@id": "https://gleaner.io/id/collection/7c1eaa1aaed95861330109026c42e57a31ecae55",
        "@type": "prov:Collection",
        "prov:hadMember": {
            "@id": "https://obis.org/dataset/9381239f-3d64-48b4-80c9-b9ebb674edc2",
            "@type": "prov:Entity",
            "prov:value": "https://obis.org/dataset/9381239f-3d64-48b4-80c9-b9ebb674edc2",
            "prov:wasAttributedTo": {
                "@id": "https://www.re3data.org/repository/obis",
                "@type": "prov:Organization",
                "http://www.w3.org/1999/02/22-rdf-syntax-ns#name": "Ocean Biodiversity Information System",
                "http://www.w3.org/2000/01/rdf-schema#seeAlso": "https://obis.org"
            }
}
        }
    }
}
Nano Prov#
This is a basic nanoprov example. Note, this is a draft and
the ID connections and examples have not been made yet.
 1{
 2    "@context": {
 3        "gleaner": "https://voc.gleaner.io/id/",
 4        "np": "http://www.nanopub.org/nschema#",
 5        "prov": "http://www.w3.org/ns/prov#",
 6        "xsd": "http://www.w3.org/2001/XMLSchema#"
 7    },
 8    "@set": [
 9        {
10            "@id": "gleaner:nanopub/XID",
11            "@type": "np:NanoPublication",
12            "np:hasAssertion": {
13                "@id": "gleaner:nanopub/XID#assertion"
14            },
15            "np:hasProvenance": {
16                "@id": "gleaner:nanopub/XID#provenance"
17            },
18            "np:hasPublicationInfo": {
19                "@id": "gleaner:nanopub/XID#pubInfo"
20            }
21        },
22        {
20            }
21        },
22        {
23            "@id": "gleaner:nanopub/XID#assertion",
24            "@graph": {
25                "@id": "DataSetURI",
26                "@type": "schema:Dataset",
27                "description": "This is where you would put corrections or annotations",
28                "identifier": [
29                    {
30                        "@type": "schema:PropertyValue",
31                        "name": "GraphSHA",
32                        "description": "A SHA256 sha stamp on the harvested data graph from a URL",
33                        "value": "{{SHA256 HASH HERE}}"
34                    },
35                    {
36                        "@type": "schema:PropertyValue",
37                        "name": "ProviderID",
38                        "description": "The id provided with the data graph by the provider",
39                        "value": "{{re3 or URL noted in config}}"
40                    },
41                    {
42                        "@type": "schema:PropertyValue",
43                        "name": "URL",
44                        "description": "The URL harvested by gleaner",
45                        "value": "{{The URL the JSON-LD came from}}"
46                    }
47                ]
48            }
49        },
50        {
46                    }
47                ]
48            }
49        },
50        {
51            "@id": "gleaner:nanopub/XID#provenance",
52            "@graph": {
53                "@id": "URIforprovondataset",
54                "prov:wasGeneratedAtTime": {
55                    "@value": "dateDone",
56                    "@type": "xsd:dateTime"
57                },
58                "prov:wasDerivedFrom": {
59                    "@id": "IDHERE"
60                },
61                "prov:wasAttributedTo": {
62                    "@id": "IDHERE"
63                }
64            }
65        },
66        {
67            "@id": "gleaner:nanopub/XID#pubInfo",
68            "@graph": {
69                "@id": "IDHERE",
70                "prov:wasAttributedTo": {
71                    "@id": "gleaner:tool/gleaner"
72                },
73                "prov:generatedAtTime": {
74                    "@value": "2019-10-23T14:38:00Z",
75                    "@type": "xsd:dateTime"
76                }
77            }
78        }
79    ]
80}
import json
77            }
78        }
79    ]
80}
import json
from pyld import jsonld
import os, sys
currentdir = os.path.dirname(os.path.abspath(''))
parentdir = os.path.dirname(currentdir)
sys.path.insert(0, parentdir)
from lib import jbutils
with open("../../../odis-in/dataGraphs/indexing/prov/graphs/nanoprov.json") as dgraph:
    doc = json.load(dgraph)
context = {
    "@vocab": "https://schema.org/",
}
compacted = jsonld.compact(doc, context)
jbutils.show_graph(compacted)
Refs#
Nanopubs Guidance
Tooling#
About#
The tooling section is a collection of tools, scripts, notebooks and other software that could
be of use to the various personas of Ocean InfoHub
OpenRefine#
In this section you will find some details around
the Open Refine project and how to use it to
generate JSON-LD documents.
Notebooks#
In this section you will find some Jupyter Notebooks
that demonstrate working with JSON-LD in various
ways.  These can be copied and used locally to
explore or implement workflows.
On-line tooling#
Schema.org Validator
json-ld playgroud
SHACL Playground
json-lint
f-uji
Dev#
json-ld
f-uji dev
jq
jello (python) and
Practical JSON at the command line using jello
Python Extruct
Personas#
About#
During the design process of the Ocean InfoHub (OIH), many of the design approached leverage three
personas that help define the various archtypes of people who engage with OIH.  It should not be assumed
these scope all the potential persona or that a person or organization scope only one.  It is quite possible
to many.   These are simply design approaches representing potential models or characters.   They
are tools used in the design process of OIH.
Publisher: A key persona whose activities are covered in detail in Publishing patterns for OIH
Aggregator: Leverages web architecture to retrieve structured data on the web and generate usable indexes.
User: The end user of the publishing and aggregation activities.  May leverage
the web for discovery or tools such as Jupyter for analytics and visualization.
Persona: Publisher#
In OIH the Publisher is engaged authoring the JSON-LD documents and publishing them
to the web.  This persona is focused on describing and presenting structured data on the web
to aid in the discovery and use the resources they manage.
Details on this persona can be found in the Publisher section.
Additionally, this persona would be leveraging this encoding described in the JSON-LD Foundation section and the
profiles described in the Thematic Patterns.
Persona: Aggregator#
In OIH the Aggregator is a person or organization who is indexing resources on the