fils/oihBook huggingface dataset:
Structured Data on the Web#
About#
Structured data on the web is a way to provide semantics and linked data in an
approachable manner.  This approach expresses concepts in JSON-LD which is a
JavaScript notation popular among developers which easily expresses  concepts
(terms) and links to related resources (things).   This structured data on the
web approach has been popularized by the large commercial search providers like
Google, Bing, Yandex and others via schema.org  As described at schema.org:
“Schema.org is a collaborative, community activity with a mission to create,
maintain, and promote schemas for structured data on the Internet, on web pages,
in email messages, and beyond.”
The popularity of leveraging the schema.org approach in the earth sciences can
be attributed to both this ease of developer adoption and also to its
foundational use of web architecture. A web architecture foundation aids
adoption by the operations side as well as the developer side.  It also takes
advantage of the scale and resilience of the web.
The broad nature of schema.org even scopes to the concepts of Datasets.  It is
the existence of schema.org/Dataset that was a focus of several EarthCube
projects (Project 418, Project 419 and the Resource Registry) from which spun up
the ESIP Science on Schema work.
Additionally, Google leveraged schema.org/Dataset to develop and populate the
Additionally, Google leveraged schema.org/Dataset to develop and populate the
Google Data Set Search and provides guidance to developers to facilitate this.
Web architecture approach#
OIH is focused on leveraging the web architecture as the foundation for this
approach.  There are several key reasons for this vs approaches like OAI-PMH or
others.
A key point is that in the processes of establishing a web presence, a standard
step for groups, they have already begun to build the infrastructure needed for
structured data on the web.  Setting up special servers or establishing and
maintaining special APIs to support harvesting is not required.
Also, a large collection of tooling already exists around JSON that is directly
usable in JSON-LD.  That scale extends to the use of schema.org patterns which
have become common in the commercial web.  Allowing us to bring those same
patterns and the tooling to the science community.
Additionally, this approach keeps the metadata and its representation a product
of the data providers.  The actor in the life cycle most aware of needed edits,
new records or other events.  That same record then serves multiple consumers
able to generate various value add products.  This benefits the provider by
facilitating multiple and varied discovery vectors for their holdings.
Another key factor is the web native and semantic nature of this representation
Another key factor is the web native and semantic nature of this representation
of metadata.  Traditional metadata, such as ISO, by itself does not express a
web referenceable instance of concepts.  In doing this, structured data on the
web allow connections to be made and discovered by people and machines across
many holdings.  This aids in both serendipitous discovery and can also be
leveraged to aid discovery via semantic relations.
Terminology#
A CSV file is a text file containing spreadsheet information following a data
model that is encoded using a convention of rows and commas defining columns.
A JSON-LD fle is a text file containing graph information following the RDF data
model that is encoded using a convention based on JSON syntax.
JSON-LD is a way to serialize RDF that uses JSON notation.  It is really no
different then than RDF-XML, turtle, n-triples, etc.  There are several ways to
represent the RDF data model in text files (and some emerging binary ones like
CBOR and parquet patterns).
Schema.org is a vocabulary for describing things similar to DCAT, FOAF, Dublin
Core.  It does this by using RDF as the underlying data model to represent this
“ontology”.
The confusion comes from the collision of outcomes.  JSON-LD came about, partly,
to allow the use of the RDF data model by a broader audience.  This is done by
to allow the use of the RDF data model by a broader audience.  This is done by
leveraging a more popular notation for the data model, JSON, in the form of
JSON-LD.  Schema.org also wanted to advance the use of structured [meta]data by
making it easier to use and connecting structured data to web pages.  At the
start, there were three approaches; RDFa, microformats and JSON-LD, to putting
schema.org in web pages.  However, the JSON-LD approach to incorporating this
structured data has grown in popularity far beyond the others. As the popularity
of both JSON-LD and schema.org grew,  they often got conflated with each other.
The term  “structured data on the web” is perhaps a more neutral way to discuss
the use of vocabularies encoding in JSON-LD used in web pages.  However,  the
phrase “schema.org” is starting to become the term for “structured data on the
web using JSON-LD as a serialization”.    Even in cases where you combine other
vocabularies such as DCAT with JSON-LD with no schema.org involved, it seems the
way to convey this is to say: “We will use the schema.org ‘pattern’ with DCAT”.
It is arguably not the best or most accurate communications strategy.  It can
It is arguably not the best or most accurate communications strategy.  It can
conflate data models, serialization and vocabularies.  However, it is concise
and ubiquitous and not likely to change.
Intellectual Merit#
OIH leverages structured data o the web patterns in the form of of
Schema.org and JSON-LD encoding.  This means that much of what is done
to address OIH implementation by providers also is available both to existing
commercial indexing approaches as well as emerging community practices
Additionally, both the publishing and indexing approaches are based
on several web architecture patterns.  Meaning that existing organization skills
are leveraged and staff experience is enhanced.   This helps to address both
the sustainability of the OIH connection and the efficiency of
organizational operation.
Broader Impacts#
By leveraging existing technology and approaches a larger community is enabled
to engage and make more samples discoverable and usable.
The nature of structured data on the web also provides the ability to apply
semantic context to samples.  This means richer discovery and information about
samples, the past uses and potential future uses is more readily available.
Simplified architecture also means easier development of tools and interfaces to
present the data.  Allowing the presentation of samples and their information in
a manner aligned with a given community’s needs.    A simplified architecture
aids sustainability from both a technical and financial perspective.
Search
    Please activate JavaScript to enable the search
    functionality.
  
    Searching for multiple words only shows matches that contain
    all words.
The Ocean InfoHub Project#
Introduction#
The Ocean InfoHub aims to build a sustainable, interoperable, and inclusive digital ecosystem for all Ocean data centres. Existing and emerging data systems are linked, with the ultimate goal of coordinating action and capacity to improve access to Ocean data and knowledge.
The following video provide brief high level context to the Ocean InfoHub project.
Broucher
Organizations are increasingly exposing data and resources on the Web.   A popular approach to this is using web architecture to expose structured data on the web using the schema.org vocabulary.   Doing this makes resources discoverable by a range of organizations leveraging this architecture to build indexes.  These include major commercial indexes, large domain focused groups and community focused services.
The Ocean Data and Information System (ODIS) will provide a schema.org based interoperability layer and supporting technology to allow existing and emerging ocean data and information systems, from any stakeholder, to interoperate with one another. This will enable and accelerate more effective development and dissemination of digital technology and sharing of ocean data, information, and knowledge. As such, ODIS will not be a new portal or centralized system, but will provide a collaborative solution to interlink distributed systems for common goals. Together with global project partners and partners in the three regions, a process of co-design will enable a number of global and regional nodes to test the proof of concept for the ODIS.
The ODIS-architecture development is being supported by the Ocean InfoHub Project, and it has been tested initially on IOC and partner databases. However, the system and standards are open for any institution or initiative that is interested in accessing the global data ecosystem to adopt and implement.
Guidance for the implementation of the ODIS-architecture#
OIH is providing guidance on the various stages of such an
architecture including authoring, publishing, indexing and interfaces.
The basics of this approach can be described as:
Providers publish HTML pages for a resource.  This may be a publication, course description, research instrument or other.   The core themes for OIH are described in the Authoring section below.
A HTML page then has a small JSON based snippet added to the HTML.  This is
described in the Including JSON-LD in your resource page in the Publishing
resource below.
If you wish a resource to be included in the OIH index, then you need to
include it in a sitemap file.  This is a small XML document that lists links
to the resources you wish to be part of the index.  This approach is shown in
the sitemap.xml section of the Publishing resource.
Once the above is done the publishing phase is over.  At this point, OIH or
other groups can now access and index your resources.   OIH is using some
other groups can now access and index your resources.   OIH is using some
existing software to index and generate the graph and expose a simple
reference interface to them.  This software is open and available and others
are free to implement the approach with other software.  Links to other
software are at the repository.
The OIH index/graph and a simple interface is current at a development site
and in a later phase of OIH a production interface will be developed.
Additionally, software to aid in validating and checking the resources is under
development and will be available at the repository.   This will aid providers
in expressing the information needed to address interfaces and services of
interest to the community.
The result is a sustainable architecture to address discovery and access to
various resources published by the community and a shared graph of these
resources.  That shared graph can be used by all members to link and discover
across groups.
Key links to the OIH GitHub repository#
Interested groups can review material addressing these stages at the OIH GitHub
repository.  Links and descriptions of these stages are described below.
Authoring Thematic Patterns#
The ODIS OIH is working across five major thematic areas; Experts and
Institutions, Documents, Projects, Training, Vessels.   Examples of these
thematic concepts are being hosted and developed with input from the community.
thematic concepts are being hosted and developed with input from the community.
Additionally, methods for validation and simple tooling for authoring and
testing are hosted at this repository.  Alongside these five thematic topics
guidance on connecting services and spatial context on resources.
Publishing#
Guidance on implementation the web architecture approach is also available.
This includes approaches on leveraging robots.txt and sitemaps.xml file for
expressing hosted resources to the net.
Indexing#
The architecture approach is open and standards based.  As such, many
organizations will be able to leverage the authoring and publishing approaches
above to index a providers resources.  OIH will be providing reference
implementations of software that can generate the index.
Interfaces and Services#
During the development of the OIH a basic reference implementation for an
interface has been generated.  This is a development site meant to test and
exercise the above elements.   It serves to demonstrate how others could also
implement this approach and how future interfaces could be developed.
An example of the value of implementing a lightweight can be seen with the
Government of Canada:
The Federal Geospatial Platform is a intra-governmental data catalogue
implementing the Harmonized North American Profile of ISO 19115 (HNAP), with
content exposed externally via OGC CSW (Catalogue Services Web).
This content is harvested by the public facing Open Maps platform, which
This content is harvested by the public facing Open Maps platform, which
includes a catalogue component that is fed in part by the Federal Geospatial
Platform. DCAT-based metadata is derived from the original ISO 19115 based
metadata. As this markup is recognized by web crawlers such as those hosted by
Google, content is harvested and is subsequently visible through Google
Dataset Search. Furthermore, the cited publication for the data is also link
via a complementary link to Google Scholar.
Publisher#
About#
This page describes the publishing process for structured data
on the web approach OIH will use.
Note many software packages you are using might already
implement this approach.  See the section:
Existing support in software at the bottom of this document.
See also
We also recommend reviewing the document:
Schema.org for Research Data Managers: A Primer
Architecture Implementation#
The Ocean Info Hub (OIH) will leverage structured data on the web and web
architecture patterns to expose metadata about resources of interest to the
community.  The primary tasks include:
Authoring JSON-LD documents (https://json-ld.org/) aligned with ODIS OIH
guidance to express the structured metadata for a resource.  This step will
require experience with using the existing metadata resources within an
organization.  So any necessary skills needed to access or query existing
facility data systems will be needed to assemble the information to populate
the JSON-LD data graph.  The JSON-LD documents need to be generated using the
tools/languages at the previous reference or through other means.
Within the system architecture of the site, a JSON-LD document needs to be
placed into the HTML DOM as a SCRIPT tag within the HEAD tag of each
published resource.   The SCRIPT tag pattern is:
<script type="application/ld+json">JSON_LD content</script>
<script type="application/ld+json">JSON_LD content</script>
Additionally these resources that are marked up with these tags and JSON-LD
documents should be expressed in an XML sitemap file.  This should follow the
guidance at https://www.sitemaps.org/.  It should also include a lastmod node
element as described at https://www.sitemaps.org/protocol.html which should
indicate the date the resource metadata was last updated and published to the
web.
The process of aligning the JSON-LD is iterative at this stage as the OIH
profile is evolved.  To aid this we can leverage existing validation tools
including JSONSchema, W3C SPARQL and more to communicate structure changes.
These tools exist and need only be implemented using knowledge of command
line environments.  The results will then indicate revisions needed in the
JSON-LD.  OIH will provide the necessary templates for the tools to use
against the authored JSON-LD documents.
Information on the sources, standards and vocabularies to be used can be found
at: https://github.com/iodepo/odis-arch/tree/schema-dev/docs
Including JSON-LD in your resource page#
To provide detailed and semantically described details on a resource, OIH uses
a JSON-LD snippet or data graph.  This small document
a JSON-LD snippet or data graph.  This small document
provides details on the resource.  It can also express any explicate
connections to other resources an author may wish to express.  The semantic
nature of the document also means that connections may later be discovered
through graph queries.
Pages will need a JSON-LD data graph placed in it via a typed script
tag/element in the document head element like the following.
<script type="application/ld+json"></script>
An example data graph can be seen below.   However, check the various
thematic sections for more examples for a given thematic area.
{
    "@context": {
        "@vocab": "https://schema.org/",
        "endDate": {
            "@type": "http://www.w3.org/2001/XMLSchema#dateTime"
        },
        "startDate": {
            "@type": "http://www.w3.org/2001/XMLSchema#dateTime"
        }
    },
    "@id": "https://foo.org/url/to/metadata/representation",
    "@type": "Course",
    "description": "In this course you will get an introduction to the main tools and ideas in the data scientist's toolbox...",
    "hasCourseInstance": {
        "@type": "CourseInstance",
        "courseMode": [
            "MOOC",
            "online"
"courseMode": [
            "MOOC",
            "online"
        ],
        "endDate": "2019-03-21",
        "startDate": "2019-02-15"
    }
}
This example is from the training and courses thematic
section.  To view all the types
being developed reference
the Thematic section.
These JSON-LD documents leverage schema.org as the primary vocabulary.
The examples in the thematic section provide examples for the various type.
JSON-LD Tools and References#
A key resource for JSON-LD can be found at JSON-LD.
There is also an interactive playground hosted there.  The JSON-LD
Playground is useful when testing or
exploring approaches for JSON-LD data graphs.  It will catch basic errors of
syntax and use.  Note, it will not catch semantic issues such as using
properties on types that are out of range.  Tools like the Structured Data
Testing Tool are
better at that.  Also the documents and validation material created here OIH
will also allow for that sort of testing and feedback.
Providers may also wish to provide content negotiation for type application/ld+json
for these resources. Some indexers,  like Gleaner, will attempt to negotiate for
the specific serialization and this will likely lighten the load on the servers going forward.
the specific serialization and this will likely lighten the load on the servers going forward.
Validation With SHACL or ShEx#
To help facilitate the interconnection of resource, some application focused validation
will be developed. Note, this validation does not limit what can be in the graphs.
Rather, it simply provides insight on to how well a given graph can be
leveraged for a specific application.  For this project, the application will
be the OIH search portal.
Some initial development work for this can be found in the
validation directory
Validation Tools and References#
SHACL playground
Schemarama
Schimatos.org
demo
Comparing ShEx and SHACL
Validation Leveraging JSON Schema#
We have been exploring the potential to use JSON Schema combined with various
on-line JSON editors (JSON Schema driven) to provide a potential approach to a
more visual editing workflow. The workflow presented here is very ad hoc but
exposes a potential route a group might take to develop a usable tool. Such a
tool might, for example, leverage the Electron app dev environment to evolve
this approach in a more dedicated tool/manner.
Use a JSON-LD document (Example) one could
load this into something like
the JSONschema.net tool.
The results of the above can then been loaded into the online JSON-Editor at
https://json-editor.github.io/json-editor/. (Ref:
https://json-editor.github.io/json-editor/. (Ref:
https://github.com/json-editor/json-editor)
The results of this then can be loaded into https://json-ld.org/playground/ to
validate that we have well formed JSON-LD.
Though this workflow is rather crude and manual it exposes a route to a defined
workflow based around established schema that leverages other tools and
software libraries to generate a workable tool.
Basics#
The basic activity can be seen in the following diagram:
Elements in detail#
robots.txt#
OPTIONAL: Providers may decide to generate or modify their robots.txt
file to provide guidance to the aggregators.
The plan is to use the Gleaner software (gleaner.io) as well as some
Python based notebooks and a few other approaches in this test.
Gleaner uses an agent string of EarthCube_DataBot/1.0 and this can be
used a robots.txt file to specify alternative sitemaps and guidance.
This also allows a provider to provide guidance to Google and other potential
indexers both for allow and disallow directives.
Sitemap: http://samples.earth/sitemap.xml
User-agent: *
Crawl-delay: 4
Allow: /
User-agent: Googlebot
Disallow: /id
User-agent: EarthCube_DataBot/1.0
Allow: /
User-agent: EarthCube_DataBot/1.0
Allow: /
Sitemap: https://example.org/sitemap.xml
sitemap.xml#
Providers will need to expose a set of resource
landing pages using a sitemap.xml file. As noted above, providers
can expose a sitemap file to just the target agent
to avoid indexing test pages by commercial providers.  You may wish
to do this during testing or for other reasons.  Otherwise,
a sitemap.xml file exposed in general from somewhere in your site is
perfectly fine.
Information on the sitemap structure can be found at sitemaps.org.
It is encouraged to use the sitemap lastmod node
to provide guidance to indexers on page updates.
Additionally indexers may test ways to evaluate additions and
removals from the sitemap URL set to manage new or removed resources.
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/site0.9">
   <sitemap>
      <loc>http://samples.earth/sitemap_websites_sampleseaxml</loc>
      <lastmod>2004-10-01T18:23:17+00:00</lastmod>
   </sitemap>
   <sitemap>
</sitemap>
   <sitemap>
      <loc>http://samples.easitemap_doclouds_igsndatagraphs.xml</loc>
      <lastmod>2005-01-01</lastmod>
   </sitemap>
</sitemapindex>
Full Workflow#
The architecture defines a workflow for objects seen in the above diagram.
The documents flow from; authoring, publishing and indexing to
storage for the objects and the resulting graph.  These resources are
then ready for use in search and other functions.
Moving left to right we can review the image.
Providers are engaged in the process of developing the OIH example
documents.  These provide a profile to follow to represent the semantic
metadata.  Note, these are not limiters, simply guidance on minimum and
recommend elements to address the functional goals of the OIH portal.
Providers use these documents to generate the JSON-LD data graphs.
These can be either static documents or generated and placed in pages
dynamically with Javascript or server side templates.  These are the
existing web pages for the resoruces, not enhanced with the
semantic metadata snippets in the HTML source.
These are published to the web and referenced in the sitemap.xml
document that is also made available.  At this point this material is
available to anyone who may wish to index it and provide discovery
for these resources.
available to anyone who may wish to index it and provide discovery
for these resources.
OIH Portal will then index and validate these resources on a
recurring bases to maintain a current index.  This index will include
both the JSON-LD objects and the graph they form.  This graph can
be used for search, connections and other value add services for the
community. The graph is also directly available to the community for them
to use in support of services they may wish to provide.
Existing support in software#
Many content management systems other web based data interfaces
may already have support for the structured data on the web pattern and
schema.org specifically. While it is beyond the scope of this project to
detail each one, a few starting points for exploration are provided below
for some of the more common ones.
Drupal
CKAN
DSpace
DKAN
ERDDAP (native support)
OPeNDAP (native support)
GeoNode
schema.org issue ref